Saturday, July 05, 2014

Lost in space

Lost in space

The following blog entry is a bunch of my own thoughts regarding a 3D engine I’m writting.

Foreword

It’s been a while now I’ve been trying to represent a 3D scene the proper way. I’ve found dozens of solutions, but still none of them is likely to be as proper nor accurate as I’d like to. The problem is actually quite simple: a 3D scene is composed of several objects. You can encounter two types of objects:

  • value-semantic object
  • entity-semantic object

A value object is an object you can think of by its value. For instance, money, color, position, a file configuration, and so on.
An entity object is an object you cannot think of by it its value. It’s a whole unique object. The simplest example is you. A human. Even if we find a way to make perfect clones, two perfect similar humans remain different. We also capture this concept in programming.

Now, what are 3D scene’s objects? Let’s select of few you could encounter in almost all engines:

  • a vertex
  • a mesh, which is a bunch of vertices with primitives construction information
  • a shader
  • a material, which holds a sharable shader
  • a model, which is a mesh associated to a material
  • a light
  • entities, that bind any value to space information (position, orientation and scale, for instance)

As you might guess, some objects in the above non-comprehensive list of scene objects are value-semantics, others are entity-semantic. A vertex is obviously a value. A mesh could be both depending on how you design your scene. In my way, it’s an entity. So is a shader, a material, a model and a light. Funny things, scene entities are values since they link an entity-semantics object to space information, which is value.

With all that said, how would you link objects between each other?

The simple way: identifiers

My first thought was a simple one: let’s introduce a higher-order type called ID:

newtype ID a = ID Int deriving (Eq,Ord,Show)

It’s a phantom type in order to know what we are refering to. ID :: Mesh refers to a Mesh. So, we can use our new identifiers type to add indirection in our types:

data Mesh = Mesh {-  -}
data Material = Material {-  -}
data Model = Model (ID Mesh) (ID Material)

That requires us to be able to generate names. I came up with a simple solution to do so:

class Identify a m where
  ident :: a -> m (ID a)
  reify :: ID a -> m a
  imod  :: ID a -> (a -> a) -> m ()

This typeclass is a bit like MonadState on steroids since it adds indirection. Then, given your own monad based on SatetT, you could do something like:

instance Identify Mesh MySceneMonad where
  ident = {-  -}
  reify = {-  -}
  imod  = {-  -}

instance Identify Material MySceneMonad where
  {-  -}

instance Identify Model MySceneMonad where
  {-  -}

foo :: MySceneMonad (ID Model)
foo = do
  msh <- ident $ Mesh {-  -}
  mat <- ident $ Material {-  -}
  ident $ Model msh mat

However, this suffers from some issues. First, the implementation of the ID supply is in your hand. If you do it wrong, everything will
just go… wrong. Furthermore, IDs are escapable. What would happen to if you escape an identifier and reify in another monad?

getID :: MySceneMonad (ID Mesh)
getID = ident $ Mesh {-  -}

foo = do
    mshid <- runMySceneMonad getID defaultState
    runMySceneMonad $ reify mshid -- oops!

The less simple way: AST

An AST would be great to represent our scene. We could find at the top objects which have the less dependencies, and at the bottom the
more dependent objects. Let’s try to write down the AST for a few objects only:

data E a
  = V a
  | Let (E a) (Scope () E a)
  | Lam (Scope () E a)
  | E a :@ E a
  | LitShader Shader
  | LitMesh Mesh
  | LitMaterial Material
  | LitModel Model
    deriving (Eq,Show)

Shader, Mesh, Material and Model would only care about core features. So you don’t find a Shader in the implementation of a Material
since the link is done through the AST. But in our case, how? We could use the Let constructor, let’s try out:

Let (LitShader shaderDef) (LitMaterial materialDef)

As you can see, that is stupid. The normal form of that expression discards the shader and turns into LitMaterial materialDef. I see here a
solution: we need a special branch in the AST to represent link. Let’s add that into our E expression type only for shaders:

WithShader :: E a -> E a -> E a

So we can do:

WithShader (LitShader shaderDef) (LitMaterial materialDef)

Notice we could even write:

WithShader (V "myShaderDef") {- any expression using that shader -}

Now I see another issue: the first expression is untyped. We could write this, it’d be perfectly valid:

WithShader (LitMaterial materialDef) (V "oshit")

And even V "myShaderDef is untyped. So we’re hitting ourselves in the foot with such an AST. We could, though, think it’s perfectly legit
and write a typechecker that would validate an AST for us, like:

typecheck :: E a -> Either ASTError (E a)

The harder way: AST via GADT

We don’t want typecheck, it’ll complicate the way we use an AST. So let’s turn E into a GADT.

data E :: * -> * -> * where
  V :: a -> E l a
  Let :: E l a -> Scope () (E l) a -> E q a
  Lam :: Scope () (E l) a -> E q a
  Apply :: E l a -> E q b -> E l a
  LitShader :: Shader -> E Shader a
  LitMesh :: Mesh -> E Mesh a
  LitMaterial :: Material -> E Material a
  LitModel :: Model -> E Model a

But now, go and try to make that a monad. You won’t be able to, since it’s not. Even less an applicative functor, so not foldable nor
traversable. I think that solution is the more complicated, the more obfuscated, and the more… solution, in the end!

The hope solution: typeclass

Typeclasses could save us all, since the interpreters could be the typeclass implementation. Let’s turn the GADT into a typeclass:

class AST expr where
  var :: a -> expr l a
  let_ :: expr l a -> a -> expr q a -> expr q a
  -- and so on and so forth

This is nice, I like that way of doing, even more than data, because we can let the compiler choose the best type for us (or force it).
An AST would be something like this:

ast :: (AST e) => e l a

But unfortunately, we have the same problem as GADT: writing a proper implementation will be… writing the same madness GADT…

So?

So… I still don’t know. I’m looking for alternatives. Up to know, I believe the ID solution is my shortest way to get things done. I’m open
to better solutions, because it might have plenty.

Tuesday, February 25, 2014

kwak – release 0.2.0.1

The Apocalypse

A few days ago, freenode – an IRC server – was injured by some people performing a DDoS. It was very messy and apocalypse day, but we all ended up quite okay. Except for one little guy: kwak, my IRC tellbot.

What happened

During a netsplit or a DDoS, you might be disconnected from the server. All IRC clients might implement something against that. As you may have already noticed, my tellbot is not really an IRC client (laughter). But it acts like it was one. You can /whois it, you’ll get something – nothing interesting though. Then during a netsplit or DDoS, the tellbot is disconnected, but doesn’t reconnect since I didn’t foresee it could happen.

The patch

Then I wrote a patch – it took me five minutes; Haskell <3. The patch introduces some fixes against disconnections. I also added an anti-kick behavior – kick the tellbot, it’ll come back! It was not mandatory, but it was something like ten words of code, so…

The patch is not deployed yet. I’ll do the change tonight if I have some spare time.

In the next release, 0.2.1.0, I’ll add a support for a minimalistic admin session through direct connection to the bot. It’ll be used to correctly shutdown the bot – useful when maintaining – and then backup all stories in order to resume them later.

Note: a bug was discovered by LLB. Using !tell in private always succeeds whereas it mustn’t. I’ll push another patch for that issue.

Friday, February 21, 2014

Logging flow activity

Foreword

Everyone needs to log things in a program. Have you already wondered what kind of logs you were dealing with? I don’t mean debug, warning or error logs. I mean semantic here.

Basically, I’ll put apart two concepts:

  • tracing an algorithm’s behavior
  • tracing a flow activity

Tracing an algorithm’s behavior is something we often do in Haskell with the Writer or WriterT monads. Those are very convenient because they allow the use of a monoid along with your computation. You can then compute things while mappending a monoidal value. Once the algorithm is done, you get a pair of values: the value from your computation, and the monoidal log. This is great when you want to launch something and analyze trace later.

This Writer stuff doesn’t fit flow activity. Why? Because tracing an algorithm’s behavior is what we could call deferred log handling whereas the later is on-the-fly log handling. There could be a lot of reasons for prefering on-the-fly over deferred:

  • if you’re planning to log in a function that may take long time to run (you’d like the logs as soon as possible, not in an hour)
  • if you’re planning to log a lot (you don’t want your application to poorly crash because your system runs out of memory due to the huge and nasty accumulating monoid memory print)
  • you have a reactive ecosystem to logs
  • and so on…

You still can do on-the-fly log handling with Writer. There are two functions called listen and listens that will grant you access to the monoidal value of a sub Writer. Then you can do something like that:

runWriterT $ do
  (_,w) <- listen $ do
    tell "Hello, this is dawg!"
    return 314
  liftIO (print w) -- print it out here

But all of this has a huge and important drawback: because the call to listen runs in Writer or WriterT, its monoid will be accumulated in its parent. So we can treat stuff on the fly, but we can’t purge logs. Imagine what happens if your application logs intensively and runs for a long time. Bam! :)

Tracing flow activity

We want to easily accumulate monoidal values. We also want to have a solution to access the monoid and do something with it – for instance print it out on screen or in the terminal. Finally we want to clear the monoid – the purge thing.

MonadJournal

Here MonadJournal comes. I’ve chosen the term journal because it’s straight-forward and a lot of applications call that that way (look at systemd’s journalctl program for instance). Let’s have a look:

class (Monoid w, Monad m) => MonadJournal w m | m -> w where
  journal :: w -> m ()
  history :: m w
  clear   :: m ()

And these useful functions:

sink   :: (MonadJournal w m, MonadIO m) => (w -> IO ()) -> m ()
absorb :: (Monadjournal w m) => (a,w) -> m a

The MonadJournal typeclass defines three functions. journal, history and clear. The former is used to trace activity. It’s quite the same thing as tell, but in MonadJournal. history is used to retrieve the history of logs. The latter is used to clear the history.

sink is a log handler. It takes a function that takes the monoid – i.e. the logs history – and does something with it in IO, and purge the history. The simpliest thing you can do with logs is print them out! And this is quite ultra simple to do. If your monoid is also in the Show class, you can simply print them all with this:

sink print

absorb is to MonadJournal what writer is to MonadWriter. It takes a value with a monoid, absorbs the monoid, and pass the value around.

JournalT

The problem with that is that it’s a typeclass, not something directly usable. There’s no implementation of journal nor sink anywhere yet. That’s why a JournalT monad transformer was introduced. This is its type:

JournalT w m a

It’s then the monad transformer version of MonadJournal, just as WriterT is the monad transformer version of MonadWriter. It has a function to run it:

runJournalT :: JournalT w m a -> m (a,w)

Now we can write a complete example:

main :: IO ()
main = do
    void . runJournalT $ do
      -- here we are tracing IO flow activity
      journal ["hello!"]
      v <- maybe (return 0) absorb . runJournalT $ test
      journal ["just read " ++ show v]
      sink print

-- and here we are tracing Maybe flow activity, which is pure
test :: JournalT [String] Maybe Int
test = do
    journal ["lol"]
    return 314

See how it’s unified thank to the typeclass!

Conclusion

I’ve been wandering around logging in Haskell for a while now. I’ve been using Writer, WriterT, passing logging action (like a function foo :: (Errorable e, MonadIO m) => (e -> IO ()) -> a -> b -> … -> m ()) and so on. I came up with the MonadJournal solution recently while working on my 3D engine, and it’s quite nice. It’s already in hackage, so give it a try, and leave your feedback! :)

If you think some combinators are missing, please let me know! :)

Thursday, February 20, 2014

Regular imperative statements versus FPL abstractions

When having talks over the Internets about FPL (Functional Programming Language), it’s not uncommon reading people exposing the same perennial misconceptions. They’re actually several sides around such a topic:
  • the ones who think FP is a toy and have poor arguments
  • the ones who think FP is not that good and have poor arguments
  • the ones who think FP sucks and can’t say why (no argument at all)
  • the ones who think FP is a “way of thinking the software” and that it’s not language-related
  • the ones who’s gone beyond the stupid toy / research toy / masturbating toy / brainfuck-like language stuffs and are now interested and want to discover more
  • the ones who think FP is great
  • the ones who have no idea what the heck FP is
Something interesting: in most cases, a lot of guys never actually experienced FP seriously. I mean “use it in a software different than Fibonacci generation or other school cases”. I don’t want to start a debate about whether FP is the best or the worse way of programming. If you’re there it’s because you’re from the guys who’s overstepped the barrier of FP and you want to dig in. If you’re there, you now understand that “it’s slow”, “it’s ugly”, “it doesn’t fit production requirements and serious business” and so on were valid arguments some decades ago, but not anymore – some people would even say some of those points weren’t actually close to be valid at all.

In this thread, I put forward an analogy between regular imperative – C/C++, Java, C#, Perl and so on – statements and FPL abstractions. The requirements for you to understand this thread are quite low: basic statements (for, if, else if, else, switch, and so on).

Loops
Looping over a collection is natural for all programmers. It’s a very common situation and is used to break the controlflow of your application. Without them, you’d have a program acting like a math function. It would take input, do some stuff on it, then exit. Iterations enables you to go backward, repeat the same functions and instructions a certain times or even do some exotic and weird things.

That was just the a “programmer” simple way of defining what looping could be. For a mathematician, regarding functions, looping does really not make any sense. You can’t apply a function and rewind.

Consider this C++ snippet:

std::vector; a = { 1, 8, 2, 34, 314 };
int sum = 0;

for (auto it = std::begin(a), e = std::end(a); it != e; ++it);
  sum += *it;
This is a typical case of looping in C++ using an iterator, which could be considered as a raw pointer in that previous example.

Now, how would you do the same thing in FPL? One say “well let’s just write a recursive function sumArray that would sum all elements!”. This would work, but it’s not that way we think in FPL. Instead, we come up with abstractions.

Abstracting is the process of generalizing something true to one concept to several other ones along a certain, well fixed, axis of abstraction. For instance, you know how to sum integers. You can generalize integers sum along some axis. If you choose the type axis, you can generalize integers sum to numbers sum. If you choose the operation axis, you can generalize integers sum into integers binary operation. And so on.

“Looping”. Try to forget programming. Does it still have any meaning to you? One of the hardest things to deal with when trying to get into functional programming is to think like a human again – yeah you’re all filfthy machines! Try reasoning about that collection, and forget looping. Ok, let’s give it a go. I’d say “I want to do something for each elements”. That’s a great start! But this is very abstract. Let’s try something else. “Well, I want to construct a value while traversing a collection”. This is much better! “In this case, I just want to sum all elements in the collection, then I just want to build a value by summing it to each element of the collection”. Quite concise. In C++, what was that again? “I want to loop over a collection and alter a variable by summing.”.

In Haskell – but also in perhaps all other FPL – it’s very common to use the abstract concept of a fold to iterate over a collection, building a value along. The collection is now called a foldable type and the iteration a fold. Folding a value is just accumulating a value using a function on each element. The idea is quite simple:
  • you have a function a -> b -> b
  • you have a foldable value with several a inside of it
  • you have an initial value for b
  • folding is just applying the function to each of the elements, passing along the previous value of the accumulator to the current call
Folding is then a special kind of iteration in which you accumulate a value. Because of polymorphic type, Haskell can handle any types of accumulator when using a folding function. That means you can fold a list and generate any kind of data.

Let’s write the same example as above, but in Haskell:

let a    = [1,8,2,34,314]
result = foldl1 (+) a
That’s all. You might say “ahah, foldl1, it’s barly readable” but in fact, it is. We find fold in foldl1. Then, we find a l. What could it be? Well, it’s a directional fold from left to right. You could also find a r as in foldr1 for instance – it would be a directional fold from right to left. What’s the 1? As mentionned above, folding needs an initial value to be able to start. Imagine the sum, the initial value is 0. foldl1 just uses the first value of the foldable type as initial value for the accumulator – it has an impact of the type of the folding function, which is now a -> a -> a; if you don’t understand why, it’s not really a matter).

Then, foldl1 (+) is a function that sums all elements of a collection (a list, actually). It could be a first implementation of the sum function, that does the same thing.

Conditional execution
This is quite the same thing in FPL, but it’s a bit different as well. In typical imperative languages, conditionals are control statement. The single exception might be the ternary operator – not all languages have it. You know, that’s operator with ? and : keywords. Well, for Haskell, we only have that one. We don’t use the same keywords though, but instead, we have if and else.

Consider this C++ snippet:

string name;

std::cin << name;
if (name.empty()) {
  std::cout << "woh, empty name!" << std::endl;
} else if (name.size() < 4) {
  std::cout << "this is a short name you have" << std::endl;
} else {
  std::cout << "what a long name!" << std::endl;
}
This code is quite simple. You enter your name, and if it’s null, you’ll told it’s empty, if it’s less than four characters, you’ll be told it’s short and more than four characters you have a long name.

Now the same thing, but in Haskell:

name <- getLine
putStrLn $
  if null name
  then
    "woh, empty name!"
    else if length name < 4
    then
      "this is a short name you have"
    else
      "what a long name!"
You don’t have to insert as much newlines and indents I do, but I think it’s clearer that way. As you can see, the test is now done inside the value construction we’re passing to putStrLn, which takes a String and outputs it into your terminal. In Haskell, conditionals yields values. You could then write something like that:

a <- getLine
let r = if a == 0 then 314 else a
Haskell also has a lot of ways doing conditional execution. For instance, it has native support for partial function implementation depending on a predicate:

foo :: String -> IO ()
foo = putStrLn . go . length
  where
    go l
        | l == 0    = "woh, empty name!"
        | l < 4     = "this is a short name you have"
        | otherwise = "what a long name"
Those | are called guards in Haskell. We also have pattern-matching to implement a function differently depending on the “form” of its argument. For instance:

bar :: String -> String
bar []     = "empty string"
bar (x:[]) = "singleton string
bar _      = "long string"
I won’t go into complex details, but we also have monadic conditional function that enables us to do pretty things like:

testSomething :: IO ()
testSomething =
    runMaybeT $ do
      userInput <- lift getLine
      guard (length userInput /= 0)
      lift (putStrLn "ok!")
In that last snippet, if the user enters an empty string, the string “ok!” won’t be printed out.

As I already told you, abstractions are everywhere in FPL and especially in Haskell.

Error handling
One last example to give you an idea: exception handling. In C++, you have the keywords throw and catch and the type std::exception to inherits from. One big mistake a lot of folks do is thinking that error-prone code should be handled through exception. This is not really a good idea since exceptions are exceptional.

In Haskell, you can handle errors via dozens of ways. For instance, just like the testSomething function just above, you can use the Maybe type or use it as a monad. Even better, you can use its monadic transformer form, MaybeT, (what I did here), in order to extend your code with failable with no message errors code. Maybe express this concept: “maybe there’s nothing, maybe there’s just something”. And well, the Maybe type has two constructors. Try to guess what. Just, and Nothing. It can express a value that can be absent, ill, misformed, and so on.

For errors with message, we have a lovely type, monad and monad transformer with corresponding combinators: Either. It shows this: “I can have either this type or this one”. This is the theory about Either. Pratically, we don’t use it that way. We use this way: “it can has a right value, or a value that represents something wrong”. Two constructors for that type: Right, for a right value, and Left, for a wrong value.

If you just want to compute a value, and express a potential error with a String, you’re eligible to use an Either String a type! You could then do that:

testSomething :: IO ()
testSomething = do
    e <- runEitherT $ do
      userInput<- lift getLine
      when (length userInput == 0) . left $
        "oh god, error! empty string!"
      lift (putStrLn "ok!")
    either err skip e
stderr. Pretty straight-forward huh? :)

Conclusion
I could have written much more lines about comparison, but I think it’s quite enough for today. I’ll go into further details if you want.

The last thing to keep in mind is that all you can do in imperative languages can be achieved in FPL as well.

Kwak – the drama tellbot – was released in version 0.2.0.0!

I few days ago, I wrote a tellbot for my fellow French demosceners. It was designed to run on our public channel (#demofr on IRCNet). I wrote it Haskell and it took me something like only six hours to write it and release the version 0.1.0.0.

Only one hour after the first release was taking place a drama that’s split up the channel into smithereens. I won’t be going into any further details since everyone now knows what really happened in there. The important thing is that I fixed some issues – directly or indirectly related to that drama – and released another version.

Yesterday, I spent some time on changing the way the bot works. In the first place I designed it to be able to read your messages. If you want to leave a message to the user foo, you just had to type that in your favorite IRC client:

!tell foo I just wanted to tell you that […]

That message was delivered later when foo said something – no matter whether he’s already there, or joining, or whatever, he had to say something.

This behavior is quite convenient and I still think it’s a good thing. Some people have IRC autoconnect hooks when starting their machine so they can join their channels without any effort – I think it’s wrong but it’s another debate – but in fact they don’t read immediately messages nor hilights on IRC. So passing along messages when the recipient says something is a good idea. The drawback is about lurkers: if you’re lurking, and reading someone who’s using the !tell command to tell you something, if you don’t right answer back, and talk the day after that, you’ll be told the message anyway. This is normal to me, but might be quite annoying for some folks.

Then comes the 0.2.0.0 version of my tellbot. Now, the bot refuses to record your message if the recipient is already there, in the same room you and the bot are. It only enables you to leave a message to another folk if he’s not there. The message is delivered when the guy joins the channel.

You can find the source code here: