This story starts with me doing Advent of Code for the first time in my life. I hadn't written a single line of code for two years, busy, as I was, writing my [sci-fi novel](https://www.amazon.com/Wohpe-English-Rimmel-Salvatore-Sanfilippo-ebook/dp/B0BQ3HRDPF/). I wanted to start coding again, but without a project in my hands, what to do? The AoC puzzles helped quite a lot, at first, but they become repetitive and a bit futile quite soon. After completing day 13, a puzzle about comparing nested lists, I saw many other solutions resorting to `eval`. They are missing the point, I thought. To me, the puzzle seemed a hint at writing parsers for nested objects.
The gentle reader should be aware that I've a soft spot for [little languages](http://oldblog.antirez.com/page/picol.html). However, Picol was too much of a toy, while [Jim](http://jim.tcl.tk/index.html/doc/www/www/index.html) was too big as a coding example. Other than interpreters, I like writing small programs that serve as [examples](https://github.com/antirez/kilo) of how you could design bigger programs, while retaining a manageable size. Don't take me wrong: it's not like I believe my code should be taken as an example, it's just that I learned a lot from such small programs, so, from time to time, I like writing new ones, and while I'm at it I share them in the hope somebody could be interested. This time I wanted to obtain something of roughly the size of the Kilo editor, that is around ~1000 lines of code, showing the real world challenges arising when writing an actual interpreter for a programming language more complex than Picol. That's the result, and as a side effect I really started programming again: after Aocla I wrote more and more code, and now [I've a new project, too](https://github.com/antirez/protoview).
This README will first explain the language briefly. Later we will talk extensively about the implementation and its design. Without counting comments, the Aocla implementation is shorter than 1000 lines of code, and the core itself is around 500 lines (the rest of the code is the library implementation, the REPL, and other accessory parts). I hope you will find the code easy to follow, even if you are not used to C and to writing interpreters. I tried to keep all simple, as I always do when I write code, for myself and the others having the misfortune of reading or modifying it in the future.
Not every feature I desired to have is implemented, and certain data types, like the string type, lack any useful procedure to work with them. This choice was made in order to avoid making the source code more complex than needed, and also, on my side, to avoid writing too much useless code, given that this language will never be used in the real world. Besides, implementing some of the missing parts is a good exercise for the willing reader, assuming she or he are new to this kind of stuff. Even with all this limitations, it is possible to write small working programs with Aocla, and that's all we need for our goals.
Floating point numbers are not provided for simplicity (writing an implementation should not be too hard, and is a good exercise). Aocla programs are valid Aocla lists, so the language is [homoiconic](https://en.wikipedia.org/wiki/Homoiconicity). While Aocla is a stack-based language, like FORTH, Joy and Factor, it introduces the idea of *local variables capturing*. Because of this construct, Aocla programs look a bit different (and simpler to write and understand in my opinion) compared to other stack-based languages. Locals capturing is optional: any program using locals can be rewritten to avoid using them, yet the existence of this feature deeply affects the language in many ways.
Since all the programs must be valid lists, and thus are enclosed between `[` and `]`, both the Aocla CLI (Command Line Interface) and the execution of programs from files are designed to avoid needing the brackets. Aocla will put the program inside `[]` for you, so the above program should be written like that:
Programs are executed from left to right, *word by word*. If a word is not a symbol nor a tuple, its execution results into pushing its value on the stack. Symbols will produce a procedure call: the symbol name will be looked up in the table of procedures, and if a procedure with a matching name is found, it gets called. So the above program will perform the following steps:
*`5`: the value 5 is pushed on the stack. The stack will contain `(5)`.
*`dup`: is a symbol. A procedure called `dup` is looked up and executed. What `dup` does is to take the top value on the stack and duplicate it, so now the stack will contain `(5 5)`.
*`*`: is another symbol. The procedure is called. It will take the last two elements on the stack, check if they are integers, multiply them together and push the result on the stack. Now the stack will contain `(25)`.
If an Aocla word is a tuple, like `(x y)`, its execution has the effect of removing a corresponding number of elements from the stack and binding them to the local variables having the specified names:
10 20 (x y)
After the above program is executed, the stack will be empty and the local variables x and y will contain 10 and 20.
Finally, if an Aocla word is a symbol starting with the `$` character and a single additional character, the object stored at the specified variable is pushed on the stack. So the program to square 5 we wrote earlier can be rewritten as:
The ability to capture stack values into locals allow to make complex stack manipulations in a simple way, and makes programs more explicit to read and easier to write. Still locals have the remarkable quality of not making the language semantically more complex (if not for a small thing we will cover later -- search `upeval` inside this document if you want to know ASAP, but if you know the Tcl programming language, you already understood from the name, that is similar to Tcl's `uplevel`). In general, while locals help the handling of the stack in the local context of the procedure, words communicate via the stack, so the main advantages of stack-based languages are untouched.
*Note: why locals must have just single letter names? The only reason is to make the implementation of the Aocla interpreter simpler to understand. This way, we don't need to make use of any dictionary data structure. If I would design Aocla to be a real language, I would remove this limitation.*
We said that symbols normally trigger a procedure call. But symbols can also be pushed on the stack like any other value. To do so, symbols must be quoted, with the `'` character at the start.
The `printnl` procedure prints the last element on the stack and also prints a newline character, so the above program will just print `Hello` on the screen. You may wonder what's the point of quoting symbols. After all, you could just use strings, but later we'll see how this is important in order to write Aocla programs that write Aocla programs.
Symbol not bound to procedure: 'foobar' in unknown:0
## Working with lists
Lists are the central data structure of the language: they are used to represent programs and are useful as a general purpose data structure to represent data. So most of the very few built-in procedures that Aocla offers are lists manipulation procedures.
There are a few more list procedures. There is `get@` to get a specific element in a given position, `sort`, to sort a list, and if I remember correctly nothing
*Note: programs like the above show that, after all, maybe the `->` and `<-` operators should expect the arguments in reverse order. Maybe I'll change my mind.*
Certain times, programs that write programs can be quite useful. They are a
to make writing programs that write programs a lot simpler than the above. Anyway, as you saw earlier, when we implemented the `repeat` procedure, in Aocla
it is possible to do interesting stuff without using this programming paradigm.
1. We now use reference counting. When the object is allocated, it gets a *refcount* of 1. Then the functions `retain()` and `release()` are used in order to increment the reference count when we store the same object elsewhere, or when we want to remove a reference. Finally, when the references drop to zero, the object gets freed.
2. The object types now are all powers of two: single bits, in binary representation. This means we can store or pass multiple types at once in a single integer, just performing the *bitwise or*. It is useful in practice. No need for functions with a variable number of arguments just to pass many times at once.
3. There is information about the line number where a given object was defined in the source code. Aocla can be a toy, but a toy that will try to give you some stack trace if there is a runtime error.
Note that in this implementation, deeply nested data structures will produce many recursive calls. This can be avoided using *lazy freeing*, but that's not needed for something like Aocla. However some reader may want to search *lazy freeing* on the web.
Thanks to our parser (that is just a more complex version of the initial day 13 puzzle parser, and is not worth showing here), we can take an Aocla program, in the form of a string, parse it and get an Aocla object (`obj*` type) back. Now, in order to run an Aocla program, we have to *execute* this object. Stack based languages are particularly simple to execute: we just go form left to right, and depending on the object type, we do different actions:
* If the object is a symbol (and is not quoted, see the `quoted` field in the object structure), we try to lookup a procedure with that name, and if it exists we execute the procedure. How? By recursively executing the list bound to the symbol.
The function responsible to execute the program is called `eval()`, and is so short we can put it fully here, but I'll present the function split in different parts, to explain each one carefully. I will start showing just the first three lines, as they already tell us something.
Here there are three things going on. Eval() takes a context and a list. The list is our program, and it is scanned left-to-right, as Aocla programs are executed left to right, word by word. All should be clear but the context. What is an execution context for our program?
The stack frame has a pointer to the previous stack frame. This is useful both in order to implement `upeval` and to show a stack trace when an exception happens and the program is halted.
We can continue looking at the remaining parts of eval() now. We stopped at the `for` loop, so now we are inside the iteration doing something with each element of the list:
The essence of the loop is a `switch` statement doing something different depending on the object type. The object is just the current element of the list. The first case is the tuple. Tuples capture local variables, unless they are quoted like this:
So if the tuple is not quoted, we check if there are enough stack elements
according to the tuple length. Then, element after element, we move objects
from the Aocla stack to the stack frame, into the array representing the locals. Note that there could be already an object bound to a given local, so we `release()` it before the new assignment.
For symbols, as we did for tuples, we check if the symbol is quoted, an in such case we just push it on the stack. Otherwise, we handle two different cases. The above is the one where symbol names start with a `$`. It is, basically, the reverse operation of what we saw earlier in tuples capturing local vars. This time the local variable is transferred to the stack. However **we still take the reference** in the local variable array, as the program may want to push the same variable again and again, so, after pushing the object on the stack, we have to call `retain()` to increment the reference count of the object.
implementing a procedure, otherwise the procedure is *user defined*, that menas it is written in Aocla, and we need to evaluate it. We do this with a nested `eval()` call. As you can see, recursion is crucial in writing interpreters.
*A little digression: if we would like to speedup procedure call, we could cache the procedure lookup directly inside the symbol object. However in Aocla procedures can be redefined, so the next time the same procedure name may be bound to a different procedure. To still cache lookedup procedures, a simple way is to use the concept of "epoch". The context has a 64 bit integer called epoch, that is incremented every time a procedure is redefined. So, when we cache the procedure lookup into the object, we also store the current value of the epoch. Then, before using the cached value, we check if the epoch maches. If there is no match, we perform the lookup again, and update the cached procedure and the epoch.*
Sorry, let's go back to our `eval` function. Another important thing that's worth noting is that each new Aocla procedure call has its own set of local variables. The scope of local variables, in Aocla, is the lifetime of the procedure call, like in many other languages. So, in the code above, before calling an Aocla procedure we allocate a new stack frame using `newStackFrame()`, then we can finally call `eval()`, free the stack frame and store the old stack frame back in the context structure. Procedures implemented in C don't need a stack frame, as they will not make any use of Aocla local variables. The following is the last part of the `eval()` function implementation:
Here I cheated: the code required to implement each math procedure separately would be almost the same. So we bind all the operators to the same C function, and check the name of the procedure called inside a single implementation (see the above function). Here is where we register many procedures to the same C function.
So if the object is already not shared (its *refcount* is one), just return it as it is. Otherwise create a copy and remove a reference from the original object. Why, on copy, we need to remove a reference from the passed object? This may look odd at a first glance, but think at it: the invariant here should be that the caller of this function is the only owner of the object. We want the caller to be able to abstract totally what happens inside the `getUnsharedObject()` function. If the object was shared and we returned the caller a copy, the reference the caller had for the old object should be gone. Let's look at the following example:
I'll not show the `deepCopy()` function, it just allocates a new object of the specified type and copy the content. But guess what? It's a recursive function, too. That's why it is a *deep* copy.
That's it, and thanks for reading that far. To know more about interpreters you have only one thing to do: write your own, or radically modify Aocla in some crazy way. Get your hands dirty, it's super fun and rewarding. I can only promise that what you will learn will be worthwhile, even if you'll never write an interpreter again.
I believe the Fibonacci implementation written in Aocla, versus the implementation written in other stack-based languages, is quite telling about the jump forward in readability and usability provided by this simple feature:
[(n)
[$n 1 <=]
[
$n
]
[
$n 1 - fib
$n 2 - fib
+
] ifelse
] 'fib def
10 fib
printnl
So, while Aocla is a toy language, I believe this feature should be looked more carefully by actual stack-based language designers.
