Madin

Favorite Books

2024-01-05T00:00:00+00:00

Favorite Quotes

2023-12-30T00:00:00+00:00

“Computer Science is the continuation of logic by other means”

– Georg Gottlob

“Young man, in mathematics you don’t understand things. You just get used to them”

– John von Neumann

“God does not play dice with the universe”

– Albert Einstein

Algebra is the offer made by the devil to the mathematician. The devil says: “I will give you this powerful machine, it will answer any question you like. All you need to do is give me your soul: give up geometry and you will have this marvellous machine.”

– Sir Michael Atiyah

“All good problem solvers occasionally admit defeat. An important part of the problem solver’s art is knowing when to give up.”

– Paul Zeitz, The Art and Craft of Problem Solving

Code Generation

2023-09-28T00:00:00+00:00

Just give me the codez

Introduction#

Welcome to the fourth and final step in our journey. This step is what differentiates a compiled language from an interpreted one¹

In interpreted languages, the interpreter is responsible for executing the program. It does so by reading the source code, parsing it into an AST and executing the AST directly. JavaScript, Python and Ruby are examples of interpreted languages.

In compiled languages, the compiler reads the source code, parses it into an AST then transforms² the AST into an Intermediate Representations (IR), assembly code or directly into machine code. The end user can then run the resulting binary directly using their operating system. C, C++ and Rust are examples of compiled languages.

Low Level Virtual Machine (LLVM)#

Most compilers used to implement code generation directly, but as the number of platforms (Windows, OSx, *Nix) kept growing and the number of compute architectures became even larger (think x86_64, arch64 etc…), it became harder to maintain a single code generator that targets all platforms.

LLVM³ was born out of the need to solve this problem⁴. It is a collection of tools and libraries that can be used to build compiler backends.

Everything that we have worked on so far, i.e. the lexer, parser and type checker are part of the frontend of a compiler.

The frontend is distinct per programming language and implemented specifically for that language. The frontend is the thing that gives a language its “identity” and provides its unique features that separate it from other languages.

LLVM – High Level Overview#

I would love to sit here and give you an introduction to LLVM, but I think this article does a an amazing job at that, so please read it before continuing. It is sufficient to read up to and including the Types and Constants section.

Translating NTLC to LLVM IR#

Read LLVM Tutorial before proceeding

NTLC is a very simple language, so translating it to LLVM IR is not that hard.

We will translate an entire NTLC program into a function, conveniently named main that takes no arguments and returns an integer. The return value of the function will be the result of evaluating the NTLC program. This is very similar to how main works in most programming languages.

The function’s body – implementation – will be the result of translating each NTLC construct to LLVM IR.

Let’s go over each each construct in NTLC and see how it maps to LLVM IR.

Integer & Boolean Constants#

Integers and Booleans are the only two types in NTLC. In LLVM IR, integers are represented by the i32 type and booleans are represented by the i1 type. Very straightforward.

Successor & Predecessor#

succ and pred are both builtin functions in NTLC. They are used to increment and decrement integers respectively. We only need to define both of these functions once since they are kind of “global functions” that can invoked by NTLC code anywhere.

In LLVM IR, we can implement them roughly as:

define i32 @succ(i32 %x) {
  %1 = add i32 %x, 1
  ret i32 %1
}

and

define i32 @pred(i32 %x) {
  %1 = sub i32 %x, 1
  ret i32 %1
}

IsZero#

iszero is another builtin function in NTLC. It is used to check if an integer is zero or not. In LLVM IR, we can implement it as follows:

define i1 @izzero(i32 %x) {
  %1 = icmp eq i32 %x, 0
  ret i1 %1
}

Now that succ, pred and iszero are all defined, we can invoke (call) them in LLVM by using their name and the appropriate arguments.

An example invocation of succ would look like this:

%1 = call i32 @succ(i32 0)

Conditionals#

The final and most complex construct in NTLC is the conditional expression. It is used to alter the execution path of a program based on the value of a boolean expression. In LLVM IR, conditionals are implemented using labels and jumps.

Labels and jumps were very popular in the early days of programming, even when working with what was deemed “high level languages” back then. But, the code they can result in can be hard to read and thus not easy to maintain. So, most language introduced constructs such as for loops, while loops and if statements that abstract the developer from working with labels and jumps directly.

I say “abstract” because under the hood, compilers are still converting those language features to labels and jumps.

If you aren’t familiar with labels and jumps, here is a small example to illustrate the concept in pseudo code:

main() {
    // A label is just a name for a sequence of statements

    // Indented statements are "part of the label": they are executed
    // when the program "jumps" to execute the label

    // In that sense, a label is similar to a function
    // except that it doesn't take any arguments and doesn't return anything
    here: // <------- `here` is a label
        print("Hello World")

    // This is another label
    there:
        print("Goodbye World")

    if (true) {
        // This is a jump
        goto here
        // It causes the program to go and execute the label statements
        // The program then continues executing the statements after the label one by one
        // WE NEVER REACH THIS LINE
    } else {
        // This is another jump
        goto there
    }
}

If the previous code snippet was actual code, it would cause an infinite loop. The program would print “Hello World” forever.

Conditionals in LLVM IR#

Even with my understanding of labels and jumps, I had a hard time implementing them initially for NTLC. Why you might ask?

Because I could not see how would nested conditionals work! Let me illustrate with an example:

if true then if false then true else true else false

Here is the problem:

Label names must be unique
You cannot nest labels

You might be thinking: “Why not just generate a unique name for each label?”

The short answer is: You are right. I just got confused with the LLVM API.

When you translate source code to LLVM IR, you don’t simple map your source language constructs to LLVM IR. That would be too error prone and would require a lot of boilerplate code.

You use the LLVM API to create constructs that represent your code. So, I thought that I would have to maintain a lookup table for label names I have generated within a function, create a label, check if it exists in the table and if it does then create a new unique name for it. I would have to do so recursively as conditionals can be nested infinitely – at least in theory.

The solution turned out to be way simpler than I thought. When using the LLVM API to create a label, if the label doesn’t exist, LLVM will create it for you. If it does exist, LLVM will generate a new unique name for you and everything will work as expected.

That’s why the LLVM API gives you the context and IRBuilder types. They are used internally by LLVM to keep track of things like label names and other state.

If this still doesn’t make sense, don’t worry. It will become clearer when we look at the code.

LLVM Optimizations#

LLVM can be configured to perform optimizations on the generated IR. This is a very powerful feature that can result in significant performance improvements.

Those optimizations tend to “rewrite” your generated IR into a functionally equivalent but more efficient IR.

I have deliberately chosen to disable those optimizations, because:

NTLC is not really meant for any real world usage
And, this allows you (and me) to see how the generated LLVM instructions map to their corresponding NTLC constructs.

Putting it all together#

If you have cloned the ntlc repository and followed the instructions mentioned in the readme to install all the required dependencies, then running the compiler via cargo run would:

Compile the example file /example/good.ntlc
It would print the output of the lexer, parser and type checker and code generator to the console
Generate a binary called good in the /bin directory.

Try executing that binary on your operating system by running ./bin/good in your favorite shell. If everything went well you should not see any output. That’s because our NTLC programs do not have the capability to print to the console.

But in Bash, Zsh, PowerShell and other shells, one can check the exit code of the last executed command/program by running echo $?

Go ahead and do so and you will see the return value of the NTLC program. Try changing the NTLC program in /example/good.ntlc and see how the return value changes.

Congratulations! You have just written your first compiler!

Closing Thoughts#

That’s it. You have made it to the end of the journey. I hope you enjoyed it as much as I did and you actually learned something useful from it.

Where to go from here? If you want to learn more, then I would refer you to some of the resources I mentioned in the introduction article.

If you feel like you have a good grasp of most concepts and would like to challenge yourself a bit more, here are some ideas:

Add support for more types such as floating point numbers, strings, characters etc…
Add support for user defined functions
Add support for user defined types

The possibilities are endless really. Have fun and happy hacking!

Some interpreted languages perform some compilation to speed things up at execution time, and some even support both interpretation and compilation which makes it hard to draw a distinction
Visitor pattern is a common OOP pattern that is used to implement both interpretation and transformation of AST
Cranelift seems to be another option. It is written in Rust, but it was/is still new and not as well documented as LLVM at the time I looked into it
This is the same as the idea behind LSPs whereby a single Language Server can be used by multiple "frontend" client editors

Code#

use inkwell::builder::Builder;
use inkwell::context::Context;
use inkwell::module::{Linkage, Module};

use crate::parser::Term;

/**
 * `MainFunc` represents how our NTLC program
 * looks like after it's compiled to LLVM IR.
 *
 * It is a function that takes no arguments
 * and returns an integer.
 *
 * You might think, hey where the the boolean
 * return value?
 *
 * Well, on the OS level there is no such thing
 * Everything is represented as numbers.
 *
 * So we will map boolean values like true and false to 0 and 1 respectively.
 *
 * That does make it hard to distinguish between booleans and integers
 * such as when the program returns an integer 1 but since
 * our language is simple we don't need to worry about that.
 *
 * I mean after all this is just a shortcut not to link with libc
 * or provide a way to print to the console.
 */
pub type MainFunc = unsafe extern "C" fn() -> i32;

#[derive(Debug)]
pub enum CodegenError {
    LlvmNative(String),
}

/**
 * This module contains the internals of the code generator.
 *
 * I chose to wrap things in an isolated module because the code generator
 * needs to keep track of some state as part of translation process.
 *
 * This way consumers of the "codegen" module don't need to worry about
 */
mod internals {
    use inkwell::values::IntValue;

    use crate::parser::Term;

    use super::*;

    /**
     * `context`, `module` and `builder` are the state that the code generator
     * needs to keep track of.
     *
     * These are constructs provided by the LLVM library.
     */
    pub struct CodeGenerator<'ctx> {
        pub context: &'ctx Context,
        pub module: Module<'ctx>,
        pub builder: Builder<'ctx>,
    }

    /**
     * Here we are implementing the actual
     * functionality of the code generator
     *
     * These functions appear in an impl block
     * which makes them methods of the CodeGenerator struct
     * defined above.
     */
    impl<'ctx> CodeGenerator<'ctx> {
        /**
         * This function adds the builtin `izzero` function to the module.
         */
        fn add_builtin_izzero(&self) -> Result<(), CodegenError> {
            // This is a representation of the integer type in LLVM IR
            let i32_type = self.context.i32_type();

            // This is a representation of the boolean type in LLVM IR
            let bool_type = self.context.bool_type();

            // This is the type of the `izzero` function
            // It takes an integer and returns a boolean
            let iz_zero_fn_type = bool_type.fn_type(&[i32_type.into()], false);

            // Now we add the function declaration to the module
            // We give it the name "izzero"
            let iz_zero_fn =
                self.module
                    .add_function("izzero", iz_zero_fn_type, Some(Linkage::Internal));

            // Now we add a basic block to the function
            // A basic block is a sequence of instructions
            // that execute sequentially
            // This is where we implement our function's body
            let basic_block = self.context.append_basic_block(iz_zero_fn, "entry");

            self.builder.position_at_end(basic_block);

            // Now we get the value of the first parameter of the function
            // which is the integer we want to check if it's zero
            let x = iz_zero_fn
                .get_nth_param(0)
                .ok_or(CodegenError::LlvmNative(
                    "Failed to get nth param".to_string(),
                ))?
                .into_int_value();

            // Now we create a constant integer with the value 0
            let zero = i32_type.const_int(0, false);

            // Now we compare the value of the parameter with the constant zero
            let iz_zero =
                self.builder
                    .build_int_compare(inkwell::IntPredicate::EQ, x, zero, "iz_zero_cmp");

            // Now we return the result of the comparison
            self.builder.build_return(Some(&iz_zero));

            Ok(())
        }

        /**
         * This function adds the builtin `pred` function to the module.
         *
         * It follows a similar construction to the `izzero` function.
         */
        fn add_builtin_pred(&self) -> Result<(), CodegenError> {
            let i32_type = self.context.i32_type();

            let pred_fn_type = i32_type.fn_type(&[i32_type.into()], false);

            let pred_fn = self
                .module
                .add_function("pred", pred_fn_type, Some(Linkage::Internal));

            let basic_block = self.context.append_basic_block(pred_fn, "entry");

            self.builder.position_at_end(basic_block);

            let x = pred_fn
                .get_nth_param(0)
                .ok_or(CodegenError::LlvmNative(
                    "Failed to get nth param".to_string(),
                ))?
                .into_int_value();

            let constant_one = i32_type.const_int(1, false);

            let pred_value = self.builder.build_int_sub(x, constant_one, "minus_one");

            self.builder.build_return(Some(&pred_value));

            Ok(())
        }

        /**
         * This function adds the builtin `succ` function to the module.
         *
         * It follows a similar construction to the `izzero` function.
         */
        fn add_builtin_succ(&self) -> Result<(), CodegenError> {
            let i32_type = self.context.i32_type();

            let succ_fn_type = i32_type.fn_type(&[i32_type.into()], false);

            let succ_fn = self
                .module
                .add_function("succ", succ_fn_type, Some(Linkage::Internal));

            let basic_block = self.context.append_basic_block(succ_fn, "entry");

            self.builder.position_at_end(basic_block);

            let x = succ_fn
                .get_nth_param(0)
                .ok_or(CodegenError::LlvmNative(
                    "Failed to get nth param".to_string(),
                ))?
                .into_int_value();

            let constant_one = i32_type.const_int(1, false);

            let succ_value = self.builder.build_int_add(x, constant_one, "plus_one");

            self.builder.build_return(Some(&succ_value));

            Ok(())
        }

        /**
         * new is similar to constructors in Object Oriented Languages
         * It initializes a new LLVM module and injects our builtin
         * functions and constants into it.
         *
         * LLVM modules are the smallest unit of compilation.
         * You can think of them as a single source file in C or Rust.
         */
        pub fn new(ctx: &'ctx Context, module_name: &str) -> Result<Self, CodegenError> {
            let module = ctx.create_module(module_name);

            let code_generator = CodeGenerator {
                context: ctx,
                module,
                builder: ctx.create_builder(),
            };

            code_generator.add_builtin_izzero()?;
            code_generator.add_builtin_pred()?;
            code_generator.add_builtin_succ()?;

            Ok(code_generator)
        }

        /**
         * This is the main function of the code generator.
         * It takes an AST and compiles it to LLVM IR.
         *
         * It does it the same way as we did we the type checker
         * by recursively traversing the AST and compiling each node
         */
        fn compile_internal(&self, ast: Term) -> IntValue {
            match ast {
                /*
                 * Terminal nodes map to simple constants in LLVM
                 */
                Term::True => {
                    let bool_type = self.context.bool_type();
                    bool_type.const_int(1, false)
                }
                Term::False => {
                    let bool_type = self.context.bool_type();
                    bool_type.const_zero()
                }
                Term::Zero => {
                    let int_type = self.context.i32_type();
                    int_type.const_zero()
                }
                /*
                 * Here we are not building the `succ` function
                 * Remember, we already added it as a builtin function
                 *
                 * This is how we call a function in LLVM IR
                 */
                Term::Successor(inner_term) => {
                    // First we get the function from the module
                    let succ_fn = self
                        .module
                        .get_function("succ")
                        .ok_or(CodegenError::LlvmNative(
                            "Failed to get succ function".to_string(),
                        ))
                        .unwrap();

                    // Then we compile the inner term
                    // which could be any valid NTLC term
                    let succ_value = self.compile_internal(*inner_term);

                    // Then we call the function with the value of the inner term
                    let succ_call =
                        self.builder
                            .build_call(succ_fn, &[succ_value.into()], "succ_call");

                    // Then we get the return value of the function call
                    let succ_call_value = succ_call.try_as_basic_value().left().unwrap();

                    // Then we return the value
                    succ_call_value.into_int_value()
                }
                // Same as above
                Term::Predecessor(inner_term) => {
                    let pred_fn = self
                        .module
                        .get_function("pred")
                        .ok_or(CodegenError::LlvmNative(
                            "Failed to get pred function".to_string(),
                        ))
                        .unwrap();

                    let pred_value = self.compile_internal(*inner_term);

                    let pred_call =
                        self.builder
                            .build_call(pred_fn, &[pred_value.into()], "pred_call");

                    let pred_call_value = pred_call.try_as_basic_value().left().unwrap();

                    pred_call_value.into_int_value()
                }
                // Same as above
                Term::IsZero(inner_term) => {
                    let iz_zero_fn = self
                        .module
                        .get_function("izzero")
                        .ok_or(CodegenError::LlvmNative(
                            "Failed to get izzero function".to_string(),
                        ))
                        .unwrap();

                    let iz_zero_value = self.compile_internal(*inner_term);

                    let iz_zero_call = self.builder.build_call(
                        iz_zero_fn,
                        &[iz_zero_value.into()],
                        "iz_zero_call",
                    );

                    let iz_zero_call_value = iz_zero_call.try_as_basic_value().left().unwrap();

                    iz_zero_call_value.into_int_value()
                }
                /*
                 * Conditionals are a bit more involved
                 * If you have seen goto statements before they might seem familiar
                 *
                 * Essentially we are creating a "label" for each possible
                 * branch of the conditional.
                 *
                 * And then we use jump instructions to execute one or the other
                 * depending on the value of the condition.
                 */
                Term::Conditional {
                    condition,
                    consequence,
                    alternative,
                } => {
                    let condition_value = self.compile_internal(*condition);

                    let bool_type = self.context.bool_type();

                    let true_value = bool_type.const_int(1, false);

                    let condition_cmp = self.builder.build_int_compare(
                        inkwell::IntPredicate::EQ,
                        condition_value,
                        true_value,
                        "condition_cmp",
                    );

                    let then_block = self.context.append_basic_block(
                        self.builder
                            .get_insert_block()
                            .unwrap()
                            .get_parent()
                            .unwrap(),
                        "then",
                    );

                    let else_block = self.context.append_basic_block(
                        self.builder
                            .get_insert_block()
                            .unwrap()
                            .get_parent()
                            .unwrap(),
                        "else",
                    );

                    let merge_block = self.context.append_basic_block(
                        self.builder
                            .get_insert_block()
                            .unwrap()
                            .get_parent()
                            .unwrap(),
                        "merge",
                    );

                    self.builder
                        .build_conditional_branch(condition_cmp, then_block, else_block);

                    self.builder.position_at_end(then_block);

                    let consequence_value = self.compile_internal(*consequence);

                    self.builder.build_unconditional_branch(merge_block);

                    let then_block = self.builder.get_insert_block().unwrap();

                    self.builder.position_at_end(else_block);

                    let alternative_value = self.compile_internal(*alternative);

                    self.builder.build_unconditional_branch(merge_block);

                    let else_block = self.builder.get_insert_block().unwrap();

                    self.builder.position_at_end(merge_block);

                    /*
                       phi is a special construction in LLVM
                       that tells us which branch of the conditional
                       was executed to get us to this point
                    */
                    let phi = self.builder.build_phi(self.context.i32_type(), "phi");

                    phi.add_incoming(&[
                        (&consequence_value, then_block),
                        (&alternative_value, else_block),
                    ]);

                    phi.as_basic_value().into_int_value()
                }
                _ => {
                    let int_type = self.context.i32_type();
                    int_type.const_int(0, false)
                }
            }
        }

        /*
         * This is the public interface of the code generator.
         * It takes an AST and compiles it to LLVM IR.
         *
         * We basically we just create a main function
         * and we put compile our NTLC program to be the body
         * of that function
         *
         * The result of evaluating the NTLC program is the return value
         * of the main function
         */
        pub fn compile(&self, ast: Term) -> Result<(), CodegenError> {
            let i32_type = self.context.i32_type();
            let main_fn_type = i32_type.fn_type(&[], false);

            let main_fn = self
                .module
                .add_function("main", main_fn_type, Some(Linkage::External));

            let basic_block = self.context.append_basic_block(main_fn, "entry");

            self.builder.position_at_end(basic_block);

            let return_value = self.compile_internal(ast);

            self.builder.build_return(Some(&return_value));

            Ok(())
        }
    }
}

/**
 * This is the only function exported from the code generator
 * Check `main.rs` to see how it's used.
 *
 * As you can see, it delegates all of its work to the code generator module
 * defined above.
 */
pub fn generate(context: &Context, ast: Term) -> Result<Module<'_>, CodegenError> {
    let codegen = internals::CodeGenerator::new(context, "main")?;

    codegen.compile(ast)?;

    Ok(codegen.module)
}

#[cfg(test)]
mod code_generator_tests {
    use inkwell::{execution_engine::JitFunction, OptimizationLevel};

    use crate::{lexer::scan, parser::parse};

    use super::*;

    #[test]
    fn test_codegen_true() {
        let input = "true";

        let tokens = scan(input).unwrap();

        let ast = parse(tokens).unwrap();

        let context = Context::create();

        let codegen = internals::CodeGenerator::new(&context, "test_codegen_true").unwrap();

        codegen.compile(ast).unwrap();

        let execution_engine = codegen
            .module
            .create_jit_execution_engine(OptimizationLevel::None)
            .unwrap();

        let main =
            unsafe { execution_engine.get_function("main").unwrap() as JitFunction<MainFunc> };

        unsafe {
            assert_eq!(main.call(), 1);
        }
    }

    #[test]
    fn test_codegen_false() {
        let input = "false";

        let tokens = scan(input).unwrap();

        let ast = parse(tokens).unwrap();

        let context = Context::create();

        let codegen = internals::CodeGenerator::new(&context, "test_codegen_false").unwrap();

        codegen.compile(ast).unwrap();

        let execution_engine = codegen
            .module
            .create_jit_execution_engine(OptimizationLevel::None)
            .unwrap();

        let main =
            unsafe { execution_engine.get_function("main").unwrap() as JitFunction<MainFunc> };

        unsafe {
            assert_eq!(main.call(), 0);
        }
    }

    #[test]
    fn test_codegen_zero() {
        let input = "0";

        let tokens = scan(input).unwrap();

        let ast = parse(tokens).unwrap();

        let context = Context::create();

        let codegen = internals::CodeGenerator::new(&context, "test_codegen_zero").unwrap();

        codegen.compile(ast).unwrap();

        let execution_engine = codegen
            .module
            .create_jit_execution_engine(OptimizationLevel::None)
            .unwrap();

        let main =
            unsafe { execution_engine.get_function("main").unwrap() as JitFunction<MainFunc> };

        unsafe {
            assert_eq!(main.call(), 0);
        }
    }

    #[test]
    fn test_codegen_succ() {
        let input = "succ 0";

        let tokens = scan(input).unwrap();

        let ast = parse(tokens).unwrap();

        let context = Context::create();

        let codegen = internals::CodeGenerator::new(&context, "test_codegen_succ").unwrap();

        codegen.compile(ast).unwrap();

        let execution_engine = codegen
            .module
            .create_jit_execution_engine(OptimizationLevel::None)
            .unwrap();

        let main =
            unsafe { execution_engine.get_function("main").unwrap() as JitFunction<MainFunc> };

        unsafe {
            assert_eq!(main.call(), 1);
        }
    }

    #[test]
    fn test_codegen_pred() {
        let input = "pred 0";

        let tokens = scan(input).unwrap();

        let ast = parse(tokens).unwrap();

        let context = Context::create();

        let codegen = internals::CodeGenerator::new(&context, "test_codegen_pred").unwrap();

        codegen.compile(ast).unwrap();

        let execution_engine = codegen
            .module
            .create_jit_execution_engine(OptimizationLevel::None)
            .unwrap();

        let main =
            unsafe { execution_engine.get_function("main").unwrap() as JitFunction<MainFunc> };

        unsafe {
            assert_eq!(main.call(), -1);
        }
    }

    #[test]
    fn test_codegen_izzero() {
        let input = "iszero 0";

        let tokens = scan(input).unwrap();

        let ast = parse(tokens).unwrap();

        let context = Context::create();

        let codegen = internals::CodeGenerator::new(&context, "test_codegen_izzero").unwrap();

        codegen.compile(ast).unwrap();

        let execution_engine = codegen
            .module
            .create_jit_execution_engine(OptimizationLevel::None)
            .unwrap();

        let main =
            unsafe { execution_engine.get_function("main").unwrap() as JitFunction<MainFunc> };

        unsafe {
            assert_eq!(main.call(), 1);
        }
    }

    #[test]
    fn test_codegen_if_then_branch() {
        let input = "if true then true else false";

        let tokens = scan(input).unwrap();

        let ast = parse(tokens).unwrap();

        let context = Context::create();

        let codegen = internals::CodeGenerator::new(&context, "test_codegen_if").unwrap();

        codegen.compile(ast).unwrap();

        let execution_engine = codegen
            .module
            .create_jit_execution_engine(OptimizationLevel::None)
            .unwrap();

        let main =
            unsafe { execution_engine.get_function("main").unwrap() as JitFunction<MainFunc> };

        unsafe {
            assert_eq!(main.call(), 1);
        }
    }

    #[test]
    fn test_codegen_if_else_branch() {
        let input = "if false then true else false";

        let tokens = scan(input).unwrap();

        let ast = parse(tokens).unwrap();

        let context = Context::create();

        let codegen = internals::CodeGenerator::new(&context, "test_codegen_if").unwrap();

        codegen.compile(ast).unwrap();

        let execution_engine = codegen
            .module
            .create_jit_execution_engine(OptimizationLevel::None)
            .unwrap();

        let main =
            unsafe { execution_engine.get_function("main").unwrap() as JitFunction<MainFunc> };

        unsafe {
            assert_eq!(main.call(), 0);
        }
    }

    #[test]
    fn test_codegen_nested_successors() {
        let input = "succ(succ(0))";

        let tokens = scan(input).unwrap();

        let ast = parse(tokens).unwrap();

        let context = Context::create();

        let codegen = internals::CodeGenerator::new(&context, "test_codegen_nested_succ").unwrap();

        codegen.compile(ast).unwrap();

        let execution_engine = codegen
            .module
            .create_jit_execution_engine(OptimizationLevel::None)
            .unwrap();

        let main =
            unsafe { execution_engine.get_function("main").unwrap() as JitFunction<MainFunc> };

        unsafe {
            assert_eq!(main.call(), 2);
        }
    }

    #[test]
    fn test_codegen_nested_predecessors() {
        let input = "pred(pred(0))";

        let tokens = scan(input).unwrap();

        let ast = parse(tokens).unwrap();

        let context = Context::create();

        let codegen = internals::CodeGenerator::new(&context, "test_codegen_nested_pred").unwrap();

        codegen.compile(ast).unwrap();

        let execution_engine = codegen
            .module
            .create_jit_execution_engine(OptimizationLevel::None)
            .unwrap();
        let main =
            unsafe { execution_engine.get_function("main").unwrap() as JitFunction<MainFunc> };

        unsafe {
            assert_eq!(main.call(), -2);
        }
    }

    #[test]
    fn test_nested_if() {
        let input = "if true then if false then false else true else false";

        let tokens = scan(input).unwrap();

        let ast = parse(tokens).unwrap();

        let context = Context::create();

        let codegen = internals::CodeGenerator::new(&context, "test_codegen_nested_if").unwrap();

        codegen.compile(ast).unwrap();

        let execution_engine = codegen
            .module
            .create_jit_execution_engine(OptimizationLevel::None)
            .unwrap();
        let main =
            unsafe { execution_engine.get_function("main").unwrap() as JitFunction<MainFunc> };

        unsafe {
            assert_eq!(main.call(), 1);
        }
    }

    #[test]
    fn test_complex_nesting() {
        let input = "if iszero(pred(succ(0))) then succ(succ(0)) else pred(pred(0))";

        let tokens = scan(input).unwrap();

        let ast = parse(tokens).unwrap();

        let context = Context::create();

        let codegen =
            internals::CodeGenerator::new(&context, "test_codegen_complex_nesting").unwrap();

        codegen.compile(ast).unwrap();

        let execution_engine = codegen
            .module
            .create_jit_execution_engine(OptimizationLevel::None)
            .unwrap();
        let main =
            unsafe { execution_engine.get_function("main").unwrap() as JitFunction<MainFunc> };

        unsafe {
            assert_eq!(main.call(), 2);
        }
    }

    #[test]
    fn test_most_complex_example() {
        let input = "if iszero(pred(succ(0))) then if iszero(pred(0)) then succ(succ(0)) else pred(pred(pred(0))) else succ(pred(pred(0)))";

        let tokens = scan(input).unwrap();

        let ast = parse(tokens).unwrap();

        let context = Context::create();

        let codegen =
            internals::CodeGenerator::new(&context, "test_codegen_complex_nesting").unwrap();

        codegen.compile(ast).unwrap();

        let execution_engine = codegen
            .module
            .create_jit_execution_engine(OptimizationLevel::None)
            .unwrap();
        let main =
            unsafe { execution_engine.get_function("main").unwrap() as JitFunction<MainFunc> };

        unsafe {
            assert_eq!(main.call(), -3);
        }
    }
}

#[cfg(test)]
mod code_generator_builtins_tests {
    use inkwell::{execution_engine::JitFunction, OptimizationLevel};

    use super::*;

    type IzZeroFunc = unsafe extern "C" fn(i32) -> bool;
    type SuccFunc = unsafe extern "C" fn(i32) -> i32;
    type PredFunc = unsafe extern "C" fn(i32) -> i32;

    #[test]
    fn test_builtin_izzero() {
        let context = Context::create();

        let codegen = internals::CodeGenerator::new(&context, "test_builtin_izzero").unwrap();

        let execution_engine = codegen
            .module
            .create_jit_execution_engine(OptimizationLevel::None)
            .unwrap();

        let iszero =
            unsafe { execution_engine.get_function("izzero").unwrap() as JitFunction<IzZeroFunc> };

        unsafe {
            assert!(!(iszero.call(1)));
            assert!(iszero.call(0))
        }
    }

    #[test]
    fn test_builtin_succ() {
        let context = Context::create();

        let codegen = internals::CodeGenerator::new(&context, "test_builtin_succ").unwrap();

        let execution_engine = codegen
            .module
            .create_jit_execution_engine(OptimizationLevel::None)
            .unwrap();

        let succ =
            unsafe { execution_engine.get_function("succ").unwrap() as JitFunction<SuccFunc> };

        unsafe {
            assert_eq!(succ.call(0), 1);
        }
    }

    #[test]
    fn test_builtin_pred() {
        let context = Context::create();

        let codegen = internals::CodeGenerator::new(&context, "test_builtin_pred").unwrap();

        let execution_engine = codegen
            .module
            .create_jit_execution_engine(OptimizationLevel::None)
            .unwrap();

        let pred =
            unsafe { execution_engine.get_function("pred").unwrap() as JitFunction<PredFunc> };

        unsafe {
            assert_eq!(pred.call(0), -1);
        }
    }
}

Language Server Protocol

2023-09-24T00:00:00+00:00

Just give me the codez: VSCode Client Side and Server

Intro#

I assume you’ve probably worked before in an IDE environment or a “smart” editor such as VSCode, Vim, Emacs or IntelliJ just to name a few.

All of these editors provide you with smart programming language features such as syntax highlighting (coloring), auto-completion, go to definition, find all references, rename symbol, etc.

Back in the day, each editor had to implement these features for each programming language. This was a lot of work and it was hard to keep up with the latest features of each language.

So, one day, a group of people got together and decided to create a protocol that would allow editors to support many languages without having to implement support for each of them over and over again for each editor.

That protocol is the Language Server Protocol (LSP).

Overview#

The way one decouples an editor from a language is by:

Having a server that implements the LSP for the language
- The server is written once using whatever language the author desires
- The server is responsible for providing the language features. It has intimate knowledge of the language
- The different editors can use the same server because all parties agree on the protocol
Each editor or IDE provides a mechanism to implement a client. Most IDEs do this by providing a plugin mechanism
- The client is usually responsible for starting the server
- The client is responsible for communicating with the server
- The client is usually written per editor

The Protocol#

LSP is a JSON-RPC based protocol:

JSON because the messages exchanged back and forth between the client and server are formatted as JSON text
RPC stands for Remote Procedure Call which means the client and server communicate by calling functions on each other.

The editor typically sends a request to the server, e.g. user opened a file, and the server responds with something that is appropriate for that particular message, e.g. the the server might choose to analyze the file and return a list of grammatical errors to the editor.

The protocol essentially defines all sorts of supported messages, what kind of fields each message has and other features of the protocol.

Implementing the Protocol#

Implementing the protocol, at least the server side of it, means you have to be able to parse the messages sent from the editor: basically deserializing the JSON payload and converting it to a class, struct or whatever in-memory representation that is suitable for your language – the one you’re writing the LSP server in, not the one you’re writing the server for.

The same has to happen when you’re sending the response back to client - editor.

As you can imagine, this is repetitive and people don’t implement it everyday because the protocol does not change much. Also, that’s not really the interesting part of the server. You want to spend your time providing language features, not implementing the protocol details.

So, typically there are libraries that implement the protocol for you. You just have to provide the actual language features. For my use case and since I used Rust to implement the compiler, I decided ot use Tower LSP to write the server so that I can reuse part of the compiler code.

Tower LSP#

Tower LSP is a Rust library that implements the LSP protocol. It basically gives you an interface and you have to fill in the methods that implement the relevant language features for your language, NTLC in our case.

For example, if the user hovers over a variable, the editor will send a request to the server asking for the type of the variable.

Tower LSP will receive that message, validate it, deserialize it and then call the method you implemented to handle that specific request. You job is to focus on implementing the logic that retrieves the type of said variable.

Syntax Highlighting#

Syntax highlighting is something that is usually also implemented by the server, but some editors like VSCode allow you to supply your syntax highlighting rules in another form.

For NTLC, this is implemented here

The reason editors offer this alternative mechanism in addition to language servers is mainly performance. When the user opens a file, the editor can highlight (color) almost instantly without having to wait for the the extension/plugin to activate the language server and for the language server to analyze the file and send the results back.

When the server does finally start and analyzes the file, it can contribute more “accurate” semantic syntax highlighting.

NTLC is very simple and so there is no need for the server to provide any “semantic” highlighting.

The Hard Parts#

Two things:

You want the server to be super fast and have as little latency as possible: as you can imagine, when working in a project with many files (some which might be thousands of lines) where you are constantly switching between them, inserting, editing and deleting text; the editor will be sending the server a message about each of those events and the server has to, in many cases, re-scan the source code, re-parse and type check it and maybe even more.
You optimally want the server to be able to do all of that incrementally, i.e. without having to read the entire file. You can opt-in and tell VSCode to send only information about exactly what had changed in the file. This is called incremental synchronization. But that also means the server has to be able to figure out what to re-analyze. So servers need to keep state and cache intelligently.

In our implementation we do neither of those things:

NTLC is a small language that I used to learn all of the stuff I have been writing about in this series
NTLC programs are single line programs. There is barely any performance overhead
Implementing incremental lexing, parsing and type checking is something you have to consider when you start out. Adding them later is not easy. I did not know of them when I started and so I didn’t implement them.

Code#

The code is made out of two parts:

The client side: essentially a VSCode extension. NTLC is super simple. The only thing the extension does is start the corresponding server when the user is working on a file that has an .ntlc extension.
The server side: a Rust tower-lsp server. I chose to support only two features:
- Inlay hints: show the type of the entire program
- Diagnostics: show errors (lexing, parsing or type checking errors) in the code

Notice that in the server implementation we reuse bits of the compiler code, namely the lexer, parser and type-checker. This is the main reason I decided to use Rust to implement the language server.

VSCode TypeScript Client#

/* --------------------------------------------------------------------------------------------
 * Copyright (c) Microsoft Corporation. All rights reserved.
 * Licensed under the MIT License. See License.txt in the project root for license information.
 * ------------------------------------------------------------------------------------------ */

import { workspace, window } from "vscode";

import {
  Executable,
  LanguageClient,
  LanguageClientOptions,
  ServerOptions,
} from "vscode-languageclient/node";

let client: LanguageClient;

export async function activate() {
  const traceOutputChannel = window.createOutputChannel(
    "NTLC Language Server trace"
  );
  const command = process.env.SERVER_PATH || "ntlc_lsp";
  const run: Executable = {
    command,
    options: {
      env: {
        ...process.env,
        RUST_LOG: "debug",
      },
    },
  };
  const serverOptions: ServerOptions = {
    run,
    debug: run,
  };
  // If the extension is launched in debug mode then the debug server options are used
  // Otherwise the run options are used
  // Options to control the language client
  let clientOptions: LanguageClientOptions = {
    // Register the server for plain text documents
    documentSelector: [{ scheme: "file", language: "ntlc" }],
    synchronize: {
      // Notify the server about file changes to '.clientrc files contained in the workspace
      fileEvents: workspace.createFileSystemWatcher("**/.ntlc"),
    },
    traceOutputChannel,
  };

  traceOutputChannel.appendLine(
    `Starting NTLC Language Server: [${process.env.SERVER_PATH}]`
  );
  // Create the language client and start the client.
  client = new LanguageClient(
    "ntlc_lsp",
    "NTLC language server",
    serverOptions,
    clientOptions
  );
  client.start();
}

export function deactivate(): Thenable<void> | undefined {
  if (!client) {
    return undefined;
  }
  return client.stop();
}

Server#

/**
 * This code was adopted from example code from the tower-lsp crate.
 * https://github.com/ebkalderon/tower-lsp
 */
use dashmap::DashMap;
use ntlcc::parser::parse;
use ntlcc::type_checker::TypedTerm;
use serde::{Deserialize, Serialize};
use tower_lsp::jsonrpc::Result;
use tower_lsp::lsp_types::notification::Notification;
use tower_lsp::lsp_types::*;
use tower_lsp::{Client, LanguageServer, LspService, Server};

/**
 * There is many Rust constructs that are new in this file.
 * Ignore them. I didn't understand every bit when I started
 *
 * Essentially Rust doesn't support async/await completely on its own
 * You need a runtime and that runtime is provided by Tokio (https://tokio.rs/)
 * in this project
 *
 * TowerLSP is a Rust library that depends on Tokio
 *
 * What does it do you might ask?
 *
 * Essentially language server protocol is a client-server protocol
 * that allows you to write a language server that can be used by
 * any editor that supports the protocol
 *
 * The protocol is defined here: https://microsoft.github.io/language-server-protocol/specifications/specification-current/
 * It uses JSON RPC to exchange messages between the client and the server
 * Just like how the web uses HTTP to exchange messages between the browser and the server
 *
 * So instead of us worrying about implementing the protocol and its details
 * we can just use TowerLSP to do that for us
 *
 * TowerLSP leaves "blanks" for us to fill in.
 * Like what to do when the code editor sends us a message
 * saying the user hovered over a piece of code
 *
 * Or what to do when the code editor sends us a message
 * saying user opened a file
 *
 * You can see the list of all supported messages that the code editor can send
 * in the protocol documentation
 */

#[derive(Debug)]
struct Backend {
    // We use the client to talk back to the editor/IDE
    client: Client,

    // This is a map that maps a file path to its AST
    // We use this to get the AST of a file when the editor
    // asks us for inlay or diagnostics
    ast_map: DashMap<String, TypedTerm>,
}

#[tower_lsp::async_trait]
impl LanguageServer for Backend {
    /*
        Here we initialize the language server
        We tell the editor what features the language server supports
    */
    async fn initialize(&self, _: InitializeParams) -> Result<InitializeResult> {
        Ok(InitializeResult {
            server_info: Some(ServerInfo {
                name: "NTLC-LSP".to_string(),
                version: Some("0.0.1".to_string()),
            }),
            offset_encoding: None,
            capabilities: ServerCapabilities {
                inlay_hint_provider: Some(OneOf::Left(true)),
                text_document_sync: Some(TextDocumentSyncCapability::Kind(
                    TextDocumentSyncKind::FULL,
                )),
                workspace: Some(WorkspaceServerCapabilities {
                    workspace_folders: Some(WorkspaceFoldersServerCapabilities {
                        supported: Some(true),
                        change_notifications: Some(OneOf::Left(true)),
                    }),
                    file_operations: None,
                }),
                ..ServerCapabilities::default()
            },
        })
    }

    async fn initialized(&self, _: InitializedParams) {
        self.client
            .log_message(MessageType::INFO, "initialized!")
            .await;
    }

    async fn shutdown(&self) -> Result<()> {
        Ok(())
    }

    async fn did_open(&self, params: DidOpenTextDocumentParams) {
        self.client
            .log_message(MessageType::INFO, "file opened!")
            .await;

        self.on_change(TextDocumentItem {
            uri: params.text_document.uri,
            text: params.text_document.text,
            version: params.text_document.version,
        })
        .await
    }

    async fn did_change(&self, mut params: DidChangeTextDocumentParams) {
        self.on_change(TextDocumentItem {
            uri: params.text_document.uri,
            text: std::mem::take(&mut params.content_changes[0].text),
            version: params.text_document.version,
        })
        .await
    }

    async fn did_save(&self, _: DidSaveTextDocumentParams) {
        self.client
            .log_message(MessageType::INFO, "file saved!")
            .await;
    }

    async fn did_close(&self, _: DidCloseTextDocumentParams) {
        self.client
            .log_message(MessageType::INFO, "file closed!")
            .await;
    }

    /**
     * We are using inlay hints to show the type of our entire
     * NTLC program
     */
    async fn inlay_hint(
        &self,
        params: tower_lsp::lsp_types::InlayHintParams,
    ) -> Result<Option<Vec<InlayHint>>> {
        self.client
            .log_message(MessageType::INFO, "inlay hint")
            .await;

        let uri = &params.text_document.uri;

        if let Some(ast) = self.ast_map.get(uri.as_str()) {
            let term = [ast.value()];

            let inlay_hint_list = term
                .iter()
                .map(|v| {
                    (
                        0,
                        u32::MAX,
                        match v {
                            ntlcc::type_checker::TypedTerm::Boolean(_) => "BOOL".to_string(),
                            ntlcc::type_checker::TypedTerm::Integer(_) => "INT".to_string(),
                            ntlcc::type_checker::TypedTerm::Void => "VOID".to_string(),
                        },
                    )
                })
                .map(|item| InlayHint {
                    text_edits: None,
                    tooltip: None,
                    kind: Some(InlayHintKind::TYPE),
                    padding_left: None,
                    padding_right: None,
                    data: None,
                    position: Position {
                        line: 0,
                        character: item.1,
                    },
                    label: InlayHintLabel::LabelParts(vec![InlayHintLabelPart {
                        value: item.2,
                        tooltip: None,
                        location: Some(Location {
                            uri: params.text_document.uri.clone(),
                            range: Range {
                                start: Position::new(0, 0),
                                end: Position::new(0, u32::MAX),
                            },
                        }),
                        command: None,
                    }]),
                })
                .collect::<Vec<_>>();

            return Ok(Some(inlay_hint_list));
        }

        Ok(None)
    }

    async fn did_change_configuration(&self, _: DidChangeConfigurationParams) {
        self.client
            .log_message(MessageType::INFO, "configuration changed!")
            .await;
    }

    async fn did_change_workspace_folders(&self, _: DidChangeWorkspaceFoldersParams) {
        self.client
            .log_message(MessageType::INFO, "workspace folders changed!")
            .await;
    }

    async fn did_change_watched_files(&self, _: DidChangeWatchedFilesParams) {
        self.client
            .log_message(MessageType::INFO, "watched files have changed!")
            .await;
    }
}
#[derive(Debug, Deserialize, Serialize)]
struct InlayHintParams {
    path: String,
}

enum CustomNotification {}

impl Notification for CustomNotification {
    type Params = InlayHintParams;
    const METHOD: &'static str = "custom/notification";
}
struct TextDocumentItem {
    uri: Url,
    text: String,
    version: i32,
}

impl Backend {
    /**
     * Whenever a file is opened, changed, saved, etc by the user
     * the editor sends us a message
     *
     * We use this message to to lex, parse, and type check the program
     * and then publish the diagnostics to the editor
     *
     * We also store the AST of the program in the ast_map
     * This is a performance optimization
     *
     * We don't really need this in NTLC because our programs are small
     */
    async fn on_change(&self, params: TextDocumentItem) {
        let program_source = params.text.trim();

        let lex_result = ntlcc::lexer::scan(program_source).map_err(|e| e.to_string());

        let parse_result = lex_result.and_then(|tokens| parse(tokens).map_err(|e| e.to_string()));

        let type_checker_result = parse_result
            .and_then(|ast| ntlcc::type_checker::infer(&ast).map_err(|e| e.to_string()));

        let (ast, errors) = match type_checker_result {
            Ok(ast) => (Some(ast), vec![]),
            Err(e) => (None, vec![e]),
        };

        let diagnostics = errors
            .into_iter()
            .map(|item| {
                let start_position = Position {
                    line: 0,
                    character: 0,
                };
                let end_position = Position {
                    line: 0,
                    character: u32::MAX,
                };
                Diagnostic::new_simple(Range::new(start_position, end_position), item)
            })
            .collect();

        self.client
            .log_message(MessageType::INFO, "before publishing diagnostics")
            .await;

        self.client
            .publish_diagnostics(params.uri.clone(), diagnostics, Some(params.version))
            .await;

        if let Some(ast) = ast {
            self.ast_map.insert(params.uri.to_string(), ast);
        }
    }
}

#[tokio::main]
async fn main() {
    env_logger::init();

    let stdin = tokio::io::stdin();
    let stdout = tokio::io::stdout();

    let (service, socket) = LspService::build(|client| Backend {
        client,
        ast_map: DashMap::new(),
    })
    .finish();

    Server::new(stdin, stdout, socket).serve(service).await;
}

Type Checker

2023-09-23T00:00:00+00:00

Just give me the codez

Expectation vs Reality#

The type checker was the part I was mostly excited about when I decided to start this project.

Why you might ask? there are a couple of reasons:

Every time I sat off to write a compiler, I stopped at the parsing phase.
Most courses on compilers usually stop here and instead implement an interpreter.

Spoiler alert though: our type checker is very basic. I did not realize that when I started off but for a type checker to do any meaningful work the language needs to, at the least, have variables. NTLC does not, and so there is barely any work for the type checker to do.

Before we start though, let’s clear up some terminology.

Type Checking vs Type Inference#

The difference is actually quite easy to explain. Let’s do that with a JavaScript example and pretend as if JavaScript had type inference built in.

const x = 1;
const y = x + 3;

Notice that in the previous example, we did not have to specify the type of x nor y. The compiler would infer that x is an integer because it is assigned the value 1, and so y is also an integer since it is equal to INTEGER + INTEGER.

Notice that JavaScript does not have type inference and the previous example would still have ran even if you did something like:

const x = 1;
const y = "hello" + 3;

This is not about type inference but about static vs dynamic typing, which we will get to in a bit.

You might think all of this is very obvious, and it is in this silly example, but this can get really tricky as you add functions and more complex types to a language¹.

If JavaScript was statically typed, then we would have to specify the types of x and y explicitly.

// Fictitious JavaScript with static typing
const x: number = 1;
const y: number = x + 3;

Now, the compiler would not have to do any sort of inference. It just checks that the types are correct.

For example, since y = x + 3, y must be a number because we can only do addition with numbers. So, if the developer would have declared y as a string for example, the compiler would have errored.

It is not uncommon to see languages that have a mixture of both, where they infer the types of some expressions and require the developer to specify the types of others.

Static vs Dynamic Typing#

Type inference algorithms in some languages are pretty powerful that you might have mistaken those languages for being dynamically typed, OCaml is one example.

JavaScript is a dynamically typed language. This means that the type of a variable is not known until runtime, or at least not checked until then. This is why you can do things like:

const x = 1;
const y = "hello" + 3;

And the compiler would not complain. It is only at runtime that the interpreter would realize that you are trying to add a string to a number and throw an error.

Contrast this to a statically typed language like Rust where the compiler would not even compile the program, let alone let you run it.

// This will not compile in Rust
let x: i32 = 1;
let y: i32 = "hello" + 3;
//           ^^^^^^^^^^^ expected integer, found `&str`

Type Checking NTLC#

As I mentioned earlier, NTLC is a very simple language. It does not have variables, so there is no room for type inference.

But, we can still do type checking to prevent nonsensical programs such as:

if succ(true) then 0 else succ(0)

How are we supposed to evaluate succ(true)? We could potentially do an implicit cast from true to 1 and pretend as if the previous expression is succ(1) but that sounds weird, at least to me.

Notice that if we don’t add type checking to NTLC, that would make a dynamic language and the previous program would be valid, and you would be able to compile and run it - only to have the program crash at runtime.

Type Checking Algorithm#

Having NTLC grammar at hand:

TERM ::=
    true
    false
    if TERM then TERM else TERM
    0
    succ TERM
    pred TERM
    iszero TERM

The idea is very simple:

We already know the types of true, false and 0, they are Boolean, Boolean and Integer, respectively.
We know that succ and pred take an Integer and return an Integer.
We know that iszero takes an Integer and returns a Boolean.
We know that an if expression must have something that evaluates to a Boolean as a condition and the consequence and alternative terms must be of the same type.

These will form the base cases of our type checking algorithm.

The algorithm is very similar to the recursive descent algorithm we used to parse the AST. We will traverse the AST and check that the types of the nodes match what we expect them to be.

Recursion would always stop at terminal nodes since those are always going to be either 0, true or false which types we know.

Again, have a look at the code and I hope all of this would make perfect sense.

Hindley-Milner seems to be the most widely used type-inference algorithm. Type Inference for Beginners and Intro to Type Checking do a great job of explaining the concepts

Code#

/*
 *    t ::= // terms
 *        true  // constant true
 *        false // constant false
 *        if t then t else t // conditional
 *        0 // constant zero
 *        succ t // successor
 *        pred t // predecessor
 *        iszero t // zero test
 *        */

use std::fmt::Display;

use crate::parser::Term;

#[derive(Debug, Clone)]
/**
 * This is like a Typed AST.
 *
 * There are two real possibilities for an NTLC program.
 * The whole program either evaluates to a boolean or an integer.
 * Check the grammar above to see why this is the case.
 *
 * `Void` is just a convenience to represent an empty program.
 */
pub enum TypedTerm {
    Boolean(Term),
    Integer(Term),
    Void,
}

/**
 * We implement equality manually for `TypedTerm` because we want to
 * ignore the inner term when comparing.
 *
 * If two expressions are of the same type, then they are equal.
 * We don't care about the inner term.
 */
impl PartialEq for TypedTerm {
    fn eq(&self, other: &Self) -> bool {
        matches!(
            (self, other),
            (TypedTerm::Boolean(_), TypedTerm::Boolean(_))
                | (TypedTerm::Integer(_), TypedTerm::Integer(_))
                | (TypedTerm::Void, TypedTerm::Void)
        )
    }
}

impl std::fmt::Display for TypedTerm {
    fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
        match self {
            TypedTerm::Boolean(_) => write!(f, "BOOLEAN"),
            TypedTerm::Integer(_) => write!(f, "INTEGER"),
            TypedTerm::Void => write!(f, "<VOID - EMPTY PROGRAM>"),
        }
    }
}

#[derive(Debug, PartialEq)]
pub enum TypeError {
    TypeMismatch {
        expected: Vec<TypedTerm>,
        found: TypedTerm,
    },
}

impl Display for TypeError {
    fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
        match self {
            TypeError::TypeMismatch { expected, found } => {
                write!(f, "Type error! Expected one of: ")?;
                for (i, t) in expected.iter().enumerate() {
                    if i == expected.len() - 1 {
                        write!(f, "{}", t)?;
                    } else {
                        write!(f, "{}, ", t)?;
                    }
                }
                write!(f, " but found: {}", found)
            }
        }
    }
}

/**
 * This is the entry point of the type checker.
 * Although I called it `infer` it doesn't really do any inference.
 * The reason being that type inference is actually not needed
 *
 * For our grammar since we know the type of everything.
 * There is no inference needed.
 *
 * It just checks if the program is "well typed".
 * "Well typed": means you are not doing things like
 * iszero true or if 0 then true else false.
 *
 * It uses the same algorithm as recursive descent
 * traversing the AST and checking the types of the nodes.
 */
pub fn infer(ast: &Term) -> Result<TypedTerm, TypeError> {
    match ast {
        // The terminals which are also the base cases of the recursion
        // are the easiest to type check.
        Term::True => Ok(TypedTerm::Boolean(Term::True)),
        Term::False => Ok(TypedTerm::Boolean(Term::False)),
        Term::Zero => Ok(TypedTerm::Integer(Term::Zero)),
        /*
           We need to make sure that the inner term of the successor
           is an integer.
        */
        Term::Successor(t) => {
            let t = infer(t)?;
            match t {
                TypedTerm::Integer(inner_term) => Ok(TypedTerm::Integer(inner_term)),
                _ => Err(TypeError::TypeMismatch {
                    expected: vec![TypedTerm::Integer(Term::Empty)],
                    found: t,
                }),
            }
        }
        Term::Predecessor(t) => {
            let t = infer(t)?;
            match t {
                TypedTerm::Integer(inner_term) => Ok(TypedTerm::Integer(inner_term)),
                _ => Err(TypeError::TypeMismatch {
                    expected: vec![TypedTerm::Integer(Term::Empty)],
                    found: t,
                }),
            }
        }
        Term::IsZero(t) => {
            let t = infer(t)?;
            match t {
                TypedTerm::Integer(inner_term) => Ok(TypedTerm::Boolean(inner_term)),
                _ => Err(TypeError::TypeMismatch {
                    expected: vec![TypedTerm::Integer(Term::Empty)],
                    found: t,
                }),
            }
        }
        Term::Conditional {
            condition,
            consequence,
            alternative,
        } => {
            let condition_type = infer(condition)?;
            let conseq_type = infer(consequence)?;
            let alt_type = infer(alternative)?;

            /*
             * For the conditional we need to make sure that the condition
             * is a boolean and that the consequence and alternative are of the same type.
             *
             * This is to avoid things like:
             * if true then 0 else false
             *
             * While there is nothing wrong with the above program
             * We don't have a way to represent it in our grammar
             * or in our type system.
             *
             * The resulting type would be what is typically known
             * as a union type.
             *
             * So we simply disallow it in our case
             */
            match condition_type {
                TypedTerm::Boolean(_) => {
                    if conseq_type == alt_type {
                        Ok(conseq_type)
                    } else {
                        let expected_type = match conseq_type {
                            TypedTerm::Boolean(_) => TypedTerm::Boolean(Term::Empty),
                            TypedTerm::Integer(_) => TypedTerm::Integer(Term::Empty),
                            TypedTerm::Void => TypedTerm::Void,
                        };
                        Err(TypeError::TypeMismatch {
                            expected: vec![expected_type],
                            found: alt_type,
                        })
                    }
                }
                _ => Err(TypeError::TypeMismatch {
                    expected: vec![TypedTerm::Boolean(Term::Empty)],
                    found: condition_type,
                }),
            }
        }
        Term::Empty => Ok(TypedTerm::Void),
    }
}

#[cfg(test)]
mod test_type_checker_happy_path {
    use super::*;

    use crate::lexer::scan;
    use crate::parser::parse;

    #[test]
    fn test_empty_program() {
        let input = "";
        let tokens = scan(input).unwrap();
        let ast = parse(tokens).unwrap();
        let typed_ast = infer(&ast).unwrap();

        assert_eq!(typed_ast, TypedTerm::Void);
    }

    #[test]
    fn test_true() {
        let input = "true";
        let tokens = scan(input).unwrap();
        let ast = parse(tokens).unwrap();
        let typed_ast = infer(&ast).unwrap();

        assert_eq!(typed_ast, TypedTerm::Boolean(Term::True));
    }

    #[test]
    fn test_false() {
        let input = "false";
        let tokens = scan(input).unwrap();
        let ast = parse(tokens).unwrap();
        let typed_ast = infer(&ast).unwrap();

        assert_eq!(typed_ast, TypedTerm::Boolean(Term::False));
    }

    #[test]
    fn test_zero() {
        let input = "0";
        let tokens = scan(input).unwrap();
        let ast = parse(tokens).unwrap();
        let typed_ast = infer(&ast).unwrap();

        assert_eq!(typed_ast, TypedTerm::Integer(Term::Zero));
    }

    #[test]
    fn test_successor() {
        let input = "succ 0";
        let tokens = scan(input).unwrap();
        let ast = parse(tokens).unwrap();
        let typed_ast = infer(&ast).unwrap();

        assert_eq!(typed_ast, TypedTerm::Integer(Term::Zero));
    }

    #[test]
    fn test_predecessor() {
        let input = "pred 0";
        let tokens = scan(input).unwrap();
        let ast = parse(tokens).unwrap();
        let typed_ast = infer(&ast).unwrap();

        assert_eq!(typed_ast, TypedTerm::Integer(Term::Zero));
    }

    #[test]
    fn test_is_zero() {
        let input = "iszero 0";
        let tokens = scan(input).unwrap();
        let ast = parse(tokens).unwrap();
        let typed_ast = infer(&ast).unwrap();

        assert_eq!(typed_ast, TypedTerm::Boolean(Term::Zero));
    }

    #[test]
    fn test_conditional() {
        let input = "if true then 0 else 0";
        let tokens = scan(input).unwrap();
        let ast = parse(tokens).unwrap();
        let typed_ast = infer(&ast).unwrap();

        assert_eq!(typed_ast, TypedTerm::Integer(Term::Zero));
    }

    #[test]
    fn test_boolean_conditional() {
        let input = "if true then false else true";
        let tokens = scan(input).unwrap();
        let ast = parse(tokens).unwrap();
        let typed_ast = infer(&ast).unwrap();

        assert_eq!(typed_ast, TypedTerm::Boolean(Term::True));
    }

    #[test]
    fn test_nested_conditional() {
        let input = "if true then if true then 0 else 0 else 0";
        let tokens = scan(input).unwrap();
        let ast = parse(tokens).unwrap();
        let typed_ast = infer(&ast).unwrap();

        assert_eq!(typed_ast, TypedTerm::Integer(Term::Zero));
    }

    #[test]
    fn test_condition_with_function_calls() {
        let input = "if iszero 0 then succ 0 else pred 0";
        let tokens = scan(input).unwrap();
        let ast = parse(tokens).unwrap();
        let typed_ast = infer(&ast).unwrap();

        assert_eq!(typed_ast, TypedTerm::Integer(Term::Zero));
    }
}

#[cfg(test)]
mod test_type_checker_unhappy_path {
    use super::*;

    use crate::lexer::scan;
    use crate::parser::parse;

    #[test]
    fn test_successor() {
        let input = "succ true";
        let tokens = scan(input).unwrap();
        let ast = parse(tokens).unwrap();
        let typed_ast = infer(&ast);

        assert_eq!(
            typed_ast,
            Err(TypeError::TypeMismatch {
                expected: vec![TypedTerm::Integer(Term::Empty)],
                found: TypedTerm::Boolean(Term::True)
            })
        );
    }

    #[test]
    fn test_predecessor() {
        let input = "pred true";
        let tokens = scan(input).unwrap();
        let ast = parse(tokens).unwrap();
        let typed_ast = infer(&ast);

        assert_eq!(
            typed_ast,
            Err(TypeError::TypeMismatch {
                expected: vec![TypedTerm::Integer(Term::Empty)],
                found: TypedTerm::Boolean(Term::True)
            })
        );
    }

    #[test]
    fn test_is_zero() {
        let input = "iszero true";
        let tokens = scan(input).unwrap();
        let ast = parse(tokens).unwrap();
        let typed_ast = infer(&ast);

        assert_eq!(
            typed_ast,
            Err(TypeError::TypeMismatch {
                expected: vec![TypedTerm::Integer(Term::Empty)],
                found: TypedTerm::Boolean(Term::True)
            })
        );
    }

    #[test]
    fn test_conditional() {
        let input = "if true then true else 0";
        let tokens = scan(input).unwrap();
        let ast = parse(tokens).unwrap();
        let typed_ast = infer(&ast);

        assert_eq!(
            typed_ast,
            Err(TypeError::TypeMismatch {
                expected: vec![TypedTerm::Boolean(Term::Empty)],
                found: TypedTerm::Integer(Term::Zero)
            })
        );
    }

    #[test]
    fn test_nested_conditional() {
        let input = "if true then if true then true else 0 else 0";
        let tokens = scan(input).unwrap();
        let ast = parse(tokens).unwrap();
        let typed_ast = infer(&ast);

        assert_eq!(
            typed_ast,
            Err(TypeError::TypeMismatch {
                expected: vec![TypedTerm::Boolean(Term::Empty)],
                found: TypedTerm::Integer(Term::Zero)
            })
        );
    }
}

Parser

2023-09-22T00:00:00+00:00

Just give me the codez

Intro#

Parsing – for me – was the most mysterious, and to a certain extent the most scary, part of compilers when I started.

Coming across names and acronyms like Shift-Reduce Parsers, Recursive Descent Parsers, LL(1), LR(1), LALR(1) and more made it feel like an intimidating topic.

But, let’s take a step back and consider what a parser is and why we need one.

Parsing#

In the previous post, we talked about lexing and how it takes a stream of characters and converts it into a stream of tokens.

The concrete example we used was:

if iszero(succ(0)) then true else false

which resulted in the following list of tokens:

[
  Token::If,
  Token::IsZero,
  Token::LeftParenthesis,
  Token::Succ,
  Token::LeftParenthesis,
  Token::Zero,
  Token::RightParenthesis,
  Token::RightParenthesis,
  Token::Then,
  Token::True,
  Token::Else,
  Token::False,
]

Now, what do we want to do with these tokens?

Well, these tokens are a representation of our program’s source code. That source code is something we want to eventually execute.

Now, how would we go about executing – evaluating – this source code to get the correct result?

We need to somehow extract “meaning” from the source code, and the way we extract meaning is by extracting structure.

Language and Grammar#

The source code is written in a language, and we need to understand that language. The best solution that we “humans” came up with so far to “understand” ¹ language was to extract structure from it, and the way we extract structure is through following a systematic grammar.

For example, in the case of natural languages such as English, we have a grammar rule that tells us that a sentence is composed of a subject, a verb, and an object.

Computer scientists adopted the same approach to allow computers to “understand” programs written in a programming language. They define grammars for said languages.

The parser is the machinery that carries out the task of extracting structure from source code by following the rules of the grammar.

Theory and Practice#

As far as I have seen, parsing theory which lends itself to the study of grammars and languages seems to be one of the most studied topics in compilers and computer science in general. There is so much literature and academic work on the topic² that it is impossible to give it justice in a single blog post³.

For our purposes, I am going to make a very gross simplification and say that grammar is just a bunch of substitution rules. For our language, NTLC, the grammar is:

TERM ::=
    true
    false
    if TERM then TERM else TERM
    0
    succ TERM
    pred TERM
    iszero TERM

How would you read this?

Every word that is made of capital letters – technically known as a non-terminal – is something that can be replaced by what it equates to on the right-hand side.
- In our case, we only have one non-terminal called TERM, and it can be replaced by any of the values on the right-hand side.
The ::= is just a notation for saying “can be replaced by”
To generate a program in NTLC, you start with TERM and keep replacing non-terminals until you have only non-capital words left – technically those are called terminals.

Example#

Following the previous grammar we can make sentences (NTLC programs) such as:

true: It is just a program that evaluates to true
0: It is just a program that evaluates to 0
succ(TERM): in this case we have one non-terminal that we still need to replace. We can choose anything that is a valid TERM. Notice that we can essentially repeat this almost indefinitely. For example, we can have succ(succ(succ(0)))

Another observation, the grammar allows us to construct nonsensical sentences such as:

succ(true)

Not all hope is lost though. You see, you can do the same in English with “just” the grammar:

The cat   is the table
^^^^^^^   ^^ ^^^^^^^
SUBJECT   VERB OBJECT

Evidently, this is also nonsensical.

Semantic Analysis#

The way we typically prevent such nonsensical programs is through another stage in the compiler pipeline called “Semantic Analysis”. This is usually where type-checkers – a topic we cover in the next post – live.

“Semantic Analysis” is where most compiler engineers seem to spend most of their time. It is the place where the compiler (at least in statically typed languages) can help the end user, developers, the most.

You may have seen such messages when programming:

“You are trying to add a number to a string”
“You are trying to call a function with the wrong number of arguments”
“You are trying to access an undefined variable”

These and many others are the workings of the Semantic Analyzer.

I digress, let’s get back to parsing.

Parsing Techniques#

As I said earlier, there is a lot of theory on parsing and so many techniques, but we are going to use one technique known as Recursive Descent Parsing.

Why did I choose this technique?

Honestly, it is the only kind of parsing technique that people actually implement by hand⁴. Every other technique is typically implemented by Parser Generators⁵

Recursive Descent Parsing#

Recursive Descent Parsing works by following the rules of the grammar in a recursive fashion, hence the name. And it does it from top to bottom, hence the “Descent” part.

What does that really mean? It is actually very simple, but, it is one of those things that seem to click only when you actually implement them.

Before that though, it might help if we actually see what the parser output looks like.

Abstract Syntax Tree (AST)#

Abstract Syntax Trees are so famous that people rarely refer to them by their full name. They are usually referred to as ASTs.

The reason being, my guess, is that once you have an AST representation of your source code, you can do so many things with it:

You can evaluate it
You can compile it to another language
You can optimize it
You can pretty print it

And the list goes on. They are pretty much a canonical representation of your source code.

So what would the AST for our NTLC program look like if it was parsed?

if iszero(succ(0)) then true else false

Would produce the following AST:

So the goal of the parser is to take the stream of tokens and convert it into such AST.

Step by step#

Let’s go through an example step by step. First the grammar for reference:

TERM ::=
    true
    false
    if TERM then TERM else TERM
    0
    succ TERM
    pred TERM
    iszero TERM

and let’s say we have the following source code:

if true then succ(0) else pred(0)

The corresponding tokens would be:

[
  Token::If,
  Token::True,
  Token::Then,
  Token::Succ,
  Token::LeftParenthesis,
  Token::Zero,
  Token::RightParenthesis,
  Token::Else,
  Token::Pred,
  Token::LeftParenthesis,
  Token::Zero,
  Token::RightParenthesis,
]

The way recursive descent would work on the preceding list of tokens is by following the grammar rules from top to bottom:

We start with TERM, and we see that the first token is Token::If, so we can use the rule if TERM then TERM else TERM
- Notice that once the parser goes down this path, it kinda “remembers” that it is in the middle of parsing an if expression and so then TERM else TERM must follow. Otherwise, the parser would either have to backtrack (try another possible path) or fail.
We see that the next token is Token::True, so we can use the rule true
We see that the next token is Token::Then, which matches our expectation so continue parsing the if expression.
We see that the next token is Token::Succ, so we can use the rule succ TERM
We see that the next token is Token::LeftParenthesis, so we can use the rule succ (TERM)
- We did not mention parenthesis ( and ) in our grammar explicitly. They are optional and help make the code clearer in some cases.
We see that the next token is Token::Zero, so we can use the rule 0
We see that the next token is Token::RightParenthesis, which matches our expectation.
We see that the next token is Token::Else, again, it matches our expectation.
Likewise, we see that the next token is Token::Pred, so we can use the rule pred TERM
We see that the next token is Token::LeftParenthesis, so we can use the rule pred (TERM)
We see that the next token is Token::Zero, so we can use the rule 0
We see that the next token is Token::RightParenthesis, so we can use the rule pred (0)
Finally, we see that we have no more tokens, so we are done

Problems and solutions#

Now our grammar is simple, but in some cases there could be multiple possible paths (choices) that are possible and so recursive descent would have to backtrack and try different alternatives.

I also did not mention two possible problems that could rise in different grammars:

Ambiguity: There are multiple possible correct paths to parse a given input, but usually only one of them is “semantically” correct. For example, 1 + 2 * 3 could be parsed as (1 + 2) * 3 or 1 + (2 * 3). The latter one is the correct one.
Left Recursion: A non-terminal can be replaced by itself. Imagine if the grammar had a rule like TERM ::= TERM + TERM. This would result in infinite recursion. People usually solve this by rewriting the grammar rule into a “semantically equivalent” rule without the recursion on the left.

For a better understanding on both issues and their solutions, I refer you to Pratt Parsers: Expression Parsing Made Easy and Simple but Powerful Pratt Parsing.

Okay, with all this talk you might think the implementation is pretty complex⁶, but actually it almost rolls off the tongue.

So, without further ado, I leave you with the code.

Recent advances in AI and specifially Largage Language Models (LLMs) might present a different approach to understanding language. After all, many people communicate and understand others although they have never taken a single grammar class
There does not seem to be much recent research on the topic though. I think it's mainly because there aren't any stones which has not been turned. i.e., it is a solved problem
I highly recommend Parsing Techniques - A Practical Guide for a very good understanding of both grammar and parsing techniques. It is an easy read even if you don't have a technical background
Funny enough, it is also the least studied theoratically. Most research covers the other techniques that are implemented by Parser Generators. Read Which Parsing Approach for a recap
Tree Sitter seems to be one of the most powerful and recent Parser Generators
The implementation of recursive descent is simple. What turned out to be difficult when implementing parsers in general is two topics that we don't cover in this post: error recovery and error reporting, also incremental parsing

Code#

use std::iter::Peekable;

/*
 *    t ::= // terms
 *        true  // constant true
 *        false // constant false
 *        if t then t else t // conditional
 *        0 // constant zero
 *        succ t // successor
 *        pred t // predecessor
 *        iszero t // zero test
 *        */
use crate::lexer::Token;

#[derive(PartialEq)]
pub enum SyntaxError {
    UnexpectedToken { expected: Vec<Token>, found: Token },
    UnexpectedEndOfInput { expected: Vec<Token> },
}

impl std::fmt::Debug for SyntaxError {
    fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
        match self {
            SyntaxError::UnexpectedToken { expected, found } => {
                write!(
                    f,
                    "Unexpected '{:?}'. expected one of: {:?}",
                    found, expected
                )
            }
            SyntaxError::UnexpectedEndOfInput { expected } => {
                write!(f, "Unexpected <EOF>. expected one of: {:?}", expected)
            }
        }
    }
}

impl std::fmt::Display for SyntaxError {
    fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
        match self {
            SyntaxError::UnexpectedToken { expected, found } => {
                write!(
                    f,
                    "Unexpected '{:?}'. expected one of {:?}",
                    found, expected
                )
            }
            SyntaxError::UnexpectedEndOfInput { expected } => {
                write!(
                    f,
                    "Unexpected end of stream. expected one of {:?}",
                    expected
                )
            }
        }
    }
}

impl std::error::Error for SyntaxError {}

#[derive(Debug, PartialEq, Eq, Hash, Clone)]
/**
 * This is our Abstract Syntax Tree (AST)
 * It is a recursive data structure that represents the program
 * Ignore those `Box<>` for now, they are just there to make the
 * Rust borrow checker happy.
 *
 * This should be self explanatory as it literally
 * follows the structure of our grammar.
*/
pub enum Term {
    Conditional {
        condition: Box<Term>,
        consequence: Box<Term>,
        alternative: Box<Term>,
    },
    Successor(Box<Term>),
    Predecessor(Box<Term>),
    IsZero(Box<Term>),
    True,
    False,
    Zero,
    Empty,
}

impl std::fmt::Display for Term {
    fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
        match self {
            Term::Conditional {
                condition,
                consequence,
                alternative,
            } => write!(
                f,
                "COND[{}] -> CONSEQUENCE[{}] ;; ALT[{}]",
                condition, consequence, alternative
            ),
            Term::Successor(expr) => write!(f, "Successor[{}]", expr),
            Term::Predecessor(expr) => write!(f, "Predcessor[{}]", expr),
            Term::IsZero(expr) => write!(f, "ZeroAssertion[{}]", expr),
            Term::True => write!(f, "T"),
            Term::False => write!(f, "F"),
            Term::Zero => write!(f, "ZERO"),
            Term::Empty => write!(f, "<EMPTY>"),
        }
    }
}

fn match_left_parantheses<T>(tokens: &mut Peekable<T>) -> Result<(), SyntaxError>
where
    T: Iterator<Item = Token>,
{
    match tokens.next() {
        Some(Token::LeftParenthesis) => Ok(()),
        Some(token) => Err(SyntaxError::UnexpectedToken {
            expected: vec![Token::LeftParenthesis],
            found: token,
        }),
        None => Err(SyntaxError::UnexpectedEndOfInput {
            expected: vec![Token::LeftParenthesis],
        }),
    }
}

fn match_right_parenthesis<T>(tokens: &mut Peekable<T>) -> Result<(), SyntaxError>
where
    T: Iterator<Item = Token>,
{
    match tokens.next() {
        Some(Token::RightParenthesis) => Ok(()),
        Some(token) => Err(SyntaxError::UnexpectedToken {
            expected: vec![Token::RightParenthesis],
            found: token,
        }),
        None => Err(SyntaxError::UnexpectedEndOfInput {
            expected: vec![Token::RightParenthesis],
        }),
    }
}

fn match_then<T>(tokens: &mut Peekable<T>) -> Result<(), SyntaxError>
where
    T: Iterator<Item = Token>,
{
    match tokens.next() {
        Some(Token::Then) => Ok(()),
        Some(token) => Err(SyntaxError::UnexpectedToken {
            expected: vec![Token::Then],
            found: token,
        }),
        None => Err(SyntaxError::UnexpectedEndOfInput {
            expected: vec![Token::Then],
        }),
    }
}

fn match_else<T>(tokens: &mut Peekable<T>) -> Result<(), SyntaxError>
where
    T: Iterator<Item = Token>,
{
    match tokens.next() {
        Some(Token::Else) => Ok(()),
        Some(token) => Err(SyntaxError::UnexpectedToken {
            expected: vec![Token::Else],
            found: token,
        }),
        None => Err(SyntaxError::UnexpectedEndOfInput {
            expected: vec![Token::Else],
        }),
    }
}

/**
 * This is the recursive function that does most of the work
 * It essentially follows the structure of our grammar
 */
fn parse_term<T>(tokens: &mut Peekable<T>) -> Result<Term, SyntaxError>
where
    T: Iterator<Item = Token>,
{
    // Get the next token
    match tokens.next() {
        // If it is `true` then our AST contains a single node `Term::True`
        Some(Token::True) => Ok(Term::True),
        // If it is `false` then our AST contains a single node `Term::False`
        Some(Token::False) => Ok(Term::False),
        Some(Token::Zero) => Ok(Term::Zero),
        /*
         * If it is `iszero` then we need to check if there is a left parenthesis
         * as our grammar allows both `iszero 0` and `iszero(TERM)`
         *
         * The same applies for other builtin functions
         */
        Some(Token::IsZero) => {
            let match_right = match tokens.peek() {
                Some(&Token::LeftParenthesis) => {
                    match_left_parantheses(tokens)?;
                    true
                }
                _ => false,
            };

            /*
               This is the heart of how recursive descent works
               We call `parse_term()` again to parse the inner expression
            */
            let inner_expr = parse_term(tokens)?;

            // If we matched a left parenthesis, we need to match a right parenthesis
            if match_right {
                match_right_parenthesis(tokens)?;
            }

            Ok(Term::IsZero(Box::new(inner_expr)))
        }
        Some(Token::Succ) => {
            let match_right = match tokens.peek() {
                Some(&Token::LeftParenthesis) => {
                    match_left_parantheses(tokens)?;
                    true
                }
                _ => false,
            };

            let inner_expr = parse_term(tokens)?;

            if match_right {
                match_right_parenthesis(tokens)?;
            }

            Ok(Term::Successor(Box::new(inner_expr)))
        }
        Some(Token::Pred) => {
            let match_right = match tokens.peek() {
                Some(&Token::LeftParenthesis) => {
                    match_left_parantheses(tokens)?;
                    true
                }
                _ => false,
            };

            let inner_expr = parse_term(tokens)?;

            if match_right {
                match_right_parenthesis(tokens)?;
            }

            Ok(Term::Predecessor(Box::new(inner_expr)))
        }
        /*
           If terms might seem more complex but they follow the same
           pattern of invoking `parse_term()` recursively

        */
        Some(Token::If) => {
            let match_right = match tokens.peek() {
                Some(&Token::LeftParenthesis) => {
                    match_left_parantheses(tokens)?;
                    true
                }
                _ => false,
            };

            let condition = parse_term(tokens)?;

            if match_right {
                match_right_parenthesis(tokens)?;
            }

            match_then(tokens)?;

            let match_right = match tokens.peek() {
                Some(&Token::LeftParenthesis) => {
                    match_left_parantheses(tokens)?;
                    true
                }
                _ => false,
            };

            let consequence = parse_term(tokens)?;

            if match_right {
                match_right_parenthesis(tokens)?;
            }

            match_else(tokens)?;

            let match_right = match tokens.peek() {
                Some(&Token::LeftParenthesis) => {
                    match_left_parantheses(tokens)?;
                    true
                }
                _ => false,
            };

            let alternative = parse_term(tokens)?;

            if match_right {
                match_right_parenthesis(tokens)?;
            }

            Ok(Term::Conditional {
                condition: Box::new(condition),
                consequence: Box::new(consequence),
                alternative: Box::new(alternative),
            })
        }
        Some(Token::EOF) => Ok(Term::Empty),
        Some(token) => Err(SyntaxError::UnexpectedToken {
            expected: vec![
                Token::If,
                Token::True,
                Token::False,
                Token::Zero,
                Token::Pred,
                Token::Succ,
                Token::IsZero,
            ],
            found: token,
        }),
        None => Ok(Term::Empty),
    }
}

/**
 * Entry point of our parser
 * This is the implementation of the Recursive Descent algorithm
 *
 * It takes a vector of tokens and returns a Term (AST) or a SyntaxError
 * if the program is not valid.
 *
 * It delegates most of the work to `parse_term()` which is a recursive function
 *
 * After `parse_term()` returns, we check if there are any tokens left in the stream
 * At this stage, there must be none left otherwise the program is not valid.
 * For example, succ(0) 0 is not valid because there is a 0 at the end of the program
 */
pub fn parse(tokens: Vec<Token>) -> Result<Term, SyntaxError> {
    let mut tokens = tokens.into_iter().peekable();

    let ast = parse_term(&mut tokens)?;

    match tokens.next() {
        Some(Token::EOF) => (),
        Some(token) => {
            return Err(SyntaxError::UnexpectedToken {
                expected: vec![Token::EOF],
                found: token,
            })
        }
        None => (),
    }

    Ok(ast)
}

#[cfg(test)]
mod tests_parser_happy_path {
    use super::*;
    use crate::lexer::scan;

    #[test]
    fn test_parse_empty_program() {
        let input = "";

        let tokens = scan(input).unwrap();

        let ast = parse(tokens).unwrap();

        assert_eq!(ast, Term::Empty);
    }

    #[test]
    fn test_parse_literal_true() {
        let input = "true";

        let tokens = scan(input).unwrap();

        let ast = parse(tokens).unwrap();

        assert_eq!(ast, Term::True);
    }

    #[test]
    fn test_parse_literal_false() {
        let input = "false";

        let tokens = scan(input).unwrap();

        let ast = parse(tokens).unwrap();

        assert_eq!(ast, Term::False);
    }

    #[test]
    fn test_parse_literal_zero() {
        let input = "0";

        let tokens = scan(input).unwrap();

        let ast = parse(tokens).unwrap();

        assert_eq!(ast, Term::Zero);
    }

    #[test]
    fn test_parse_conditional() {
        let input = "if true then false else true";

        let tokens = scan(input).unwrap();

        let ast = parse(tokens).unwrap();

        assert_eq!(
            ast,
            Term::Conditional {
                condition: Box::new(Term::True),
                consequence: Box::new(Term::False),
                alternative: Box::new(Term::True),
            }
        );
    }

    #[test]
    fn test_parse_successor() {
        let input = "succ 0";

        let tokens = scan(input).unwrap();

        let ast = parse(tokens).unwrap();

        assert_eq!(ast, Term::Successor(Box::new(Term::Zero),));
    }

    #[test]
    fn test_parse_predecessor() {
        let input = "pred 0";

        let tokens = scan(input).unwrap();

        let ast = parse(tokens).unwrap();

        assert_eq!(ast, Term::Predecessor(Box::new(Term::Zero),));
    }

    #[test]
    fn test_parse_iszero() {
        let input = "iszero 0";

        let tokens = scan(input).unwrap();

        let ast = parse(tokens).unwrap();

        assert_eq!(ast, Term::IsZero(Box::new(Term::Zero),));
    }

    #[test]
    fn test_parse_nested() {
        let input = "if true then succ   (0) else pred( 0 )";

        let tokens = scan(input).unwrap();

        let ast = parse(tokens).unwrap();

        assert_eq!(
            ast,
            Term::Conditional {
                condition: Box::new(Term::True),
                consequence: Box::new(Term::Successor(Box::new(Term::Zero))),
                alternative: Box::new(Term::Predecessor(Box::new(Term::Zero)))
            }
        );
    }
}

#[cfg(test)]
mod tests_sad_path {
    use super::*;
    use crate::lexer::scan;

    #[test]
    fn test_parse_unexpected_true() {
        let input = "true true";

        let tokens = scan(input).unwrap();

        let ast = parse(tokens);

        assert_eq!(
            ast,
            Err(SyntaxError::UnexpectedToken {
                expected: vec![Token::EOF],
                found: Token::True,
            })
        );
    }

    #[test]
    fn test_parse_unexpected_false() {
        let input = "true false";

        let tokens = scan(input).unwrap();

        let ast = parse(tokens);

        assert_eq!(
            ast,
            Err(SyntaxError::UnexpectedToken {
                expected: vec![Token::EOF],
                found: Token::False,
            })
        );
    }

    #[test]
    fn test_parse_unbalanced_parenthesis() {
        let input = "iszero(true";

        let tokens = scan(input).unwrap();

        let ast = parse(tokens);

        assert_eq!(
            ast,
            Err(SyntaxError::UnexpectedToken {
                expected: vec![Token::RightParenthesis],
                found: Token::EOF,
            })
        );
    }

    #[test]
    fn test_unbalanced_if() {
        let input = "if true then false iszero 0";

        let tokens = scan(input).unwrap();

        let ast = super::parse(tokens);

        assert_eq!(
            ast,
            Err(SyntaxError::UnexpectedToken {
                expected: vec![Token::Else],
                found: Token::IsZero,
            })
        );
    }

    #[test]
    fn test_invalid_syntax_starting_with_else() {
        let input = "else 0 if true";

        let tokens = scan(input).unwrap();

        let ast = super::parse(tokens);

        assert_eq!(
            ast,
            Err(SyntaxError::UnexpectedToken {
                expected: vec![
                    Token::If,
                    Token::True,
                    Token::False,
                    Token::Zero,
                    Token::Pred,
                    Token::Succ,
                    Token::IsZero
                ],
                found: Token::Else,
            })
        );
    }

    #[test]
    fn test_parse_unexpected_token() {
        let input = "if true then false else true 0";

        let tokens = scan(input).unwrap();

        let ast = super::parse(tokens);

        assert_eq!(
            ast,
            Err(SyntaxError::UnexpectedToken {
                expected: vec![Token::EOF],
                found: Token::Zero,
            })
        );
    }
}

Introduction

2023-09-20T00:00:00+00:00

Intro#

We are going to build a super tiny programming language called NTLC, and, we will do it from scratch in Rust.

Why another programming language?#

It’s not a real programming language. It’s just a toy language that will we will use for fun and learning. It’s not meant to be used for anything except learning and exploring.

If you do end up falling in love with programming languages, or you wish to learn more about them, or, perhaps you would like to create your own real language then then I highly recommend:

Crafting Interpreters by Bob Nystrom
Writing an Interpreter in Go by Thorsten Ball
Or if you are up for some adventures in OCaml and C++, the amazing Bolt language series by Mukul Rathi

Why from scratch, and, why Rust?#

Well, that’s how one really learns in my opinion – by doing.

As for Rust, it’s a language that I have been wanting to explore for a while now – the hype you know – and this seemed like a good opportunity to do so (spoiler alert: it might not have been the wisest choice).

In my opinion, Rust is not a beginner friendly language. It’s a very powerful one, but, it’s also a complex one, especially if you haven’t done any C or C++ before. Nevertheless, I will do my best to explain things as we go.

Having already planned to write about NTLC; I should’ve perhaps chosen a language that is more beginner friendly or one that I am really proficient with but where is the challenge in that!

The Language#

NTLC stands for “Neo Typed Lambda Calculus”. The name was inspired from the book “Types and Programming Languages” by Benjamin C. Pierce, which is a great book on type theory and programming languages in general.

The book uses a language called “Simply Typed Lambda Calculus” to demonstrate the concepts of type theory, and, we are building a typed version of a simpler precursor to it known as “Untyped Arithmetic Expressions”. It is just that for some reason my mind was already fixated on the name “Neo Typed Lambda Calculus” so I stuck with it.

End to End#

The language itself is so simple that one might not even call it a language, but that also makes for a great learning playground.

Also, I have never built a language before so this felt like a great place to start, and, it will allow us to build the full compiler pipeline¹: including a type checker, simple IDE integration and native code generation from start to finish.

Small Interlude#

I recommend this approach for anything you wish to build or learn:

Get a feeling for the landscape you’re about to enter
Build a fully functioning thing. It doesn’t have to be perfect or even good. It just has to work
Now, iterate and refine. You will have a much better understanding of the problem space and will be able to make better decisions

What does it look like?#

Here is a program/snippet written in NTLC:

if iszero(succ(0)) then true else false

which evaluates, you guessed it, to false.

iszero() is a builtin function that checks if the given argument is zero. succ() (successor) is a function that increments its argument by one. true, false and 0 are predefined constants.

There is one more builtin function called pred() (predecessor) that decrements its argument by one.

You can also nest things, here is another contrived example:

if iszero(succ(pred(0))) then if true then false else true else false

Yikes! And, that’s it. That’s the whole language.

There are no user defined types (think classes, structs, or even variables) or functions. There are no loops. There are no strings. There aren’t even any other numbers beside the predefined 0 constant. The language doesn’t even allow empty lines. The whole program must be typed in one single line.

Again, this keeps things simple and allows us to build things end to end. Also, adding those – albeit tricky for some and near impossible for other – constructs can be done iteratively and incrementally.

What’s next?#

In the next chapter, we will build the first stage of our language compiler – the lexer.

See you there!

It is as full of a pipeline as possible with such simple language that doesn't have many constructs but it is more than enough for a start.

Lexer

2023-09-20T00:00:00+00:00

Just give me the codez

What is a lexer?#

The lexer¹, also known as the tokenizer or the scanner, is the first stage in the compilation process. It takes source code (NTLC source code in our case) and converts it into a stream of “tokens”.

First stage you say, are there more? Definitely!

Actually, the compiler is more like a pipeline. Each stage takes the output of the previous one and transforms it into something that is ready for the next stage to consume. The output of the lexer is the input of the parser, and so on.

Here is a very typical compiler pipeline²:

Why do we need a lexer?#

Strictly speaking, we don’t. It just allows us to work with the source code on a higher conceptual level. Later stages, namely the parser, will have way easier time reasoning about the structure of the source code if it doesn’t have to worry about string indices, whitespace characters, and/or the nuances of character sets and text encoding.

Actually this is true of every stage in the pipeline. If the source language was so simple, you probably would not even need a compiler. You could get away with a few regular expressions and a bunch of string manipulation functions.

The compiler is essentially a systematic and deterministic way to convert a program written in one language to another. Remember the good old divide and conquer technique: it advises you to break a big problem into smaller manageable ones! Well, that’s what the compiler really does with all its stages.

Lexer output#

Remember when we said earlier that the lexer converts source code to tokens. It is time we talk about those tokens.

The lexer typically outputs an array/vector/list of tokens (technically speaking, that’s if it is not a streaming lexer but let’s ignore that for now).

Now, most of the time a token³ is a structure that has a type and a value. Here is an example to illustrate how our NTLC snippet from the previous post would map to a list of tokens:

Notice that because our language is very simple, things almost map one-to-one without any additional data.

But, say for example our language was to support arbitrary numbers or strings, then we would probably extend our token type to include a value field that would hold the actual value of the token.

For instance, a string token might look like this:

pub enum Token {
    StringLiteral(String),
    //            ^^^^^^
    //            This is the value field
    //
    //
    // More token types ...
}

or in TypeScript:

class StringLiteral {
  constructor(public readonly value: string) {}
}

type Token = StringLiteral | ...

When the lexer scans an input that has a string literal like “Hello World”, it would emit a token of type StringLiteral with a value of “Hello World” like so:

Token::StringLiteral("Hello World")

One final thing, you might be thinking: wait, where did we get our token types from? Did we just make them up?

Actually, sort of. Remember, this is our language and we get to define the rules, but, we won’t get into those until the next article on parsers.

How do lexers work?#

Before we describe the workings of the lexer, and parser for that matter, it is worth mentioning that there exists tools known as Parser Generators (yacc and Tree Sitter are such tools) that can actually generate code that implements the lexer and parser for your given you provide them with your programming language definition in some form or another, but hey, where is the fun in that!

Okay, getting back to how lexers work. They basically scan the input character by character⁴ (and most of the time, they take a sneak peek ahead at the next character or characters) and try to match those characters against a set of rules. If a rule matches, the lexer would emit a token and move on to the next character. If it comes across an unexpected character, it would throw an error.

It’s good to know that in NTLC we have the following token types:

pub enum Token {
    True, // Literal true
    False, // Literal false
    If, // Keyword if
    Then, // Keyword then
    Else, // Keyword else
    Zero, // Literal zero
    Succ, // builtin function succ()
    Pred, // builtin function pred()
    IsZero, // builtin function iszero()
    LeftParenthesis, // (
    RightParenthesis, // )
    EOF, // End of file (this is not something that ones types in but it's a handy way to represent
         // the end of the input)
}

Okay, enough with words. I feel that describing the workings of a compiler makes things sound more complicated than they actually are. So, let’s just look at the code. I have left comments all over the place to explain what is going on.

The word "lexer" is derived from the Latin word "lexicon" which refers to the vocabulary of a language, including its words and their meanings.
Actually, there is something quite satisfying about compilers: they are as "pure" as they can be. The compiler is essentially a function that takes a string and returns an alternative representation of that string -- no network calls, no database transactions, no file storage etc.
Sometimes the token encodes additional information such as the line number and column of the corresponding source code. Choosing what to store in a token depends on your application and requirements
Most of the time, lexers are defined in terms of one or more more regular expressions and you will see why soon!

Full Lexer code listing#

HINT: check the tests at the bottom of the file for more examples on how the lexer works

Again, the full source code for the project is available here!

The scan()function at line 271 is the entry point of the lexer module and the place where the main logic resides, so, make sure you start there first as everything else is more or less helper utilities.

  1use std::error::Error;
  2use std::fmt::Debug;
  3use std::vec::Vec;
  4
  5// I keep a copy of the grammar here for reference
  6/*
  7 *    t ::= // terms
  8 *        true  // constant true
  9 *        false // constant false
 10 *        if t then t else t // conditional
 11 *        0 // constant zero
 12 *        succ t // successor
 13 *        pred t // predecessor
 14 *        iszero t // zero test
 15 *        */
 16
 17static KEYWORD_TRUE: &str = "true";
 18static KEYWORD_FALSE: &str = "false";
 19static KEYWORD_IF: &str = "if";
 20static KEYWORD_THEN: &str = "then";
 21static KEYWORD_ELSE: &str = "else";
 22static KEYWORD_IS_ZERO: &str = "iszero";
 23static KEYWORD_ZERO: &str = "0";
 24static KEYWORD_SUCC: &str = "succ";
 25static KEYWORD_PRED: &str = "pred";
 26
 27// These are directives. Similar to annotations or decorators in other programming languages
 28// clippy is Rust's linter and we are just telling it to be quite about EOF casing
 29#[allow(clippy::upper_case_acronyms)]
 30// In Rust one can ask the compiler to implement certain interfaces for
 31// a type by using the `derive` keyword. In this case we are asking the
 32// compiler to implement the `PartialEq` (Partial Equality) interface for the `Token` type
 33// which is needed/used when we compare two values of the same type for equality using ==.
 34#[derive(PartialEq)]
 35/*
 36 `enum` is, roughly speaking, similar to how unions work in TypeScript where you can say:
 37 `type Token = True | False`
 38 And so token can either be True or False
 39 Notice that `True` and `False` are not the literal boolean values `true` and `false`
 40 but rather the names of the variants of the enum. They are types in their own right.
 41 Tokens are the output of the lexer and are the input of the parser.
 42*/
 43pub enum Token {
 44    True,
 45    False,
 46    If,
 47    Then,
 48    Else,
 49    Zero,
 50    Succ,
 51    Pred,
 52    IsZero,
 53    LeftParenthesis,
 54    RightParenthesis,
 55    EOF,
 56}
 57
 58/**
 59 * Rust doesn't have classes but structs which are similar to classes in other languages, just without "inline" methods.
 60 * This is how one implements "traits" (which roughly correspond to interfaces in other languages)
 61 * for a struct.
 62 * Essentially we are saying the Token type implements the Debug trait.
 63 * The Debug trait allows us to print the value of a Token when used in a format string (akin to string interpolation).
 64 */
 65impl Debug for Token {
 66    fn fmt(
 67        &self, /* similar to `this` in object oriented languages */
 68        f: &mut std::fmt::Formatter,
 69    ) -> std::fmt::Result {
 70        /*
 71         * I hope this very readable and self explanatory
 72         */
 73        match self {
 74            Token::True => write!(f, "TRUE"),
 75            Token::False => write!(f, "FALSE"),
 76            Token::If => write!(f, "IF"),
 77            Token::Then => write!(f, "THEN"),
 78            Token::Else => write!(f, "ELSE"),
 79            Token::Zero => write!(f, "0"),
 80            Token::Succ => write!(f, "SUCC"),
 81            Token::Pred => write!(f, "PRED"),
 82            Token::IsZero => write!(f, "IS_ZERO"),
 83            Token::LeftParenthesis => write!(f, "("),
 84            Token::RightParenthesis => write!(f, ")"),
 85            Token::EOF => write!(f, "<EOF>"),
 86        }
 87    }
 88}
 89
 90/**
 91 * While the `Debug` trait is used for debugging purposes, the `Display` trait is used for printing
 92 * a human readable or nice string representation of a value.
 93 * It is similar in that regard to how one might override `toString()` in Java.
 94 */
 95impl std::fmt::Display for Token {
 96    fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
 97        match self {
 98            Token::True => write!(f, "TRUE"),
 99            Token::False => write!(f, "FALSE"),
100            Token::If => write!(f, "IF"),
101            Token::Then => write!(f, "THEN"),
102            Token::Else => write!(f, "ELSE"),
103            Token::Zero => write!(f, "0"),
104            Token::Succ => write!(f, "SUCC"),
105            Token::Pred => write!(f, "PRED"),
106            Token::IsZero => write!(f, "IS_ZERO"),
107            Token::LeftParenthesis => write!(f, "("),
108            Token::RightParenthesis => write!(f, ")"),
109            Token::EOF => write!(f, "<EOF>"),
110        }
111    }
112}
113
114#[derive(PartialEq)]
115/*
116* We use an enum to represent the different kind of errors the lexer could run into
117*/
118pub enum LexingError {
119    UnrecognizedToken {
120        found: String,
121        column: usize,
122        line: usize,
123    },
124    UnexpectedCharacter {
125        found: char,
126        expected: Vec<String>,
127        column: usize,
128        line: usize,
129    },
130    UnexpectedEOF {
131        expected: Vec<String>,
132        column: usize,
133        line: usize,
134    },
135}
136
137/**
138 * This is a helper function that is used to format Lexer error message
139 */
140fn format_expectation_error(expected: &Vec<String>) -> String {
141    let mut result: Vec<String> = Vec::new();
142
143    for e in expected {
144        match e {
145            s if s == " " => result.push("<SPACE>".to_string()),
146            s if s == "\t" => result.push("<TAB>".to_string()),
147            s if s == "\0" => result.push("<EOF>".to_string()),
148            s => result.push(format!("'{}'", s)),
149        }
150    }
151
152    result.join(", ")
153}
154
155impl std::fmt::Display for LexingError {
156    fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
157        match self {
158            LexingError::UnrecognizedToken {
159                found,
160                column,
161                line,
162            } => {
163                write!(
164                    f,
165                    "Unrecognized token '{}' at line {}, column {}",
166                    found,
167                    line + 1,
168                    column
169                )
170            }
171            LexingError::UnexpectedCharacter {
172                found,
173                expected,
174                column,
175                line,
176            } => {
177                write!(
178                    f,
179                    "Found '{}' at line {}, column {} when perhaphs you meant '{}'?",
180                    found,
181                    line,
182                    column + 1,
183                    format_expectation_error(expected)
184                )
185            }
186            LexingError::UnexpectedEOF {
187                expected,
188                column,
189                line,
190            } => {
191                write!(
192                    f,
193                    "Unexpected end of file at line {}, column {} when you perhaps meant '{}'?",
194                    line,
195                    column + 1,
196                    format_expectation_error(expected)
197                )
198            }
199        }
200    }
201}
202
203impl std::fmt::Debug for LexingError {
204    fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
205        match self {
206            LexingError::UnrecognizedToken {
207                found,
208                column,
209                line,
210            } => {
211                write!(f, "Unknown token '{}' at ({}, {})", found, line, column)
212            }
213            LexingError::UnexpectedCharacter {
214                found,
215                expected,
216                column,
217                line,
218            } => {
219                write!(
220                    f,
221                    "Unexpected token '{}' at ({}, {}). Expectation context {}",
222                    found,
223                    line,
224                    column,
225                    format_expectation_error(expected)
226                )
227            }
228            LexingError::UnexpectedEOF {
229                expected,
230                column,
231                line,
232            } => {
233                write!(
234                    f,
235                    "Unexpected EOF looking for lexeme '{}' at ({}, {})",
236                    format_expectation_error(expected),
237                    line,
238                    column
239                )
240            }
241        }
242    }
243}
244
245/**
246 * The `Error` trait is like a marker interface
247 * It is just tells the compiler that `LexingError` is, well, kinda of an error
248 */
249impl Error for LexingError {}
250
251/**
252 * This is the heart of the lexer. It basically takes NTLC program source code
253 * as input and returns a vector/list of tokens of type `Token` as output.
254 *
255 * It iterates the input string character by character from left to right and tries to match them
256 * against the known tokens of the language.
257 *
258 * Most of the time, the lexer is actually made part of the parser module and is implemented
259 * as a streaming lexer. That is, it is implemented as a function that returns the next
260 * token only when asked for explicitly by the parser and not all at once.
261 * This is done to avoid having to read the entire input into memory at once.
262 * Also, when working within a text editor or IDE, those tools can send you incomplete
263 * pieces of code to avoid resending the entire source file over and over again.
264 * We will see this when we are working on the language server.
265 *
266 *
267 * You might see the `&str` type and wonder what it is. It is just a reference to a string.
268 * Or to be more technically correct, a readonly reference to a string slice (what a mouthful).
269 * This avoids copying the string around in memory. Don't worry about it too much.
270 *
271 * The `Result<>` type is a probably implemented as an `enum` under te hood. It says this function
272 * either returns a vector of tokens or an error
273 * This is a concept from functional programming and it's very idiomatic to see the Result<>
274 * type all over the place in Rust code.
275 * If you worked with a language that uses exceptions, this might be feel strange at first
276 * but, I personally think it is way cleaner to explicitly state that your function might
277 * return an error. This reminds me to some extent of checked exceptions in Java (at least last time I used it).
278 */
279pub fn scan(input: &str) -> Result<Vec<Token>, LexingError> {
280    let mut tokens: Vec<Token> = Vec::new();
281
282    /*
283    This line might seem to be doing a bit of magic but essentially it is just a convenient
284    way to line up the input with itself shifted by one character, essentially giving us each character
285    and the one following it as a pair (char, next_char). Let me demonstrate:
286
287    let's say that the input is "succ(0)" this will give us the following:
288    s u c c ( 0 )
289    u c c ( 0 ) \0
290
291    and as pairs comes out as:
292    [('s', 'u'), ('u', 'c'), ('c', 'c'), ('c', '('), ('(', '0'), ('0', ')'), (')', '\0')]
293
294    This is handy because a lot of the time we need to "lookahead" when we are scanning the input
295    to decide what the next token is. Keep reading.
296    */
297    let mut chars = input
298        .chars()
299        .zip(input.chars().skip(1).chain(['\0'])) // we are adding a null character at the end of the shifted string so both strings have the same length, this prevents the iterator from stopping prematurely
300        .peekable();
301
302    // We are tracking the line number although our lexer doesn't support multiple lines of input
303    // This is useful for error reporting if we were to support new lines.
304    // The same with the `index` which is the column number within the input string
305    let line = 1usize;
306    let mut index = 0usize;
307
308    // This is the main loop of the lexer. It iterates over the input characters
309    // and tries to match them against known tokens of the language
310    // The `let Some(&(c, next)) = chars.peek()` is known as a pattern match expression
311    // It is also something that is very common in functional languages
312    // It is similar to destructuring in JavaScript or Python but more powerful
313    // Here we are saying, if there is a next character in the input, assign it to `c`
314    // and assign the character after it to `next`
315    while let Some(&(c, next)) = chars.peek() {
316        // This is the main match statement. It is similar to a switch statement in other languages
317        // but more powerful because of pattern matching
318        // Now, the code basically and does the following:
319        match c {
320            // if the current character is 't' then check the next character
321            // because in our grammar, there are two tokens that start with 't'
322            // `true` and `then` and we need to know which one it is
323            // if the `t` is followed by an `r` then it is `true` otherwise it is `then`
324            // and we can move on to the next token
325            // the reset of the rules work in a similar fashion
326            // now you see why lexers typically use regular expressions to match tokens
327            't' => match next {
328                'r' => {
329                    /*
330                    We use the consume function to "consume" the rest of the keyword
331                    and advance the iterator to the next character
332                     */
333                    index += consume(&mut chars, KEYWORD_TRUE, line, index)?;
334                    tokens.push(Token::True);
335                    /*
336                    We use the match_one_of function to check if the next character is
337                    one of the characters we expect to see after the keyword
338                    Imagine that the input might be the invalid program `truefalse`
339                    This is incorrect and we we use match_one_of to detect that
340                    Here we are saying we expect to see either a space, a tab, a right parenthesis
341                    Notice that this would accept the input `true)` which is also incorrect
342                    but that would be detected by the parser
343                    Choosing which errors to catch where is a matter of taste and maybe
344                    a bit of performance
345                    */
346                    match_one_of(
347                        chars.peek().map(|&(x, _)| x),
348                        &[' ', '\t', ')'],
349                        true,
350                        line,
351                        index,
352                    )?;
353                }
354                'h' => {
355                    index += consume(&mut chars, KEYWORD_THEN, line, index)?;
356                    tokens.push(Token::Then);
357                    match_one_of(
358                        chars.peek().map(|&(x, _)| x),
359                        &[' ', '\t'],
360                        false,
361                        line,
362                        index,
363                    )?;
364                }
365                '\0' => {
366                    return Err(LexingError::UnexpectedEOF {
367                        expected: vec!["true".to_string(), "then".to_string()],
368                        column: index,
369                        line,
370                    })
371                }
372                c => {
373                    return Err(LexingError::UnexpectedCharacter {
374                        found: c,
375                        expected: vec!["true".to_string(), "then".to_string()],
376                        column: index,
377                        line,
378                    })
379                }
380            },
381            'f' => {
382                index += consume(&mut chars, KEYWORD_FALSE, line, index)?;
383                tokens.push(Token::False);
384                match_one_of(
385                    chars.peek().map(|&(x, _)| x),
386                    &[' ', '\t', ')'],
387                    true,
388                    line,
389                    index,
390                )?;
391            }
392            'i' => match next {
393                'f' => {
394                    index += consume(&mut chars, KEYWORD_IF, line, index)?;
395                    tokens.push(Token::If);
396                    match_one_of(
397                        chars.peek().map(|&(x, _)| x),
398                        &[' ', '\t'],
399                        false,
400                        line,
401                        index,
402                    )?;
403                }
404                's' => {
405                    index += consume(&mut chars, KEYWORD_IS_ZERO, line, index)?;
406                    tokens.push(Token::IsZero);
407                    match_one_of(
408                        chars.peek().map(|&(x, _)| x),
409                        &[' ', '\t', '('],
410                        false,
411                        line,
412                        index,
413                    )?;
414                }
415                '\0' => {
416                    return Err(LexingError::UnexpectedEOF {
417                        expected: vec!["if".to_string(), "iszero".to_string()],
418                        column: index,
419                        line,
420                    })
421                }
422                c => {
423                    return Err(LexingError::UnexpectedCharacter {
424                        found: c,
425                        expected: vec!["if".to_string(), "iszero".to_string()],
426                        column: index,
427                        line,
428                    })
429                }
430            },
431            'e' => {
432                index += consume(&mut chars, KEYWORD_ELSE, line, index)?;
433                tokens.push(Token::Else);
434                match_one_of(
435                    chars.peek().map(|&(x, _)| x),
436                    &[' ', '\t'],
437                    false,
438                    line,
439                    index,
440                )?;
441            }
442            '0' => {
443                index += consume(&mut chars, KEYWORD_ZERO, line, index)?;
444                tokens.push(Token::Zero);
445                match_one_of(
446                    chars.peek().map(|&(x, _)| x),
447                    &[' ', '\t', ')'],
448                    true,
449                    line,
450                    index,
451                )?;
452            }
453            's' => {
454                index += consume(&mut chars, KEYWORD_SUCC, line, index)?;
455                tokens.push(Token::Succ);
456                match_one_of(
457                    chars.peek().map(|&(x, _)| x),
458                    &[' ', '\t', '('],
459                    false,
460                    line,
461                    index,
462                )?;
463            }
464            'p' => {
465                index += consume(&mut chars, KEYWORD_PRED, line, index)?;
466                tokens.push(Token::Pred);
467                match_one_of(
468                    chars.peek().map(|&(x, _)| x),
469                    &[' ', '\t', '('],
470                    false,
471                    line,
472                    index,
473                )?;
474            }
475            '(' => {
476                chars.next();
477                index += 1;
478                tokens.push(Token::LeftParenthesis);
479            }
480            ')' => {
481                chars.next();
482                index += 1;
483                tokens.push(Token::RightParenthesis);
484            }
485            /*
486            We don't care about whitespace characters so we just skip them
487            */
488            '\t' | ' ' => {
489                chars.next();
490                index += 1;
491            }
492            /*
493            This is the case where we don't recognize the current character
494            as part of the language so we just error
495            */
496            ut => {
497                return Err(LexingError::UnrecognizedToken {
498                    found: ut.to_string(),
499                    column: index,
500                    line,
501                })
502            }
503        }
504    }
505
506    // We add an EOF token at the end of the token stream to indicate we are done
507    // This is used later by the parser to know when to stop or error
508    tokens.push(Token::EOF);
509
510    Ok(tokens)
511}
512
513/*
514This is a helper function that consumes a literal from the input
515It is used to match keywords and other literals in the language
516 */
517fn consume<T>(
518    input: &mut T,
519    literal: &'static str,
520    line: usize,
521    index: usize,
522) -> Result<usize, LexingError>
523where
524    T: std::iter::Iterator<Item = (char, char)>,
525{
526    let mut i = index;
527
528    for expected in literal.chars() {
529        match input.next() {
530            Some((c, _)) => {
531                if c != expected {
532                    return Err(LexingError::UnexpectedCharacter {
533                        found: c,
534                        expected: vec![literal.to_string()],
535                        column: i,
536                        line,
537                    });
538                } else {
539                    i += 1;
540                }
541            }
542            None => {
543                return Err(LexingError::UnexpectedEOF {
544                    expected: vec![literal.to_string()],
545                    column: i,
546                    line,
547                });
548            }
549        }
550    }
551
552    Ok(literal.len())
553}
554
555/**
556 * This is another helper function that is used to match a character against a list of characters
557 * Sometimes we might expecting one character or another so this function tries to do that
558 * and if it cannot it errors
559 * For example, let's say our input program is `pre(0)`
560 * The `scan` function sees a `p` and tries to match it against the `pred` keyword
561 * but when it reaches `e` it cannot find `d` so it errors
562 */
563fn match_one_of(
564    input: Option<char>,
565    literals: &[char],
566    allow_eof: bool,
567    line: usize,
568    index: usize,
569) -> Result<usize, LexingError> {
570    match input {
571        Some(next_next) => {
572            let eof_expectation = if allow_eof {
573                vec!["<EOF>".to_string()]
574            } else {
575                vec![]
576            };
577
578            if !literals.contains(&next_next) {
579                return Err(LexingError::UnexpectedCharacter {
580                    found: next_next,
581                    expected: literals
582                        .iter()
583                        .map(|c| format!("{}", c))
584                        .chain(eof_expectation)
585                        .collect(),
586                    column: index,
587                    line,
588                });
589            }
590
591            Ok(1)
592        }
593        None => {
594            if !allow_eof {
595                return Err(LexingError::UnexpectedEOF {
596                    expected: literals.iter().map(|c| format!("{}", c)).collect(),
597                    column: index,
598                    line,
599                });
600            }
601
602            Ok(0)
603        }
604    }
605}
606
607#[cfg(test)]
608mod lexer_tests_happy_path {
609    use super::*;
610
611    #[test]
612    fn test_scan_empty_string() {
613        let input = "";
614
615        let tokens = scan(input).unwrap();
616
617        assert_eq!(tokens[0], Token::EOF);
618    }
619
620    #[test]
621    fn test_scan_basic() {
622        let input = "true false if then else 0 succ pred iszero ()";
623
624        let tokens = scan(input).unwrap();
625
626        assert_eq!(tokens[0], Token::True);
627        assert_eq!(tokens[1], Token::False);
628        assert_eq!(tokens[2], Token::If);
629        assert_eq!(tokens[3], Token::Then);
630        assert_eq!(tokens[4], Token::Else);
631        assert_eq!(tokens[5], Token::Zero);
632        assert_eq!(tokens[6], Token::Succ);
633        assert_eq!(tokens[7], Token::Pred);
634        assert_eq!(tokens[8], Token::IsZero);
635        assert_eq!(tokens[9], Token::LeftParenthesis);
636        assert_eq!(tokens[10], Token::RightParenthesis);
637    }
638
639    #[test]
640    fn test_scan_mixture_of_whitespaces() {
641        let input = "if\ttrue\t\t then         0 else        succ\t0";
642
643        let tokens = scan(input).unwrap();
644
645        assert_eq!(tokens[0], Token::If);
646        assert_eq!(tokens[1], Token::True);
647        assert_eq!(tokens[2], Token::Then);
648        assert_eq!(tokens[3], Token::Zero);
649        assert_eq!(tokens[4], Token::Else);
650        assert_eq!(tokens[5], Token::Succ);
651        assert_eq!(tokens[6], Token::Zero);
652        assert_eq!(tokens[7], Token::EOF);
653    }
654
655    #[test]
656    fn test_scan_parantheses() {
657        let input = "if (true) then 0 else (succ (pred(0))";
658
659        let tokens = scan(input).unwrap();
660
661        assert_eq!(tokens[0], Token::If);
662        assert_eq!(tokens[1], Token::LeftParenthesis);
663        assert_eq!(tokens[2], Token::True);
664        assert_eq!(tokens[3], Token::RightParenthesis);
665        assert_eq!(tokens[4], Token::Then);
666        assert_eq!(tokens[5], Token::Zero);
667        assert_eq!(tokens[6], Token::Else);
668        assert_eq!(tokens[7], Token::LeftParenthesis);
669        assert_eq!(tokens[8], Token::Succ);
670        assert_eq!(tokens[9], Token::LeftParenthesis);
671        assert_eq!(tokens[10], Token::Pred);
672        assert_eq!(tokens[11], Token::LeftParenthesis);
673        assert_eq!(tokens[12], Token::Zero);
674        assert_eq!(tokens[13], Token::RightParenthesis);
675        assert_eq!(tokens[14], Token::RightParenthesis);
676        assert_eq!(tokens[15], Token::EOF);
677    }
678
679    #[test]
680    fn test_scan_valid_tokens_bad_syntax() {
681        let input = "true)false)if then 0";
682
683        let tokens = scan(input).unwrap();
684
685        assert_eq!(tokens[0], Token::True);
686        assert_eq!(tokens[1], Token::RightParenthesis);
687        assert_eq!(tokens[2], Token::False);
688        assert_eq!(tokens[3], Token::RightParenthesis);
689        assert_eq!(tokens[4], Token::If);
690        assert_eq!(tokens[5], Token::Then);
691        assert_eq!(tokens[6], Token::Zero);
692        assert_eq!(tokens[7], Token::EOF);
693    }
694}
695
696#[cfg(test)]
697mod lexer_tests_sad_path {
698    use super::*;
699
700    #[test]
701    fn test_scan_unrecognized_token_error() {
702        let input = "if    true then 0   else succ    0 1";
703
704        let tokens = scan(input);
705
706        assert_eq!(
707            tokens.err(),
708            Some(LexingError::UnrecognizedToken {
709                found: String::from("1"),
710                column: 35,
711                line: 1,
712            })
713        );
714    }
715
716    #[test]
717    fn test_scan_unexpected_token_error() {
718        let input = "succ( suc ( 0 ) )";
719
720        let tokens = scan(input);
721
722        assert_eq!(
723            tokens.err(),
724            Some(LexingError::UnexpectedCharacter {
725                found: ' ',
726                expected: vec!["succ".to_string()],
727                column: 9,
728                line: 1,
729            })
730        );
731    }
732
733    #[test]
734    fn test_scan_unexpected_eof_error() {
735        let input = "tr";
736
737        let tokens = scan(input);
738
739        assert_eq!(
740            tokens.err(),
741            Some(LexingError::UnexpectedEOF {
742                expected: vec!["true".to_string()],
743                column: 2,
744                line: 1,
745            })
746        );
747    }
748
749    #[test]
750    fn test_scan_literals_no_spaces_unexpected_token_error() {
751        let input = "truefalse";
752
753        let tokens = scan(input);
754
755        assert_eq!(
756            tokens.err(),
757            Some(LexingError::UnexpectedCharacter {
758                found: 'f',
759                expected: vec![
760                    " ".to_string(),
761                    "\t".to_string(),
762                    ")".to_string(),
763                    "<EOF>".to_string()
764                ],
765                column: 4,
766                line: 1,
767            })
768        );
769    }
770}

About Me

2023-09-17T00:00:00+00:00

Hi,

I love learning new things and building interesting stuff

If you are keen, here is my Résumé