Tokenizing and parsing examples in Ruby

The ruby programs linked here show simple examples of tokenizing, checking, parsing, and translating from one language to another.

The source language we are translating from is a very simple toy language, with the specs given here: IttyBitty.

The translation approach is done in steps:

  1. tokenize the source code

  2. check the tokenized code for syntax

  3. build a parse tree from the tokenized code

  4. translate the parsed code

A coordinating ruby script is provided to translate an IttyBitty file to C (it takes the name of the input file as a command line argument, and displays the resulting C code)

A batch test script is also provided, that runs all the routines above on a collection of test cases.

Sample IttyBitty program Resulting C translation

start = 10
end = 200
index = start
check = 150
total = 0
while ( index < end ) {
   total = total + index
   if ( index == check ) {
      total = total - check
   }
   index = index + 5
}
print " the final total is "
print total



#include <stdio.h>

int main()
{
   int start, end, index, check, total;
   start = 10;
   end = 200;
   index = start;
   check = 150;
   total = 0;
   while (index < end)
   {
      total = total + index;
      if (index == check)
      {
         total = total - check;
      }
      index = index + 5;
   }
   printf("the final total is \n");
   printf("%d\n", total);
   return 0;
}