Week 2: Codegen, AST and VSCode!

My first task this week is picking up from the codegen issue last week.

So to summarize, I need some good changes for codegeneration as a whole. So a while ago, I reused some concepts from yacc (or rather any parser) and the idea was to have a stack property, and this would be modified accordingly by each statement.

code:ECLCode[];

However, this is difficult to visualize – Each node of the parse tree will have some assumptions about the stack – ie. eg. After visiting a definition statement, the select statement will assume the definition statement would have put its code on the stack, can it can be used from there. This is good, but this does really complicate the whole process.

So, its easier to have each visit() statement return an instance of ECLCode, ie everything that it has translated. So once last week’s bug was fixed and we set have set up the new Parse Tree Visitor, we can move onto the next step!

Output

Taking a detour, OUTPUT is a very important statement. Most programs process some input and give some output and, well, the OUTPUT statement allows for just that.

Working out the particular syntax, we can see one issue from the beginning, ECL has an oddity with OUTPUT statements –

// this works
OUTPUT(someSingularValue,NAMED('hello1'));
// this works
OUTPUT(someTableValue,NAMED('hello2'));

//this does not work
OUTPUT(someSingularValue,,NAMED('hello3'));
//this works
OUTPUT(someTableValue,,NAMED('hello4'));

// this, should not work
OUTPUT(someSingularValue,'~output::hello5');
// but this doesn't work too!
OUTPUT(someTableValue,'~output::hello6');

// Although this works! What
OUTPUT(someTableValue,,'~output::hello7');
// this fails to compile too
OUTPUT(someTableValue,NAMED('hello8'),'~output::hello8');
// but this works!
OUTPUT(someTableValue,,NAMED('hello9'),'~output::hello9');

So, here’s the OUTPUT statement documentation, and seeing it, it becomes rather obvious what is happening.

Tables require a record expression, which is optional as in it may be left entry. For the first table output (hello2), it gets recognized as an expression and it can pass as correct syntax. But the latter ones, it can only be the table record variant, and syntaxes with only one comma, fails.

For reference, we can use SQL. SQL is mostly only tables. So, following that, I decided that the easiest (and maybe the laziest) is to not support singular (aka not table) data outputs. So, working that out will be two parts:

1. Error on trying to output singular values
2. Warnings on trying to output possibly singular values – eg. any values.

All right, with all that done, we can get around to get OUTPUT working.

Codegeneration

With the codegeneration issues out of the place, it was a full speed ahead moment. And in no time at all, its done, OUTPUT and assignments.

VSCode

All right, this is the meat of what’s part of HSQL; An easy way to deal with HSQL Code. And usually that means a not so easy way for us to deal with HSQL code. I’d worked with some language server work before (shameless plug here), and one of the things that was interesting was the recommendation to use Webpack to bundle the extension. The idea of the language server/client is very simple. There are two components:

1. Language Client – Its the extension that plugs into the IDE. It feeds the language server any IDE contents, and what actions the user may be taking, and then takes any language server feedback and feeds it into the IDE.
2. Language Server – It is a separate program that runs (maybe independently) and listens and communicates with API

So, the way the extension in VSCode is coded, is that you just start up the language server (or connect to it) and then you can communicate with it (uses JSON RPC).

It is apparent that what I needed was two repositories under a parent repo; where the parent repo had all the extension related content, and the child repos would have the language server and the client. And with that, I had to create a webpack config, to transpile the typescript projects, and generate two outputs, client.js, which would be executed as part of the extension, and server.js, which could be separately executed, but was mainly going to be bundled with the client.

I wrote up a nice webpack config, and it threw an error. Oh no.

This, took way too long. The majority of the day in fact; and I was knee deep trying to interpret what the compilation errors meant before the fix was apparent:

One entry.

transpileOnly: true

The trick was to tell ts-loader to not bother itself with anything other than transpiling and finally, it worked!

Pretending that didn’t happen, I pulled in older resources such as the textmate grammar for HSQL (It works pretty well for now), and added in some support code and voila, a working extension (I mean it didn’t do anything much yet, but atleast it compiled).

What was really interesting is that while the whole project was consuming atleast over a 100MB of space, Webpack easily pulled it down to less than 10MB for the two JS files (combined). This is a good indicator of why packaging is important in Javascript.

Wrapping up

With all this, I plan move into the next week, with the following tasks directly lined up:

1. Have a compilation command for the HSQL extension.
2. Explore some webpack optimizations? Perhaps like chunk splitting since the client and server will share the hsqlt module in common.
3. Have some syntax checking support fixed in. This will remain as a good way to allow people to test as the compiler tool evolves.
4. Start work on SELECT’s AST. This has been long pending, and its time to start the possibly biggest section of HSQL.