Use ChatGPT for compiler error regeneration

ChatGPT is so hot that everybody is trying to use it everywhere. Recently I use it in my compiler to help users on error regeneration. Here's how I do it.

What's the compiler?

Laco is an optimized compiler for compact embedded system written by me. It supports Scheme programming language. In this post, I'm going to write some code with bugs intended. And compare the regular error message and the AI reviewer advice.

Here's a trivial Scheme program:

(let ((a (1)))
 (display b))

It means to bind variable a with the value (1), and print the undefined variable b. Well, obviously it's wrong code. (1) means to apply integer 1 as a function which is definitely wrong. If we run it with any Scheme compiler/interpreter, it may throw error like this:

Usually, the error message is not friendly for the human newbies.

Now let's try ChatGPT

First, you need an OpenAI account for API token. Now let's set the token string to an environment LACO_OPENAI_KEY;

export LACO_OPENAI_KEY=your_openai_key

The logic is simple, if the compiler encounters compiling error, and LACO_OPENAI_KEY was set, then Laco compiler will wrap the source code with certain prompt to send to the OpenAI server for advice. The ChatGPT may randomly give you different advice each time, most of the time, it tells you what's wrong with the code and how to fix it properly. Here's one possible result:

Conclusion

Is AI good enough for error regeneration? I have to say it's not what I expected, but it works somehow. I've tried many prompts to get an expected result, however, I didn't find a perfect one. Sometimes the AI is too verbose but miss the point.

The critical limit is that AI has to get the hint that "the code can't be compiled". However, for our code example, the (1) part usually detects in the runtime. Fortunately, my Laco compiler can detect this error in the compile time, so the AI correctly get to the point.

What if the compiler can't detect the runtime error? The AI may randomly (and very likely) to congratulate your code is completely correct. So it seems that the current AI (GPT 3.5 turbo) is not good enough for the work on its own. But if you've confirmed the code is wrong, the AI will give you more friendly advice to fix the bug.

Is the advice good enough? Well, it depends. AI doesn't know the correct code in your mind, so even it find out the wrong part, it may not give you the expected fix. For example, in our case, AI may tell you (1) is wrong, but the fix I expected is to get rid of the parens, say, 1. But the solution from AI is randomly to add a quote to form a list, say, '(1). I can't blame AI, but it's a wrong fix for me. Anyway, this problem is impossible to have a perfect solution unless AI was connected to my brain.

I think the advice may not very useful for veteran, but still useful for newbies. Dunno, I haven't tested complex program. IMO, the AI is more like an assistant or tool at present.

What do you think? Comments are welcome.