Chapter 1: Semantic Compression

#

In this very confusing time of “agentic coding”, I believe we don’t overthink stuff enough. We are so focused on shipping faster and faster that we basically forgot to reinvent react every week.

That is a big problem that I intend to solve.

Allow me to bring you into a mental framework I hallucinated in my brain in the attempt to create a single original thought without asking Claude to do it for me.

What Are Compilers .. Spiritually?

#

You’ve probably heard about “levels of abstraction” and that Python is a “higher level” language than “C”. While that is true, these blank CS graduate statements don’t capture well the philosophy of abstractions.

I like to think about this matter more in terms of compression. If you think about it, Python is just a more compressed version of C. Two images can show the exact same picture, but they have wildly different file sizes because they’re using different compression algorithms. Python is just a higher level of compression.

It’s important to understand that this compression is lossy, because from the point-of-view of the programmer, you loose the control over, for example, low-level memory management - same as with higher lossy compression on an image you loose more details.

Still with me? Alright, let’s continue.

What Happens When We Go Above Programing?

#

Have you ever thought about what would happen if we just kept going?

What if we make a language that’s even higher on the abstraction scale than Python? And then make another one even higher?

We now know the answer to this question - welcome to natural language programming.

Think about it this way: programming is just telling the computer what should happen. Every programming language does exactly that. What’s higher than Python? Pseudocode for visualizing the high-level descriptions of algorithms. Above that? Just verbal descriptions of the logic.

We have now entered the realm where strict syntax ceases to exist.

What Is A Prompt .. Spiritually?

#

“Verbal description of the program’s logic.” - Sound familiar, huh?

Turns out that in this compression framework, at a certain point, programming languages transition to prompts.

Prompts and source code fundamentally sit on the same scale of semantic compression. Both are descriptions of what the final program logic is, that are given to another piece of software.

Now you may wonder … why do we then generate the code as a response to the prompt when the code is more of a decompressed version of the prompt? Good question …

Chapter 2: Sequential Generation Sucks

#

For quite some time now, I’ve been having a recurring thought about the way the current SOTA coding LLMs work. The thing that troubles me is the fact that they write in a sequential fashion, one token at a time, but I have almost never written code in a linear way.

Nowadays, we have “agentic harnesses” that silently hide the fact that the LLM is linearly generating text by allowing it to make continuous diffs to files, but the fact is that the LLM still usually decides on an architecture/api/structure and sticks with it.

But the problem, in my eyes, is when those decisions are made. How can you design a good API for a component without first writing it’s use-cases. To me this feel like starting to paint a painting from the left upper corner in the highest detail possible and only after that corner is finished, moving onto the patch right next to it.

We’ve actually known that writing code is non-linear for quite some time now.

In fact, the very profession of Software Engineering is about designing the architecture without going into details of, for example, the syntax of a specific language. And it’s been known for a very long time that this process is not linear, but rather incremental in the abstraction level domain.

If Not Sequential, Then What?

#

Sequential generation is very good for open ended tasks and for tasks where what’s written is set and stone. Software development is exactly not that sort of a task and it almost feels like it would be the best to generate the entire codebase at once and then iterate on refining details that are finer and finer with iteration. This idea, in fact, is not my own - there is an entire research field into this. Most famously, the exploration of Diffusion LLMs.

What Do Pictures Have In Common With Code?

#

Think about it: iterative generation, starting with a rough draft, iteratively refine details - doesn’t that sounds familiar? If it sounds to you like stable diffusion - you’re exactly right. The idea is very similar, but just a little altered for text generation.

The LLM starts with a fixed-size sequence or random token; in each iteration, it uses it’s learned weights to “refine the details” - nudges the tokens to a more coherent version of what tokens should actually be there - and this repeats until the model is satisfied with the output.

Chapter 3: What’s Next?

#

If you’re like me, you also see the incredible snowball effect of “agentic engineering”. The bottleneck of this process is the human and it will continue to be the bottleneck until we release the human from the development loop.

So here’s some food for though: How do we scale agentic engineering?

I think we shall approach the problem using the semantic compression. What really should be focusing on the lost details of the compression, because that is exactly what the coding agent has to assume when decompressing your prompt.