I'm no longer sure you have to, actually. I mean, we do trust the assembly that compilers produce without having to read it, don't we? We're rapidly getting to that stage with LLMs, IMO.
The assembly is a deterministic transform of the input logic, and if it doesn't match then it's a bug in the compiler. If an LLM-based code generator doesn't match what you asked for, that's OK, just pull the slot-machine handle again. that's the difference.
The "pull the slot-machine handle again" is the dangerous thing here.
I can feel it sometimes, as my brain shuts down and I gamble instead of thinking. It's a reversion to what I call "monkey mind" where you just keep pressing buttons to "make it work". I took a decade training my mind away from this, and too much AI is bringing it back.
The issue here is not just the gambling aspect, but verification that you got what you asked for. Code problems can be extremely subtle, and the cost of verification is deceptively high, ruining the advantages of using AI in the first place.
I truly believe that people claiming huge productivity gains from AI are either terribly slow programmers or are skipping their due diligence. Many "vibe coders" are incapable of checking the output of the code.
And then getting bugs when they use a new version of the AI, just like people occasionally got bugs when they upgraded to new versions of the compiler...
they would get bugs on every invocation of the software, not on a new version of the AI. it's equivalent to your compiler have a RAND function in it where it chooses between a billion different options every time it compiles, it's absolutely not equivalent to a compiler having a bug.
They’re not, and will never be in their current form and architecture.
Compilers are mechanical and engineered to produce a correct output. A compiler emitting incorrect machine code is exceedingly rare, and considered a bug. They have heuristics and probabilities in them, but those are to pick between a set of known-good outputs.
An AI is a bag of weights outputting a probability of the most plausible token that follows [1]. It is inherently probabilistic in nature and its output is organic (by design, they’re designed to mimic human speech), as opposed to mechanical like a compiler.
A compiler follows hard rules. An AI does its best.
And to be fair, AIs are no better than human in this regard: humans are pretty bad at generating correct code without mechanical tools to keep them in line (compilers, linters, formatters). It’s not a wonder we use the same tools to keep LLM output in line as we do humans. (And, to be fair, LLMs are better than humans at oneshotting valid code).
[1]: to those that tell me this vision of an LLM is outdated: nope. The heavy lifting is done in the probability generation. Debates about understanding are not relevant here, and the net output of an LLM is a probability vector over raw tokens. This basic description can be contrasted to a compiler whose output is a glorified Jinja template.
Not OP, but it means nothing, because it's not "effectively" becoming a compiler.
Think about it from an information theory standpoint:
A compiler takes at least the exact amount of information it needs to produce a result, and produces exactly that result every time (unless it's bad at its job or has a bug).
An LLM always takes far less information than would actually be needed to fully describe the desired output, and extrapolates from that. It fetches contexts and such to give itself a glut of assumedly relevant information, but the prompt always contains less information than necessary to produce the code it generates. If it did fully contain enough information, then you've just written a far more verbose version of the program in human language.
Yes, I am saying there's no functional difference (for practical purposes) between a deterministic transformation like a compiler and a perfect probabilistic transformation like an LLM.
We do not have "perfect" probabilistic transformation, and we probably never will (in part because it's hard to know what exactly that even means), but the gap between the two is shrinking every day.
Ergo:
> they're becoming (effectively) more and more similar every day.
I know it’s tiring to talk about “hallucination”, but truly, models still do hallucinate
They constantly say they did a thing they didn’t, say they know how to solve something when they don’t, etc. Regardless of guard rails or tests - AI forces a constant vigilance of a new kind.
Not just “what might have gone wrong” but also “what do I think is working but isn’t actually”.
And we’re not even talking about how it chooses substandard solutions, is happy to muddy code/architectures, add spaghetti on top of spaghetti etc.
Agentic coding often feels like an army of unexperienced developers who are also incredibly eager to please.
"Still" means "it always had hallucinations, and it still does, despite people thinking that it doesn't anymore". People think we've moved past that. We haven't.
This is a really, really, really bad comparison. I used to say the same thing. But the semantic distance between compiling a for loop to equivalent assembly instructions is much smaller than the distance between "I'd like a web application that can store and retrieve todo items." The space of the latter is practically infinite in what can be "compiled."
A counterpoint, since I never made that logical jump in your latter part of your comment: programming languages are, functionally, all domain-specific languages and do a good job of either describing directly, or consistently, deterministically, providing a reasonable and unambiguous abstraction over low-level concepts expressed by assembly languages.
Human languages are mostly very bad at this, and in particular bad at mapping low-level abstraction to the human written word unambiguously in a way that is as expressive as programming languages.
Inference closes that specific gap significantly (which is why anyone at all sees LLMs as a useful option to explore), but it will never be as good as a purpose-built language designed to map to a reasonable corresponding assembly language implementation.
> we do trust the assembly that compilers produce without having to read it
Yes, because wrong assembly blows really loudly. From wrong behavior to invalid instruction errors and everything between them. Moreover, compilers are battle tested over the years, with extremely detailed test suites, and extreme testing (everyday, hundreds of thousands users test and verify them).
Also, as people said, assembly generation is deterministic. For a given source file and set of flags, you get the same thing out. Byte by byte, bit by bit. This is what we call "reproducible builds".
AI is not like that. It's randomized on purpose, it pulls from training set which contains imperfect, non-ideal code. "Yeah, it works whatever", doesn't cut it when you pull a whole function out of its connections, formed by the training data. It can and will make errors, because it's randomized from a non-ideal pool.
Next, sometimes you need tight code. Fitting into caches, running at absolute performance limit of the processor or system you have. AI is not a good fit here. Sometimes you go so far that you optimize for the architecture at hand, and it works slower on newer systems, so you need to re-optimize that thing.
For anyone who reads and murmurs "but AI can optimize", yes, by calling specific optimization routines written by real talented people for some cases; by removing their name, licenses, and context around them. This is called plagiarism in its mildest form and will get you in hot water in academia, for example. Writing closed source software doesn't make you immune from cheating and doing unethical things.
Lastly, this still rings in my ears, and I understood it over and over as I worked with more high performance, correctness critical code:
I was taking an exam, there's this tracing question. I raise my head and ask my professor: "Why do I need to trace this? Compiler is made to do this for me". The answer was simple yet deep: "If you can't trace that code, the compiler can't trace it either".
As I said, I just said "huh" at the time, but the saying came back and when I understood it fully, it was like being shocked by a Tesla coil.
Get your sleep, eat your veggies and understand your code. That's the four essential things you need to do.
I've actually taken to double-checking the assembly in some instances. There are surprising times that the compiler won't make the shortcuts and optimizations you thought it should, and I also used this method to call out an unsuitable compiler since I caught it spitting out some ridiculous 10x-long set of instructions in certain critical instances.
We are not rapidly getting to that stage with LLMs and frankly it's hilarious that you are claiming so.
For anything other than Greenfield, new code projcets without dependencies and conventions and connections to other proprietary code, it has to be reviewed. Even for that case it's not good to not review code
