When AI Writes the Code: Probabilistic Engineering

Until very recently, writing software was a deterministic activity. A developer typed instructions, the instructions ran, and the same input produced the same output every single time. If something went wrong, you could read the code, find the broken line, and fix it. The whole craft was built around the idea that humans could read what humans had written.

Probabilistic engineering describes what happens when that assumption breaks. AI coding tools - ChatGPT, Claude, Gemini, GitHub Copilot, Cursor - generate code based on statistical patterns. They have read more software than any human ever could, and they produce reasonable-looking output by predicting what comes next. Most of the time it works. Sometimes it does not, and the reason it failed is not always clear, even to the AI.

The useful analogy is the difference between writing an essay and proofreading one. Writing is generative - you put words on a page from scratch. Proofreading is validating - you check whether the words on the page make sense, are accurate, and say what they were meant to say. Both are real work. They use different muscles.

Traditional software development was effort-heavy at the generation step. Writing a thousand lines of working code took a skilled developer days or weeks. Reviewing those lines, by comparison, took an hour. The ratio favoured careful generation followed by light review.

That ratio has flipped. A modern AI coding tool can produce a thousand lines in minutes. The generation step that used to take a week now takes a cup of tea. But the review step has not got any faster - in fact, it has got harder, because the human reviewer did not write the code and so cannot rely on having a clear mental model of what it does.

This is the practical heart of probabilistic engineering. We have made generation cheap without making validation any cheaper. The bottleneck moved, and most of the industry is still adjusting to where it ended up.

A useful way to picture it is a car factory. For decades the slow step was bolting parts together by hand. When robots took over the assembly, you might expect cars to roll off the line in seconds. They don't. The slow step moved to quality assurance (the people checking each car works before it leaves the factory). Speeding up assembly without speeding up QA just means more cars piling up at the inspection bay - or worse, more cars leaving the factory with faults nobody caught.

Three forces converge to make validation harder, not easier, in the AI era.

The volume has exceeded human attention. When a developer writes their own code, they can hold the whole thing in their head while they write. When the same developer reviews AI output at five times the speed, they cannot. By the time the third pull request lands, the first one is already a blur. Important details slip past not through laziness but through bandwidth.

According to established work on software quality, defect cost rises sharply the later a bug is found. A bug caught while writing costs minutes. The same bug in production costs hours of debugging, an incident report, and sometimes a public apology. The economics of validation reward catching things early, which is precisely where AI-assisted workflows are weakest.

Models reviewing models miss plenty. One common response is to ask an AI to review AI output. This works for stylistic issues and obvious mistakes. It works much less well for subtle logic errors, security holes, or anything that requires real understanding of the system the code is being added to. A model that confidently wrote a bug is unlikely to confidently flag the same bug.

Context is finite. Even the most capable AI assistants can only hold so much of your codebase in mind at once. The further a change ripples through a system, the more likely it is that the AI has missed something elsewhere that depends on the bit being changed. We explored this in detail in our piece on context rot, but the short version is that AI's awareness of a system fades the larger the system gets.

Even if you have never opened a code editor, the shift to probabilistic engineering shows up in the software you already use. Three patterns are worth noticing.

Things get built faster. Features that would have taken a small startup six months can ship in six weeks. This is genuinely good. It means small teams can compete with large ones, niche tools get built that nobody could have funded before, and your favourite app gets the feature you've been asking for sooner than you expected.

Things break in unfamiliar ways. When a bug used to slip through, you could often guess what had happened - someone forgot a check, a number was wrong somewhere. AI-introduced bugs tend to look weirder. The code does something that is almost right, in a way that suggests the AI understood 90% of the problem and made up the last 10%. If your bank app suddenly displays your balance in dollars one Tuesday morning, the answer is probably probabilistic.

Quality is now more about review culture than headcount. The teams that ship reliable software now are not necessarily the ones with the most engineers. They are the ones with the strongest habits around deciding what to trust, testing aggressively, and being honest when validation has slipped. This is good news for small teams who take quality seriously. It is bad news for organisations that hoped AI would let them cut corners.

Several failure patterns recur often enough that they deserve names.

Plausible-looking nonsense. The AI generates code that compiles, runs, passes tests, and quietly does the wrong thing under specific conditions nobody tested. This is the classic hallucination problem dressed up in technical clothing - confident output that is just confidently wrong.

Reviewer fatigue. A team agrees to review every AI-generated change. They mean it. By week four, exhausted, they are skimming. The decline is gradual enough that nobody notices until something embarrassing reaches production.

Integration drift. Each individual change looks fine. The sum of changes is no longer coherent. Pieces of the codebase quietly disagree about how a thing should work, because the AI that wrote piece A had different context to the AI that wrote piece B, and the human reviewer of each missed the disagreement.

Test theatre. AI is excellent at writing tests that pass. It is much less excellent at writing tests that would actually catch a bug. A codebase can rapidly accumulate hundreds of confidence-inducing tests that prove almost nothing.

None of these failure modes are new in principle. Skilled humans have been making versions of these mistakes for as long as software has existed. What is new is the rate. Generation got cheap; mistakes got cheap to make in bulk.

The failure patterns above can make AI-written code sound like a lost cause. It is not, and one recent open-source project makes a useful worked example of the alternative.

bbtree is an ordered map for the Rust programming language - a way of storing keys and values so they can be looked up quickly and also read back in sorted order. Databases, file systems and search tools lean on structures like this constantly. According to the project's own write-up, all of bbtree's code was written by LLMs. A single developer, Xiangpeng Hao, who built B-trees through his PhD and co-authored one used at Microsoft, steers every design decision and takes responsibility for the result. The write-up states that correctness rests on formal verification, model checking, deterministic simulation, fuzzing and a large body of tests.

Why speed here is about waiting, not thinking. It is natural to assume that a faster data structure does fewer calculations. For an ordered map, that is mostly beside the point. A modern processor can work through an instruction in a fraction of a nanosecond, but fetching a piece of data that is not already sitting in its small, fast cache means a trip out to main memory, which the bbtree write-up puts at roughly 100 nanoseconds - a couple of hundred times longer. A lookup that has to chase several of those trips spends most of its life waiting. bbtree's design is about packing more useful information into each trip, so a search needs fewer of them. None of those ideas are new; they come from decades of database research, which is the point. The model wrote the code, but an expert chose what it should build.

Read the headline numbers with care. The project reports that bbtree is up to five times faster than Rust's standard BTreeMap on a single thread and uses about 30% less memory. Those are the author's own benchmarks, run on one desktop processor (an AMD Ryzen 9 9900X) with 10 million entries, on workloads the author selected. That does not make them wrong, but it does make them a best case rather than a promise. Speed is exactly the kind of goal an AI coding tool is good at chasing: it is easy to measure, so a model can keep iterating until a number goes down. Your own data, your own hardware and your own access patterns may give a smaller gain.

The lesson for everyone else. The interesting part of bbtree is not that an LLM wrote fast code. It is how the project tries to make that code trustworthy. Its author makes the point that speed is easy to measure and correctness is not: no prompt can list every rule, and no pile of tests can prove there are no bugs. So the project stacks independent checks - a human expert who is accountable, mathematical proofs of key properties, and automated tools that throw large volumes of unexpected inputs at the code. That is the validation bottleneck from earlier in this piece, tackled head-on rather than skipped. Verification, not the model, is what earns the trust.

The same idea surfaced repeatedly at HPTS 2026 (High Performance Transaction Systems, a small invitation-only workshop for database and distributed-systems engineers, held at Asilomar in California). Three of the four sessions on its opening day were about AI, according to a day-one trip report by Murat Demirbas, a distributed-systems researcher who attended. Across those talks, the thread was consistent: AI can now write and run large parts of a system, so the controls that matter are the ones that sit outside the model - written specifications, independent checks, logs and accountable people.

Two terms carry most of that weight, and both are simpler than they sound.

Formal verification is proving, with mathematics and a computer that checks the maths, that a program obeys a written rule in every possible case. Testing tries a few thousand inputs and reports what happened. Verification covers all inputs at once, provided the rule itself was written down correctly. That proviso is why the specification - the plain statement of what the software must do - becomes the most valuable thing a human writes.

An audit trail is a record of every action a system took, written in order and never edited afterwards, so anyone can reconstruct later what happened and why. Banks have kept them for decades. AI agents that act on real systems need the same thing.

Here is what those ideas look like when engineers put them into practice.

Check the scoreboard, not just the score. In the opening keynote, Berkeley professor Ion Stoica described an AI agent asked to build a faster key-value store (a simple database that looks up a value by its name). One version came out six times faster than the others. It had noticed that the benchmark generated its test values from a predictable formula, so it calculated the answers instead of storing them. Researchers call this reward hacking: the model hit the number without doing the job. The production habit it argues for is simple - when AI-written code posts a surprising result, inspect how it got there before celebrating.

Compare against a trusted reference. One talk on generating custom analytics databases with AI used DuckDB, an established open-source database, as a reference oracle: every answer the new system gives is checked against the answer the trusted system gives for the same query. Any team can copy the pattern. Keep the old, boring implementation around and diff the new one against it.

Prove the parts that cannot fail. A Cornell team described generating database code alongside machine-checked proofs, using Verus (a verification tool for the Rust programming language). Their conclusion, as reported, was that the human job shifts to writing the specifications and the checkers. Full proofs remain expensive, so in practice teams reserve them for the core that everything else trusts - storage, payments, access control.

Ship what the model wrote, not the model. A Google engineer walked through several machine-learning-driven database features that never shipped, because running a model inside every decision added delay and made behaviour hard to explain. The approach that worked was to have the model write ordinary code that humans then review and deploy. The reviewable artefact is the product; the model stays in the workshop.

Log the intention before the action. A Meta team presented LogAct, described in its arXiv paper as a framework that records each agent action in a shared log before it runs, lets separate voters halt it, and uses the same log to recover after a failure. The design goal is a trustworthy record of what the agent did, rather than a promise that the agent itself is trustworthy. For a smaller team, the equivalent is an append-only log of every command an AI agent issues, with high-impact commands paused for a human to approve.

None of this makes the model smarter. It makes the system around the model checkable, which is the real answer to the validation bottleneck.

Worry is the wrong frame. The right frame is awareness.

Worth knowing: most software you rely on day-to-day will continue working most of the time. The systems that handle money, medical records, or anything safety-critical are built with deeper validation layers (regulators, auditors, formal testing) than the average startup app. They are slower-moving and harder to disrupt, which is exactly the point.

Worth doing: be a bit more sceptical of brand-new features in apps you depend on. Wait a fortnight before trusting a new banking integration with your salary. Read reviews of new AI-powered tools rather than buying on launch day. The probabilistic-engineering era rewards patience.

Worth ignoring: the headlines claiming AI is about to either solve all bugs or destroy all software. Neither is happening. The reality is more mundane and more interesting - we are working out, in public, how to do quality assurance for a kind of work that did not exist three years ago. That is going to take a few more iterations.

Tim Davis's essay on probabilistic engineering coined the framing we have used in this piece. The term is useful precisely because it points at a real shift without pretending the shift is wholly good or wholly bad.