August 2, 2026

Code Hallucinations

On the dangers of trusting AI coders

Let me start by saying that I am a strong supporter of AI, at least when it comes to coding. No doubt it boosts significantly my productivity. However, this latest tech revolution is no different from previous ones. Those convinced that they can simply swap workforce for machinery will get bitten by the same disappointment as every time before. Machinery does displace work — power looms and telephone switchboards settled that argument long ago. What it does not displace is the person who can tell whether the output is right. Where that judgement is the expensive part, and in systems programming it usually is, the gain arrives as 'more or better with the same people', never as 'the same with fewer'. A word on the title, before anything else. What follows is not a hallucination in the usual sense: nothing was invented, no API was made up, no library that doesn't exist. It is worse than that. It is code that is well designed, idiomatic, compiles clean, and is wrong in three places that no compiler and no linter will ever tell you about.

The stakes

Software performance is important in any industry. In the trading industry, though, software performance is a key success (or failure) factor. Being forced to always find the most optimal way to lay down code doesn't take long to become a habit, a bone-marrow instinct that eats tranquillity and, sometimes, sleep. Concretely, I am writing some binary serialization code meant to wire domain objects in a highly performant network bus, part of a 'Strangler Fig' work to modernize an existing trading platform into a microservices architecture.

The stack

Outside of the hot paths, where latency commands C++ (and sometimes Assembly), my stack of choice is .NET. And my language of choice is F#, with C# on the few F# pain points and whenever the tooling makes it convenient. This is another point where AI has its say. It appears that thanks to code agents we are all becoming suddenly polyglots, and half of that is true. We are becoming polyglot readers, genuinely: point an agent at code in any language and it will tell you what it does. But we are becoming polyglot writers at the same time, and that half is dangerous — producing code in a language, and for a stack, whose rules we do not actually know. This article is evidence of the price of that gap, bearing in mind that F# is a language I do know.

The problem

The bus API accepts a bare byte array, and the first implementation was using ArrayBufferWriter<byte> to hand-roll the serialization rather than relying on well known binary serializers (MessagePack, Protobuf, ...) that ease life but at a cost of being not optimal. ArrayBufferWriter<> implements IBufferWriter<>, that relies on the highly optimized Span<T> and Memory<T> structs, but it is append only. This is not a limit per se, but when serializing a complex type, where fields are not of primitive type, it becomes impossible to know in advance the lengths of the chunks that will need to be laid down in the buffer. This forces to allocate multiple buffers, only to be able to retrieve the serialized field length and put that length into the final array, let alone the necessity to copy that nested serialized field and dispose of the temporary buffer putting pressure on the Garbage Collector. As such, an evident optimization is to reach out to utilities like ArrayPool<T> and create a custom binary writer with some smart caching and pooling. I reached out to my favourite code agent, wrote an exhaustive content, my requirements and let it work, until it spit out this:

[<Sealed>]
type PooledWriter private () =
    [<ThreadStatic; DefaultValue>] static val mutable private _pool: Stack<PooledWriter>
    let mutable _buffer = ArrayPool<byte>.Shared.Rent(1024)
    let mutable _pos = 0

    let ensure n =
        if _pos + n > _buffer.Length then
            let mutable cap = _buffer.Length * 2
            while cap < _pos + n do cap <- cap * 2
            let next = ArrayPool<byte>.Shared.Rent(cap)
            Buffer.BlockCopy(_buffer, 0, next, 0, _pos)
            ArrayPool<byte>.Shared.Return(_buffer)
            _buffer <- next

    let rest n = ensure n; _buffer.Length - _pos

    static member private Pool =
        if isNull (box PooledWriter._pool) then PooledWriter._pool <- Stack()
        PooledWriter._pool

    member __.Reset() = _pos <- 0

    static member Rent() =
        let pool = PooledWriter.Pool
        if pool.Count = 0 then PooledWriter() else let w = pool.Pop() in w.Reset(); w

    member __.Position = _pos
    member __.ToArray() = ReadOnlySpan(_buffer, 0, _pos).ToArray()
    member __.CopyTo(destination) = ReadOnlySpan(_buffer, 0, _pos).CopyTo(destination)
    member __.Span(n) = ensure n; Span(_buffer, _pos, n)
    member __.Advance(n) = _pos <- _pos + n
    member __.PatchInt32BigEndian(offset, value) = BinaryPrimitives.WriteInt32BigEndian(Span(_buffer, offset, 4), value)

    member this.Return() =
        this.Reset()

        if PooledWriter.Pool.Count < 8 
            then PooledWriter.Pool.Push(this) 
            else ArrayPool<byte>.Shared.Return(_buffer)

    interface IBufferWriter<byte> with
        member this.Advance(count) = this.Advance(count)
        member __.GetMemory(sizeHint) = Memory(_buffer, _pos, rest (max sizeHint 256))
        member __.GetSpan(sizeHint) = Span(_buffer, _pos, rest (max sizeHint 256))

Not bad, I'd say. Great choice to make the pool thread static, avoiding multithread races and the consequent necessity for locks or barriers — at the modest price of up to eight writers being retained by every thread that ever encodes. Also, the feature of being Patchable is exactly what saves a lot of auxiliary buffer allocations for nested fields. Note in fact the method PatchInt32BigEndian, built on BinaryPrimitives. We can simply skip 4 bytes for the length unassigned, go and serialize the nested fields in place and patch back the length after, no need to instantiate (and then GC) temporary buffers, no need for copy-back to the main buffer. It looks good, and it builds with no errors and no warnings. Nobody sensible believes an agent outsmarts a good developer. But plenty of us have quietly started believing something softer and far more dangerous: that output like this is good enough that careful review is optional. Well it turns out that this code is buggy, and not in an evident way. At first sight it looks correct. Read it with more attention and three serious defects come out, the kind that hide deep in the haystack and bite you down the path, while you are celebrating your fast serializer.

Three defects

1. A span over a dead array

member __.GetSpan(sizeHint) = Span(_buffer, _pos, rest (max sizeHint 256))

rest calls ensure, and ensure may rent a bigger array and rebind _buffer. But arguments are evaluated left to right here, so _buffer is read first — before the reallocation. The Span is constructed over the array we just abandoned, and everything the caller writes into it is silently thrown away.

It only fires once a payload outgrows the initial 1 KB. Small messages work forever.

2. A buffer returned while still in use

ArrayPool<byte>.Shared.Return(_buffer)
_buffer <- next

ensure hands the outgrown array straight back to the pool. But a caller may still be holding a Span over it — that is the whole point of IBufferWriter. Another thread rents the same array, and two pieces of code write into the same memory. Under load, in production, intermittently.

The documentation for ArrayPool<T>.Return does not leave this open to interpretation:

Once a buffer has been returned to the pool, the caller gives up all ownership of the buffer and must not use it. [...] Returning the same array reference twice or continuing to use the array reference after it has been returned is a high-severity security issue. These actions can lead to double-free and use-after-free vulnerabilities, which might result in data corruption, data leaks, and denial of service.

The fix is almost offensive in its simplicity: don't return it. Let the GC take it. One allocation, on a path that is rare by construction.

3. A release that isn't idempotent

member this.Return() =
    this.Reset()
    if PooledWriter.Pool.Count < 8
        then PooledWriter.Pool.Push(this)
        else ArrayPool<byte>.Shared.Return(_buffer)

Call this twice and you either push the same instance onto the pool twice — after which two callers Rent() the same writer and interleave their writes — or you hand the same array back to ArrayPool twice, which is the double-free the documentation quoted above calls a high-severity security issue. A guard flag fixes it, and it has to guard both branches.

Two of the three defects, then, are the two failure modes that one paragraph of Microsoft documentation warns about by name.

What the three have in common is worth stating plainly: none is a logic error you can reason about from the code in front of you. Each requires knowing an invariant that lives outside it — argument evaluation order, ArrayPool ownership semantics, the release-idempotency contract. And all three are invisible until the buffer grows or the pool wraps, which is to say: not in your unit tests.

Here is the corrected version.

[<Sealed>]
type PooledWriter private () =
    [<ThreadStatic; DefaultValue>] static val mutable private _pool: Stack<PooledWriter>
    let mutable _buffer = ArrayPool<byte>.Shared.Rent(1024)
    let mutable _pos = 0
    let mutable _returned = false

    // The outgrown buffer is dropped for the GC rather than returned to the pool:
    // a client may still hold a Span over it, and returning it would
    // let another thread rent the same array and corrupt those writes.
    let ensure n =
        if _pos + n > _buffer.Length then
            let mutable cap = (max _buffer.Length 1) * 2
            while cap < _pos + n do cap <- cap * 2
            let next = ArrayPool<byte>.Shared.Rent(cap)
            Buffer.BlockCopy(_buffer, 0, next, 0, _pos)
            _buffer <- next

    let rest n = ensure n; _buffer.Length - _pos

    static member private Pool =
        if isNull (box PooledWriter._pool) then PooledWriter._pool <- Stack()
        PooledWriter._pool

    member private __.Reuse() = _pos <- 0; _returned <- false

    static member Rent() =
        let pool = PooledWriter.Pool
        if pool.Count = 0 then PooledWriter() else let w = pool.Pop() in w.Reuse(); w

    member __.Position = _pos
    member __.ToArray() = ReadOnlySpan(_buffer, 0, _pos).ToArray()
    member __.CopyTo(destination) = ReadOnlySpan(_buffer, 0, _pos).CopyTo(destination)
    member __.Span(n) = ensure n; Span(_buffer, _pos, n)
    member __.Advance(n) = _pos <- _pos + n
    member __.PatchInt32BigEndian(offset, value) = BinaryPrimitives.WriteInt32BigEndian(Span(_buffer, offset, 4), value)

    // Return must be idempotent. The flag guards BOTH branches: without it a
    // second call either pushes this instance onto the pool twice -- two callers
    // then rent the same writer -- or hands the same array to ArrayPool twice.
    member this.Return() =
        if not _returned then
            _returned <- true
            _pos <- 0
			
            if PooledWriter.Pool.Count < 8 
            then PooledWriter.Pool.Push(this)
            else
                ArrayPool<byte>.Shared.Return(_buffer)
                _buffer <- Array.empty

    // `rest` may reallocate _buffer, and arguments are evaluated left to right --
    // so it must be bound BEFORE _buffer is read, or the span is built over the
    // discarded array and everything written into it is lost.
    interface IBufferWriter<byte> with
        member this.Advance(count) = this.Advance(count)
        member __.GetMemory(sizeHint) = let n = rest (max sizeHint 256) in Memory(_buffer, _pos, n)
        member __.GetSpan(sizeHint) = let n = rest (max sizeHint 256) in Span(_buffer, _pos, n)

And here is the part I find hardest to write.

That corrected version was wrong too. Rent() called Reset(), which sets _pos back to zero but leaves _returned at true — so every recycled writer came out of the pool already flagged as released, its next Return() did nothing, and the pool quietly emptied itself. No crash. No corruption. Just an optimisation that had stopped happening, wearing the code of one that hadn't. I found it on the third read, after publishing the fix to myself as final.

Hence the separate Reuse() above, and no public Reset() at all — because a public Reset() is precisely the trap: it looks like it gives you a clean writer, and it doesn't.

Conclusion

I use code agents extensively every day, and am grateful they exist because they make my work easier. But not for a moment have I felt threatened by them as a software professional. They are yet another power tool, and power tools amplify whoever is holding them.

Look at what actually happened here. The agent did not fail at the hard part. The design is sound — thread-static pooling to avoid locks, back-patching to kill the temporary buffers, IBufferWriter to get the Span machinery for free. I would have arrived somewhere similar, slower. It failed at three small things, and every one of them required knowing an invariant that is not written anywhere in the code: that F# evaluates arguments left to right, that ArrayPool.Return transfers ownership, that a release method has to be idempotent.

That is exactly the knowledge the people being 'replaced' are supposed to have. Sixty lines appeared in seconds. Verifying them took me longer than writing them would have, and I still missed one on the first pass.

Note what that does to the productivity claim I opened with. On routine code the boost is real and large. On code where correctness is subtle, the agent did not reduce the cost — it moved it from writing to verifying, and here it moved it upward. The design work was still worth the price. But that is not the trade a spreadsheet imagines when it reads 'AI productivity gains'. It is more output per person, conditional on the person being able to tell working from merely plausible.

That is the thing to take away. The danger is not that an agent writes obvious rubbish you can spot and throw away. It is that it writes something better than obvious — well designed, idiomatic, clean-building, review-passing — with defects sitting in the two or three places where the language, the runtime or the framework has a rule that the code itself cannot show you. Nothing in that listing looks wrong. That is the whole problem.