Can LLMs Write TLA+? An Experiment in Modeling Real-World Systems

LLMs try to master TLA+ — and it's as graceful as a cow on roller skates.

A team of researchers decided to put modern language models (like GPT-4) to the test: can they write formal TLA+ specifications for real-world distributed systems? The results are a mixed bag — think IKEA furniture assembled with a hammer.

The Setup

The scientists picked classic algorithms (Paxos, two-phase commit, Raft) and asked LLMs to generate TLA+ specs. They evaluated not just syntax but also correctness — does the model actually capture the system's behavior? Turns out:

• Models nail boilerplate (initialization, type definitions).
• But when it comes to nontrivial logic (e.g., fault handling), LLMs start hallucinating states that don't exist.
• Without fine-tuning on TLA+ corpora, models confuse operators and forget invariants faster than a dev on Friday evening.

So, Any Practical Use?

The authors honestly admit: LLMs won't replace verification engineers anytime soon. But as a quick prototyping tool or syntax suggester — it's decent. Especially if you hate remembering whether TLA+ uses semicolons after Next.

METABYTE's take: We tried letting an AI write TLA+ once — ended up reverting the commit and grabbing a coffee. If you want your distributed systems to be reliable (and hallucination-free), better call the humans. Or at least keep Copilot on a short leash.