Parser Combinators vs. Parser Generators: Why Tooling Keeps Choosing the Former
A grammar file and a build step, or a handful of composable functions. The industry’s answer has been drifting for a decade.
For most of programming history, writing a parser meant learning a separate tool. You wrote a grammar in yacc or ANTLR’s own syntax, ran it through a generator, and got back code in your target language that you mostly didn’t read. Parser combinators offer a different deal: write the parser directly in your host language, as ordinary functions that combine smaller parsers into bigger ones, no separate grammar file and no code-generation step. Both approaches are decades old now. What has changed is which one keeps showing up in new tooling, and the reasons are visible in real usage data, not just developer opinion.
Two Old Ideas, One Clear Trend
Yacc came first, written by Stephen Johnson at Bell Labs and documented in a 1975 technical report, generating LALR parsers from a grammar file. ANTLR followed as the successor to 1989’s PCCTS, with its first release in 1992, and reinvented itself again with ANTLR4’s ALL(*) algorithm around 2013. Parser combinators trace to Hutton and Meijer’s 1996 “Monadic parser combinators,” formalized for production use in Leijen and Meijer’s 2001 paper introducing Parsec, whose design later shaped Haskell’s megaparsec, Rust’s nom and combine, and Scala’s parser-combinator libraries. The ideas are roughly contemporaries by the standard of programming language history. The usage data is not.
Rust’s ecosystem happens to have a clean, live, checkable measure of this: nom alone has been downloaded more times than pest, lalrpop, peg, and lrpar combined, and it is not close. That is a snapshot of a Rust-specific ecosystem, not a universal law, but it lines up with a pattern visible across languages once you look at where new parsers actually get written.
The Migrations Tell the Same Story
Aggregate downloads are one kind of evidence. Individual projects choosing to rip out a generator and replace it with hand-written code are another, and the public record has several documented cases. In 2024, Astral’s Ruff, a Python linter and formatter written in Rust, replaced its LALRPOP-generated parser with a hand-written recursive descent parser, citing performance and, just as importantly, the ability to recover from errors and keep parsing instead of stopping at the first mistake, something a generated parser struggled to do gracefully. HashiCorp’s HIL library made the same move in 2016, dropping a yacc-generated parser for a hand-written one after finding it too hard to extend and too vague in its error messages, a complaint echoed in the pull request that made the change. Clang’s C and C++ front ends have used a hand-written recursive descent parser since the project’s early days, a choice its authors attribute directly to how context-sensitive C++’s grammar is, a poor fit for a grammar file that assumes context-freedom.
| Project | Change | Cited reason |
|---|---|---|
| Ruff (2024) | LALRPOP generator → hand-written recursive descent | Speed, plus graceful error recovery |
| HashiCorp HIL (2016) | Yacc generator → hand-written recursive descent | Easier to extend, clearer error messages |
| Clang (since inception) | Never used a generator for C/C++ | C++ grammar is too context-sensitive for one |
| CPython (2020, PEP 617) | LL(1) generator (pgen) → PEG generator (pegen) | Removes grammar hacks; still generated, but PEG rules read like recursive descent |
CPython’s case is worth calling out precisely because it is not a clean win for combinators: PEP 617 replaced one generator with another. What it shows is that even the side that kept a grammar file and a generation step still moved toward PEG, a formalism whose rules read like functions calling functions in order, because it eliminates the “hacks” the old LL(1) grammar needed to work around ambiguity. Combinators did not have to be the only tool that benefited from that lesson.
What the Grammar File Was Costing People
The recurring complaints across these migrations are consistent: a separate grammar means a separate DSL to learn, a build step that turns debugging into reading generated code you didn’t write, and rigid error handling because the generator, not you, decides what happens when a rule fails to match. Parser combinators trade some of the declarative clarity of a grammar file for the ability to write error handling, backtracking, and context-sensitive lookahead as ordinary code, in the same language and the same debugger as everything else in the project. For compilers and language servers that increasingly need to parse broken code gracefully, the kind an IDE sees while someone is mid-sentence typing, that flexibility is not a nice-to-have, it is close to a requirement.
Reach for a combinator library over a generator when
- Error messages and error recovery matter as much as correctly recognizing valid input
- The grammar has context-sensitive corners a separate grammar file fights against, rather than expresses cleanly
- You want the parser to live in the same language, build, and debugger as the rest of the toolchain, with no codegen step
- You’re building something that needs to parse incomplete or invalid input gracefully, such as an IDE or language server
What We Learned
Parser combinators and parser generators are both decades-old ideas, born within a few years of each other by the standard of programming language history, yet the tooling world keeps choosing combinators, or hand-written recursive descent in the same spirit, for new work. Live crates.io data shows the gap in Rust’s ecosystem is large and holding steady rather than closing. Individual, documented migrations at Ruff, HashiCorp, and Clang all point to the same two reasons: a hand-written or combinator-based parser gives up some declarative clarity in exchange for direct control over error messages, recovery, and context-sensitive lookahead, control that a generated parser hands to a tool instead of the person maintaining the code. Even CPython’s move to a PEG-based generator, which kept the generator, moved its grammar closer to how a hand-written parser already reads.
