Gartner recently warned that ungoverned decisions using LLMs will cause financial or reputational loss for enterprises. The fix, Gartner said, is to redesign analytic workflows around a required evaluation stage — a step most AI pipelines skip. Another Gartner prediction estimates that by 2027, 40% of enterprises will demote or decommission autonomous AI agents, with governance gaps surfacing only after a production incident. Strip away the framing, and both predictions describe that AI systems are already making structured, machine-actionable decisions inside enterprises.
Kirill Pechurin, a senior software engineer at Arctera, has built a way to check whether any of those decisions is correct — the kind of check a pipeline can run before the output reaches the system that acts on it. He built and open-sourced structured-eval, a free tool under the Apache-2.0 license, specifically designed to catch structured LLM outputs that parse correctly but contain wrong values. He also led the development of an LLM-based system for procurement, delivery, and equipment-accounting automation, including a spec-reconciliation tool that cut manual work by roughly 30% and exposed the problem this article addresses.
Where Hallucination Benchmarks Stop Mattering
Most conversations about AI reliability center on hallucination rates in open-domain question answering — a chatbot inventing a citation or a search summary getting a fact wrong. Those numbers have improved. But those benchmarks test a model on clean questions and expect an answer in plain sentences, not what most business systems actually ask. In production, a model’s output is often a structured record — a JSON entry written straight into a database, a compliance check, or a purchase order — without a human looking at it first.
“A mistake doesn’t look like ‘the model said something false’ in a way a reader could notice anymore. It’s now one buried detail, nested deep inside an otherwise correct-looking record, that doesn’t match reality. No one skimming a chat transcript would catch it. But the system downstream will still act on it.” — Kirill Pechurin
That’s a significant blind spot. A model can pass every open-domain hallucination benchmark and still ship a record with a wrong quantity, a fabricated part number, or fields that contradict each other. Nothing in the pipeline checks individual fields against a source of truth — only whether the output parsed as valid JSON.
The Procurement Tool That Never Throws an Error
Kirill ran into this gap in back-office automation at Arctera. His procurement tools included a component that automatically reconciled technical specifications against vendors’ commercial proposals, and another that parsed vendor-supplied equipment specifications into structured records. The system cut manual operations by roughly 30% in a high-frequency decision workflow. But that also matches the risk Gartner warns about: an LLM’s structured output feeding directly into a real business decision, with no human necessarily reading the raw model response before it’s acted on.
“A reconciliation error doesn’t throw an exception. The tool just quietly matches a spec to the wrong vendor line, and by the time anyone notices, the equipment’s already been ordered.” — Kirill Pechurin
In such workflows, asking whether the output looks plausible is the wrong question. Asking whether a specific field matches the source document is the right one — and almost nobody had built the tooling to ask it systematically.
Grading Structured Output Like an Auditor
Building the procurement system left Kirill with an unsolved problem: the evaluation layer was missing. He built one and released it under an Apache-2.0 license. The result, structured-eval, is a software tool specifically designed to assess the quality of structured JSON or YAML LLM outputs. Instead of stopping at whether the output parsed into valid JSON, it:
- Compares values field by field at any level of nesting
- Checks outputs against a source document to catch fabricated or unsupported values
- Enforces logical consistency across fields through a rules layer built for the purpose
- Supports running evaluations across a full dataset rather than one output at a time
- Tests whether a model gives a stable answer when the same prompt is run repeatedly
“Running the same prompt twice and getting two different structured outputs is routine. Most evaluation tooling doesn’t even ask that question.” — Kirill Pechurin
The framework is designed to sit alongside existing LLM-evaluation tools such as deepeval and LangSmith instead of replacing them. Pechurin frames the underlying idea: “There’s no such thing as one hallucination score. A quantity field and a description field fail in completely different ways, so each field gets scored with whatever comparison actually fits it, then those get combined.”
A quantity field might need exact or near-exact numeric matching; a description field might tolerate paraphrase and reward it with a token-level similarity score; a status field might need strict enumeration matching with zero tolerance. Letting each field carry its own comparison method, then combining those into a composite score, is much closer to how a human reviewer actually audits a structured record than a single blended accuracy number ever is.
What the Fix Actually Buys You
That kind of infrastructure is the practical answer to the industry pattern. It’s cheaper to build field-level verification into a pipeline once than to discover, after a procurement error or a compliance gap, that nobody was checking. Organizations less likely to face after-the-fact governance failures — decommissioning agents after the damage is already done — are those treating structured-output verification as a first-class engineering problem rather than an afterthought.
This isn’t the first time reliability work of this kind produced a large, measurable result. Earlier in his career at Consul Group, Kirill redesigned how a 1-million-plus-document Elasticsearch index handled updates, cutting query latency from 5,000 milliseconds to 500 milliseconds — a tenfold improvement. It’s a different problem than structured-output verification, but the same idea: not to assume a system behaves correctly at scale, but to measure it and rebuild the part that doesn’t.
An LLM can be reliable in general and still be the wrong tool for a procurement pipeline if nobody checks whether its specific fields match the specific documents it was given. Kirill Pechurin built structured-eval to close this gap, giving enterprises a concrete way to check AI-driven procurement, finance, and compliance decisions before they ship.
Leave a Reply