The moment you want to use a language model inside a program rather than a chat window, you hit the same wall: you need its answer as data your code can use, not as prose. So you ask for JSON. And sometimes you get clean JSON, and sometimes you get JSON wrapped in a friendly “Sure, here’s the data:”, and sometimes you get JSON with a trailing comment that breaks your parser at 3am. Getting reliable structured output from an LLM is a solved problem now, but the solution is not “ask nicely and hope” — it is a small stack of techniques, from weakest to strongest.
Why “respond in JSON” is not enough
The naive approach is to put “respond only in JSON” in the prompt and parse whatever comes back. This works most of the time, which is exactly what makes it dangerous: it fails intermittently, on inputs you did not test, in production. The model might add a preamble, wrap the JSON in markdown code fences, include a comment, or — for a genuinely hard input — emit slightly malformed JSON. Your parser throws, and because it only happens on some fraction of requests, you discover it from error logs rather than from testing.
The deeper issue is that a plain instruction is a request, not a guarantee. The model is doing its best to comply, but nothing constrains it to valid JSON, so on the edge cases it drifts. For anything that runs unattended, “usually valid” is not good enough, because the whole point of structured output is that your code can rely on it.
The strongest tool: constrained decoding
The best solution, when your provider offers it, is to make invalid output structurally impossible rather than merely discouraged. Modern LLM APIs offer a mode — variously called structured output, JSON mode, or a response-format schema — where you supply a JSON Schema and the model is constrained at generation time to produce output matching it. This is not a prompt asking for a shape; it is the decoding process being restricted so that only tokens leading to valid, schema-conforming output can be chosen.
Provide a schema, get conformance guaranteed by construction:
{
"type": "object",
"properties": {
"sentiment": { "type": "string", "enum": ["positive", "negative", "neutral"] },
"confidence": { "type": "number" }
},
"required": ["sentiment", "confidence"]
}
// The model literally cannot emit a sentiment outside the enum,
// or forget the confidence field, or wrap the result in prose.When this is available, use it — it eliminates the entire class of parse-and-pray failures, because the output is valid by construction rather than by luck. It also removes a surprising amount of prompt-engineering effort, since you no longer have to beg the model to avoid code fences or preambles; the format is enforced beneath the prompt.
When you only have prompting: tool/function calling
If a raw schema mode is not available but function calling is, that is the next-best thing, and it is available on essentially every capable model now. Function calling was designed for the model to invoke tools, but it doubles as a reliable structured-output mechanism: you define a “function” whose parameters are the shape you want, and the model returns arguments matching that parameter schema. Under the hood it is the same idea — the model produces structured arguments rather than free text — so you get schema-shaped data even when a dedicated JSON mode is not exposed.
This is often the most portable path, because tool-use support is nearly universal while dedicated structured-output modes vary by provider. Defining a single-function schema and reading its arguments is a dependable way to get typed data out of a model that only advertises tool calling.
Prompting techniques that still matter
Even with enforcement, a few prompt-level habits improve results, and without enforcement they are your main defense. Give the model an explicit example of the exact output you want — a concrete sample is worth more than a paragraph of description, because it removes ambiguity about formatting. State clearly that the response must be only the JSON with no surrounding text, since the most common non-enforced failure is a chatty preamble. And keep the schema as flat and simple as the task allows, because deeply nested, sprawling structures are harder for the model to fill correctly and harder for you to validate.
One counterintuitive point: if a field genuinely requires the model to reason, let it reason in a dedicated field rather than forbidding it. A schema with a reasoning string before the answer field often produces better answers than one that demands only the answer, because you have given the model room to think inside the structure instead of forcing it to jump straight to a conclusion — the same reason showing work improves an LLM-as-a-judge evaluation.
Always validate, even when it is guaranteed
Whatever method you use, validate the parsed result against your schema in code before trusting it. With constrained decoding this should never fail, but “should never” is not “cannot,” and a validation layer costs almost nothing while catching the rare surprise — a provider quirk, an edge case, a schema you got subtly wrong. Treat the model’s output as untrusted input that crossed a boundary, because that is exactly what it is. This matters as much for a retrieval pipeline, where a malformed extraction quietly corrupts what you feed back in, as the diagnosis in our guide to fixing RAG retrieval shows.
Have a plan for the failure too. When validation does fail, decide in advance whether you retry the request, fall back to a default, or surface an error — silently swallowing a malformed response is how a subtle data-quality problem creeps into a system unnoticed. The combination is what makes structured output production-grade: enforce the shape at generation where you can, request it well where you cannot, and always validate before you rely on it.