Every team building on top of large language models eventually hits the same wall: the easy stuff stops working. A clever prompt got you a working demo. Then real users showed up, edge cases multiplied, and the model started doing things you didn't ask for. The instinct at this point is to reach for the next optimization technique — and the one after that, and the one after that.
That instinct is exactly what “The Staircase to Optimization Hell” is warning you about. It's a mental model, not a to-do list: ten increasingly powerful techniques for improving LLM applications, arranged in the order you should reach for them, with a simple rule underneath — each step down the staircase buys you more control and better performance, but it also costs you more in engineering time, infrastructure, and long-term maintenance burden. Nobody starts a project intending to build a custom router, a fine-tuned model, and a tree-search decoder. They arrive there one “just one more optimization” at a time, and by the time they look up, they're maintaining a system with the operational complexity of a small distributed database — for a task that maybe should have been solved with a better prompt.
The staircase isn't an argument against sophistication. Routers, fine-tuning, and search-based decoding are legitimate, sometimes necessary tools. The point is that they're expensive tools, and paying that price before you've exhausted the cheap ones is how teams end up in optimization hell: high complexity, high cost, and — because they skipped the fundamentals — often no better results than they'd have gotten from a well-written prompt.
Here's the staircase, step by step.
Step 1: Zero-shot prompting
This is the ground floor — a plain instruction with no examples, no scaffolding, just a task description and a question. “Summarize this article.” “Classify this email as spam or not spam.” Zero-shot prompting works because modern LLMs have already seen enormous volumes of instructional data during training; they don't need to be shown what a good answer looks like, they need to be told what's being asked.
It's also where almost every LLM feature should start, and where a surprising number of them should end. If a well-phrased zero-shot prompt gets you 90% of the way to a working feature, the other techniques on this staircase are solving a problem you may not actually have yet. The cost of a zero-shot prompt is close to zero: no extra latency, no extra tokens, no extra infrastructure. The failure mode is inconsistency — the model's format, tone, or reasoning depth can vary run to run — which is exactly what the next few steps exist to fix.
Step 2: Few-shot prompting
When zero-shot output is inconsistent or the task is subtle enough that “describe it in words” doesn't fully capture what you want, you show the model examples instead. Few-shot prompting embeds two, three, or a handful of input-output pairs directly in the prompt, letting the model infer the pattern rather than relying purely on instruction-following.
This is still cheap — you're paying in tokens (examples take up context) and in the design work of picking representative examples, but there's no new infrastructure. The tradeoff that starts to appear here is maintenance: your examples become a de facto specification of correct behavior, and as your task evolves, someone has to remember to keep them in sync. Bad or unrepresentative examples can also anchor the model in unhelpful ways, so few-shot prompting requires more care in curation than it might first appear.
Step 3: Chain of thought
Some tasks — arithmetic, multi-step logic, anything requiring the model to hold several constraints in mind at once — get meaningfully better when the model is instructed to reason step by step before producing a final answer, rather than jumping straight to it. Chain of thought (CoT) prompting is simple to implement (often just “think through this step by step” appended to a prompt) but changes the shape of your system: outputs get longer, latency goes up, and you now need to decide whether to show the reasoning to users, parse it out programmatically, or discard it entirely.
CoT is still squarely in “prompt engineering” territory — no new infrastructure, no new failure surface beyond longer and sometimes meandering outputs. But it's the first step where the cost is no longer nearly free: every CoT response burns more tokens and more time, and reasoning that looks plausible isn't always reasoning that's correct, which means CoT outputs need at least as much scrutiny as zero-shot ones, not less.
Step 4: Temperature and sampling parameters
Temperature (along with related knobs like top-p and top-k) controls how deterministic or exploratory the model's token selection is. Low temperature makes outputs more focused and repeatable — useful for factual extraction, classification, or code generation where you want the same input to reliably produce the same output. Higher temperature introduces more variation, useful for brainstorming, creative writing, or generating diverse candidates.
This step is cheap to implement — it's a parameter, not a system — but it's easy to misuse. Tuning temperature is often treated as a magic fix for output quality problems that actually stem from a weak prompt or an ambiguous task. It's better understood as a dial for variance, not a dial for correctness. Get the prompt right first; use temperature to shape the range of acceptable outputs around it.
Step 5: Workflows (hardcoded multi-step chains)
This is where the staircase starts to feel like real software engineering rather than prompt tuning. A workflow breaks a complex task into an explicit, hardcoded sequence of LLM calls — extract entities, then classify them, then generate a response referencing them — with the control flow written in your application code rather than left up to the model. This is sometimes called “prompt chaining.”
Workflows buy reliability: each step has a narrow, well-defined job, which is usually easier to get right than asking one giant prompt to do everything at once. They also make debugging tractable — when something goes wrong, you can point to exactly which step in the chain failed. The cost is real, though: you're now maintaining a pipeline, not a prompt. Each step adds latency and token spend, error handling has to be designed deliberately (what happens when step 2 returns something step 3 can't parse?), and the system as a whole becomes harder to reason about than a single call, even though each individual piece is simpler.
That is the work of AI Workflow Automation: a chain you can name, not a chatbot that recites answers. See also stop buying another chatbot.
Step 6: Evaluators
Once you have a workflow — or even a single nontrivial prompt — in production, “it looked good when I tried it” stops being an acceptable form of quality control. Evaluators are automated pipelines that score model outputs against a rubric, a reference answer, a set of test cases, or another model acting as a judge. They turn “does this work?” into a measurable, repeatable question.
This is the step where the project starts to require dedicated engineering investment rather than prompt-writing skill. Building good evaluators means building a test dataset, deciding what “good” means well enough to encode it into a scoring function or a judge prompt, and creating the infrastructure to run evaluations regularly — ideally in CI, so regressions get caught before they reach users. Evaluators don't improve your model's output directly; what they buy you is the ability to know, with evidence, whether any of the other changes you're making are actually helping. Skipping this step doesn't mean you don't pay its cost — it means you pay it later, in the form of shipping regressions you can't detect.
In operations, the same rule shows up as a review-and-apply loop: a human accepts, edits, or rejects the write. Don't skip the check. Don't let the model write to production.
Step 7: Agentic loops
Workflows are LLM calls chained by code you wrote. Agentic loops flip that: the model itself decides which tools to call, in what order, and when it's done, iterating autonomously rather than following a script you laid out in advance. This unlocks tasks where the right sequence of steps can't be known ahead of time — open-ended research, multi-turn debugging, tasks where the number of steps depends entirely on what the model discovers along the way.
The power here comes with a proportional jump in operational complexity. Agentic loops need guardrails: iteration limits, cost ceilings, tool-use validation, and recovery logic for when the model gets stuck in unproductive loops or calls a tool with malformed arguments. Latency and cost both become variable and harder to predict, since you no longer control how many steps a task will take. This is also usually the point where you need the evaluators from the previous step the most — an agent that occasionally goes off the rails in ways a human reviewer would catch immediately, but that no one is systematically checking for, is a liability wearing the costume of a feature.
Step 8: LLM routers
At this point in the staircase, you likely have more than one task and more than one model that could plausibly handle it — a large, expensive, high-quality model and one or more smaller, cheaper, faster ones. An LLM router uses a (typically smaller) model to classify incoming requests and dynamically send each one to whichever downstream model is best suited to handle it, trading cost and latency against quality on a per-request basis.
Done well, routing can meaningfully cut costs by reserving your most expensive model for the requests that actually need it. Done poorly, it introduces a new, opaque failure mode: a misrouted request silently gets worse quality with no obvious error to point to. Routers also add a component that itself needs training, tuning, and evaluation — you're not just managing your original task anymore, you're managing a classifier whose job is to manage your original task. This is genuinely useful at scale, and genuinely overkill below it.
Step 9: Fine-tuning
Fine-tuning means retraining an existing base model on your own domain-specific data so the desired behavior gets baked into the model's weights rather than re-specified in every prompt. It's the step where you stop steering a general-purpose model with instructions and start producing a specialized artifact of your own.
The payoff can be substantial: shorter prompts (because the model already “knows” your format and domain), better performance on narrow tasks, and behavior that's hard to achieve through prompting alone. The cost is substantial too. You need a quality training dataset, which is often the hardest part of the entire process. You need infrastructure to run the training and, afterward, to host and serve a custom model — you can no longer just call a shared API endpoint. And you now own a new maintenance burden indefinitely: as the underlying base models improve, your fine-tuned version doesn't automatically inherit those gains, and every prompt or data change upstream may require retraining. Fine-tuning should generally be a response to a clearly measured gap that prompting and workflows couldn't close — not a first resort.
Step 10: Sampling (best-of-N, tree search, and friends)
The bottom of the staircase is reserved for compute-intensive decoding strategies: generating many candidate outputs and searching over them for the best one. Best-of-N sampling generates N completions and picks the strongest by some scoring method; tree-search decoding explores and prunes a branching space of possible continuations, similar in spirit to how game-playing AI searches a tree of possible moves.
These techniques can produce meaningfully better outputs than a single generation pass, particularly on hard reasoning tasks — but the cost scales directly with how much better you want the answer to be. Generating and scoring N candidates costs roughly N times the compute and latency of generating one, and tree search compounds that further. This is the step reserved for problems where being right matters enough to justify spending 10x, 50x, or more per query — competition math, high-stakes code generation, safety-critical decisions — and where every cheaper technique on the staircase has already been tried and found insufficient.
Reading the staircase correctly
The order of the staircase matters more than any individual step on it. Zero-shot, few-shot, chain of thought, and temperature tuning are prompt-level techniques: cheap, fast to iterate on, and they should be exhausted first because they're where the highest ratio of improvement to effort lives. Workflows and evaluators are where the project starts to look like real engineering — still very achievable, but now requiring actual system design and testing discipline. Agentic loops, routers, fine-tuning, and search-based sampling are where cost, infrastructure, and ongoing maintenance escalate sharply, and where the techniques stop being generically useful and start being justified only by a specific, measured need.
The failure pattern the staircase is naming isn't “using advanced techniques.” It's skipping steps — reaching for fine-tuning to fix a problem a better prompt would have solved, or building an agentic loop for a task that was really just a three-step workflow in disguise. Every step you skip on the way down is a step you eventually have to climb back up to diagnose, because the techniques below it can't fix a problem that originates above it. A fine-tuned model trained on inconsistent, poorly-specified examples doesn't produce consistent outputs — it just produces expensive inconsistent outputs.
The healthiest way to use this framework isn't as a ladder to climb as fast as possible. It's as a checklist to climb slowly, one rung at a time, with evidence — ideally from the evaluators you built back at step six — that each additional rung is actually buying you something the ones above it couldn't. That discipline is the difference between a system that's sophisticated because it needs to be, and one that's just in optimization hell.
Frequently asked questions
Do we need an agent if the path is already known?
No. If you can write the steps in code, write the steps in code. An agentic loop is for work whose next move depends on what the model finds. A hardcoded chain is cheaper to run and easier to debug.
When is fine-tuning the right next step?
When you have a measured gap that prompting, sampling parameters, and a workflow could not close — and a training set good enough to encode the behavior you want. Fine-tuning a messy spec just makes the mess expensive.
Where should a team start this week?
Write the zero-shot prompt. Add a few honest examples. Decide what “good” means well enough to score. Then stop. Climb only when the score says the current step is not enough.
What to do next
If you are about to add another model, another loop, or a fine-tune before the cheap steps are exhausted, pause. Map the staircase you are actually on. See the automation work or talk to us before the next optimization becomes the system you have to live with.
