Designing AI Systems Around LLM Limitations

LLM limitations are often discussed as model problems. In production, they become system design problems.

The useful question is not whether a model can hallucinate. It can. The question is what the product does when uncertainty appears, when the output is malformed, when external state has changed, or when the task requires authority the model should not have.

Good AI systems are designed around those limits instead of pretending they do not exist.

Hallucination is an interface problem

Hallucination is not only false text. It is a mismatch between generated output and the system's contract with the user.

The model may invent:

• facts;
• tool results;
• capabilities;
• unavailable data;
• a level of certainty the system does not have.

The mitigation is not a single prompt instruction. The mitigation is interface design:

• make unknown state representable;
• cite sources where claims depend on retrieved data;
• separate generated reasoning from verified facts;
• avoid asking the model to produce authoritative output when the system cannot validate it.

If the interface only allows confident answers, the model will tend to produce confident answers.

Structured outputs are a boundary, not a feature

Structured output is not just nicer JSON. It is a boundary between probabilistic generation and deterministic software.

A good schema should encode the shape of allowed behavior. It should make invalid states difficult to express and easy to reject.

For example, instead of asking for:

What should the system do next?

the system can ask for:

Choose one of:
- ask_user_for_missing_information
- call_read_only_tool
- call_mutating_tool_with_confirmation
- stop_with_explanation

This reduces the decision surface. It also gives the application a place to enforce policy.

Retries need budgets

Retries are useful until they hide failure.

An LLM call can fail because of transient provider issues, malformed output, context problems, or a bad task definition. Retrying all of those in the same way creates noisy systems.

Retries should have:

• a maximum count;
• an error category;
• a reason visible in traces;
• a different prompt or strategy when appropriate;
• a clear fallback after the budget is exhausted.

If a system retries silently until it gets a valid answer, it may produce success metrics while destroying debuggability.

Validation should happen outside the model

The model can help evaluate output, but the system should not rely on the same kind of uncertainty to verify itself.

Validation should be deterministic where possible:

• schema validation;
• permission checks;
• business rule checks;
• database constraints;
• external API confirmations;
• idempotency and duplicate detection.

For subjective tasks, model-based evaluation may still be useful. But even then, it should be treated as another signal, not as final authority.

Deterministic workflows still matter

Not every step should be agentic.

Many AI products become more reliable when the workflow is mostly deterministic and the model is used at specific decision points. A model can classify, extract, rank, summarize, or choose between bounded actions. The workflow engine can handle routing, retries, permissions, and persistence.

This is less exciting than a fully autonomous agent. It is also easier to ship.

Human-in-the-loop is a product primitive

Human review is often treated as a temporary workaround. In many systems it is a permanent product primitive.

The question is not whether humans should be involved. The question is where review creates the most leverage:

• before irreversible actions;
• when confidence is low;
• when policy is ambiguous;
• when the model detects missing context;
• when the cost of being wrong is high.

A good review flow should not feel like failure. It should feel like the system knows where its authority ends.

The main pattern

The pattern I keep returning to is:

1. use the model for language and judgment;
2. encode decisions into structured outputs;
3. validate those outputs outside the model;
4. execute through narrow tools;
5. record everything in traces;
6. escalate when the system reaches uncertainty or authority limits.

This architecture is less magical than the demo version. That is the point.