Four architecture approaches for using smaller language models

Smaller language models can offer meaningful reductions in inference cost and latency, but those advantages only matter if they can be achieved without compromising the quality of the application.

In many production systems, the best approach is not to run every request through the most capable model, nor to force a smaller model to handle work beyond its capabilities. Instead, the application can decide where stronger reasoning is actually necessary and where a smaller, more specialized model is sufficient.

The goal is to move as much work as possible onto the lowest-cost model that can perform it reliably, while preserving the performance, accuracy, and robustness expected from the overall system.

There are several ways to design applications around this principle. The right architecture depends on where complexity lives in the application and how much autonomy the smaller model can reliably handle.

SLM as part of a deterministic workflow

In this pattern, the application owns the overall workflow and the SLM is responsible for one narrowly defined task inside it. The sequence of steps is already known in advance, and the model does not decide what should happen next. It simply receives an input, performs a specific language task, and returns a result that the application can validate and use.

This works particularly well for tasks such as classification, extraction, routing, normalization, or structured summarization. Because the model operates within a small problem space, the prompt can be tightly constrained and the output can often be validated against a schema or a known set of values.

A practical example is an invoice processing pipeline. An incoming document is fetched and corrected for orientation before being passed to a fine-tuned vision language model, which extracts the required fields into a structured JSON output. The extracted data is validated against expected formats and business rules before being written to the database. A similar architecture is explored in more detail in this AI invoice processing case study, a fine-tuned SLM for structured invoice extraction.

SLM orchestrator with a frontier model as advisor

In this architecture, the SLM interprets the request, selects tools, tracks progress, and decides what action to take next. For the majority of interactions, the SLM remains in control of the agent loop.

The frontier model is brought in only when additional reasoning capability is required. It might help create a plan for an unusually complex request, review a decision before an important action, verify whether the gathered information is sufficient, or suggest a recovery path when the smaller model repeatedly fails to make progress.

The important idea is that the frontier model is not processing every interaction. It acts more like a specialist that the system consults at selected points. This allows the smaller model to handle the common path while the application retains access to stronger reasoning for the difficult parts of the workflow.

Consider a customer support agent handling a user request that requires several actions. The SLM first generates a plan for resolving the request, which is reviewed by a frontier model for missing steps or potential issues. The SLM incorporates that feedback, revises the plan, and then proceeds with execution using the tools available to the support agent.

Frontier model orchestrator with SLM powered sub-agents

This pattern reverses the previous architecture. The frontier model remains responsible for the primary agent loop and handles the user interaction, planning, and higher-level reasoning. When part of the task requires a significant amount of bounded exploration, it delegates that work to a specialized sub-agent powered by an SLM.

The sub-agent receives an objective, invoke its own limited set of tools, inspect results, refine its approach, and decide when it has gathered enough information to return something useful to the main agent.

This makes the pattern particularly useful for tasks where a lot of work needs to happen underneath a relatively simple interface. The main model does not need to spend its context window and reasoning capacity performing every search, reading every intermediate result, or deciding how to reformulate every query.

For example, consider a deep search agent answering questions over a large corpus of internal information. The main agent generates the search objective and delegates the retrieval work to a search sub-agent, which can issue queries, inspect results, refine its searches over multiple rounds, and use different retrieval tools as needed. Once the search is complete, it returns a ranked set of relevant results for the main agent to reason over. This deep dive on specialised retrieval sub-agents explores the architecture in more detail.

Use case specific SLM agent

The final pattern gives the SLM responsibility for the entire agent loop. The smaller model receives the user request, determines what needs to be done, calls tools, interprets their outputs, maintains progress, and eventually produces the final response.

The key to making this architecture work is specialization. The environment needs to be designed around the tasks the model is expected to perform like reducing the number of tools, managing state outside the model, constraining the number of possible actions, and validating intermediate outputs. The system does more of the work required to keep the agent on track, which reduces the amount of reasoning the model has to perform on its own.

A text-to-SQL assistant for internal teams is a good example of the final pattern. A fine-tuned SLM takes a natural language question, generates SQL, executes it against the relevant data source, and uses the returned results to answer the user’s question. The task-specific harness can provide schema context, validate queries before execution, enforce access controls, and help the model recover from SQL errors.

Conclusion

The objective is not to replace frontier models everywhere. It is to use them where their additional reasoning capability creates value, rather than paying their cost for every step of every workflow.

In practice, this means designing for the common path. High-volume, predictable, or domain-specific work can often be handled by an SLM, while more capable models remain available for the smaller set of requests that genuinely require them. Done well, this creates systems that reduce runtime cost and latency without lowering the performance or accuracy users experience.