Improving agentic web search in small language models

Large models have an obvious advantage in closed-book question answering. They can store more knowledge, recall more facts, and often reason more reliably from what they already know.

When a model has access to search, the answer does not necessarily have to live inside its parameters. The model needs enough capability to recognize what it does not know, formulate a useful search query, inspect evidence, connect information across sources, and return the correct answer.

Parametric knowledge still helps, and model size certainly does not become irrelevant. But the bottleneck shifts from knowing the facts to executing the information-gathering process correctly.

That makes agentic web search an interesting target for small language models. We wanted to see how much agentic web search ability could be improved in SLMs.

Evaluation setup

We used the BFCL v4 Websearch split for our evaluation, which contains 100 multi-hop questions. A multi-hop question usually cannot be answered with a single search. The model has to find one piece of information, use it to determine what to search for next, and repeat this process until it can answer the original question.

For example:

“In which year was the first national park established in the birth country of the driver who won the 2024 Formula 1 World Championship?”

To answer this, the agent first needs to identify the 2024 Formula 1 World Champion, then determine the driver’s birth country, find the first national park established in that country, and finally retrieve the year it was established.

The agent operates with two simple tools:

• search_engine_query, which returns organic search results
• fetch_url_content, which takes a URL and returns the webpage content as Markdown

The model has to decide what to search for, which result to inspect, what information matters, and whether another search is needed.

Baseline results

The base model reached 24% accuracy. More importantly, on 46% of questions, it never fetched a webpage at all. It would often search briefly or skip search entirely and then try to answer from its own knowledge.

So the failure was not simply that the 2B model lacked enough knowledge. In many cases, it was failing to fully engage with the information source available to it.

When analyzing performance by hop depth, the drop from 2-hop to 4-hop questions is expected because every additional hop adds another dependency the model has to resolve correctly.

In a 2-hop question, the model might only need to find one fact and use it to search for a second. In a 4-hop question, that process has to succeed several times in a row.

This means errors compound as the chain gets longer. Even if the model handles each individual step reasonably well, maintaining that accuracy across several consecutive search and reasoning steps is much harder. As a result, deeper multi-hop questions provide a more demanding test of the agent’s ability to sustain a reliable search process from start to finish.

Building the training set

Agentic web search is already a well-solved problem for frontier models, which perform strongly at it. This gives us an opportunity to train a smaller student model to imitate the agentic search behavior of a stronger teacher.

We sampled questions from the MuSiQue dataset. It contains compositional multi-hop questions of varying difficulty, and we focused on 3-hop and 4-hop questions.

MuSiQue itself is not a web search dataset, but its questions are framed around general-knowledge facts, which made it well suited to this setting. We only needed the multi-hop question-and-answer pairs from the dataset.

A stronger teacher model, DeepSeek-v4-Pro, was allowed to solve these questions using the same agent harness. This converted the original questions into search trajectories containing the full sequence of actions required to reach an answer.

The goal of the training was to transfer behaviors such as:

• Breaking a question into intermediate steps
• Using one retrieved fact to formulate the next search query
• Fetching pages instead of relying only on search snippets
• Continuing until enough evidence had been collected

After trajectory generation and filtering, we retained 193 training examples. A relatively small number of high-quality demonstrations can be enough to teach the missing policy.

We fine-tuned the Qwen3.5–2B starting checkpoint on those 193 trajectories and reran the same BFCL v4 Websearch evaluation.

Fine-tuning results

The improvement was substantial. Completed-answer accuracy increased from 24.0% to 51.02%.

The gains were also consistent across hop depths.

The 4-hop result is particularly interesting. It remains the hardest category, but the fine-tuned model achieved 2.8× the baseline accuracy.

The behavioral changes help explain the accuracy gain. Page fetches per example increased from 1.16 to 2.26, while the percentage of examples with no page fetch dropped from 46.0% to 11.2%. Total tool calls increased from 4.43 to 6.57 per example.

Analysis of agent behavior

The most useful way to interpret this result is that fine-tuning improved the model’s search policy.

The base model already knew how to call a search tool, read text, and produce an answer. The problem was that it did not reliably connect those abilities into an effective multi-step search process.

A small amount of trajectory training changed that behavior. The model learned to search more often, inspect more pages, and continue working through intermediate dependencies instead of answering too early.

This also helps explain why training on MuSiQue transferred to BFCL Websearch. The student was not being taught the benchmark answers. It was being taught a reusable pattern for resolving multi-hop questions:

Find one piece of information, use it to locate the next one, and keep going until the original question can be answered.

Takeaway

With just 193 distilled trajectories, Qwen3.5–2B improved from 24% to about 51% accuracy on BFCL v4 Websearch.

More importantly, its behavior changed in the direction we wanted. The share of examples with no page fetch fell from 46.0% to 11.2%, and the model became much more proactive about gathering evidence before answering.

The result suggests that, for agentic search, a meaningful part of the gap between small and large models may come not from missing factual knowledge, but from the policy governing how the model searches, verifies, and continues reasoning across multiple steps.