Fine-tuning or better prompting: which one actually fixes your agent’s weak spot

When an agent gives a wrong or off-brand answer, the reflex is often to reach for fine-tuning — retraining the model on a set of company-specific examples so it "learns" the right behavior. That instinct is usually wrong as a first move. Fine-tuning is a real tool, but it is not the default answer to "the agent isn’t doing what we want," and using it as one adds cost and maintenance burden that a better prompt or a retrieval step often makes unnecessary.

What the comparison actually shows

A controlled study by researchers at Google DeepMind and Stanford, On the generalization of language models from in-context learning and finetuning (2025), compared the two approaches on matched data and found that in-context learning — giving the model the relevant information directly in the prompt — generalized better than standard fine-tuning on new, related situations the model had not seen verbatim in training. Fine-tuning tended to make the model better at reproducing patterns close to its training examples, but less flexible when a real query strayed from them.

A practical way to decide

The question to ask first is not "do we have enough data to fine-tune" but "is the problem that the model lacks information, or that it lacks a skill." A model answering wrong because it does not know your current pricing or policy is an information problem — solved by retrieval, not retraining, and updated the moment the source document changes. A model that consistently reasons badly on a narrow, stable task, at high volume, with a workflow unlikely to change, is closer to the case where fine-tuning earns its cost.

• →Start with prompting and retrieved context — no training cost, no retraining when the underlying information changes, and it is where most agent failures actually get fixed.
• →Reach for fine-tuning only for a narrow, high-volume, stable task where prompting has been tried and measurably falls short.
• →Remember fine-tuning is not "set once" — a source of truth that changes regularly means a fine-tuned model needs periodic retraining to stay current, an ongoing cost retrieval avoids.

The reflex to fine-tune usually comes from a diagnosis error, not a real need — most of the time, the agent was missing the right information at the right moment, not a new skill.