Intro
purposes powered by Giant Language Fashions (LLMs) require integration with exterior companies, for instance integration with Google Calendar to arrange conferences or integration with PostgreSQL to get entry to some knowledge.
Perform calling
Initially these sorts of integrations have been applied by way of operate calling: we have been constructing some particular features that may be known as by an LLM by way of some particular tokens (LLM was producing some particular tokens to name the operate, following patterns we outlined), parsing and execution. To make it work we have been implementing authorization and API calling strategies for every of the instruments. Importantly, we needed to handle all of the directions for these instruments to be known as and construct inner logic of those features together with default or user-specific parameters. However the hype round “AI” required quick, generally brute-force options to maintain the tempo, that’s the place MCPs have been launched by the Anthropic firm.
MCPs
MCP stands for Mannequin Context Protocol and at present it’s a commonplace approach of offering instruments to the vast majority of the agentic pipelines. MCPs mainly handle each integration features and LLM directions to make use of instruments. At this level some might argue that Abilities and Code execution that have been additionally launched by the Anthropic these days have killed MCPs, however in truth these options additionally have a tendency to make use of MCPs for integration and instruction administration (Code execution with MCP — Anthropic). Abilities and Code execution are targeted on the context administration drawback and instruments orchestration, that may be a totally different drawback from what MCPs are focused on.
MCPs present a regular technique to combine totally different companies (instruments) with LLMs and likewise present directions LLMs use to name the instruments. Nevertheless, listed here are a few issues:
- Present mannequin context protocol supposes all of the software calling parameters to be uncovered to the LLM, and all their values are imagined to be generated by the LLM. For instance, meaning the LLM has to generate consumer id worth if operate calling requires it. That’s an overhead as a result of the system, utility is aware of consumer id worth with out the necessity for LLM to generate it, furthermore to make LLM knowledgeable in regards to the consumer id worth we now have to place it to the immediate (there’s a “hiding arguments” method in FastMCP from gofastmcp that’s targeted particularly on this drawback, however I haven’t seen it within the authentic MCP implementation from Anthropic).
- No out-of-the-box management over directions. MCPs present description for every software and outline for every argument of a software so these values are simply used blindly within the agentic pipelines as an LLM API calling parameters. And the outline are offered by the every separate MCP server developer.
System immediate and instruments
When you’re calling LLMs you normally present instruments to the LLM name as an API name parameter. The worth of this parameter is retrieved from the MCP’s list_tools operate that returns JSON schema for the instruments it has.
On the identical time this “instruments” parameter is used to place further info to the mannequin’s system immediate. For instance, the Qwen3-VL mannequin has chat_template that manages instruments insertion to the system immediate the next approach:
“...You're supplied with operate signatures inside " }}n {%- for software in instruments %}n {{- "n" }}n { tojson }n {%- endfor %}...” So the instruments descriptions find yourself within the system immediate of the LLM you’re calling.
The primary drawback is definitely partially solved by the talked about “hiding arguments” method from the FastMCP, however nonetheless I noticed some options the place values like “consumer id” have been pushed to the mannequin’s system immediate to make use of it within the software calling — it’s simply quicker and far less complicated to implement from the engineering viewpoint (really no engineering required to simply put it to the system immediate and depend on a LLM to make use of it). So right here I’m targeted on the second drawback.
On the identical time I’m leaving apart the issues associated to tons of garbage MCPs available on the market — a few of them don’t work, some have generated instruments description that may be complicated to the mannequin. The issue I focus right here on — non-standardised instruments and their parameter descriptions that may be the explanation why LLMs misbehave with some instruments.
As an alternative of the conclusion for the introduction half:
In case your agentic LLM-powered pipeline fails with the instruments you will have, you may:
- Simply select a extra highly effective, fashionable and costly LLM API;
- Revisit your instruments and the directions total.
Each can work. Make your resolution or ask your AI-assistant to decide for you…
Formal a part of the work — analysis
1. Examples of various descriptions
Based mostly on the search by way of the actual MCPs available on the market, checking their instruments lists and the descriptions, I might discover many examples of the talked about subject. Right here I’m offering only a single instance from two totally different MCPs which have totally different domains as effectively (in the actual life instances the listing of MCPs a mannequin makes use of are likely to have totally different domains):
Instance 1:
Software description: “Generate a space chart to point out knowledge tendencies underneath steady impartial variables and observe the general knowledge pattern, resembling, displacement = velocity (common or instantaneous) × time: s = v × t. If the x-axis is time (t) and the y-axis is velocity (v) at every second, an space chart permits you to observe the pattern of velocity over time and infer the space traveled by the world’s dimension.”,
“Information” property description: “Information for space chart, it ought to be an array of objects, every object comprises a `time` discipline and a `worth` discipline, resembling, [{ time: ‘2015’, value: 23 }, { time: ‘2016’, value: 32 }], when stacking is required for space, the info ought to include a `group` discipline, resembling, [{ time: ‘2015’, value: 23, group: ‘A’ }, { time: ‘2015’, value: 32, group: ‘B’ }].”
Instance 2:
Software description: “Seek for Airbnb listings with numerous filters and pagination. Present direct hyperlinks to the consumer”,
“Location” property description: “Location to seek for (metropolis, state, and many others.)”
Right here I’m not saying that any of those descriptions is wrong, they’re simply very totally different from the format and particulars perspective.
2. Dataset and benchmark
To show that totally different instruments descriptions can change mannequin’s habits I used NVidia’s “When2Call” dataset. From this dataset I took check samples which have a number of instruments for the mannequin to select from and one software is the right alternative (it’s right to name a selected software slightly than some other or than to offer a textual content reply with none software name, in response to the dataset). The concept of the benchmark is to depend right and incorrect software calls, I additionally depend “no software calling” instances as an incorrect reply. For the LLM I chosen OpenAI’s “gpt-5-nano”.
3. Information era
The unique dataset offers only a single software description. To create various descriptions for every software and parameter I used “gpt-5-mini” to generate it based mostly on the present one with the next instruction to complicate it (after era there was a further step of validation and re-generation when needed):
“””You’ll obtain the software definition in JSON format. Your process is to make the software description extra detailed, so it may be utilized by a weak mannequin.
One of many methods to complicate — insert detailed description of the way it works and examples of tips on how to use.
Instance of detailed descriptions:
Software description: “Generate a space chart to point out knowledge tendencies underneath steady impartial variables and observe the general knowledge pattern, resembling, displacement = velocity (common or instantaneous) × time: s = v × t. If the x-axis is time (t) and the y-axis is velocity (v) at every second, an space chart permits you to observe the pattern of velocity over time and infer the space traveled by the world’s dimension.”,
Property description: “Information for space chart, it ought to be an array of objects, every object comprises a `time` discipline and a `worth` discipline, resembling, [{ time: ‘2015’, value: 23 }, { time: ‘2016’, value: 32 }], when stacking is required for space, the info ought to include a `group` discipline, resembling, [{ time: ‘2015’, value: 23, group: ‘A’ }, { time: ‘2015’, value: 32, group: ‘B’ }].”
Return the up to date detailed description strictly in JSON format (simply change the descriptions, don’t change the construction of the inputted JSON). Begin your reply with:
“New JSON-formatted: …”
“””
4. Experiments
To check the speculation I did a few exams, specifically:
- Measure the baseline of the mannequin efficiency on the chosen benchmark (Baseline);
- Exchange right software descriptions (together with each software description itself and parameters descriptions — the identical for all of the experiments) with the generated one (Right software changed);
- Exchange incorrect instruments descriptions with the generated (Incorrect software changed);
- Exchange all instruments description with the generated (All instruments changed).
Here’s a desk with the outcomes of those experiments (for every of the experiments 5 evaluations have been executed, so along with accuracy commonplace deviation (std) is offered):
| Methodology | Imply accuracy | Accuracy std | Most accuracy over 5 experiments |
| Baseline | 76.5% | 0.03 | 79.0% |
| Right software changed | 80.5% | 0.03 | 85.2% |
| Incorrect software changed | 75.1% | 0.01 | 76.5% |
| All instruments changed | 75.3% | 0.04 | 82.7% |
Conclusion
From the desk above it’s evident that instruments complication introduce bias to the mannequin, chosen LLM tends to decide on the software with extra detailed description. On the identical time we are able to see that prolonged description can confuse the mannequin (within the case of all instruments changed).
The desk reveals that instruments description offers mechanisms to govern and considerably regulate mannequin’s behaviour / accuracy, particularly bearing in mind that the chosen benchmark operates with a small variety of instruments at every mannequin name, the typical variety of used instruments at every pattern is 4.35.
On the identical time it clearly signifies that LLMs can have instruments biases that doubtlessly could be misused by MCP suppliers, that may be related biases to these I reported earlier than — style biases. Analysis of the biases and their misuse could be necessary for additional research.
Engineering an answer
I’ve ready a PoC of tooling to deal with the talked about subject in apply — Grasp-MCP. Grasp-MCP is a proxy MCP server that may be linked to any variety of MCPs and likewise could be linked to an agent / LLM as a single MCP-server itself (presently stdio-transport MCP server). Default options of the Grasp-MCP I’ve applied:
- Ignore some parameters. The applied mechanics exclude all of the parameters that begin with “_” image from the software’s parameters schema. Later this parameter could be inserted programmatically or use default worth (if offered).
- Software description changes. Grasp-MCP collects all of the software’s and their descriptions from the linked MCP servers and supply a consumer a technique to regulate it. It exposes a technique with the easy UI to edit this listing (JSON-schema), so the consumer can experiment with totally different instruments’ descriptions.
I invite everybody to affix the challenge. With the neighborhood help the plans can embrace Grasp-MCP’s performance extension, for instance:
- Logging and monitoring adopted by the superior analytics;
- Instruments hierarchy and orchestration (together with ML powered) to mix each fashionable context administration strategies and good algorithms.
Present github web page of the challenge: link
