LLM Tool Calling for Internal Assistants: Design Choices

LLM tool calling lets an internal assistant answer questions about live data — "what's waiting on me?", "who's on leave Friday?" — by calling functions you define and answering from their results. What matters: narrow, typed tools (not "run SQL"), run with the user's own permissions, results formatted for the model (local times, names), a cap on tools per question, and answers that conclude nothing beyond the data. Native function calling or a routing step both work under those rules.

What "tools" are

A tool is a function the model can ask you to run: a name, a description and a parameter schema. The model doesn't execute anything; it returns "call my_meetings with { when: "tomorrow" }", your code runs it, and the result goes back to the model to write an answer. Every major provider documents this — Gemini as function calling, Groq and Anthropic as tool use.

For an internal assistant, tools turn a documentation chatbot into something that knows your workspace. LetRelay's assistant has read tools like these:

Tool	Answers
my_day	What's waiting on me: requests, tasks, meetings, approvals
search	Find a request, task, person, meeting or article
find_person	Who someone is, their team, availability
my_meetings	Upcoming meetings and requests awaiting my answer
who_is_on_leave	Who's off on a date (as far as I'm allowed to see)
get_task / get_request	Details of one item

Rule 1: narrow, typed tools

A tool per question type, with a small parameter schema, is safer and more accurate than a general one. "Search requests by status and assignee" can't be talked into reading the payroll table; "run this SQL" can. Narrow tools also make the model's choice easier: good descriptions ("Upcoming meetings for the current user, and meeting requests waiting for their reply") guide it to the right one.

Rule 2: the user's permissions, always

Every tool must run as the person asking. In LetRelay, tools query the database with the user's own authenticated session, so row-level security decides what comes back: a requester asking "show me all open requests" gets their own; an agent gets their team's; nobody gets another organization's. The tool doesn't re-implement permissions, so it can't get them wrong. See row-level security vs app-layer authorization.

This is also the main defense against prompt injection: if the model is tricked into calling a tool, it can still only see what the user could see. See prompt injection in internal AI tools.

When visibility is limited, say so. LetRelay's who_is_on_leave adds a note when the user can only see part of the organization, so the model doesn't conclude "nobody is on leave" from a partial list.

Rule 3: format results for the model

Raw rows make models make mistakes. Before returning results:

• Times in the user's timezone, as readable strings ("Thu 27 Nov, 15:00"), computed in code — never leave timezone conversion to the model.
• Names, not IDs. assignee: "Jordan Lee", not a UUID.
• Only needed fields. Fewer tokens, less to misread.
• Explicit empties. "No meetings tomorrow" is clearer than an empty array.
• Order that matters first. LetRelay's my_day leads with what's waiting for the user's reply, because an early version buried a pending meeting request below routine items and the model left it out.

Rule 4: native function calling or a routing step?

Native function calling: send the tool definitions with the conversation; the model decides whether and which tools to call, possibly over several turns. Flexible, and supported by all major providers.

A separate routing step: a small, fast model returns a structured decision — which tool(s), which arguments — using a strict JSON schema; your code runs the tools; a second call writes the answer.

LetRelay uses the second, for reasons specific to running on several free providers:

• One strict schema works identically across providers, so failover between models doesn't change behaviour.
• Schema-guaranteed output (every field required, nullable when unknown) stopped a real bug where arguments were silently dropped.
• The routing model can be small and fast while the answering model is stronger.
• Deterministic code can re-check arguments — ticket numbers, dates, names — before any tool runs.

Native tool use is a good choice when you use one provider and want the model to chain calls. Either way, the other rules apply. The routing design is in intent routing with small LLMs.

Rule 5: cap the calls

Open-ended agent loops — call a tool, read, call another, repeat — cost a model call per step and multiply latency. For workplace questions, one step with up to two tools covers nearly everything ("what's on my plate and who's out tomorrow?"). LetRelay allows two tools per question, run in parallel with no extra model call. Anthropic's guidance on building agents makes the general case for starting with the simplest pattern that works.

Rule 6: answers that stay inside the data

The answer prompt needs rules that stop the model filling gaps:

• Answer only from the tool results provided.
• If results are empty or partial, say so; never conclude beyond the data ("nobody is on leave" when the tool could only see your team).
• Be complete: if the tool returned four items, mention four, or say "and 2 more".
• Reply in the user's language and script.

Test these with an evaluation set of real questions and known data, checking both what must appear and what must not. See how to evaluate an LLM feature with a golden set.

Rule 7: reads run; writes confirm

Read tools can run immediately — they only show the user what they could already see. Tools that change data are different: they should produce a proposal shown on an editable confirmation card and run only after the user confirms. See AI agent actions with human confirmation.

Rule 8: work without the model

If the answering model fails after tools ran, the results are still useful. LetRelay formats tool results with a template when every model is unavailable — a plain list of the user's meetings is better than an error. See AI that degrades gracefully.

FAQ

What is LLM tool calling?

A pattern where a language model asks your application to run a defined function — with a name, description and parameters — and then answers using the function's result. The model never executes code itself.

How do I keep tool calls secure?

Make tools narrow and typed, run every call with the requesting user's own permissions, validate arguments in code, and require confirmation for any tool that changes data.

Should I use native function calling or a router?

Native function calling is convenient with a single provider and multi-step tasks. A separate routing step with a strict schema is more predictable across providers and lets code check arguments before tools run.

How many tools should an assistant call per question?

As few as answer the question. For workplace assistants, one step with up to two tools covers most questions without the cost of open-ended loops.

Sources

• Google — Gemini function calling
• Groq — Tool use
• Anthropic — Tool use overview
• Anthropic — Building effective agents
• OWASP — LLM06:2025 Excessive Agency