LLM Features in Production: Practical Guides
Engineering notes on using language models in a product: testing them, caching their answers, controlling cost, and keeping them from doing damage.
6 articles
LLM Tool Calling for Internal Assistants: Design Choices
How to give an internal AI assistant tools — reading workspace data with the user's permissions, formatting results for the model, native function calling vs a routing step, limits on how many tools run, and answers that stay inside the data.
Caching LLM Responses Safely: What to Cache and What Not
Caching LLM answers saves tokens and latency — and can leak one user's data to another if done carelessly. A safe design: cache only answers that are identical for everyone, key on normalized question plus content version, expire, and lock the cache down.
Intent Routing with Small LLMs: Fast, Cheap and Right
How to route chat messages to the right action or tool with a small, fast language model — a flat strict schema, deterministic argument extraction, rules before the model, a short timeout, and a 51-case evaluation — using measurements from a production assistant.
How to Reduce LLM Token Usage Without Worse Answers
Practical ways to cut LLM token usage — skip the model when rules suffice, send less context, route to smaller models, cache repeated answers, use provider prompt caching and cap output — with measured numbers from a production assistant.
Prompt Injection in Internal AI Tools: A Practical Defense
Internal AI assistants read tickets, documents and messages written by other people — which makes prompt injection an internal threat too. How direct and indirect injection work, and the layered defenses that actually limit the damage.
How to Evaluate an LLM Feature with a Golden Set
A practical way to test LLM features before shipping changes — a golden set of real inputs, deterministic checks for what must and must not appear, separate evals for retrieval, routing and answers, and running them on free-tier rate limits.