The prompts that never leave the building: local LLMs for sensitive IP.

Some prompts should never leave the building. Not because the model on the other end is untrustworthy, but because the text we would be sending (an unreleased architecture, a customer’s data model, the guts of something we haven’t shipped yet) is the whole reason the work has value. Over the last stretch we’ve been quietly moving a class of our internal AI workflows off hosted APIs and onto models that run entirely on our own hardware. This is what we learned, and how we decide which prompt goes where.

Why some prompts should never leave the building

Most AI tooling is a wonderful default. For drafting, summarising, brainstorming, and the hundred small tasks that fill a working day, a hosted frontier model is faster and better than anything we would run ourselves, and we reach for it without a second thought. The problem isn’t capability. It’s that a meaningful slice of our work involves material we have a duty to keep close: proprietary source, pre-launch product designs, and information customers hand us on the understanding that it stays with us.

The instinct to paste that into whatever tool is open is completely human, and completely understandable. It is also the exact moment a piece of intellectual property leaves our control. A privacy policy is a promise about intent; it is not a guarantee about where bytes travel. For the sensitive slice, we wanted a stronger guarantee than a promise, one that holds even if we’re wrong about someone’s terms of service.

What “local” actually buys you

Running a model locally means the weights live on a machine we own and the inference happens there too. The prompt is composed, answered, and discarded without a single token crossing our network boundary. That single property, the data never leaves, changes the risk calculation completely. There is no third-party retention policy to read, no region to worry about, no question of whether a payload was logged for debugging on the far end. The honest short version is this:

• Confidentiality by construction. If the data physically cannot reach an outside server, no outside server can leak, log, or train on it.
• A clean audit story. “Where did this IP go?” has a one-word answer: nowhere. That is far easier to stand behind than a chain of vendor assurances.
• Predictable behaviour offline. A local model doesn’t change underneath us on someone else’s release schedule, and it keeps working when the network doesn’t.
• Cost that scales with hardware, not tokens. For high-volume, repetitive internal tasks, the sunk cost of a GPU beats a per-call meter.

The trade we made

None of this is free, and we’d be lying to say otherwise. The open models we can realistically host are smaller and less capable than the best hosted frontier systems. They are slower on our commodity hardware, they need more careful prompting, and getting one to run well (quantised, memory-fit, actually fast enough to use) is real engineering work rather than a checkbox. We spent more evenings than we’d like tuning context windows and watching a laptop fan spin.

The question was never “which model is smartest?” It was “what is the smallest, most private thing that gets this specific job done?”

Framed that way, the answer stopped being about leaderboards. A local seven-or-eight-billion-parameter model, pointed at a narrow, well-defined task (classify this, rename that, summarise this internal doc, refactor this private snippet) is often more than good enough. We were comparing it against “the best model in the world” when we should have been comparing it against “the risk of this leaving the building.” On that scale, good enough and private wins.

Where local models fall short (and that’s fine)

We are not romantics about this. When a task genuinely needs the strongest reasoning available (a gnarly design problem, long-context synthesis, work where the quality ceiling really matters and the inputs are not sensitive), a hosted frontier model is still the right tool, and we use it happily. The mistake would be treating “local” as a religion instead of a tool. The point isn’t to run everything ourselves; it’s to have a private option that’s good enough that reaching for it is easy, so the sensitive prompts have somewhere safe to go.

How we decide, prompt by prompt

The rule we landed on is almost embarrassingly simple, and it fits on one line: if this leaked, would we care? If the answer is yes (if the prompt carries our IP, an unreleased product, or a customer’s confidential data), it runs local, full stop, even if a hosted model would answer a little better. If the answer is no, we use the best tool for the job without ceremony. Making that judgement the default, rather than something to remember under deadline pressure, is most of the battle.

The broader lesson has outlived the specific models we’re running today. Privacy isn’t a feature you bolt on at the end; it’s a decision about where computation happens, made one workflow at a time. For anything we’d be unhappy to see on someone else’s server, the safest place for that computation to happen is a machine we control, and the engineering to make that pleasant to use is some of the most worthwhile work we’ve done this year.