When private AI makes sense — and when it doesn't

I design private AI systems for clients. It’s part of what I sell. So read this knowing it costs me something to write: a meaningful share of the companies that call me about private AI shouldn’t buy it, and I tell them that on the first call.

Private AI is an architecture decision, not an ideology. Make the decision deliberately and the right answer falls out of your data, your workflows, and your capacity to operate what you buy. Make it on fear, or on a vendor’s pitch deck, and you end up in one of two bad places: overspending on infrastructure you can’t run, or leaving sensitive data flowing into tools nobody approved because “we’ll go private eventually” became a reason to do nothing.

Here’s how I actually make the call.

What private AI actually is

Strip the marketing and private AI is four things: model infrastructure you control (on your hardware or in your private cloud), a governed knowledge layer over your documents and systems, a controlled way for your team to use it, and the policy and training wrapped around all of it.

It’s an architecture and an operating model, not a GPU box. That distinction matters because the box is the easy part. Plenty of vendors will sell you hardware this quarter. What makes the system safe and useful is everything around it: who can query which data, how source systems connect, how usage gets monitored, who owns the roadmap. A private model deployed without those controls isn’t safer than a public one. It’s just more expensive.

The problem underneath the question

When an executive asks me about private AI, the real question is usually “how do I stop worrying about what my people are pasting into AI tools right now?” That worry is grounded in data, not paranoia.

Cyberhaven’s 2026 AI Adoption & Risk Report found that 39.7% of data movements into AI tools involve sensitive data, and 32.3% of workplace ChatGPT usage happens through personal accounts, outside any corporate control. IBM’s 2025 Cost of a Data Breach research found that one in five organizations reported a breach tied to shadow AI, that heavy shadow-AI use added an average of $670,000 to breach costs, and that only 37% of the breached organizations IBM studied had policies to manage AI or detect unsanctioned use.

Look at what those numbers actually say. The damage comes from ungoverned use. Nothing in that research says commercial AI tools, governed properly, are the risk. That’s the step the private AI sales pitch skips: buying private infrastructure does not, by itself, fix ungoverned behavior. Policy and governance fix ungoverned behavior. Sometimes the architecture that policy points to is private. Often it isn’t.

When private AI is the right call

Four situations where I’ll recommend it without hesitation:

Regulated or privileged data is the work. Patient records, legal matters, client financials, audit workpapers. When the core workflow runs on data with statutory or privilege obligations, “trust the vendor’s terms” is a harder conversation with your regulator than “the data never left our environment.”

Contracts say the data can’t leave. Data-residency and processing clauses in client agreements are binding whether or not you think a commercial vendor is trustworthy. I’ve seen MSAs that flatly prohibit third-party AI processing of client material. If that’s your paper, the architecture question is already answered.

Your IP is the company. Proprietary formulations, trading logic, unpublished R&D, deal pipelines. When the cost of a single leak is existential rather than embarrassing, the premium for provable containment is cheap insurance.

Leadership needs provable control, not promised control. Some boards and some clients want to see the audit trail, the access controls, and the network diagram. “The vendor says they don’t train on our data” is a policy assertion. A private deployment is a fact you can demonstrate.

When it’s the wrong call

And the four situations where I’ll tell you to keep your money:

Your data is mostly public anyway. Marketing content, published pricing, public documentation. Running a private model to draft blog posts is buying a vault for the newspaper.

You can’t name one genuinely sensitive workflow. Not a sensitive feeling — a workflow, with named data, that runs regularly. If we sit down and can’t identify one, you don’t have a private AI problem. You have a prioritization problem, and it’s cheaper to solve.

You’re small with no IT capacity. A ten-person firm with no one to own infrastructure should not be running model servers. Private AI comes with patching, monitoring, capacity planning, and upgrades. If nobody owns that, the “secure” system rots into the least secure thing you operate.

Enterprise commercial tools meet your bar. This is the one buyers underweight. The enterprise tiers of Claude, ChatGPT, and Copilot are not the free consumer apps your employees are using in the shadows. Anthropic’s commercial products don’t use your inputs or outputs for model training by default, per their published policy, and enterprise tiers across vendors add SSO, admin controls, and audit visibility. For many companies, an enterprise agreement plus a real acceptable-use policy clears the actual security bar at a tenth of the cost of a private deployment. When that’s true, that’s what I recommend, even though I make more money building the other thing.

The test I run

Four questions, in order. I use this in every assessment:

1. What specific data can never leave, and why? Name the regulation, the contract clause, or the IP. “It feels risky” doesn’t count.
2. How many workflows touch that data weekly? One occasional workflow can often be isolated and handled separately. Three or more daily workflows start justifying dedicated architecture.
3. Who operates it in month six? Not who installs it. Who patches it, monitors it, and answers for it after the consultants leave. No name means no private AI.
4. What’s the real cost delta? Price the enterprise commercial option with proper controls against the private build including operations. If private costs 8–10x more, the sensitive workflows have to be worth 8–10x more protection. Sometimes they are. Say it out loud and check.

Strong answers to all four: build private, and I’ll help you do it right. Weak answers to two or more: buy enterprise commercial tools, write the governance, train the team, and revisit in a year.

What the fix looks like when it works

One number from my own delivery work. An ad-tech client came in with sensitive data showing up in 18% of employee AI interactions. After we put governance and the right architecture in place — delivered with Last Rev, the platform engineering firm I co-founded — that number went to zero. Measured, not estimated.

The lesson from that engagement wasn’t “private beats public.” It was that the exposure ended when someone owned the decision: which data, which tools, which controls, verified with monitoring instead of a memo. That ownership is the product. Private infrastructure is one possible component, and whether you need it is a question with a checkable answer, not a matter of belief.