AI at a Crossroads: Why Wall Street is Saying "No" to ChatGPT and Claude?

Author: Jeff @IOSG

On July 1st, Palantir CEO Alex Karp gave a 20-minute interview on CNBC, which some media described as "a mental breakdown." According to Karp, enterprises are paying a token premium to frontier labs while watching their own IP flow to the model providers. He called this leakage a transfer of alpha, occurring at the architectural layer: every request sent to a closed-source model arrives in plaintext on the service provider's servers. Just days before the broadcast, Palantir announced a partnership with NVIDIA to run the open-source Nemotron model in a customer-controlled environment, accompanied by a nine-point AI sovereignty declaration. PLTR's stock jumped 8% after the CNBC segment aired.

For the past two decades, enterprises adopted cloud software based on trust at the protocol level, and it worked. Each SaaS vendor saw only a slice of enterprise data, and most had little incentive to use customer data to enhance their core products. Salesforce saw sales pipelines, Workday saw personnel, Jira saw development iterations, and AWS provided the storage and compute foundation. Today's AI workflows, however, advocate uploading the entire corpus, along with structured context linking all departments, to maximize productivity. Regardless of intent, upstream providers can now use this data for new features rather than letting it sit idle on servers.

No one is slowing down. Anthropic's annualized revenue reached $47 billion in May, a significant jump from $9 billion at the end of 2025. OpenAI surpassed 900 million weekly active users in February. Both companies completed new funding rounds this spring, with valuations nearing $1 trillion and expected to IPO at even higher market caps. Years of privacy and IP allegations have not dented their momentum.

Some enterprises acted long ago. By February 2023, less than three months after ChatGPT's release, major Wall Street banks had restricted its use. In May 2023, after Samsung engineers leaked chip source code into ChatGPT, the company banned generative AI across its network. In response, OpenAI launched ChatGPT Enterprise in August 2023, promising not to use business data for training and offering a zero-data-retention (ZDR) agreement, which later became a standard requirement for enterprise procurement.

But contracts only lock down corporate accounts. IBM found that by 2025, shadow AI (employees feeding company data into unauthorized AI tools via personal accounts) was involved in one-fifth of data breaches, with heavy shadow AI use adding an average of $670,000 to breach costs. In a 2025 survey by security training company Anagram, 40% of employees said they were willing to violate AI usage policies to complete tasks faster.

Enterprises can at least buy their way out with ZDR contracts, non-training service tiers, and sovereign deployments for governments or Palantir's clients. For ordinary users like us, the debate over whether private AI matters continues—until a court subpoena arrives.

In May 2025, a court order forced OpenAI to retain even deleted consumer chats. In November, a judge ordered 20 million of those records to be handed over to *The New York Times*'s lawyers as evidence. Then came criminal cases: ChatGPT records of a defendant in the Palisades arson case entered evidence, and an affidavit for a Florida double homicide cited a suspect's queries about body disposal. In a July 2025 interview, Sam Altman also acknowledged that ChatGPT conversations are not protected by legal privilege, and OpenAI "could be compelled to hand over" user chat logs in litigation.

The point isn't that only criminals need private conversations. The fact that people's AI conversations are logged and subpoenable is a surveillance surface most users don't know exists. A Kolmogorov Law survey of 1,000 U.S. AI users in October 2025 found that 50% didn't know these conversations could be subpoenaed, while two-thirds believed such chats should receive protections equal to consulting a lawyer or doctor.

Self-hosted or open-source models running in verifiable environments are catching up fast, but the strongest ones still lag the frontier closed-source models by about 4 months in general capability. This puts token-maximizing enterprises and individuals at a crossroads: either forfeit a few months of model quality for privacy, or continue sending sensitive material to Anthropic's servers because that's what competitors are doing to seize productivity advantages.

Currently, there is no perfect solution. This report examines attempts to bridge the gap, observing how far frontier intelligence with provable privacy is from being delivered to enterprises and ordinary users.

Private AI is not a single engineering feat, but each mechanism in the market today addresses the same event: a prompt leaves your device, travels over the network, lands on a machine running a model, and a response returns. The differences lie in where plaintext exists on this path, who can read it there, and what verifies the privacy of the response.

At this layer, someone other than you reads your plaintext prompt. What happens next relies entirely on a promise.

• Contractual Zero-Retention is the enterprise version. The service provider knows who you are, processes your prompt, and promises not to retain it, enforced by contract and reputation.

• Anonymous Proxy erases who you are but does not encrypt what you say; the downstream provider still handles plaintext per their own policy. Terms vary: proxies like Duck.ai (DuckDuckGo's chatbot) negotiate deletion agreements with model providers, while Venice advises users to assume providers keep everything. Neither side can verify.

Every segment of the machine-to-machine path runs over TLS, which only encrypts the pipe; the receiving party can read all information. Relays typically use Oblivious HTTP (RFC 9458) to split this knowledge, like asking a friend to pass a note. The friend knows who passed it but can't read the content; the recipient can read the content but doesn't know who wrote it. OHTTP became an IETF standard in January 2024, and many companies now run production traffic over OHTTP relays rented from Cloudflare and Fastly.

This is also the privacy ceiling for accessing closed-source models, due to a matter of arithmetic. A single flagship training run now costs billions of dollars, and these labs' near-trillion-dollar valuations hinge on exclusive ownership of model weights. The duration of the capability gap sustains the premium, so labs guard the weight files like state secrets.

Meta has already unwittingly conducted this experiment. The LLaMA model released in February 2023 was initially available only to researchers, but within a week, its weights were leaked as a torrent on 4chan. Another week later, llama.cpp allowed the smallest 7B model to run locally on a MacBook. Three days after that, Stanford fine-tuned a chat assistant, Alpaca, on the same model for under $600. This leak brought the cost of running Llama down to electricity; anyone with the file could run it at home. In July 2023, Meta officially open-sourced Llama 2 under a commercial license with a 700-million-monthly-active-user exclusion clause. The weights leaked, and the premium followed.

Frontier labs could theoretically provide attestation (remote attestation) for closed-source model inference, but attestation can only prove which piece of code read the prompt, not what that code did with it. To determine if the server retained data, we would need to audit the serving code and reconstruct it to the hash reported by the hardware. But once the serving code is handed over, the lab also surrenders the batching and caching techniques underpinning its profit margins—techniques that will migrate to every future model generation. Apple and Meta can provide remote attestation for the service stacks behind iPhone and WhatsApp because their profits come from devices and ads; revealing the service code costs them almost nothing.

This is why flagship model weights and serving code don't reach external operators. Without external operators, there is no third-party attestation; without attestation, verifiable privacy only exists for open-source models.

Each mechanism in this category replaces trust promises with proofs based on hardware, cryptography, or physics. However, each pays a different price for the privacy upgrade, the primary one being that they can only run open-source models.

• TEE (Trusted Execution Environment) Confidential Computing runs inference inside a hardware enclave (a sealed compartment on the chip that even the machine operator cannot open). The chip signs an attestation stating exactly which model and which code were run.

• The prompt is only sealed at the destination. On the path relayed by the platform proxy, there remains a party that can read plaintext, and what prevents the proxy from logging or leaking the relayed content is only the protocol.

• E2EE (End-to-End Encryption) seals the readable relay. The user's device encrypts the prompt using the enclave's key; every hop in between carries a sealed envelope only the enclave can open.

• Trust is placed in the client. The code responsible for encrypting the prompt and verifying the attestation also has the power to revoke this guarantee. Therefore, verifiable E2EE requires both a proven enclave and open, reproducible client code.

• Compared to TEE's simplicity, E2EE's cost is engineering overhead, which slows feature integration. E2EE turns the proxy into a blind messenger, so all features that rely on reading plaintext must be rebuilt around the client key or reconstructed solely within the enclave.

• FHE (Fully Homomorphic Encryption, and MPC variants) removes the trusted party entirely. The server performs computations on ciphertext within a locked box it can never open; the key stays with you. MPC (Multi-Party Computation) splits the prompt into secret shares distributed among multiple parties; unless all participants collude, the effect is the same.

• The cost is speed. FHE natively only performs addition and multiplication, so the non-linear steps required for transformer operation must be rebuilt at high cost. Inference on ciphertext is 10,000 to 100,000 times more expensive than on plaintext, taking seconds to minutes per token for small models, compared to milliseconds unencrypted.

• Chips customized for encrypted operations promise to narrow the gap, but the first prototype was demoed only in early 2026, with commercial versions still years away.

• Local Inference removes the path entirely. The model runs on your own hardware. No relay, no server, no service provider, and no verification needed.

• The obvious costs are expense and model capability. gpt-oss-120b scores about half of GLM-5.2 on the Artificial Analysis index but is 65GB in size, exceeding the combined VRAM of two flagship gaming graphics cards. The full-precision GLM-5.2 can only run on 8-card data center nodes, with GPUs alone costing over $300,000.

Beyond these structural limitations, the cost of putting inference into an enclave is shrinking. For single-card inference, benchmarks from enclave cloud provider Phala show that enclave-mode H100 throughput loss averages less than 7%, and approaches zero for large models because the main cost is moving data onto the chip, not computing within it. For multi-card inference, NVIDIA's new Blackwell GPU supports direct encryption of inter-chip traffic, whereas older H100s can only achieve the same effect by routing through the CPU host at one-seventh the bandwidth. NVIDIA's own benchmarks on Blackwell show less than 8% throughput loss for a 397B model in enclave mode. With these advancements, the performance penalty of private inference itself is no longer a decisive constraint.

In fact, the enclave itself adds almost no extra operating cost for the provider. Every H100 shipped since 2023 has enclave mode built-in; the extra cost is the throughput loss from encryption, not additional chips. Currently, confidential H100 SKUs on Azure rent for $8.90 per hour, compared to $6.98 without enclave, a 27% premium over traditional cloud infrastructure. On the other hand, specialized enclave providers like Phala rent confidential-mode H100s starting at $3.80 per hour, lower than Lambda's price range of $3.99 to $4.29 for regular SXM cards. For managed API solutions, NEAR AI offers gpt-oss-120b endpoints with attestation at $0.15 per million input tokens and $0.55 per million output tokens, on par with plaintext routes from Amazon Bedrock, Together, and Groq. Even for models requiring multiple chips, NEAR AI's pricing for GLM 5.2 matches Fireworks exactly and is 15% cheaper for input and 4% cheaper for output on the larger Kimi K2.6.

While these new private inference providers may be burning profits to gain market share (a statement that holds for any company seeking growth), the structural trend is that the cost of privacy is decreasing for both consumers and operators.

Despite shrinking performance overhead, a visible gap remains between frontier models and SOTA open-source models. An entity pursuing maximum productivity must still trust frontier labs not to steal its IP to stay in the top tier.

The gap persists, but Bridgewater's AIA Labs and Thinking Machines presented a case on June 30: an open model fine-tuned with expert annotations beat frontier models in both accuracy and cost.

In the study, the team fine-tuned Qwen3-235B on Tinker (Thinking Machines' managed fine-tuning API service). They first sourced annotations from vendors, used that data for initial training, then sent disputed samples to the firm's investment professionals for re-labeling. Training used reinforcement learning (GRPO) with three modifications: round-robin batching (each task takes turns providing a batch), CISPO loss (limiting how far a single answer can shift the model), and on-policy distillation (anchoring to the current best checkpoint to prevent the model from learning from weaker copies).

Tasks were all from investment professionals' daily workflows: determining if a news article was important for C-suite investment professionals, if a central bank document hinted at future interest rate changes, and where templated language began in a document or email. Scores came from an independent test set. Frontier models averaged about 50% with simple prompts and only reached 78.2% with expert prompts, below the 80% threshold set by investment professionals. The fine-tuned Qwen achieved 84.7%, which, per the original wording, means making 29.8% fewer errors than the frontier best, at 13.8x lower inference cost.

https://thinkingmachines.ai/news/learning-to-replicate-expert-judgment-in-financial-tasks/

This case proves open-source models can win in accuracy and cost, but the training process was still not private. The expert annotations used were Bridgewater's private data, passing through Tinker's third-party service, residing at the same trust level as ZDR agreements. The fund also rented compute; the entire training ran on machines it never controlled. Buyers wanting this recipe without trust assumptions have few choices today. Renting bare GPU clusters leaves the training process readable to the cloud operator. Buying the cluster solves data hosting but sends costs skyrocketing.

Routes with attestation have just arrived. In March, Workshop Labs and Tinfoil released Silo, a post-training stack running in Tinfoil enclaves on a single 8-card node, with keys controlled only by the customer. The article states the enclave cost adds 11 minutes to a two-hour training run, and by freezing base weights and training only small adapters on top, the stack can accommodate a trillion-parameter model (Kimi K2 Thinking). The challenge is that reinforcement learning requires moving data back and forth between components, precisely where enclave costs lie.

Less than a month after Silo's release, Workshop Labs was acquired by Thinking Machines. The components needed to run the next Bridgewater-style RL loop within an enclave are now under the same roof.

Another problem exists outside all private inference mechanisms. These mechanisms manage the path from prompt to model, but every external tool call initiated by an agent opens a path the inference layer cannot touch. The recent harness engineering trend magnifies the problem: every tool, memory store, and data source connected to the model is another destination that reads its slice of the workflow in plaintext. The calendar server reads schedules, the database server reads queries. A fully local agent needing anything outside its training set must still send search terms in plaintext to a search engine; the server cannot answer without reading the plaintext.

Mainstream solutions still default to the protocol layer. Companies like Runlayer and MintMCP use a central gateway to manage all tool traffic, stripping personally identifiable information (PII) before requests leave. The gateway also decides which servers receive traffic, blocking unvetted ones, and logs each call's destination and content for forensics. Even with independent audits (SOC 2), the tool server must still read the plaintext query to respond; whether it retains a copy depends on its own retention terms, multiplied by every tool in the harness. Furthermore, the gateway itself is an additional trust-dependent reader on the path, not a verification.

Structural solutions target the middle layer. For example, Phala hosts MCP servers directly in TEEs, with directories covering wallets, code execution, and data sources; users can verify privacy claims via attestation rather than trusting the operator. However, TEE-hosted tools ultimately must pass queries in plaintext to the service provider; the enclave seals only the messenger, not the destination.

Only a few destinations have learned to respond without reading, but only for structured queries. Apple provides Private Information Retrieval for iPhones, allowing phone number checks against spam databases without exposing the number; Microsoft uses the same scheme for passwords in Edge. MongoDB's Queryable Encryption encrypts fields client-side before they leave, enabling the server to perform equality and range matches on ciphertext alone.

For open-ended search, today's best answer stops at trust; verifiable encrypted search hasn't left the lab. Brave promises zero data retention on its own 40-billion-page index (not Google's), but it's still at the protocol layer. Exa built a neural index that embeds user keywords into semantics, ranking results by semantic match, but the embedding step still computes from plaintext on Exa's servers. MIT's 2023 Tiptoe paper ranked results on 360 million web pages without exposing the query, but each search burned significant server compute, with ranking quality lagging unencrypted search. Apple's 2024 Wally paper hid real queries among decoys, reducing communication cost up to 31x, but this math only becomes cheap at millions of concurrent queries—a scale no private search system has today.

Encrypted search is possible; it's just that performance and price aren't commercially viable yet.

Demand for private AI is growing. Venice AI recently surpassed 3.5 million registered users and 1.3 trillion monthly tokens, followed by a Series A equity funding round at a $1 billion valuation. Proton is a direct competitor; its chat product Lumo surpassed 10 million users within a year of launch. On the infrastructure side, Phala currently processes 2 to 3 billion tokens daily on OpenRouter. Duck.ai routes gpt-oss-120b and Gemma into Tinfoil enclaves, providing verifiable privacy beyond proxy anonymity. This doesn't even count self-hosting, likely the largest channel for private inference, as models run on one's own hardware, leaving no usage trail.

Yet within the massive wave of mainstream AI, private AI occupies only a tiny fraction, and this gap will only close if frontier labs intentionally meet this demand. In May, Google's entire product line processed 3,200 trillion tokens; by that measure, Venice's monthly throughput equals about 18 minutes of Google's. Last November, Google launched Private AI Compute (PAC), running some Gemini-powered features in sealed TPU enclaves isolated from the company itself, with the design independently audited by NCC Group. But the issue is that PAC only covers a few Pixel features like personalized recommendations and recording summaries, not the Gemini app used by hundreds of millions. Google dares to submit the design for audit because these features monetize via devices and ads, not token sales.

Current hosting solutions are also imperfect. Users seeking maximum privacy via E2EE must wait for new features to be rebuilt where the service provider can't read. Private harnesses still rely on protocols at the service layer. Affordable post-training still requires trusting third-party vendors for the best fine-tuning results. Self-hosting sheds all service providers at once, but running the strongest open-source model locally may cost more than the house it's in.

Flaws aside, private AI is already a real and affordable option, and the remaining gaps are narrowing. For ordinary consumers, private chats with open models under no-logging promises cost nothing on Lumo and Venice, while $18–$20 subscriptions from Venice or Tinfoil seal the same chat into enclaves, no more expensive than a ChatGPT subscription. For enterprise workflows, attested endpoints are now even cheaper than plaintext routes. Endpoints like NEAR's E2EE API can already bring encrypted context into enclaves; memory, file uploads, and custom instructions can now operate over E2EE. For attested post-training, NVIDIA's upcoming Vera Rubin NVL72 will expand confidential computing from Blackwell's 8-card nodes to 72-card racks, making frontier RL loops more feasible without exposing IP.

However, the crucial value capture lies beyond these layers of price compression. Privacy is nearly free where it already exists, but it hasn't covered mainstream agentic workflows. Operators focused on renting/selling enclaves hold a switch on standard chips, not a moat, while protocol-layer gateways compete with traditional middleware. The defensible ground is the half of the problem not yet solved in this report: training loops locked in enclaves, end-to-end sealed tool calls, search indexes that cannot see terms. Whoever builds one of these first sells something no price war can commoditize. Capital chasing private AI should buy the gap, not the switch.

So, trust or verify? For execution-heavy, agent-heavy tasks, choose trust, because each tool call already hands plaintext to destinations an enclave cannot seal, and frontier models deserve their price in such loops. For the high-order thinking that distinguishes a company from its competitors, choose verification. Strategy, planning, and judgment distilled from years of professional experience are precisely the alpha in dispute. The path forward is fine-tuning open-source models with these proprietary insights within company-controlled boundaries. In areas central to a company's alpha, expert-tuned open models have already beaten the frontier in both accuracy and cost, and the infrastructure to build them in privacy is arriving node by node.