How large do you wager your moat to be? Confidential computing is something all major cloud providers either have or are about to have and from there it's a very small step to offer LLM-s under the same umbrella. First mover advantage is of course considerable, but I can't help but feel that this market will very quickly be swallowed by the hyperscalers.
Cloud providers aren't going to care too much about this.
I have worked for many enterprise companies e.g. banks who are trialling AI and none of them have any use for something like this. Because the entire foundation of the IT industry is based on trusting the privacy and security policies of Azure, AWS and GCP. And in the decades since they've been around not heard of a single example of them breaking this.
The proposition here is to tell a company that they can trust Azure with their banking websites, identity services and data engineering workloads but not for their model services. It just doesn't make any sense. And instead I should trust a YC startup who statistically is going to be gone in a year and will likely have their own unique set of security and privacy issues.
Also you have the issue of smaller sized open source models e.g. DeepSeek R1 lagging far behind the bigger ones and so you're giving me some unnecessary privacy attestation at the expense of a model that will give me far better accuracy and performance.
> Cloud providers aren't going to care too much about this. ... [E]nterprise companies e.g. banks ... and none of them have any use for something like this.
As former CTO of world's largest bank and cloud architect at world's largest hedge fund, this is exactly opposite of my experience with both regulated finance enterprises and the CSPs vying to serve them.
The entire foundation of the IT industry is based on trusting the privacy and security policies of Azure, AWS and GCP. And in the decades since they've been around not heard of a single example of them breaking this.
On the contrary, many global banks design for the assumption the "CSP is hostile". What happened to Coinbase's customers the past few months shows why your vendor's insider threat is your threat and your customers' threat.
Granted, this annoys CSPs who wish regulators would just let banks "adopt" the CSP's controls and call it a day.
Unfortunately for CSP sales teams — certainly this could change with recent regulator policy changes — the regulator wins. Until very recently, only one CSP offered controls sufficient to assure your own data privacy beyond a CSP's pinky-swears. AWS Nitro Enclaves can provide a key component in that assurance, using deployment models such as tinfoil.
I suspect Nvidia have done a lot of the heavy lifting to make this work; but it's not that trivial to wire the CPU and GPU confidential compute together.
Being gobbled by the hyperscalers may well be the plan. Reasonable bet.
GCP has confidential VMs with H100 GPUs; I'm not sure if Google would be interested. And they get huge discount buying GPUs in bulk. The trade-off between cost and privacy is obvious for most users imo.
This. Big tech providers already offer confidential inference today.
Yes Azure has! They have very different trust assumptions though. We wrote about this here https://tinfoil.sh/blog/2025-01-30-how-do-we-compare
Last I checked it was only Azure offering the Nvidia specific confidential compute extensions, I'm likely out of date - a quick Google was inconclusive.
Have GCP and AWS started offering this for GPUs?
Azure and GCP offer Confidential VMs which removes trust from the cloud providers. We’re trying to also remove trust in the service provider (aka ourselves). One example is that when you use Azure or GCP, by default, the service operator can SSH into the VM. We cannot SSH into our inference server and you can check that’s true.
But nobody wants you as a service provider. Everyone wants to have Gemini, OpenAI etc which are significantly better than the far smaller and less capable model you will be able to afford to host.
And you make this claim that the cloud provider can SSH into the VM but (a) nobody serious exposes SSH ports in Production and (b) there is no documented evidence of this ever happening.
We're not competing with Gemini or OpenAI or the big cloud providers. For instance, Google is partnering with NVIDIA to ship Gemini on-prem to regulated industries in a CC environment to protect their model weights as well as for additional data privacy on-prem: https://blogs.nvidia.com/blog/google-cloud-next-agentic-ai-r...
We're simply trying to bring similar capabilities to other companies. Inference is just our first product.
>cloud provider can SSH into the VM
The point we were making was that CC was traditionally used to remove trust from cloud providers, but not the application provider. We are further removing trust from ourselves (as the application provider), and we can enable our customers (who could be other startups or neoclouds) to remove trust from themselves and prove that to their customers.
You are providing the illusion of trust though.
There are a multitude of components between my app and your service. You have secured one of them arguably the least important. But you can't provide any guarantees over say your API server that my requests are going through. Or your networking stack which someone e.g. a government could MITM.
I don't know anything about "secure enclaves" but I assume that this part is sorted out. It should be possible to use http with it I imagine. If not, yeah it is totally dumb from a conceptual standpoint.
Confidential computing as a technology will become (and should be) commoditized, so the value add comes down to security and UX. We don’t want to be a confidential computing company, we want to use the right tool for the job of building private & verifiable AI. If that becomes FHE in a few years, then we will use that. We are starting with easy-to-use inference, but our goal of having any AI application be provably private