I've been poking around the medical / ehr LLM space and gently asking people how they're preserving privacy and everyone appears to be just shipping data to cloud providers based solely on a BAA. Kinda baffling to me, my first step would be to set up local models even if they're not as good, data breaches are expensive.
Same, and I've just sent an email up the chain to our exec saying 'hey remember those trials we're running and the promises the vendors have made? Here is why they basically can't be held to that anymore. This is a risk we highlighted at the start'
Even Ollama + a 2K gaming computer (Nvidia) gets you most of the way there.
Technically you could probably just run it on EC2, but then you’d still need HIPPA compliance