Physical access has always been game over. Having a networked computer means your threat model is literally everyone on the planet, which is a much bigger problem than keeping people from physically getting access.
Direct physical access by the attacker isn't strictly necessary (i.e. operation Olympic Games) to "network" a computer you otherwise believe isn't networked. Unless you're bootstrapping from nothing attackers have tons of potential "ins" (firmware, the operating system, application software) to introduce backdoors or side-channels.
I've very nearly reached the point of just assuming all "modern" computers are effectively "networked", even if only by ultra-low bandwidth, exceedingly high-latency unidirectional side channels. Just bringing an "untrusted" computer into proximity of a "trusted" computer (say, having a smartphone in your pocket) might be enough to allow for exfiltration of data from the "trusted" system (assuming there's a side-channel in the "trusted" computer you're unaware of).
Ooh! This is a fascinating approach. I'm still skeptical that this is widespread enough of an issue to warrant the same level of caution as connecting a computer to the Internet, but I'd love to read more about examples of this actually happening in the real world (ie not researchers with full control of the environment) if you have any.