Had the same question, lmk if you find the magical "off" switch
edit: apparently it's turning into a bit of a disaster for Google, cos everyone wants it off:
https://bsky.app/profile/litbowl.bsky.social/post/3lfvlw656x...
Workspace admins have a bunch of Gemini checkboxes but none of them actually seem to disable Gemini.
They only gave me access to the 'Gemini for Google Workspace' settings that allow me to turn it off when I complained to support.
They're making it a forced upgrade so they can charge more for email accounts. +$2/user/month is what they want for Gemini.