Ah yeah you're right. Does the "Audio MIDI Setup" Mac utility app help you here at all?
It gets close, but no way to truly pin it still. It effectively does the same thing that System Settings > Sound > Output & Input does but with a better UI making it clearer that you are making a change to the primary. But the change is still just as unpinned as it would be from the other location.