But for the intrusive one you have the visibility into its membership in the other structures, since you're holding a reference to the object and its intrusive fields.

In the non-intrusive case, all you know is how you got there, you don't have return info to the other structures— particularly annoying if you're trying to destroy an object and have to speculatively look for references to it in the other lists in order to get rid of them.