Overriding models breaks ability to interact with objects

When I override the hand models either for either SteamVR or for all SDKs I lose the ability to interact with objects.

If I stop playback and switch off the Override models checkbox in NVRPlayer and play the application it all works again using the standard models.

So to be clear, I override with controller models for each hands and mesh colliders as well (the colliders are present on the prefab GO I put in all 4 references in NVRPlayer). The colliders work, I cant push my hand through walls and I can collide with the buttons - but I cant interact with the drawer or anything using grip.