Technicaly i dont know why this happens.
There are many problems with the physics, especialy when animations or subworlds are involved.
I have the same problem in one of my ages under construction.
I noticed that when i place a second collider just above or under the existing one, it does not happen any longer.
The same dull solution for cameras that go through walls.
I know its a workaround, but in most cases it is sufficient.
May be this helps you a bit or others to find a real solution.