So I took the driver side out, tightened the screws down really tight on the enclosure and it stopped the vibrating
I tried it on the passenger side and while it did reduce the vibrating, it still does it a little.
But on the whole it's like an entirely new system, I can finally listen to sounds with any kind of bass!