Just playing Super Mario All Stars on an RGB matrix panel as you do.
To keep things together better, I 3D printed a bracket in two pieces, the subject of a work in progress playground note: https://adafruit-playground.com/u/jepler/pages/3d-printed-bracket-for-4-64x64-2mm-rgb-matrix-panels
This is using a 3rd party breakout board with 3 HUB75 connectors. I recently added the ability to drive up to 3 connectors to Adafruit Piomatter, as well as temporal dithering.
In this specific interest, I'm using 2 connectors to drive a total of 4 64x64 panels. --num-planes=6 --num-temporal-planes=2
means I'm getting effective 18 bit color, albeit with the least significant 2 bits being shown on alternate refreshes, creating a tiny amount of 44Hz color shimmer.
All these "go faster" tricks together give me about 88Hz refresh rate on the panel, despite that we're still limited to about 10MB/s of data between the PI's main CPU and the PIO peripheral that's acting as the LED controller.