In the end, I used an SD adapter - which juts out of the board quite far.
So, my suggestion would be to use a micro SD port rather than the full version in future revisions.
- Mechanically identical, so hopefully not too hard to rejig the layout.
- Smaller, so would be harder to knock and break.
- More common. Almost every phone uses these memory cards - so people should have an old one laying around. (Same rationale as using microUSB for charging)