I managed to solve this after a few more hours of tinkering, I am putting the fix here for other's reference.
To get the xinput-calibrator to fit the screen run
Code: Select all
xinput_calibrator --geometry 480x320
My 16:9 3.5" screen worked well with 480x320, but this might need modification for other sizes or aspect ratios.