Thanks for the replies.
However, I think you guys might be misunderstanding what I'm trying to achieve. Advance MAME is a program that emulates old arcade games, mostly from the eighties and nineties. Most arcade games from that era had a much lower resolution than modern TVs and computer monitors. Therefore, the games generally have to be scaled up to fit on a modern display. This can either by done by Advance MAME itself (i.e. in software), or Advance MAME can program the framebuffer to create a custom display mode that exactly matches the resolution of the original game. The latter is generally considered to be the preferred option as it means that more processor power is freed up for other aspects of the emulation. It's a bit crazy to use the CPU for scaling when there's a perfectly good GPU available to do the same thing.
A traditional framebuffer driver on a PC version of Linux can create custom display modes by setting the pixel clock, vertical sync, horizonal sync timings, etc. However, it appears that on the Raspberry Pi, those parameters are set in config.txt and cannot easily be changed after the RPi has been booted. Instead, custom display modes are created purely by using GPU scaling. That would be OK except that the GPU driver always forces pixels to be square, and some games don't have square pixels.
This is probably best illustrated by an example. One game that causes problems is Ghouls 'n Ghosts. This game was originally played on an arcade monitor with a 4:3 aspect ratio. However, its resolution is 384x224. This means that each individual pixel has a 7:9 aspect ratio (i.e. (4*224)/(384*3)).
I'm using a wide screen 16:9 aspect ratio monitor with a resolution of 1920*1080. So, to determine what display resolution I need to use, I perform the following calculation:
Yres = 224
Xres = 384*(3*16)/(4*9) = 512
That should, in theory, give me a resolution that's big enough to display the game, and with each individual pixel having a 7:9 aspect ratio. My calculation assumes that the usable display area completely fills the screen. The trouble is, it doesn't. The display driver realises that the pixels won't be square, and compensates by creating big margins at the top and bottom of the screen.
This is very frustrating. There must surely be a simple setting somewhere to change this behaviour. However, despite a lot of googling, I'm not finding it.