Okay so eventually : I swapped the 17" monitor (1280x1024) I had acquired especially for this use (and an Amiga) with my main 24" 1080p screen...
Surprisingly, it really looks better now even though it should look worse from the same distance (blockier) !
So I disabled the bilinear filtering and enabled scanlines. I'm pretty satisfied with the result. Sure, it's not as nice as using complicated filters on a regular PC with MAME but that's the most the Pi can handle anyway. And it's pretty good !
So thanks to you guys for suggesting the issue was the monitor. (btw : forcing the resolution didn't give satisfactory results on the 17" anyway).
Now, I'm really hoping someone will carry on Squid's work as his 3 ports are the best ones on the Pi, it's sad to see there's no more progress anymore
BTW out of curiosity I tried retropie... It has a FBA version that is MUCH slower (but more compatible). Sadly Squid's FBA port doesn't support everything (R-Type, Toki, later DDP, PCE stuff are missing for instance
Also, sound on Pulstar is broken, boohoo