I am experiencing similar problems (see viewtopic.php?f=44&t=270030
Has anybody found values that work or dividers that work, apart from integer dividers of the Oscillator clock (19.2MHz)?
It's a nightmare to use custom mode in HDMI, everything is scarcely documented and there are many references to "defaults" or "normal" or "standard for" which are not normal for laypeople not accustomed to HDMI standards.
BTW I think a clear guide with clear instructions (i.e. put '1' if clock is active low, or put '0' if HSYNC is active High, without ambiguities, should be a nice thing).