The only reason that I considered using the HDMI is that you thought that you already had a driver. But I would go back to my original idea of using the CSI-2 port. You tell me what works best for you.
I don’t know for a fact if your HDMI port is an input only, output only, or configurable to be either. I know for a fact the CSI-2 port is an input.
To avoid trying to make this all “automatic”, just provide a method where the user has to enter all of the configuration information:
• Aspect ratio
• RGB or Y Cb Cr
Also, let the user select the encoding algorithm MPEG2 or H.264 (MPEG only works if you buy the key)
Encoding details H.264 profile, level, etc
But what only somebody with access to the non public sections of the Broadcom data sheet can do is route the video from the CSI-2 connector, through the VIP to the encoder and select the encoder parameters.
Then the GPU can generate the correct ”container” file that specifies which encoder, profile, level, etc was used.
All the code to do the above is present, you can specify resolution, framerate, effects etc. In fact most of the settings you would find on a compact camera. You can use OpenMAX components to tunnel the capture/video around to storage, encode, display etc.
What won't be there is the driver for the specific camera module used, plus tuning to actually make the output from the camera look half decent. That's the stuff that needs to be done for each new camera supported, and all that code is on the GPU. It also takes a long time to get right. Camera modules are always a PITA, the documentation is never sufficient, and the tuning takes ages, unless its a camera module that has an internal ISP, which is passed through and bypasses the Videocore ISP.