High definition video capture using PiCapture HD1

[mwlinder]

We have completed development of the high-definition version of our PiCapture solution, which supports capture of progressive video up to 1080p 60fps using the Raspberry Pi camera interface. It has both HDMI and Component video inputs - the component video processing is one approach to dealing with HDCP protected sources.

You can see examples of 1080p video at the Lintest Systems website: https://lintestsystems.com/hd1-examples

Like the SD1, there is very low latency and minimal load on the Raspberry Pi processor because it is processed in real time by the GPU just like the camera.

The PiCapture HD1 supports 480p, 720p, and 1080p video, and will be shipping in December.

[6by9]

supports capture of progressive video up to 1080p 60fps using the Raspberry Pi camera interface

I'm not going to knock your achievement at all (it's really pretty impressive), but 1080P60?

Seeing as you appear to be reusing the OV5647 camera modes those are set up to only allow up to 1080P30. Now you can feed data in faster and the SoC will try to handle it - I can't think of anything in the imaging pipe that looks at the image timestamps/frame rate. But while the ISP may be able to process that amount of data, the codec block can't, so I'm not sure what you're expecting to do with incoming data at that rate. Doing anything useful with that amount of data on the ARM is unlikely to fly either.
Any client software will also have to be careful to select the "sensor" mode that achieves the desired resolution, regardless of frame rate.

[mwlinder]

I'm not going to knock your achievement at all (it's really pretty impressive), but 1080P60?

Seeing as you appear to be reusing the OV5647 camera modes those are set up to only allow up to 1080P30. Now you can feed data in faster and the SoC will try to handle it - I can't think of anything in the imaging pipe that looks at the image timestamps/frame rate. But while the ISP may be able to process that amount of data, the codec block can't, so I'm not sure what you're expecting to do with incoming data at that rate. Doing anything useful with that amount of data on the ARM is unlikely to fly either.
Any client software will also have to be careful to select the "sensor" mode that achieves the desired resolution, regardless of frame rate.

I understand what you are saying about the high data rate, but the bottom line is that it works just fine! The GPU doesn't seem to have any issue with 1080p 60fps, encoding it just fine.

You are absolutely right that the "sensor mode" needs to match the incoming data. We have been using Mode 6 for 480p, Mode 5 for 720p, and Mode 1 for 1080p. If there is a mismatch between what the PiCapture is sending and what the Raspberry Pi is expecting you will either get buffer garbage for part of the picture or sometimes an MMAL error. Since Mode 5 is a few pixels larger than 720p in each dimension we add the extra to the right and bottom of the frame (which can be removed by selecting the ROI)

The one problem with 60fps data is that encoded H.264 video will play back in slow motion because the computer thinks it is at 30fps. There are two solutions to this that we are using: MP4Box will put an MP4 "wrapper" on the video that specifies the correct 60fps frame rate (this is what was done with the linked examples), or we can drop half the frames and record at 30fps.

This entire project has been a bit of an adventure because so much of what happens in the SOC GPU is a "black box", but the GPU is very capable and the implementation is tolerant of the differences between what it expects from the camera module and what PiCapture is sending. The ARM processor is able to deal with it just fine as well, with very low latency, because the GPU does the heavy lifting.

[mikronauts]

Will you be adding support for 1080i?

[6by9]

I understand what you are saying about the high data rate, but the bottom line is that it works just fine! The GPU doesn't seem to have any issue with 1080p 60fps, encoding it just fine.

I suspect the GPU is dropping frames and you're just not noticing it.
Assuming you're using raspivid or similar
(a) you'll need to select H264 level 4.2 to allow 1080P60 ("-lev 4.2"). The default level 4.0 only allows up to 245760 macroblocks/s, or 1920x1080@30.1fps.
(b) use the "-pts <filename>" option to save the timestamps of the frames. Those are usec accurate timestamps from when the CSI2 receiver generates the frame start interrupt. Those should all be nice and uniform, but I suspect that they aren't.

It may take a little while for the system to get far enough behind to actually drop buffers. Memory says that by default there are 4 image buffers allocated on the output side of the ISP (input side has 3). The codec needs to hold on to one for use as the reference frame, and get the new one in. It has an input FIFO that will happily swallow as many buffers as are going. Once all the buffers are inside the codec or in the FIFO, the ISP side will eventually get to a stage where it has no output buffer to process an input buffer into before a new input buffer comes along. One of those two input buffers will be dropped, and you'll see a glitch in the time stamps.
The regression tests that we used to run at Broadcom ran the camera, ISP, and codec at 1080P33 as that was judged to be within expectations of working reliably. There wasn't a huge amount of headroom above that, mainly due to the codec block.

You are absolutely right that the "sensor mode" needs to match the incoming data. We have been using Mode 6 for 480p, Mode 5 for 720p, and Mode 1 for 1080p. If there is a mismatch between what the PiCapture is sending and what the Raspberry Pi is expecting you will either get buffer garbage for part of the picture or sometimes an MMAL error. Since Mode 5 is a few pixels larger than 720p in each dimension we add the extra to the right and bottom of the frame (which can be removed by selecting the ROI)

The one problem with 60fps data is that encoded H.264 video will play back in slow motion because the computer thinks it is at 30fps. There are two solutions to this that we are using: MP4Box will put an MP4 "wrapper" on the video that specifies the correct 60fps frame rate (this is what was done with the linked examples), or we can drop half the frames and record at 30fps.

The -pts option is really what you want again.

Code: Select all

raspivid -pts timecodes.txt -o video.h264
mkvmerge -o video.mkv --timecodes 0:timecodes.txt video.h264

should produce a correct file based on the timestamps gathered from the SoC.

You will probably find that the bitrate control has gone a bit crazy in the codec - you told it to expect 30 frames per second and have given it ~60, so all the predictions it has made for how many bits to allocate to a frame will be out by a factor of 2.

Purely from memory, I'd hope that the following command should capture a stream accurately described as 1080P60 from your board:

Code: Select all

raspivid -o stream.h264 -pts timestamps.txt -md 1 -w 1920 -h 1080 -fps 60 -b 20000000 -t 30000

I'm guessing you need to add at least "-awb=off -awbg 1.0,1.0" to stop it messing with the colour balance. I'd be interested to see the raw output files from that.

This entire project has been a bit of an adventure because so much of what happens in the SOC GPU is a "black box", but the GPU is very capable and the implementation is tolerant of the differences between what it expects from the camera module and what PiCapture is sending. The ARM processor is able to deal with it just fine as well, with very low latency, because the GPU does the heavy lifting.

I'm working on getting the CSI2 receiver exposed via V4L2 to make it slightly less of a black box, but time is a scarce commodity at the moment Things will be changing next year.
It's already done via MMAL, but not in a state that can be supported long term. The ISP is now also exposed as a generic resize/format conversion for YUV and RGB input formats to YUV and RGB output formats. That opens up a fair few possibilities for these capture devices, but needs the software effort to support the various input devices. I'm hoping to shift that effort on to the existing kernel drivers for things such as the TC358743, ADV7282, etc (basically anything with a CSI2 output).

The ARM will cope with a suitable bitrate H264 stream. It will struggle to do anything useful if given 60 YUV420 images at 1920x1080 every second. That's why I'm querying what can be achieved when given a 1080P60 source.

[mwlinder]

Will you be adding support for 1080i?

The PiCapture SD1 has RAM for deinterlacing NTSC and PAL sources, but we don't have that on the HD1 - it is designed for progressive video only.

We have implemented a mode on the HD1 that works with 1080i by presenting each half-frame to the GPU at 960x540 resolution.
This takes the 60 "fields per second" of 1080i, which corresponds to 30 "frames per second" of 1920x1080 data and interpolates it to 960x540 at 60 "frames per second". Because there isn't a corresponding camera mode for that resolution, we use camera mode 5 and set the region of interest (ROI) for the transmitted data. It looks OK, but costs resolution.

We are trying to decide whether to support this mode or not - do you think it is useful to capture 1080i at 960x540 resolution?

[mikronauts]

1080i capture is only useful as 1920x1080

You should be able to treat it as 1920x540 @ 60Hz with post-processing to change it to 1920x1080i

The reason it is useful is older gear that outputs component video often has 1080i hdmi output as well, and it would be a shame to lose half the horizontal resolution.

Will you be adding support for 1080i?

The PiCapture SD1 has RAM for deinterlacing NTSC and PAL sources, but we don't have that on the HD1 - it is designed for progressive video only.

We have implemented a mode on the HD1 that works with 1080i by presenting each half-frame to the GPU at 960x540 resolution.
This takes the 60 "fields per second" of 1080i, which corresponds to 30 "frames per second" of 1920x1080 data and interpolates it to 960x540 at 60 "frames per second". Because there isn't a corresponding camera mode for that resolution, we use camera mode 5 and set the region of interest (ROI) for the transmitted data. It looks OK, but costs resolution.

We are trying to decide whether to support this mode or not - do you think it is useful to capture 1080i at 960x540 resolution?

[6by9]

1080i capture is only useful as 1920x1080

You should be able to treat it as 1920x540 @ 60Hz with post-processing to change it to 1920x1080i

The reason it is useful is older gear that outputs component video often has 1080i hdmi output as well, and it would be a shame to lose half the horizontal resolution.

Deinterlace of 1080i is a pretty onerous task even on the GPU. It has to adopt an alternate algorithm and use the QPUs to achieve 30fps.
The tricky bit is actually signalling top/bottom field, which I don't think is possible from where the PiCapture is sitting.

That all pushes it back to being done within the HD1, and the answer there has already been given.

[mwlinder]

The -pts option is really what you want again.

Thanks for pointing me to this - it is a big help to understand what is happening in the encoder. I ran some experiments, and you are correct that at 1080p 60fps the SoC is dropping some of the frames. The captured video plays seemed faithful to the source and so I didn't recognize that we were running into this limitation. The timestamps are perfectly regular at 30fps.

I'm guessing you need to add at least "-awb=off -awbg 1.0,1.0" to stop it messing with the colour balance.

Yes - that is important, as well as turning off the exposure control "-ex off" - the camera driver doesn't like scene changes in the source video and will work to adjust the brightness if exposure control is on. Those two switches are the most important.

Thanks again for your help.

[schwami7488]

Could anyone tell me if the same sort of thing would be possible using the Auvidea HDMI to CSI bridge (https://auvidea.com/b101-hdmi-to-csi-2- ... 5-pin-fpc/)?

I have a fixed 1080p60 source that I'm trying to input into the RasPi 3. I'm not concerned with dropping frames. Right now it appears to be capturing something, but the image is extremely distorted.

Thanks!

[6by9]

Could anyone tell me if the same sort of thing would be possible using the Auvidea HDMI to CSI bridge (https://auvidea.com/b101-hdmi-to-csi-2- ... 5-pin-fpc/)?

I have a fixed 1080p60 source that I'm trying to input into the RasPi 3. I'm not concerned with dropping frames. Right now it appears to be capturing something, but the image is extremely distorted.

viewtopic.php?f=38&t=120702
The app in https://github.com/6by9/userland/blob/h ... tc358743.c (thanks to Orbital6 for a load of the updates) provides some better support. It won't do 1080P60 as the CSI bus as configured can't handle the data rate, but 1080P30 should be OK.

Using raspivid etc with the B101 is NOT a supported block of code. It will only really take 720P, and that isn't going to change.
The PiCapture is doing some significant signal processing to convert the incoming data into a form that the GPU believes to be the camera module. All support requests for that need to go to the manufacturer too.