MRV
Posts: 29
Joined: Thu Jul 19, 2018 3:09 am

Max resolution for stereo?

Thu Jan 16, 2020 1:34 pm

Hello.

We are using a Compute Module 3+ attached to a custom carrier board with two cameras - this allows for the acquisition of stereo imagery using Raspivid. Currently we are restricted to 1920 x 1080 encoding. Is there a way, either on PI3+ or PI4, to increase this size to at least 2048 x 1024?

Thanks.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7888
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Max resolution for stereo?

Thu Jan 16, 2020 3:07 pm

No. The codec hardware has an internal line buffer that is of a fixed size.
I had expected it to be 2048 pixels wide to accomodate 2,048×1,[email protected] that is specified as part of level 4.0, but something goes wrong in the config of the last macroblock column, and (memory says) 2032 is the highest width that works.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

MRV
Posts: 29
Joined: Thu Jul 19, 2018 3:09 am

Re: Max resolution for stereo?

Thu Jan 16, 2020 3:24 pm

Is 2032 x 1016 possible?
6by9 wrote:
Thu Jan 16, 2020 3:07 pm
No. The codec hardware has an internal line buffer that is of a fixed size.
I had expected it to be 2048 pixels wide to accomodate 2,048×1,[email protected] that is specified as part of level 4.0, but something goes wrong in the config of the last macroblock column, and (memory says) 2032 is the highest width that works.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7888
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Max resolution for stereo?

Thu Jan 16, 2020 3:45 pm

MRV wrote:
Thu Jan 16, 2020 3:24 pm
Is 2032 x 1016 possible?
It should be for simple encode, but won't be for stereoscopic as it doesn't meet the requirement for the width of each image being a multiple of 128.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

MRV
Posts: 29
Joined: Thu Jul 19, 2018 3:09 am

Re: Max resolution for stereo?

Thu Jan 16, 2020 4:51 pm

Is there an active bug ticket regarding this? (config of the macroblock column)
Would like to see the PI actually being able to get what the level 4 spec says it should. 2048x1024 would be excellent.

[
quote=6by9 post_id=1596450 time=1579189555 user_id=92334]
MRV wrote:
Thu Jan 16, 2020 3:24 pm
Is 2032 x 1016 possible?
It should be for simple encode, but won't be for stereoscopic as it doesn't meet the requirement for the width of each image being a multiple of 128.
[/quote]

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7888
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Max resolution for stereo?

Thu Jan 16, 2020 5:09 pm

MRV wrote:
Thu Jan 16, 2020 4:51 pm
Is there an active bug ticket regarding this? (config of the macroblock column)
Would like to see the PI actually being able to get what the level 4 spec says it should. 2048x1024 would be excellent.
It's so far down the priorities list I don't think it is being actively tracked - sorry.
You could check for or create an issue at https://github.com/raspberrypi/firmware/issues, but it's not going to boost the priority.

We no longer have developers who know the intimate detail of the codec code, and it may well be the hardware itself that isn't capable of doing it. From extracting the motion vectors we know that we get an extra column of vectors, so it may be the setup there that either requires it or is wrong, but you're probably talking a couple of man days of time to determine if that is the case, and then you need to understand whether that is necessity or accident.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

MRV
Posts: 29
Joined: Thu Jul 19, 2018 3:09 am

Re: Max resolution for stereo?

Thu Jan 16, 2020 5:17 pm

And the PI3+ and PI 4 both have this issue (limitation)?

6by9 wrote:
Thu Jan 16, 2020 5:09 pm
MRV wrote:
Thu Jan 16, 2020 4:51 pm
Is there an active bug ticket regarding this? (config of the macroblock column)
Would like to see the PI actually being able to get what the level 4 spec says it should. 2048x1024 would be excellent.
It's so far down the priorities list I don't think it is being actively tracked - sorry.
You could check for or create an issue at https://github.com/raspberrypi/firmware/issues, but it's not going to boost the priority.

We no longer have developers who know the intimate detail of the codec code, and it may well be the hardware itself that isn't capable of doing it. From extracting the motion vectors we know that we get an extra column of vectors, so it may be the setup there that either requires it or is wrong, but you're probably talking a couple of man days of time to determine if that is the case, and then you need to understand whether that is necessity or accident.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7888
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Max resolution for stereo?

Thu Jan 16, 2020 5:50 pm

MRV wrote:
Thu Jan 16, 2020 5:17 pm
And the PI3+ and PI 4 both have this issue (limitation)?
All versions of the Pi share the same codec block.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

MRV
Posts: 29
Joined: Thu Jul 19, 2018 3:09 am

Re: Max resolution for stereo?

Sat Jan 18, 2020 6:05 pm

Is there a reason that the RAM, CPU frequency and kernels have improved yet there has been no improvement regarding the video encoder (GPU)?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7888
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Max resolution for stereo?

Sat Jan 18, 2020 10:16 pm

MRV wrote:
Sat Jan 18, 2020 6:05 pm
Is there a reason that the RAM, CPU frequency and kernels have improved yet there has been no improvement regarding the video encoder (GPU)?
Mainly expertise and ownership.

Cranking up ARM cores is relatively simple as you buy in the relevant IP from ARM. Interfacing to alternate RAM providers depends on finding a suitable package and wiring it up.

The H264/MPEG4/H263/MPEG2/VC-1 codec block is a Broadcom IP block, so support really needs to come from them. They no longer sell any other products using this block, so they have no interest in investing effort into updating it. Most of their multimedia devices are for set top boxes, so decode is their requirement, not encode.
AFAIK ARM don't sell a video encode IP block, so you can't easily buy one off the shelf and plug it to your chip. (Licencing and avoiding cross-contamination would get entertaining anyway with Broadcom making the chip and designing their own codec blocks)

CPU performance and software is catching up to the extent that the need for hardware acceleration is significantly reduced. There are parts that can be usefully hardware accelerated (eg CABAC, and motion searching), but the rest can be run on the CPU. Someone observed recently that video decode through the hardware blocks achieved around 60fps, when software decode could hit about 110fps.

We have invested in video decode for HEVC on Pi4. The block does have some other options, but I don't recall the details.

The ISP isn't rated significantly higher than the video_encode block anyway - somewhere around 150MPix/s in reality for Bayer images, so 75fps at 1080p. ARM definitely don't do a decent ISP, therefore without a new design from scratch for that you'd be struggling to create the content to encode.

It's all an area that is being looked at and the use cases considered.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

dickon
Posts: 702
Joined: Sun Dec 09, 2012 3:54 pm
Location: Home, just outside Reading

Re: Max resolution for stereo?

Sun Jan 19, 2020 3:24 pm

6by9 wrote:
Sat Jan 18, 2020 10:16 pm
AFAIK ARM don't sell a video encode IP block, so you can't easily buy one off the shelf and plug it to your chip.
I believe they do as part of their Mali video blocks, and AIUI they're available as IP cores for integration. 10b 8Kp30 HEVC / H.264 / VP[89] at the top end, apparently. 8K seems ambitious for mobile phones...

asavah
Posts: 367
Joined: Thu Aug 14, 2014 12:49 am

Re: Max resolution for stereo?

Sun Jan 19, 2020 6:45 pm

dickon wrote:
Sun Jan 19, 2020 3:24 pm
6by9 wrote:
Sat Jan 18, 2020 10:16 pm
AFAIK ARM don't sell a video encode IP block, so you can't easily buy one off the shelf and plug it to your chip.
I believe they do as part of their Mali video blocks, and AIUI they're available as IP cores for integration. 10b 8Kp30 HEVC / H.264 / VP[89] at the top end, apparently. 8K seems ambitious for mobile phones...
I have no idea how many SoC models in the wild actually use Mali-Vxx VPU, I couldn't find any useful references.
And knowing how ARM does its business I doubt its driver/software is opensource and I'm pretty sure the licensing costs are very pricey.
All the major Arm Soc vendors I know of have their own video codec IPs and implementations, there is no standard.
However in mainline linux kernel some reverse engineered drivers are starting to get implementations under v4l2 hood.
Amlogic has its own (amcodec), Rockchip has its own (mpp), Allwinner has its own (cedar), Qualcomm Snapdragon has its own (don't know the codename) , edit: MTK has its own, they all seem to develop their own codecs or license 3rd party stuff but not Mali VPU.
ARM world is a zoo ...

>8K seems ambitious for mobile phones...
The target market for these would be android tv boxes.

dickon
Posts: 702
Joined: Sun Dec 09, 2012 3:54 pm
Location: Home, just outside Reading

Re: Max resolution for stereo?

Sun Jan 19, 2020 7:07 pm

The point still stands: if you're designing a SoC -- which is what 6by9 was talking about -- the IP blocks are available. I agree that nobody seems to be using it -- likely for the reasons you cite -- but the cores seem to be available if you want to design your own.

8k *en*code on an STB?

Return to “Advanced users”