raspivid raw RGB data is incorrect?

[jwatte]

It seems like I'm getting wrong data buffers when using the splitter to convert the RAW output data to RGB.
I'm setting the size to 320x240.
Raspivid should be writing 230400 bytes per frame, but it actually just writes 115200 bytes per frame.
When I take the first 320x240x3 bytes out of the stream, and convert to a PNG, I get an image that looks like this:



You can see that there's actually two images in there. Also, each image seems to be semi-planar, in that there are three scanlines per actual row. Additionally, there's a bunch of empty space below those three images. It should just be one image, in RGB color filling the 320x240 frame.

I'm reporting this bug in raspivid, because I'm using the raspivid code as a base for a program I'm writing on my own, and I'm seeing the same bad data/format mismatch in my own code, too.

Something's very much not right here. What's going on, and how can I debug this?

[HermannSW]

You can compare against raw output from raspiraw command from 6by9. I played a lot with that in this recent thread, and get 640x480=307200 raw bytes per frame, which is exactly 300KB.

So your raw data for 320x240 should be exactly 75KB, and in RGB 225KB or 320*240*3=230400 bytes.

raspiraw output can be converted to viewable .ppm image by 6by9's hacked dcraw, maybe raspivid raw output can be converted by "normal" dcraw, otherwise build 6by9's dcraw, instructions here:
viewtopic.php?f=43&t=188401#p1189442

You can rewrite my program "rawvga" from end of recent thread and do the conversion yourself, just do the necessary changes 640=>320, 480=>240.

Hermann.

[ethanol100]

All correct in raspivid, but not what you want. Raw in raspivid means raw yuv420 images. One Y plan(x*y), one (1/4*x*y) U plane and (1/4*x*y) V plane. (Or rgb= x*y r,g,b tupple if you use -rf rgb, or x*y gray values if you use -rf gray). It has nothing to do with raw bayer pattern. You can't get bayer raw data fron raspivid.

[6by9]

Ethanol100 is correct - raspivid doesn't produce bayer.
raspiraw is only really useful for those wanting to do really weird things at the lowest level.

However, from the raspivid help text (though one does sometimes wonder whether it is worth bothering to write it seeing as it never gets read)

--raw-format/ -rf: Specify output format for raw video. Default is yuv

Other values include "gray" and "rgb".
yuv (4:2:0 to be precise) is a planar format, and with the 2x2 chroma subsampling it does result in effectively 12 bits per pixel.
320x240x12(bits/pixel)/8(bits per byte) = 115200 bytes as produced by raspivid. No bug.

Add "-rf rgb" to your raspivid command and it should produce 320x240x24/8 = 230400 bytes per frame.

[jwatte]

All correct in raspivid, but not what you want. Raw in raspivid means raw yuv420 images.

It's clear that the data in this picture is not I420, because the three "little" images are the same size; I420 has one plane of Y at full resolution, then two planes of U and V at half resolution (quarter as many pixels.) In fact, I420 at the resolution I use uses 115200 bytes per picture, which may be one reason why the splitter gets the buffer size wrong.

I am also not interested in Bayer pattern data, because I'm scaling down by a large factor, and would rather just let the hardware deal with that.

I am using -raw rgb. Here is the entire command line:

Code: Select all

raspivid -w 320 -h 240 -t 30000 -td 1000,1000 --raw raw -rf rgb -o foo

I then generate the PNG by simply reading the first 230400 bytes into RAM and calling stbi_write_png("output.png", 320, 240, 3, data, 0) from the fabulous "stb" single-header library. This function works fine for other images, and the image it shows on disk is the same as the image I draw from RAM, so I trust that this is the correct representation.

When using the yuv format, I get data exactly in the format I would expect for the mode, and I can manually convert that to RGB. That is my work-around for now. However, that uses more CPU.

Why is the "-raw RGB" output data from raspivid, and thus from the video splitter, wrong? Can it be fixed?

What I really need is:
- camera capture at 320x240 or another similar small resolution (640x480 is probably pushing it) at 60 Hz
- hardware binning
- hardware encoding to H.264 using opaque buffers (no round-trip to CPU memory)
- separate stream of RGB data in the same resolution at the same frame rate going to CPU memory

I'm currently using the same code as raspivid (and, in fact, using un-modified raspivid for verifying this bug.)
Could some other mechanism be used? For example, can the "still" port be pressed into 60 Hz service with RGB format, while the video capture port is still I420?

Perhaps a thing I could do would be to add a "resizer" on top of the "splitter" (or on top of the "preview"?) that converts to RGB, and keep the splitter "tap" port in I420 mode?

[jamesh]

Quick glance at code, seems OK. Do you get the correct setup information for the raw capture format when you use --verbose?

[6by9]

If you only give half the story we have to guess. Generally the guess will be the most likely error/omission. In this case it sounds like you have asked for the correct thing.

The only thing I can think of is an oversight that bgr888 is supported by video splitter, but rgb888 is not (or vice versa). Raspivid switches between them based on whether a fix was done in the firmware to the camera component because it was originally implemented the wrong way round. James may be able to do a quick check on that tomorrow.
What version of firmware are you using? ("vcgencmd version")

[6by9]

Oh, and don't try to use the stills port as it won't work like that, and resizer will compromise performance. If there is a bug then it will get fixed, either in the firmware or raspivid

[jamesh]

If you only give half the story we have to guess. Generally the guess will be the most likely error/omission. In this case it sounds like you have asked for the correct thing.

The only thing I can think of is an oversight that bgr888 is supported by video splitter, but rgb888 is not (or vice versa). Raspivid switches between them based on whether a fix was done in the firmware to the camera component because it was originally implemented the wrong way round. James may be able to do a quick check on that tomorrow.
What version of firmware are you using? ("vcgencmd version")

Not going to have time this week, and then on hols! Does sound like an issue somewhere in there though.

[cpunk]

Interesting... I will try to recreate the effect on my system, since at some point, I would also be interested in shifting the work of producing RGB to hardware instead of Gstreamer (which I'm using currently).

I'll mention if I see something that draws attention.

[cpunk]

I have recreated it - definitely not the kind of RGB8 that Gstreamer outputs. Much more likely YUV or some similar format.

I will try to search for it tomorrow, although I am very new to "raspivid" source code, having modified it only once before (so please don't count on me to solve it, I am more likely to just provide more info).

The command line I used:
raspivid -f -n -t 0 -w $1 -h $2 -fps 5 -fli 50hz -ex off -o /dev/null -r - -rf rgb -x !

..where $1 was 320 and $2 was 320 and my application read the standard output of raspivid. Results were visualized by sending each scanline (tagged with scanline number) over UDP to a PC where they got drawn using a method that was already well tested by drawing RGB frames from GStreamer.

Result:

[jwatte]

Thanks for the answers! Regarding "posting the whole story," there's the danger of "I was born right after the Apollo landing ..." so I prefer to post the data I actually have that is actually verified, and if someone needs something else, then I'm happy to fill in details.

I actually don't think it's YUV, for two reasons:
1) As I said above, YUV has quarter-size planes for U and V; the three images have the same size here.
2) The image is actually straight grayscale, even though it's three bytes per pixel. This means each of the three bytes have the same value, else there would be rainbow/shimmering across edges in the image.

version

Code: Select all

pi@robot-raspberrypi:~ $ vcgencmd version
Jul  3 2017 14:20:31 
Copyright (c) 2012 Broadcom
version 4139c62f14cafdb7d918a3eaa0dbd68cf434e0d8 (tainted) (release)

[cpunk]

I get the same output if I tell Gstreamer to decode H264 into I420. So I think it must be I420 ("8 bit Y plane followed by 8 bit 2x2 subsampled U and V planes", according to: https://www.fourcc.org/pixel-format/yuv-i420/ ).

My versions over here are:

Code: Select all

# vcgencmd version
Feb 20 2017 14:01:24 
Copyright (c) 2012 Broadcom
version 272c4de2358f2469414f38a1dea7179b5f09656c (clean) (release)

When I use the -v argument, it prints (among other data):

"Raw output enabled, format rgb"

[cpunk]

I am confident that "format->encoding" is set by "create_splitter_component(RASPIVID_STATE *state)". I think this is not a command line parsing problem.

[cpunk]

I can additionally confirm that even if I override the choice between RGB and BGR, and tell it to always output BGR, it still gives me YUV.

The nice thing is that I can decode the YUV like this:

Code: Select all

public class YuvToRGB {
	private static int R = 0;
	private static int G = 1;
	private static int B = 2;
	public static byte[] I420ToRGB(byte[] src, int width, int height){
		int numOfPixel = width * height;
		int positionOfV = numOfPixel;
		int positionOfU = numOfPixel/4 + numOfPixel;
		byte[] rgb = new byte[numOfPixel*3];

		for(int i=0; i<height; i++){
			int startY = i*width;
			int step = (i/2)*(width/2);
			int startV = positionOfV + step;
			int startU = positionOfU + step;
			for(int j = 0; j < width; j++){
				int Y = startY + j;
				int V = startV + j/2;
				int U = startU + j/2;
				int index = Y*3;		
				rgb[index+B] = (byte)((src[Y]&0xff) + 1.4075 * ((src[V]&0xff)-128));
				rgb[index+G] = (byte)((src[Y]&0xff) - 0.3455 * ((src[U]&0xff)-128) - 0.7169*((src[V]&0xff)-128));
				rgb[index+R] = (byte)((src[Y]&0xff) + 1.779 * ((src[U]&0xff)-128));
			}
		}
		
		return rgb;
	}
}

...with only minor oddities and defects on the left side of the frame (no idea, and no time to figure it out today)... and since that occurs on the desktop, it doesn't noticeably hurt performance. :) But yes, on a Pi, I would definitely not like to decode YUV in Java. RGB would be nice to have, or if that has become impossible, then I'd GRAY8 (equivalent to the luma or Y channel from YUV) would be OK too.

I also have the suspicion that I'm currently feeding my decoder function oversized buffers, fortunately it doesn't seem to break from that.

[cpunk]

Also, GRAY8 (-rf gray) is working as intended. :) However, since the source code suggests that GRAY8 is produced by actually running in I420 mode and sending the leading part of the buffer instead of the whole buffer, I guess I should not be surprised. :)

[AsgeirOM]

So if anyone is interested in some additional information or wants to test this in the console, here are some quick steps:

Code: Select all

mkfifo test

1. Setups pipes so you can run the commands in separate consoles.

Code: Select all

ffmpeg -f rawvideo -r 3 -s 1920x1088 -pix_fmt yuv420p -vcodec rawvideo -i test -y test_rgb.mp4

2. Notice how I specify a larger resolution, as well as the yuv420-planar input.

Code: Select all

raspivid -w 1920 -h 1080 -n -fps 3 -o /dev/null -vf -r test -rf rgb -v

3. This generates correct video on the versions pasted below.

Code: Select all

raspivid Camera App v1.3.12

ffmpeg version n3.2.2-8-g64bb329

Jul  3 2017 14:20:31
Copyright (c) 2012 Broadcom
version 4139c62f14cafdb7d918a3eaa0dbd68cf434e0d8

[cpunk]

Thanks for trying... it is strange to hear that somewhere, it is outputting RGB.

So I tried recreating that (but could not). First I reinstalled the following packages:

libraspberrypi-bin
libraspberrypi0
raspi-config

Output was:

Code: Select all

(Reading database ... 70121 files and directories currently installed.)
Preparing to unpack .../libraspberrypi-bin_1.20170703-1_armhf.deb ...
Unpacking libraspberrypi-bin (1.20170703-1) over (1.20170703-1) ...
Preparing to unpack .../libraspberrypi0_1.20170703-1_armhf.deb ...
Unpacking libraspberrypi0 (1.20170703-1) over (1.20170703-1) ...
Preparing to unpack .../raspi-config_20170705_all.deb ...
Unpacking raspi-config (20170705) over (20170705) ...
Processing triggers for systemd (215-17+deb8u7) ...
Setting up libraspberrypi0 (1.20170703-1) ...
Setting up libraspberrypi-bin (1.20170703-1) ...
Setting up raspi-config (20170705) ...
Processing triggers for libc-bin (2.19-18+deb8u10) ...
Press Return to continue.

Then I switched my Pi off and replaced the 2.1 camera with a no-name 1.3 fish-eye camera. Modified my testing code to not require motion vectors, rebooted and retried with the command line:

raspivid -t 0 -w 320 -h 320 -fps 3 -o /dev/null -r - -rf rgb

...and I still get YUV-looking output instead of RGB like this:



Then for good measure, I also did:

git clone https://github.com/raspberrypi/userland.git
cd userland
./buildme
reboot

....and tried again, and nope, it still produces YUV. I am not successful in recreating the situation of getting good RGB. But I can't try with the resolution you mentioned, as it exceeds the size of the largest screen I have available. I only tried at 320 x 320 as my drone uses that.

[cpunk]

P.S. I also have:

Code: Select all

raspivid Camera App v1.3.12

[jwatte]

FWIW, I'm now agreeing that the "bad data" we're seeing is actually YUV planar interpreted as RGB. Each scanline is repeated 3x horizontally, because a 1-byte-per-pixel pixel is split into 3-bytes-per-pixel. So you convinced me
What I think is freaky is that there's so little change between R/G/B pixels that there are no "rainbow shimmers" along edges -- where R, G and B should be sufficiently different to cause blue or red tints. But that's a different question.
For what it's worth, I'm making do with CPU-side conversion for now; it takes about 2 milliseconds per frame for 320x240.

[cpunk]

P.S. Maybe this problem of getting YUV instead of RGB is specific to low resolutions?

[6by9]

I'll be looking at this in the morning - I've been on holiday all week.
I will agree that the video splitter is doing something wrong - it's logging asserts on the Gpu due to something going wrong in the format selection.

[6by9]

Issue found, now just a matter of fixing it.
Calling mmal_port_format_commit on the input port of video_splitter copies all settings through to the output port, including the colour format.

Raspivid is trying to set everything up in video_splitter first, and then uses a mmal_connection to connect it to the camera component. Creating the tunnel does a mmal_port_format_commit in order to make sure the sink is configured in the same way as the source (raspivid doesn't need to do this manually).

I think I have a two line firmware change that means that if the colour format has been set on an output port when the input port is updated then the colour format is retained but resolution and buffer size is updated.
The alternative is to modify raspivid to set up the components and connections in order, setting the output formats once the input(s) are connected. The connections don't need to be enabled until you're ready to run. That may be a better option jwatte as he's writing his own code.

[cpunk]

Wow, nice. :) If you need testers at any point, just drop a line here - I would definitely be interested in trying (although I don't think I have ever compiled firmware for anything GPU-related - only to put the WiFi card in packet injection mode).

[jwatte]

That sounds good!

Another option: I don't need the on-screen preview, but I'm running it so that the auto-level-magic can do its work.
Can I connect a memory buffer pool to the preview output?
If so, can that output have RGB format when the main video encoder uses YUV? Or can the encoder use YUV?
If so, I don't need the splitter at all!