Raspiraw problems with NoIR V2

[6by9]

Code: Select all

pi@raspberrypi:~/raspiraw $ ./raspiraw --help
-?, --help	: This help information
-md, --mode	: Set sensor mode <mode>
-hf, --hflip	: Set horizontal flip
-vf, --vflip	: Set vertical flip
-e, --ss	: Set the sensor exposure time (not calibrated units)
-g, --gain	: Set the sensor gain code (not calibrated units)
-o, --output	: Set the output filename
-hd, --header	: Write the BRCM header to the output file <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<
-t, --timeout	: Time (in ms) before shutting down (if not specified, set to 5s)
-sr, --saverate	: Save every Nth frame
-b, --bitdepth	: Set output raw bit depth (8, 10, 12 or 16, if not specified, set to sensor native)
-c, --cameranum	: Set camera number to use (0=CAM0, 1=CAM1).
-eus, --expus	: Set the sensor exposure time in micro seconds.
-y, --i2c	: Set the I2C bus to use.
-awbg, --awbgains	: Set the AWB gains to use.
-r, --regs	: Change (current mode) regs
-hi, --hinc	: Set horizontal odd/even inc reg
-vi, --vinc	: Set vertical odd/even inc reg
-f, --fps	: Set framerate regs
-w, --width	: Set current mode width
-h, --height	: Set current mode height
-lt, --left	: Set current mode left
-tp, --top	: Set current mode top
-hd0, --header0	: Sets filename to write the BRCM header to
-hdg, --headerg	: Sets filename to write the .pgm header to
-ts, --tstamps	: Sets filename to write timestamps to
-emp, --empty	: Write empty output files
-m, --metadata	: Decode register metadata
-awb, --awb	: Use a simple grey-world AWB algorithm
-n, --nopreview	: Do not send the stream to the display
-P, --processing	: Pass images into an image processing function
-p, --preview	: Preview window settings <'x,y,w,h'>
-fs, --fullscreen	: Fullscreen preview mode
-op, --opacity	: Preview window opacity (0-255)
-PY, --processing_yuv	: Pass processed YUV images into an image processing function
-oY, --output_yuv	: Set the output filename for YUV data

So:
"raspiraw -o foo%u.raw" writes just the raw data
"raspiraw -hd -o foo%u.raw" writes the header and raw data
"raspiraw -hd0 foo.hdr -o foo%u.raw" writes the header to foo.hdr, and just the raw data to the remaining files.

[RDX705]

Thanks. Is there any way to use .raw data in openCV?

[6by9]

Thanks. Is there any way to use .raw data in openCV?

What are you wanting to do with it?

Whilst it does appear to have some support for Bayer images, the first thing it is likely to do is a VERY expensive conversion to RGB (demosaicing).
If you tell raspiraw to unpack to 16bpp, then you could load it as greyscale, but the colour separation is going to be odd.

[RDX705]

I want to use raspiraw output for feature extraction. Currently extraction algorithm uses dcraw to convert raw frame to ppm, which is then processed by opencv, but dcraw is a bit slow, especially with higher resolutions, so I wondered if its possible to use raw directly. Tried to create Mat with data extracted from .raw file, but got chaotic mess of pixels.

I tried outputting .pgm, but got colored images with this code (from one of your threads):

Code: Select all

static void callback(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buffer)
{
	static int count = 0;
	vcos_log_error("Buffer %p returned, filled %d, timestamp %llu, flags %04X", buffer, buffer->length, buffer->pts, buffer->flags);
	if (running)
	{
		RASPIRAW_PARAMS_T *cfg = (RASPIRAW_PARAMS_T *)port->userdata;

		if (!(buffer->flags&MMAL_BUFFER_HEADER_FLAG_CODECSIDEINFO) &&
                    (((count++)%cfg->saverate)==0) && (count > 1))
		{
			// Save every Nth frame
			// SD card access is too slow to do much more.
			FILE *file;
			char *filename = NULL;
			if (create_filenames(&filename, cfg->output, count) == MMAL_SUCCESS)
			{
				
				file = fopen(filename, "wb");
				if (file)
				{
					if (cfg->ptso)  // make sure previous malloc() was successful
					{
						cfg->ptso->idx = count;
						cfg->ptso->pts = buffer->pts;
						cfg->ptso->nxt = malloc(sizeof(*cfg->ptso->nxt));
						cfg->ptso = cfg->ptso->nxt;
					}
					if (!cfg->write_empty)
					{
					
						if (cfg->write_header)
							fwrite(brcm_header, BRCM_RAW_HEADER_LENGTH, 1, file);
							
								/////
				int i,j,k,t;
                   unsigned char *p, *q;
 
                   p = q = buffer->data;
 
                   for(i=1; i<480; i+=2) 
                   {
                     p += 800;
  
                     for(j=0; j<640; j+=4, p+=5)        // convert raw10 to .pgm
                     {
                       k = (((int) p[0])<<2) + ((p[4]>>0)%0x03);
                       *q++ = (k>=50) ? 255 : 0;                    // b/w filter
 
                       k = (((int) p[2])<<2) + ((p[4]>>4)%0x03);
                       *q++ = (k>=50) ? 255 : 0;
                     }
                   }
				  
 
				//////
						fwrite(buffer->data, buffer->length, 1, file);
					}
					fclose(file);
				}
				
				free(filename);
			}
		}
		buffer->length = 0;
		mmal_port_send_buffer(port, buffer);
	}
	else
		mmal_buffer_header_release(buffer);
}

Am I doing something wrong here?

[HermannSW]

You need to make sure that this does not get written to pgm file (eg. by "-hd0 filename"):

Code: Select all

fwrite(brcm_header, BRCM_RAW_HEADER_LENGTH, 1, file);

You need to write the correct (converted) buffer length (640x480 raw10 gets converted to 320x240 grey8 (q - buffer->data is 320*240)):

Code: Select all

fwrite(buffer->data, 320*240, 1, file);

And you are missing writing pgm header, add this line before writing the grey8 (b&w) data:

Code: Select all

    fprintf(file,"P5\n320 240\n255\n");

[RDX705]

I did everything like this:

Code: Select all

static void callback(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buffer)
{
	static int count = 0;
	vcos_log_error("Buffer %p returned, filled %d, timestamp %llu, flags %04X", buffer, buffer->length, buffer->pts, buffer->flags);
	if (running)
	{
		RASPIRAW_PARAMS_T *cfg = (RASPIRAW_PARAMS_T *)port->userdata;

		if (!(buffer->flags&MMAL_BUFFER_HEADER_FLAG_CODECSIDEINFO) &&
                    (((count++)%cfg->saverate)==0) && (count > 1))
		{
			// Save every Nth frame
			// SD card access is too slow to do much more.
			FILE *file;
			char *filename = NULL;
			if (create_filenames(&filename, cfg->output, count) == MMAL_SUCCESS)
			{
				
				file = fopen(filename, "wb");
				if (file)
				{
					if (cfg->ptso)  // make sure previous malloc() was successful
					{
						cfg->ptso->idx = count;
						cfg->ptso->pts = buffer->pts;
						cfg->ptso->nxt = malloc(sizeof(*cfg->ptso->nxt));
						cfg->ptso = cfg->ptso->nxt;
					}
					if (!cfg->write_empty)
					{
				
							
								/////
				int i,j,k,t;
                   unsigned char *p, *q;
 
                   p = q = buffer->data;
 
                   for(i=1; i<480; i+=2) 
                   {
                     p += 800;
  
                     for(j=0; j<640; j+=4, p+=5)        // convert raw10 to .pgm
                     {
                       k = (((int) p[0])<<2) + ((p[4]>>0)%0x03);
                       *q++ = (k>=50) ? 255 : 0;                    // b/w filter
 
                       k = (((int) p[2])<<2) + ((p[4]>>4)%0x03);
                       *q++ = (k>=50) ? 255 : 0;
                     }
                   }
				  
 
				//////
						
						fprintf(file,"P5\n320 240\n255\n");
						fwrite(buffer->data, 320*240, 1, file);
					}
					fclose(file);
				}
				
				free(filename);
			}
		}
		buffer->length = 0;
		mmal_port_send_buffer(port, buffer);
	}
	else
		mmal_buffer_header_release(buffer);
}

However, it outputs blank white images now (even though each .pgm file contains correct header and some data). Is the code still incorrect or I have to use special software to view 320x240 .pgm?

[HermannSW]

However, it outputs blank white images now

That is caused by the b&w filter. Please replace

Code: Select all

q++ = (k>=50) ? 255 : 0;

with

Code: Select all

q++ = k;

Then you will get an unchanged 320x240 grey8 image (256 shades of grey for each pixel).
In reality you get a pure green pixel image, the G pixel of rg/Gb 2x2 squares.
But because those are by far the brightest pixels, it should give a good grey image approximation.


The left is 320x240 left top blue pixel image, the right 320x240 top right g pixel image, with colors instead of grey8:
viewtopic.php?t=189661#p1218763

[RDX705]

Great, thank you, it outputs greyscale now.

It works fine on less than 200 fps, then images begin to appear "torn apart" at lower section, with higher FPS "tear" goes higher as well, with part of next image visible below.

Is it possible to fix this or its caused by camera's sensor itself?

[HermannSW]

Hi,

It works fine on less than 200 fps, then images begin to appear
...
Is it possible to fix this or its caused by camera's sensor itself?

it is the camera sensor itself (202fps for 640x480 v2, 95fps for v1).
But you can get higher framerates when reducing vertical resolution, see v2 column in this diagram:



P.S:
Since you are interested in v2 camera, here are the v2 camera approximation formulas between framerate and height:

[RDX705]

Hello again. After testing the system with 320x200 .pgm output I got two more questions:

1) The camera works well with moderate lighting, but some bright objects (for example, sunlit white paper box) appear as a mess of black and white zones, making feature extraction impossible. Is it possible to fix this, or its inherent flaw of b/w conversion inside raspiraw?

2) I remember mentions that raw data can be converted to .ppm with addition of header. Is it possible to do it inside raspiraw in a similar way to to b/w conversion?

[HermannSW]

On 1) Did you do change to "q++ = k;" as I proposed? In that case you would have 256 shades of grey (grey8) and not b/w. With grey8, what you described should not happen.

On 2), "rawvga -c" outputs .ppm color image directly from the 640x480 captured raw data, as a 640x480 .ppm file.:
viewtopic.php?t=189661#p1192548
Not sure what you want to do with that kind of image, here a sample from that posting, red/gren/blue pixels are kept "as is" from the sensor. If you zoom in with eg. gimp, you can see the rg/Gb pixels in each 2x2 sub image:

[RDX705]

Yes, i did it like this:

Code: Select all

static void callback(MMAL_PORT_T *port, MMAL_BUFFER_HEADER_T *buffer)
{
	static int count = 0;
	vcos_log_error("Buffer %p returned, filled %d, timestamp %llu, flags %04X", buffer, buffer->length, buffer->pts, buffer->flags);
	if (running)
	{
		RASPIRAW_PARAMS_T *cfg = (RASPIRAW_PARAMS_T *)port->userdata;

		if (!(buffer->flags&MMAL_BUFFER_HEADER_FLAG_CODECSIDEINFO) &&
                    (((count++)%cfg->saverate)==0) && (count > 1))
		{
			// Save every Nth frame
			// SD card access is too slow to do much more.
			FILE *file;
			char *filename = NULL;
			if (create_filenames(&filename, cfg->output, count) == MMAL_SUCCESS)
			{
				
				file = fopen(filename, "wb");
				if (file)
				{
					if (cfg->ptso)  // make sure previous malloc() was successful
					{
						cfg->ptso->idx = count;
						cfg->ptso->pts = buffer->pts;
						cfg->ptso->nxt = malloc(sizeof(*cfg->ptso->nxt));
						cfg->ptso = cfg->ptso->nxt;
					}
					if (!cfg->write_empty)
					{
				
							
								/////
				int i,j,k,t;
                   unsigned char *p, *q;
 
                   p = q = buffer->data;
 
                   for(i=1; i<400; i+=2) 
                   {
                     p += 800;
  
                     for(j=0; j<640; j+=4, p+=5)        // convert raw10 to .pgm
                     {
                       k = (((int) p[0])<<2) + ((p[4]>>0)%0x03);
                       *q++ = k;
 
                       k = (((int) p[2])<<2) + ((p[4]>>4)%0x03);
                       *q++ = k;
                     }
                   }
				  
 
				//////
						
						fprintf(file,"P5\n320 200\n255\n");
						fwrite(buffer->data, 320*200, 1, file);
					}
					fclose(file);
				}
				
				free(filename);
			}
		}
		buffer->length = 0;
		mmal_port_send_buffer(port, buffer);
	}
	else
		mmal_buffer_header_release(buffer);
}

I write it in 320x200 to reduce vertical tear from 250 fps.

[HermannSW]

I write it in 320x200 to reduce vertical tear from 250 fps.

Ok, you do writre gre8 .pgm file.
Can you please show/attach a frame captured with that code as you described, and a smartphone photo of same scene for comparison?

[RDX705]

Sure, here's the .pgm picture. Smartphone picture is on imgur, as its "too large" for attachment: https://imgur.com/a/ArVKsi7
The white flagpole is glitched.

[HermannSW]

Thanks, the grey8 image looks nice besides the flagpole.

My code seems to have a bug for very bright pixels (int k holds a 10bit number, and *q stores least 8bit of that).
Please replace two times

Code: Select all

*q++ = k;

with either

Code: Select all

*q++ = (k>255 ? 255 : k);

or

Code: Select all

*q++ = (k >> 2);

and let us know which is better (perhaps attach both option grey8 .ppm here and upload corresponding smartphone image).
If the latter option turns out better, then replacing this

Code: Select all

                       k = (((int) p[0])<<2) + ((p[4]>>0)%0x03);
                       *q++ = k;
 
                       k = (((int) p[2])<<2) + ((p[4]>>4)%0x03);
                       *q++ = k;

with

Code: Select all

                       *q++ = p[0];
 
                       *q++ = p[2];

simplifies and speeds up a bit the code.


P.S:
I did not realize that bug because my interest was on robot being able to follow black line, and therefore anything brighter was converted to white by the b/w conversion I used:



P.P.S:
I will continue working on my robot platform when having received T101 caterpillar robot chassis from aliexpress. I will replace the 330rpm gear motors with same form factor 1500rpm gear motors I already have. With 65mm wheel diameter that robot will achieve 5.1m/s (18.36km/h) speed. I expect to see quite some rolling shutter effect for curves (ahead) with robot running along straight line with 5m/s. If that is true, I will go with global shutter camera. Since v2 camera does not allow for global shutter, and v1 camera global external shutter only works for dark scene, I will go with ov7251 Mipi monochrome global shutter camera in that case (allows for 640x480@135fps, more fps for smaller resolutions). Otherwise I likely will go with v1 camera to keep robot platform cheap in total (Pi0W on robot).

[RDX705]

That's interesting.

Code: Select all

*q++ = (k>255 ? 255 : k) 

makes bright areas in image even brighter. If lighting is moderate, its fine (image 1-1.pgm), if its too light, big parts of image become completely white (1-2.pgm).

Code: Select all

*q++ = (k >> 2)

makes image darker (2.pgm) if lighting is moderate or normal if lighting is bright (2-2.pgm)
But they both fix the b/w mess at least!

[RDX705]

Added missing file

Also, I tried using pipe for communication between raspiraw and my algorithm and it turned out to be extremely effective, both fast, reliable and simple to implement.

[HermannSW]

OK, so option two is preferable, for bright images definitely, and for normal lit images you might need to increase brightness. Perhaps multiplying each pixel with (255.0/maxGreyValueInFrame). Another option would be to make Raspberry v2 camera send raw8 Bayer data instead of raw10 and you would get better scaled brightness, @6by9 might know how.


P.S:
I have no excuse anymore, the new 2nd T101 caterpillar robot chassis arrived, I will start assembling after having posted v1 camera global external shutter tools and documentation later today:

[6by9]

The sensor can send 8bit Bayer. You'll need to amend the register set (try 0x3034), and amend the image_id configuration from 0x2B to 0x2A as per the CSI2 spec.
That may alter the pixel timings significantly as you're sending less data over the CSI bus. I doubt that allows you to increase the frame rate though as you'll be hitting the limit of the A/D converters, not the CSI bus.