jacklab3
Posts: 6
Joined: Wed Jun 09, 2021 2:53 am

Some questions about image buffers

Wed Jun 09, 2021 3:01 am

Hello, all.
I have read the Camera Hardware Documentation on https://picamera.readthedocs.io/en/latest/fov.html, I have some questions about the pipeline. To the best of my knowledge, there will be a flag at the start/end of an image buffer to indicate the start/end of one frame. I wonder what will happen if there is something wrong that can make the size of the image buffer smaller as expected. For example, 10 wrong lines are dropped in streaming frame lines, the image(3280*2464) will be shown in the wrong resolution(3280*2454) or still be the true resolution(3280*2464)? If the resolution is also 3280*2464, where did the 10 missing rows of data in the buffer come from?
Thanks a lot if anyone can give me some suggestions!
Image
Attachments
pipeline.png
pipeline.png (64.28 KiB) Viewed 529 times

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11271
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Some questions about image buffers

Wed Jun 09, 2021 1:32 pm

jacklab3 wrote:
Wed Jun 09, 2021 3:01 am
I have read the Camera Hardware Documentation on https://picamera.readthedocs.io/en/latest/fov.html, I have some questions about the pipeline. To the best of my knowledge, there will be a flag at the start/end of an image buffer to indicate the start/end of one frame.
Yes, there are frame start and frame short messages in the CSI2 protocol.
jacklab3 wrote:I wonder what will happen if there is something wrong that can make the size of the image buffer smaller as expected. For example, 10 wrong lines are dropped in streaming frame lines, the image(3280*2464) will be shown in the wrong resolution(3280*2454) or still be the true resolution(3280*2464)? If the resolution is also 3280*2464, where did the 10 missing rows of data in the buffer come from?
How do you believe these 10 lines get lost?
Interference on the CSI2 bus resulting in multiple bit errors would typically be the most common issue, but that is not an expected behaviour as it is essentially game over at that point anyway.

No the CSI2 receiver doesn't flag this as a short frame (it does flag it as corrupt), and the rest of the image pipeline will handle it as if it were complete.
You'll typically get significant image corruption due to the disruption of the Bayer pattern in these conditions too.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

jacklab3
Posts: 6
Joined: Wed Jun 09, 2021 2:53 am

Re: Some questions about image buffers

Thu Jun 10, 2021 2:26 am

Thank you very much, 6by9.
How do you believe these 10 lines get lost?
Interference on the CSI2 bus resulting in multiple bit errors would typically be the most common issue, but that is not an expected behaviour as it is essentially game over at that point anyway.
According to the MIPI CSI-2 protocol, a 1-bit error can be corrected and 2-bit errors can be detected. Is it possible for multiple bit errors to result in a line dropped? If possible, what will happen after dropping the error lines?
What do you mean in 'it is essentially game over at that point anyway'?
Will the capture stop if multiple bit errors happen?
No the CSI2 receiver doesn't flag this as a short frame (it does flag it as corrupt), and the rest of the image pipeline will handle it as if it were complete.
What does 'short frame' mean? To my understanding, what you want to say is 'short packet' which is the polar opposite of 'long packet'?
Despite the fact that the rest of the picture pipeline will treat it as if it were complete, the corrupted data will not support the resolution. Where did the frame's missing rows of data come from?

cleverca22
Posts: 3796
Joined: Sat Aug 18, 2012 2:33 pm

Re: Some questions about image buffers

Thu Jun 10, 2021 2:36 am

jacklab3 wrote:
Thu Jun 10, 2021 2:26 am
According to the MIPI CSI-2 protocol, a 1-bit error can be corrected and 2-bit errors can be detected. Is it possible for multiple bit errors to result in a line dropped? If possible, what will happen after dropping the error lines?
Will the capture stop if multiple bit errors happen?
if multiple bit errors occur, the entire scanline is lost
jacklab3 wrote:
Thu Jun 10, 2021 2:26 am
What does 'short frame' mean? To my understanding, what you want to say is 'short packet' which is the polar opposite of 'long packet'?
due to the loss of a scanline, you now have fewer scanlines in a frame then your supposed to, so the frame is short
jacklab3 wrote:
Thu Jun 10, 2021 2:26 am
What do you mean in 'it is essentially game over at that point anyway'?
because the lost scanline was in bayer format, and the system is assuming no lost scanlines, all scanlines after the lost one will have the color decoded incorrectly

an example from another thread: viewtopic.php?f=43&t=311529&p=1863123
Image

a line new the top got lost, messing things up, and turning the image pink
nearish to the bottom, a couple more got lost, but it stayed pink
then another one got lost, re-aligning the remaining scanlines, fixing the colors

jacklab3
Posts: 6
Joined: Wed Jun 09, 2021 2:53 am

Re: Some questions about image buffers

Thu Jun 10, 2021 3:33 am

Code: Select all

if multiple bit errors occur, the entire scanline is lost
Is there any reference to confirm this? I don't find it in the Documentation of 'MIPI Alliance Specification for Camera Serial Interface 2 (CSI-2)'
due to the loss of a scanline, you now have fewer scanlines in a frame then your supposed to, so the frame is short
Consider capturing a full resolution image (3280*2464) using a Camera module v2, if there are some scanlines dropped, where do the loss lines come from when display the image captured?

cleverca22
Posts: 3796
Joined: Sat Aug 18, 2012 2:33 pm

Re: Some questions about image buffers

Thu Jun 10, 2021 4:25 am

jacklab3 wrote:
Thu Jun 10, 2021 3:33 am
Is there any reference to confirm this? I don't find it in the Documentation of 'MIPI Alliance Specification for Camera Serial Interface 2 (CSI-2)'
i would assume its specific to the CSI receiver that broadcom put into the chip

the CSI specs just define that a 1bit error can be corrected, and a 2bit error can only be detected
its up to whoever designs the CSI receiver, to decide what to actually do in the event of a 2bit error

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11271
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Some questions about image buffers

Thu Jun 10, 2021 7:17 am

cleverca22 wrote:
Thu Jun 10, 2021 4:25 am
jacklab3 wrote:
Thu Jun 10, 2021 3:33 am
Is there any reference to confirm this? I don't find it in the Documentation of 'MIPI Alliance Specification for Camera Serial Interface 2 (CSI-2)'
i would assume its specific to the CSI receiver that broadcom put into the chip

the CSI specs just define that a 1bit error can be corrected, and a 2bit error can only be detected
its up to whoever designs the CSI receiver, to decide what to actually do in the event of a 2bit error
Correct.
If you have multiple bit errors, then you have image corruption to some level - game over. Behaviour at that point is up to the hardware implementation.
The Pi SoC discards the line, sets a status bit in the STA register (I'd need to check which bit), but does not increment the write pointer so the next line received without multiple bit errors will be written to that point in the image buffer.

CSI2 is designed for relatively reliable links, generally up to 30cm around a mobile phone or tablet. If you're getting multiple bit errors then you need to be looking at your hardware to determine why you are getting so many errors. No amount of software will correct for this corruption.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

jacklab3
Posts: 6
Joined: Wed Jun 09, 2021 2:53 am

Re: Some questions about image buffers

Thu Jun 10, 2021 8:06 am

the CSI specs just define that a 1bit error can be corrected, and a 2bit error can only be detected
its up to whoever designs the CSI receiver, to decide what to actually do in the event of a 2bit error
Besides the corruption image, how to ensure if there are multiple bit errors?
How does CSI check the bit error, the outlier data value or something related to the ECC?
If you have multiple bit errors, then you have image corruption to some level - game over. Behaviour at that point is up to the hardware implementation.
What does 'game over' mean?
The Pi SoC discards the line, sets a status bit in the STA register (I'd need to check which bit), but does not increment the write pointer so the next line received without multiple bit errors will be written to that point in the image buffer.
Hope you can check which bit and reply to me.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11271
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Some questions about image buffers

Thu Jun 10, 2021 8:30 am

jacklab3 wrote:
Thu Jun 10, 2021 8:06 am
the CSI specs just define that a 1bit error can be corrected, and a 2bit error can only be detected
its up to whoever designs the CSI receiver, to decide what to actually do in the event of a 2bit error
Besides the corruption image, how to ensure if there are multiple bit errors?
How does CSI check the bit error, the outlier data value or something related to the ECC?
See section 9.5.4 Applying ECC on RX side" of the CSI2 spec (I'm looking at v1.01). The basic syndrome detector gives 3 outputs - no error, corrected (ie 1-bit) error, or error
jacklab3 wrote:
If you have multiple bit errors, then you have image corruption to some level - game over. Behaviour at that point is up to the hardware implementation.
What does 'game over' mean?
It means you have image corruption at an unknown level. You can not trust any of the data, therefore you should give up on that frame and discard it.
Game over. Thanks for playing. Try again (ie wait for the next frame and hope it's better).
jacklab3 wrote:
The Pi SoC discards the line, sets a status bit in the STA register (I'd need to check which bit), but does not increment the write pointer so the next line received without multiple bit errors will be written to that point in the image buffer.
Hope you can check which bit and reply to me.
Looking at https://github.com/raspberrypi/linux/bl ... icam.h#L91
SBE = Single Bit Error corrected in packet header
PBE = Parity Bit Error detected in ECC of packet header
HOE = Higher Order Error detected in packet header
Exactly as per the spec 9.5.4's description:
Checking if a single error has occurred by searching Table 5, if the syndrome is one of the entries in the table, then a single bit error has occurred and the corresponding bit is affected, thus this position in the data stream needs to be complemented. SBE
Also, if the syndrome is one of the rows of the identity matrix I, then one of the parity bits are in error. PBE
If the syndrome cannot be identified,then a higher order error has occurred and the error flag will be set (the stream is corrupted and
cannot be restored). HOE
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

jacklab3
Posts: 6
Joined: Wed Jun 09, 2021 2:53 am

Re: Some questions about image buffers

Fri Jun 11, 2021 3:12 am

6by9 wrote:
Thu Jun 10, 2021 8:30 am
jacklab3 wrote:
Thu Jun 10, 2021 8:06 am
the CSI specs just define that a 1bit error can be corrected, and a 2bit error can only be detected
its up to whoever designs the CSI receiver, to decide what to actually do in the event of a 2bit error
Besides the corruption image, how to ensure if there are multiple bit errors?
How does CSI check the bit error, the outlier data value or something related to the ECC?
See section 9.5.4 Applying ECC on RX side" of the CSI2 spec (I'm looking at v1.01). The basic syndrome detector gives 3 outputs - no error, corrected (ie 1-bit) error, or error
Thank you very much, 6by9. I have found it on the documentation.
The Pi SoC discards the line, sets a status bit in the STA register (I'd need to check which bit), but does not increment the write pointer so the next line received without multiple bit errors will be written to that point in the image buffer.
It is the mechanism the SoC write data into the buffer, I have two questions about how can SoC write and read buffer as follows:
* For example, the SoC is configured to read 3280*2464 data from the buffer, but there are only 3280*2000 data new written into the buffer (caused by the multiple bit errors detected at the CSI receiver) at the clock t, do the remained 3280*464 data come from the clock (t-1)?
* If there is the synchronization clock to write and read data from the buffer?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11271
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Some questions about image buffers

Fri Jun 11, 2021 8:18 am

jacklab3 wrote:
Fri Jun 11, 2021 3:12 am
The Pi SoC discards the line, sets a status bit in the STA register (I'd need to check which bit), but does not increment the write pointer so the next line received without multiple bit errors will be written to that point in the image buffer.
It is the mechanism the SoC write data into the buffer, I have two questions about how can SoC write and read buffer as follows:
* For example, the SoC is configured to read 3280*2464 data from the buffer, but there are only 3280*2000 data new written into the buffer (caused by the multiple bit errors detected at the CSI receiver) at the clock t, do the remained 3280*464 data come from the clock (t-1)?
* If there is the synchronization clock to write and read data from the buffer?
You have two independent hardware blocks - CSI receiver (Unicam) and ISP.

- A buffer in SDRAM will be allocated for your 3280x2464 image.
- The start (and end) address of this buffer are passed to Unicam, along with a line pitch.
- Unicam waits for the Frame Start short packet, and then writes all subsequent data into this buffer.
- For each Line End short packet, IFF the line is valid (no uncorrectable errors), the write pointer for the next line will be advanced based on the programmed line pitch . If the line is invalid then the write pointer for the next line will not be advanced, and the next line will overwrite the previous (corrupt) line(*).
- When the Frame End short packet is received the buffer is signalled as "complete" and passed to the ISP. If there have been uncorrectable errors, then the last N lines of the buffer will not have been written to SDRAM based on this frame, so they will either be uninitialised or hold image data from a previous complete frame.

Once Unicam has signalled the frame as complete, the ISP will be told that a frame of the defined size is available, and it will process the defined image buffer with whatever data is in there.

I will stress again - if you are having to worry about multiple bit errors on a CSI2 link, then you are in a bad place and should be looking closely at the hardware side, including whether CSI2 with D-PHY are the correct interface to be using.
If you're getting errors due to distance between sensor and receiver, then you may be better off looking at THine’s V-by-One (available as a module for the Pi and reviewed in MagPi 106 page 80) or possibly TI's FPD-Link.

(*) I don't have detail of the actual hardware implementation, but this is the behaviour of it.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

jacklab3
Posts: 6
Joined: Wed Jun 09, 2021 2:53 am

Re: Some questions about image buffers

Tue Jun 15, 2021 6:17 am

6by9 wrote:
Fri Jun 11, 2021 8:18 am
- When the Frame End short packet is received the buffer is signalled as "complete" and passed to the ISP. If there have been uncorrectable errors, then the last N lines of the buffer will not have been written to SDRAM based on this frame, so they will either be uninitialized or hold image data from a previous complete frame.
What will happen if they are unitialized?
‘hold image data from a previous complete frame’ means the last N lines of the buffer will be replaced with those from a previous complete frame?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11271
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Some questions about image buffers

Tue Jun 15, 2021 8:17 am

jacklab3 wrote:
Tue Jun 15, 2021 6:17 am
What will happen if they are unitialized?
Uninitialised memory will have undefined contents. It could be all zeros, it could be all 1's, or it could be old data should that region have been used for something else in the past. (If the Linux kernel has done the allocation then it will normally have memset it to 0 to avoid any security issues of allowing a process to see stale data from an older user of that memory block).
jacklab3 wrote:‘hold image data from a previous complete frame’ means the last N lines of the buffer will be replaced with those from a previous complete frame?
Not replaced, but just not overwritten.
I have a canvas and some paints, and I paint a picture. I then paint over the top 90% of the picture, but the bottom 10% still shows the old picture.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Return to “Camera board”