jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23709
Joined: Sat Jul 30, 2011 7:41 pm

Re: Raw sensor access / CSI-2 receiver peripheral

Tue Jun 02, 2015 5:57 pm

EpsiOne wrote:Hello 6by9 and everyone else who is doing all the hard work!

I found my way here after reading a discussion on DIY Drones (http://diydrones.com/forum/topics/3-km- ... y-hardware) where the Paspberry is used to stream video using a technique called "Wifibroadcast".

Me and a lot of other drone pilots would be more than happy to be able to use our GoPros and other HDMI cameras with this new low latency "Wifibroadcast", instead of the Raspberry Pi camera module.

Being a beginner I've tried to understand this very interesting thread without really understanding anything... So I'm just going to ask: Will this progress that is being done right know enable us to use an module like this, B100 HDMI to CSI-2 Bridge: http://www.auvidea.com/index.php/theme- ... i-2-bridge to connect an HDMI camera to our Raspberry any time soon?

If I'm completely lost, I'm sorry for taking your time. Keep up the good work!
Doesn't answer your question, but why would a GoPro attached to a Pi be better than the much lighter Pi camera attached to a Pi? Is it just optics?
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

EpsiOne
Posts: 2
Joined: Tue Jun 02, 2015 7:37 am
Location: Malmö, Sweden

Re: Raw sensor access / CSI-2 receiver peripheral

Tue Jun 02, 2015 8:07 pm

jamesh wrote: Doesn't answer your question, but why would a GoPro attached to a Pi be better than the much lighter Pi camera attached to a Pi? Is it just optics?
Sorry for slowly drifting off topic, but yes better optics is a major reason. I'm just guessing that a lot of people would love to be able to live stream HD with any HDMI camera, perhaps this one: https://www.blackmagicdesign.com/produc ... nemacamera Basically I'm looking for a poor man's DJI Lightbridge https://www.dji.com/product/dji-lightbridge but with all the possibilities that an open platform like the Raspberry offers.

greenarchon
Posts: 5
Joined: Thu May 28, 2015 1:46 pm

Re: Raw sensor access / CSI-2 receiver peripheral

Tue Jun 02, 2015 8:26 pm

rascol wrote:At the moment the only i2c device on my RPi 2 is /dev/i2c-1 and so far no luck in getting a /dev/i2c-0 device working. Is it possible to get raw sensor access working on ic2-1 instead? Or do I have to first get ic2-0 going?
Did you set dtparam=i2c0=on (and not dtparam=i2c=on) in /boot/config.txt? Then a reboot and a modprobe i2c_dev and running the gpio script 6by9 provided should work.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7317
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw sensor access / CSI-2 receiver peripheral

Tue Jun 02, 2015 9:54 pm

flu wrote:my questions:
  • you wrote "therefore you've got one fixed exposure time and analogue gain." I suppose to choose fixed exposure time and analogue gain means changing the appropriate entries of struct sensor_regs ov5647[] - do you know which ones / how to find out? I am looking for maximum analogue gain.
https://github.com/6by9/userland/blob/r ... iraw.c#L87
flu wrote:[*]I found a spec from Omnivision which seems rather official (marked "preliminary Specification", november 2009, 140 pages). Is this the "copy floating around", or is there something better?[/list]
That sounds like it may well be a spec sheet for the OV5647 - when you don't link to it then it's hard to say. As my access to the datasheet is under NDA, I can not make comment about any potential differences between that doc and the latest version released to the Foundation.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7317
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw sensor access / CSI-2 receiver peripheral

Tue Jun 02, 2015 10:05 pm

EpsiOne wrote:Me and a lot of other drone pilots would be more than happy to be able to use our GoPros and other HDMI cameras with this new low latency "Wifibroadcast", instead of the Raspberry Pi camera module.

Being a beginner I've tried to understand this very interesting thread without really understanding anything... So I'm just going to ask: Will this progress that is being done right know enable us to use an module like this, B100 HDMI to CSI-2 Bridge: http://www.auvidea.com/index.php/theme- ... i-2-bridge to connect an HDMI camera to our Raspberry any time soon?
Yes, you should be able to support that board and similar from userland using the new component - I don't have one of these to test with, and the price is very steep for what it is. If I find I have lots of time (unlikely), then I believe the Foundation has a very similar board that I may try to borrow. AIUI they considered it not economically viable to manufacture it, but all the design work was done, and software support added to the GPU firmware.
I suspect your latency will be higher by going via HDMI, which may be a showstopper for you if you're trying to stream it for control.

I haven't checked, however I suspect that there will currently be a minor hiccup in that the board will spit out YUYV, but currently the video codec won't accept that. I do have changes in progress that should be able to support YUYV into the video_encode component - timescales to complete currently unknown.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
jbeale
Posts: 3494
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: Raw sensor access / CSI-2 receiver peripheral

Tue Jun 02, 2015 11:04 pm

EpsiOne wrote:
jamesh wrote: Doesn't answer your question, but why would a GoPro attached to a Pi be better than the much lighter Pi camera attached to a Pi? Is it just optics?
Sorry for slowly drifting off topic, but yes better optics is a major reason. I'm just guessing that a lot of people would love to be able to live stream HD with any HDMI camera, perhaps this one: https://www.blackmagicdesign.com/produc ... nemacamera Basically I'm looking for a poor man's DJI Lightbridge https://www.dji.com/product/dji-lightbridge but with all the possibilities that an open platform like the Raspberry offers.
I'm also hoping for HDMI input eventually, just to play with. In the meantime, as you probably know there are aftermarket versions of RPi-compatible cameras using the same sensor (= software compatible) but allowing for different optics; eg. M12 lens mount boards so you can select your own lens. I have a bunch of cheap M12 mount "megapixel" lenses in various focal lengths from m12lenses.com, one 6mm $50 lens from sunex.com which is a little bit better, and you can spend many hundreds for the better lenses, although maybe not justified with the 1/4" format sensor.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23709
Joined: Sat Jul 30, 2011 7:41 pm

Re: Raw sensor access / CSI-2 receiver peripheral

Wed Jun 03, 2015 9:08 am

A Pi camera is always going to be a lot lighter than a GoPro, and with the right optics should be at least, as good if not better. I'd suggest looking in to that a bit more.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

rascol
Posts: 9
Joined: Sat May 30, 2015 1:48 pm

Re: Raw sensor access / CSI-2 receiver peripheral

Wed Jun 03, 2015 3:03 pm

greenarchon wrote:
rascol wrote:
At the moment the only i2c device on my RPi 2 is /dev/i2c-1 and so far no luck in getting a /dev/i2c-0 device working. Is it possible to get raw sensor access working on ic2-1 instead? Or do I have to first get ic2-0 going?


Did you set dtparam=i2c0=on (and not dtparam=i2c=on) in /boot/config.txt? Then a reboot and a modprobe i2c_dev and running the gpio script 6by9 provided should work.
Thanks for responding greenarchon. Your suggestion is correct. But I had one additional problem. I was using a real-time kernel, http://docs.emlid.com/Downloads/Real-time-Linux-RPi2/. This one does not appear to support /dev/i2c-0. After reverting back to the current Raspian kernel, 3.18.14-v7+, both I2C devices appear.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7317
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw sensor access / CSI-2 receiver peripheral

Wed Jun 03, 2015 4:39 pm

rascol wrote:Thanks for responding greenarchon. Your suggestion is correct. But I had one additional problem. I was using a real-time kernel, http://docs.emlid.com/Downloads/Real-time-Linux-RPi2/. This one does not appear to support /dev/i2c-0. After reverting back to the current Raspian kernel, 3.18.14-v7+, both I2C devices appear.
From some of the docs on that page (mainly the mention of /etc/modprobe.d/i2c.conf) it sounds like that kernel image hasn't been updated to use device tree. In which case you probably need to go back to the old route of adding bcm2708.vc_i2c_override=1 to cmdline.txt and possibly editing blacklists.
Don't ask me why they haven't enabled device tree - it's possible that they just haven't run the mkknlimg script against the kernel.img after building, so there is no DT appended to the image, and that disables all DT stuff.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

rascol
Posts: 9
Joined: Sat May 30, 2015 1:48 pm

Re: Raw sensor access / CSI-2 receiver peripheral

Wed Jun 03, 2015 10:07 pm

6by9 wrote:
rascol wrote:
Thanks for responding greenarchon. Your suggestion is correct. But I had one additional problem. I was using a real-time kernel, http://docs.emlid.com/Downloads/Real-time-Linux-RPi2/. This one does not appear to support /dev/i2c-0. After reverting back to the current Raspian kernel, 3.18.14-v7+, both I2C devices appear.
From some of the docs on that page (mainly the mention of /etc/modprobe.d/i2c.conf) it sounds like that kernel image hasn't been updated to use device tree. In which case you probably need to go back to the old route of adding bcm2708.vc_i2c_override=1 to cmdline.txt and possibly editing blacklists.
Don't ask me why they haven't enabled device tree - it's possible that they just haven't run the mkknlimg script against the kernel.img after building, so there is no DT appended to the image, and that disables all DT stuff.
Thanks for the tip 6by9. You nailed it. From the cross-compiling setup, https://github.com/emlid/linux-rt-rpi, it's evident that the mkknlimg script wasn't run on the compiled kernel. After doing that, both I2C devices appear with the real-time kernel.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7317
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw sensor access / CSI-2 receiver peripheral

Fri Jun 12, 2015 10:05 pm

Just a bit of Googling for OV5647 drivers brought up https://android.googlesource.com/kernel ... 5647_reg.c which has register settings for a claimed "/*2608*1952 Reference Setting 24M MCLK 2lane 280Mbps/lane 30fps*/".
Might be useful if anyone either wants the masked pixels for black level calibration, or in analysing register setups.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

flu
Posts: 7
Joined: Sun May 31, 2015 8:10 pm

Re: Raw sensor access / CSI-2 receiver peripheral

Sat Jun 27, 2015 1:33 pm

I would like to share and discuss with you my findings on
  • exposure time setting
  • analogue gain setting
  • noise
  • reproducibility
Experimental setting:
I modified raspiraw so I can pass parameters for exposure time (Registers 0x3500-0x3502) and analogue gain (Registers 0x350A-0x350B) as command line options. And I extract 256 Pixel of each frame to make some statistics on them (calculation of Mean and deviation ) and write statistics to file.
Mysetting is a dark room, power LED and Camera-LED turned dark as well. The Camera captures only light of one LED (green, red, blue or white)
For a while I have worked with a diffusor between Camera an LED (to reduce differences of brightness).
More recently I skipped the diffusor, to achieve more brightness in the center and to employ minimal exposure times.
With this experimental setting I am able to study the effect of variations of exposure time and analogue gain very precisely.

Summary I am still far away from my goal to identify shot noise in my data. I am happy about any help.

Details of my findings:
Effect of Exposure setting is more complicated then expected. I expected a rather linear relation between mean of raw data and exposure time. I learned to keep an eye on „dark pixels“ which are not illuminated and which are not effected by this variation (as the signal they carry seems to be a kind of „noise" which is not dependent of exposure time.) And of course one has to keep an eye on saturated pixels, which are not effected by this variation as well.
below numeric value of 8 variations don’t have any effect. The Step from 8 to 16 does more then the expected increase by 2 of pixel values. Without precise measurement on this, I would say: The Step from 8 to 16 equals to an increase by 3.
The Exposuretime which equals to numeric value of 16 has to be clarified yet. I am aware of the following statement of 6by9: "rawcam as checked in gives an exposure time of 36.403ms.“ see https://github.com/raspberrypi/userland ... -101806950 I translate this to "numeric value of 1120 equals 36.403ms“ or "numeric value of 1 equals 32,5us"
This would fit, if tROW equaled to 32,5us, and if registers 0x3500-0x3502 defined exposure time as multiples of trow.
But it think registers 0x3500-0x3502 define exposure time as multiples of 1/16 tROW.
This is the case for OV5642, a sensor which seems very similar to OV5647, and of which a - in some details - better data sheet is available at: http://www.dragonwake.com/download/came ... 642-DS.pdf
The following source code (6by9, thanks for hinting to it in this thread!) takes into account a factor of 1/16 as well:
https://android.googlesource.com/kernel ... 5647_reg.c
tROW should be something like 1/(15*1868)=34us in case of full resolution, 15 frame per second.
I assume for now, that numeric value of 8 sets exposure time to 1/2 tROW = 17us, and I assume that some special register setting would be needed to choose even faster exposure times. This setting ist OK to me, so I am not searching for such a register setting.
But it would be nice to find confirmation of exact values somewhere.
Does anybody know more about this matter? (e.g. experimental results. If one takes a shot of a LED which is switched on/off with a known frequency of some 10 Khz, the resulting image should verify exposure time)

Effect of analogue gain setting:
I am rather sure that my Specimen produces at "gain 128“ (This means a numeric value of 128 written into the registers) one code increment per photon. At gain 256 it produces two code increments per photon and at gain 512 four code increments per photon. "per photon“ means: for each photon which is absorbed in the pixel and which does „produce“ an electron-hole pair. Of course not every photon is detected (quantum efficiency is below 1, but this does not matter here).
I can derive this from Diagrams, were I printed for every possible pixel value (0..1023) the number of pixels, which took the value. These Diagrams show „missing“ codes for gain > 128, and clear comb structures at gain around 256 and 512. Example: At gain 150 I observe Missing codes at 16, 33, … 101,118, 135,152,169. This means no Pixel (or nearly no Pixel) took these values while there were plenty of Pixels which took the values in between. (In fact I can't be sure, that gain 64 produces 1 code per photon, an that there may be other reasons why I don't observe missing codes with gain below 128. I would be glad to find somebody who is expert of this matter)
To find powers of 2 here indicates, that gain was set very precisely and for purpose at this level. I see no reason, why the manufacturer should be very careful with linearity and precision of analogue gain. So these findings astonished me very much. Thats why I focused on this matter, and tried many variations of gain to see what happens. I can’t help, these were my results.
I observe a strange side effect: when I increase analogue gain, then the output of „dark pixels“ decreases. At gain 250 they are typically at values around 10-20 (which is horribly much!) but at higher gain „0“ becomes very frequent, and mean values go down. This makes sense when we assume that „analogue gain“ setting does control Pixel reset circuit as well (the voltage, each pixel capacitor is charged to before exposure has to be smaller with higher analogue gain, so fewer electron-hole pairs are needed to discharge it )

Effect of Noise is much more complicated then I anticipated.
I tried to follow instructions from this paper http://journals.aps.org/prx/pdf/10.1103 ... X.4.031056 which basically tells go down with exposure time, then much of technical noise goes down as well, but that is not what I did experience.
My approach to get rid of fixed pattern noise: I take a series of up to 1024 frames, and calculate for each of my 256 chosen "test pixel“ the mean of 1024 measurements, and the deviation of its mean. And I calculate one covariance values between neighboring pixels of same color (covariance = what is meant in general when „correlation“ is said)
My biggest Trouble is: in the well illuminated areas, I observe very high covariance values. Shot noise would not show this.
In my general setting, the LED is powered by a GPIO Pin, so there could be some Mains hum. This could have an effect. But when I powered LED by a battery, I still found very high covariance values.

Problems with reproducibility.
I have serious doubts wether the camera’s state is fully controlled by the register writing at start of raspiraw, as I don’t always get same results with same setup. I am not sure about that. May be, my troubles comes from differences in LED positioning. Of course there may be Bugs in my code as well.

User avatar
jbeale
Posts: 3494
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: Raw sensor access / CSI-2 receiver peripheral

Sat Jun 27, 2015 5:21 pm

I am interested in these experiments, can you show us any of your data?

Also, I am curious about "...I extract 256 Pixel of each frame to make some statistics on them (calculation of Mean and deviation ) and write statistics to file." In particular, have you checked if the statistics depend on which areas of the frame you extract the 256 pixels from? I would like to think that every pixel would show the same statistics, but then again, 256 pixels is a very small fraction of the total sensor.

flu
Posts: 7
Joined: Sun May 31, 2015 8:10 pm

Re: Raw sensor access / CSI-2 receiver peripheral

Sun Jun 28, 2015 4:50 am

hi jbeale,

for now, my choice of 256 Pixels is simple and static. I take first 4 rows, center 8 rows, and last 4 rows. I take first 4 columns, center 8 columns, and last 4 columns. When I put this together, I get an Array where the center 8x8 Pixels are from the very center of the image sensor. The 4x4 Corners are from the corners of the image sensor. And I get 4 pieces of border.
For a while I really saved "full Data files", containing data of every single Pixel. But I don't have Software to process these files in a useful way. So this does not make much sense.
jbeale wrote:256 pixels is a very small fraction of the total sensor.
Yes, this is true. My spreadsheet application can do with 256 columns and many more rows. And processing is very fast. So I could process bigger array sizes. And probably some day I will do it. But for now I am very happy with the small data files. They open very fast, and I capture there content with one glance - no scrolling around. Most important inconvenience: I always have to check that LED illuminates the center. Without diffusor, the LED illuminates only a small fraction of the sensor. ( I could use lenses to change this, but I don't see the need to do this yet)
jbeale wrote:"have you checked if the statistics depend on which areas of the frame you extract the 256 pixels from?
Obviously, statistics depend on whether pixel data are from the illuminated center, or the dark borders. But within these categories, I did not observe differences yet. Specifically, I did not identify bad pixels yet. But I did not search for them either.
jbeale wrote:can you show us any of your data?
As a starter, I would like to post 4 files which show my results for an experiment with green LED, exposure 8 and gain 128. 5th File shows Terminal Output. Unfortunately, I am hindered to upload these files, regardless of file extension txt, csv or none. Error message reads: "The extension is not allowed". According do this post, one must be "given the appropriate permissions to create attachments": faq.php?mode=bbcode#f5r1
If this is the reason, I hope somebody gives this permission to me soon.
And in the longer run I would suggest the error message should be changed to some text which describes the situation more precisely. But no, please don't give me permission to do such administration stuff like changing error messages. That's nothing I am good at!

flu
Posts: 7
Joined: Sun May 31, 2015 8:10 pm

Re: Raw sensor access / CSI-2 receiver peripheral

Sun Jun 28, 2015 1:28 pm

Hi jbeale,

I have uploaded my results from an experiment with green LED, exposure 8, gain 128 and 1024 frames to pastebin:
  1. Terminal output The Lines "mmal: Frame #### starts:..." give values of first 4 Pixels. The Image is swapped horizontally or vertically? I forgot. Anyway, because of the swapping they are ordered green, red, green, red
  2. Histogram data. "green1" indicates the pixel which are in same row as red pixel. "green2" indicates the pixel which are in same row as blue pixel.
  3. mean values.
  4. deviation from mean. Calculated as Average absolute deviation, see https://en.wikipedia.org/wiki/Average_a ... _deviation
  5. covariance. Each entry of this Array represents covariance of 2 points of same row, but two columns (of my small 256 pixel pattern) apart. So the interpretation of data of first row goes as follows: first value represents covariance of 2 green "corner"-pixels very close to each other. Second value represents covariance of 2 red "corner"-pixels very close to each other. Third value represents covariance of 2 green pixels, which are more then 1000 pixels distant from each other. and so on. When comparing deviation and covariance, one has to keep in mind that calculation of covariance gives a "squared value", see: https://en.wikipedia.org/wiki/Covarianc ... covariance.
And for comparison I have uploaded Histogram data at gain 150.
This should give an impression of what I call "missing codes". (Conditions: green LED, exposure 8, gain 128 and 1024 frames)

PS: I read more about file attachment at this Forum. Conclusion: file attachments are discouraged, one should rather use external services like pastebin. That's why I registered at pastebin today.

User avatar
jbeale
Posts: 3494
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: Raw sensor access / CSI-2 receiver peripheral

Tue Jun 30, 2015 1:56 pm

Thank you for your data, I had a quick look. Now I want to try getting some measurements myself...

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7317
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw sensor access / CSI-2 receiver peripheral

Tue Jun 30, 2015 2:43 pm

I'm glad some people are finding this useful. And welcome to the wonderful world of mobile phone sensors where things aren't quite as one would hope.

I can confirm that in the line times for the various modes as configured in the firmware are:
- 1080P30 cropped: 29.584us
- 5MPix 15fps: 32.503us (this is the mode captured by jbeale and used in raspiraw)
- 5MPix 1/6-1fps: 183.789us
- 1296x976 binned: 23.216us
- 1296x730 binned: 23.216us
- 640x480 30-60fps: 31.749us
- 640x480 60-90fps: 21.165us
Those numbers were computed using an OVT provided spreadsheet and should be accurate.

From https://github.com/raspberrypi/userland ... -101806950 I had linked to www.viionsystems.com/UbuntuImages/ov5647_mipi.c
The GPU driver for OV5647 looks very similar for setting OV5647_set_shutter() and OV5647_set_gain16() Neither try to do fractions of a line, so I can't say whether that does or does not work on OV5647. The Brcm code actually has a defined minimum exposure time of 1 integration line (partly boilerplate code as some sensors have a minimum of several lines).

Seeing as the register settings were captured without requesting flips, I believe the expected Bayer order is GBRG. Flips/mirror (0x3820 and 0x3821) do alter the Bayer order as has been previously noted. (Register settings in raspiraw indicate an HFLIP - that appears to be defined in the GPU platform defines for OV5647 possibly to compensate for it being a BSI sensor. Base Bayer order for no flips at all should be BGGR).

Reproducibility: The registers in raspiraw are all that configures the OV5647. As we're fixing the powerdown GPIO the sensor doesn't get a reset between runs, but the register map should set everything anyway.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

flu
Posts: 7
Joined: Sun May 31, 2015 8:10 pm

Re: Raw sensor access / CSI-2 receiver peripheral

Thu Jul 02, 2015 6:19 am

Hello 6by9,

thanks for your comment. It is great to have a collection of the relevant values of tROW here.
Do you think a numeric value of 1120 as checked in with rawcam gives an exposure time of 1120 * tROW (approx 36.4 ms) or rather 1120/16 * tROW (approx 2 ms)?
I suppose the latter, but I lack confirmation.
Regarding Reproducibility:
If nobody else reports trouble with this, I clearly should focus on looking for reasons in my experimental setting. (And I already learned to pay more attention to Reproducibility of illumination)

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7317
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw sensor access / CSI-2 receiver peripheral

Thu Jul 02, 2015 6:46 am

Ye, my error. Register values of
{ 0x3500, 0x00 },
{ 0x3501, 0x04 },
{ 0x3502, 0x60 },
in raspiraw are an exposure of 1120 * 1/16 * 32.503us, or 2.275ms.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
jbeale
Posts: 3494
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: Raw sensor access / CSI-2 receiver peripheral

Sun Jul 12, 2015 5:22 am

Starting from a fresh Raspbian install, I did the below steps and was able to get a set of rawXXXX.raw files, and view them. So far, so good.

I now have a OV5647 board with FREX and STROBE pinned out, but so far been unable to get either FREX or STROBE to become outputs (they remain high impedance inputs while the code is running). My feeble attempt so far was to add { 0x3b00, 0x81 }, to the initialization register set struct sensor_regs ov5647[] in raspiraw.c, I thought that is supposed to enable STROBE REQUEST in LED 1 mode; when send a signal in FREX and get a signal out STROBE. But possibly the trigger is not via FREX, only via I2C register set while the camera runs?. I don't know if I need to set other registers too. As mentioned elsewhere the "preliminary" full datasheet available is a little light on the details.

Code: Select all

sudo apt-get update
sudo apt-get upgrade
sudo rpi-update
sudo reboot

git clone git://git.drogon.net/wiringPi
cd wiringPi/
git pull origin
./build

cd
wget https://raw.githubusercontent.com/6by9/userland/rawcam/camera_i2c
chmod +x camera_i2c

sudo mv /boot/start_x.elf /boot/start_x_OLD.elf
wget https://raw.githubusercontent.com/6by9/RPiTest/blob/master/rawcam/start_x.elf
sudo mv start_x.elf /boot/start_x.elf

insert these lines at end of /boot/config.txt
  start_file=start_x.elf
  dtparam=i2c_vc=on
  dtparam=i2c_arm=on

insert these lines at end of /etc/rc.local
  # enable i2c interface for raw camera access
  sudo modprobe i2c-dev

sudo reboot
  
sudo apt-get install cmake
git clone https://github.com/6by9/userland.git
cd userland; git fetch origin; git checkout -b rawcam origin/rawcam  
sudo ./buildme
sudo ln -s /opt/vc/bin/raspiraw /usr/bin/raspiraw

~/camera_i2c     # setup GPIO pins for camera operation
raspiraw         # run camera and record raw frames

wget https://raw.githubusercontent.com/6by9/RPiTest/master/dcraw/dcraw
chmod +x dcraw
./dcraw raw0016.raw          # convert one RAW data file into ppm format
sudo apt-get install fim     # 'fim' not installed by default
fim raw0016.ppm             # display image on framebuffer

Refer to datasheet http://www.seeedstudio.com/wiki/images/3/3c/Ov5647_full.pdf
Not too inspiring, my first raw photo from raspiraw / dcraw / imagemagick (downscaled to 25%)
Image

flu
Posts: 7
Joined: Sun May 31, 2015 8:10 pm

Re: Raw sensor access / CSI-2 receiver peripheral

Tue Jul 21, 2015 8:10 pm

Hello 6by9,

I can confirm the issue with reproducibility now.
I had another session, where the mean sensor values did not follow exposure setting as they should have. Only solution: power down, power up again.

I don't think, that this issue is related to your code, 6by9. But I think the symptoms may be more evident (more obvious) when raspiraw is invoked. As raspistill is based on exposure / gain regulation, some symptoms may be less obvious then.

My understanding now is that the sensor is physically reset only at powerup, but the timing at power up does not always comply with the sensor's needs. Result may be an unpredicted state. (with "mysterious symptoms")

According to my experience reset issues are not so rare. It always helps to actively trigger a reset line. Especially if there is no control of voltage raise time (volts per second ratio) as it is the case here.

So I think I should open a seperated thread in a hardware related category. But before I do this I just wanted to ask you about your remark:
6by9 wrote: Reproducibility: ... As we're fixing the powerdown GPIO the sensor doesn't get a reset between runs, but the register map should set everything anyway.
fixing ... does this mean, that there is a change scheduled to get a reset between runs? In this case I would not need to raisw an issue.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7317
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw sensor access / CSI-2 receiver peripheral

Tue Jul 21, 2015 10:13 pm

Rejigging your questions slightly.
flu wrote:
6by9 wrote: Reproducibility: ... As we're fixing the powerdown GPIO the sensor doesn't get a reset between runs, but the register map should set everything anyway.
fixing ... does this mean, that there is a change scheduled to get a reset between runs? In this case I would not need to raisw an issue.
The sensor has two hardware lines - PWDN and RESETB. I don't have the circuit diagrams, but one of those is controlled by GPIO 41 (on B+ and Pi2s anyway). That GPIO also controls the oscillator, and I suspect an LDO for powering the sensor. (GPIO 32 should solely control the camera LED, but on some clone boards it appears to do other things too).

My startup script that does the pinmuxing includes https://github.com/6by9/userland/blob/r ... era_i2c#L8

Code: Select all

gpio -g write 41 1
which is setting that line high and bringing the sensor out of reset/powerdown. There is no script set up to drop the line low again after running raspiraw. That's what I meant by "fixing the line high"

If you manually do:

Code: Select all

gpio -g write 41 0
gpio -g write 41 1
you will power cycle the sensor. You may be able to use that instead of power cycling the whole Pi.

There is also a software shutdown via register 0x0100 doing the same as the PWDN line, although the datasheet says "All register content is maintained in both modes" which I wasn't quite expecting.
flu wrote:I can confirm the issue with reproducibility now.
I had another session, where the mean sensor values did not follow exposure setting as they should have. Only solution: power down, power up again.
Hopeful reset solution above.
flu wrote:I don't think, that this issue is related to your code, 6by9. But I think the symptoms may be more evident (more obvious) when raspiraw is invoked. As raspistill is based on exposure / gain regulation, some symptoms may be less obvious then.
The GPU code is solely receiving the data from the sensor. There is no processing going on (other than the packing/unpacking/compression/decompression options which aren't used by default)
raspiraw is solely setting the defined values. raspistill is running a control loop, so if the sensor doesn't give the expected results for the requested exposure and gain values, the control loop will just adjust them further to compensate. The two apps doing different things, so I'd say it's hard to compare them.
flu wrote:My understanding now is that the sensor is physically reset only at powerup, but the timing at power up does not always comply with the sensor's needs. Result may be an unpredicted state. (with "mysterious symptoms")
I'm still surprised if it is values left from a previous run causing issues. The register list sets almost all the registers, even if just with the same values as they should default to, and, with the exception of the exposure and gain value, they're being set to the same values on each run anyway. The I2C should be robust enough not to have transmission errors, or at least report it if there are errors (there is an ACK/NAK response on any I2C transaction).
flu wrote:According to my experience reset issues are not so rare. It always helps to actively trigger a reset line. Especially if there is no control of voltage raise time (volts per second ratio) as it is the case here.

So I think I should open a seperated thread in a hardware related category. But before I do this I just wanted to ask you about your remark:
Answered above.
It may be best to hive off this discussion to a new thread anyway. Stick with this forum as I don't think there is a better option - it seems to all be related to the OV5647 camera board in particular.

One thing comes to mind - which frame are you analysing after you set your exposure and gain values? These style of sensors require at least one frame, sometimes two to adjust to new settings. Some of your comments imply that you are analysing a large number of frames, in which case that may not be relevant.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

flu
Posts: 7
Joined: Sun May 31, 2015 8:10 pm

Re: Raw sensor access / CSI-2 receiver peripheral

Wed Jul 22, 2015 9:30 pm

Hi 6by9,

thanks a lot for the fast reply
GPIO 41 - that's what I needed.
So I will just add a proper reset,
and everything should be fine.

danielm5
Posts: 2
Joined: Wed Jul 29, 2015 8:13 pm

Re: Raw sensor access / CSI-2 receiver peripheral

Wed Jul 29, 2015 8:23 pm

Hi 6by9,
I'm using rawcam and it seems to work well so far. Thank you for your effort on having this done.

On the side, I'm fairly new to MMAL and I've noticed that I receive two callbacks for each image, one with and the other without MMAL_BUFFER_HEADER_FLAG_CODECSIDEINFO flag. Could anyone tell me what extra information is in the second call?

Thanks,
Daniel./

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7317
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raw sensor access / CSI-2 receiver peripheral

Wed Jul 29, 2015 9:49 pm

danielm5 wrote:On the side, I'm fairly new to MMAL and I've noticed that I receive two callbacks for each image, one with and the other without MMAL_BUFFER_HEADER_FLAG_CODECSIDEINFO flag. Could anyone tell me what extra information is in the second call?
Er, try reading my first post under the heading "What is this not doing?"
The handling of the sensor non-image data path is not tested. You will find that you always get a pair of buffers back with the same timestamp. One has the MMAL_BUFFER_HEADER_FLAG_CODECSIDEINFO flag set and should be the non-image data. I have not tested this at all as yet, and the length will always come through as the full buffer size at the moment.
Read the CSI2 spec and you'll see that all data coming in has an ID field, normally dictated by the format of the image data (eg 0x2B for 10 bit Bayer data). In the peripheral, you tell it the ID you expect for all your image data and it saves that into an image buffer. Any data received for other ID values are passed in a data buffer, and that is what is passed out in the MMAL buffer with CODECSIDEINFO set.
Typically this is used to convey metadata from the sensor back to the system, and often includes a full register dump from the sensor as it makes extracting the exposure and gain for the frame easy. AFAIK the OV5647 does not use this, but whilst I was writing the component, it made sense to expose the functionality.

Glad it's of use.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Return to “Camera board”