RasPi as a camcorder


14 posts
by macarelo » Sun Oct 30, 2011 3:30 am
Hello everyone!
I'm new here, I'm not really a programmer or anything like it, I'm more like an enthusiastic nerd. And a director of photography (guy in charge of camera and lighting in a motion picture).

I read about the Raspberry Pi and was wondering, how much benefit can someone take out of the CSI2 interface? Just as a science project, and just to see how far technology has come in the last decade, I want to try to build a camcorder with a cellphone camera (a rather good one), but I don't want to use high compression rates, I actually want to try to capture RAW 10-bit or 8-bit RGB data as some high end (USD 100.000) digital film cameras can. Of course I don't mean to compare such footage with real movie making cameras, I just want to try to make the most out of the footage, and then compare it to cameras from 5 or 10 years ago (like DVCAM cams).
Its funny to think a 100 dollar camera today can have better or same quality than a 8000 dollar camera from 2002.

So, just as the Raspberry Pi is intended, I am now interested in programming. But, as a former Windows nerd, and an actual Mac nerd, I really don't know where or how to start. How do you program a camera? how does the interface behave? is it the same for all cameras that use a CSI2 connector?
Where can I start to learn these things?

Any help would be appreciated!

PS: Sorry about my rough english, I'm from Argentina.
Posts: 17
Joined: Sun Oct 30, 2011 2:16 am
by Nexy » Tue Nov 01, 2011 11:14 am
@macarelo: Unfortunately I probably know less about cameras than you do, but I did find these so I may as well link them here in case you or others hadn't seen them (and if they're even relevant.)

MIPI Alliance Standard for Camera Serial Interface CSI-2 overview

The OMAP3 Camera-ISP driver page on OMAPpedia - While I appreciate we're not using an OMAP chip it may give some assistance.

I hope that the interface is exposed to users under Linux in an easy-to-use way on one of the distributions like Arch or Fedora but we may have to wait and see. :)
Posts: 72
Joined: Sun Oct 09, 2011 9:03 pm
by jamesh » Tue Nov 01, 2011 11:43 am
I work on cameras on the Videocore GPU used in the Raspberry Pi.

A good example of the sort of quality you could expect can be seen on the Nokia N8 phone (plenty of cljps on YouTube)- this uses a slightly older videocore GPU so the quality isn't quite as good as the latest version which is used in the latest Nokia phones (e.g. 701). However, the lenses on the latest ones are not as good as the N8.

Although the N8 doesn't do 1080p, the Raspi can do it, 25mb/s data rate is possible, with a decent camera.

However, at the moment, there is no access to the GPU code for third parties, which means that although the CSI2 and CCP2 pins are available, you won't be able to write a camera driver for any arbitrary camera. We do hope to make a camera module available to purchase some time next year which will have a driver. We also need to get some more software in place to allow to to get at the data stream from Linux.

Note that writing camera drivers is not simple - they are quite awkward little buggers that require a lot of tweaking and in many cases conversations with the manufacturers to get the settings right. I've never met a camera datasheet that had everything needed in it. In addition you also need to tune the camera once the driver is done - that takes many man months to get a decent image and you really need access to a image lab with proper lighting set up etc.

Capturing the raw 10 or 8 data stream is difficult without very fast storage - SD cards won't be fast enough. You need about 600mb/s for 10 bit raw data at 30fps.
Moderator
Moderator
Posts: 10528
Joined: Sat Jul 30, 2011 7:41 pm
by Nexy » Tue Nov 01, 2011 12:16 pm
@jamesh: Thanks. At the risk of asking a dumb question, what would be the advantages of that over using a high-end webcam plugged into USB?
Posts: 72
Joined: Sun Oct 09, 2011 9:03 pm
by jamesh » Tue Nov 01, 2011 12:27 pm
The camera attached directly to the GPU can use the GPU's image processing pipeline to do white balance, gain control, denoise, sharpening, and many other stages (all at 30fps) - including encode to H264. Of course if you want raw you don't need that stuff, but the picture will suck. A high end Webcam may provide some of that processing internally, but not all (and not as good). Also, USB2 cannot handle the bandwidth of 1080p30 raw data and the Arm won't be able to keep up either, so the webcam would be compressed, often with MJPEG, which also sucks.
Moderator
Moderator
Posts: 10528
Joined: Sat Jul 30, 2011 7:41 pm
by Nexy » Tue Nov 01, 2011 1:00 pm
Ah, awesome, thanks. I look forward to the module's release at a later date! :D
Posts: 72
Joined: Sun Oct 09, 2011 9:03 pm
by macarelo » Tue Nov 01, 2011 7:34 pm
Jamesh, great info. I suspected RAW would be impossible because of transfer bandwith, the goal for me is to make a camera that can get the most out of the sensor. If RAW is out of the question (I was thinking about SATA SSD through USB), then I will look into compressions. I know H264 is the easy way out probably, and at first I would use that in order to get any image at all, but after that (if I succeed), I will probably look into something better.

It does suck that we can't access the GPU, which sets a limit on what we can do with the CSI2 connector... so, what can we do with it? if we buy, for instance, a Nokia N8 camera module, what can we do with it?
Posts: 17
Joined: Sun Oct 30, 2011 2:16 am
by NoSuchNick » Tue Nov 01, 2011 9:44 pm
but after that (if I succeed), I will probably look into something better.

Don't dismiss h264 to fast. If you use one of the high profile versions (for example Hi10P or even Hi444PP) you will probably have to worry more about the lense then the compression scheme. If you want to make sure that you can easily do frame accurate cutting, use the "intra" version of the profiles.
The big advantage will be, that besides motion jpeg, h264 will probably be the best supported codec for the camera. (Though the high profiles might not be supported.)

One feature that i will probably miss greatly, will be a bigger imaging sensor to get nice depth effects.
Posts: 40
Joined: Tue Sep 20, 2011 1:38 pm
by jamesh » Tue Nov 01, 2011 10:20 pm
To answer two of the above posts.

The N8 camera would probably just work if Broadcom allowed use of the existing driver, (I think I ported it to the VC4). Not sure as that may 'belong' to Nokia. However, buying that sensor might be very difficult indeed. Should Raspi be able to sell their own sensor it probably won't be the N8 sensor - it's expensive as it has an autofocus lens, and I'm not sure it can do 1080p30 (The N8 is 720p). That said, some of the hyperfocal sensors out there are pretty good.

We still need software to get the image/h264 from the GPU to Linux or SD card - that is being thought about as we speak.

H264 is a pretty good compression scheme (30x with little quality loss) and the best the VC4 can do at the moment. It's High Profile level 4, which is pretty good (Blu-ray uses it). The higher ones don't really improve much unless you are really really picky. See http://en.wikipedia.org/wiki/H.....4#Profiles
Moderator
Moderator
Posts: 10528
Joined: Sat Jul 30, 2011 7:41 pm
by cadack » Wed Nov 02, 2011 9:06 am
Sorry, slightly off topic. With the r-pi handling 1080p@30fps, is it possible to reduce the resolution and increase the frame rate? Eg 540@120fps or does it just not work that way.

If it did, it would be great for motion capture at sport labs ect..
Posts: 5
Joined: Mon Aug 15, 2011 8:09 pm
by jamesh » Wed Nov 02, 2011 9:20 am
Quote from cadack on November 2, 2011, 09:06
Sorry, slightly off topic. With the r-pi handling 1080p@30fps, is it possible to reduce the resolution and increase the frame rate? Eg 540@120fps or does it just not work that way.

If it did, it would be great for motion capture at sport labs ect..


Yes. I think we have had a VC3 up to 120fps on lower res images (so the VC4 should be faster. Its also dependent on the sensor being able to frame and output at that speed, but most ones I have worked with can I think.

It's basically how cameras like the Casio Exilim get the high frames rates - just make the frame smaller.
Moderator
Moderator
Posts: 10528
Joined: Sat Jul 30, 2011 7:41 pm
by cadack » Wed Nov 02, 2011 1:22 pm
The r-pi just keeps on giving!
Posts: 5
Joined: Mon Aug 15, 2011 8:09 pm
by radu » Mon Nov 21, 2011 9:48 pm
I recently got a Logitech 1080p Webcam Pro C910 for my desktop, and it seems like a very capable little camera. It can be set to capture RAW images, but it has no h.264 hardware compression. However, it has a huge sensor for a web cam (1/2.5") and I think only 5MP, so the pixels are pretty big for a small camera.
I don't know if the Raspberry Pi has enough juice to compress the video properly, but there are some other cool applications I can think of. For example, making a rig of 30 or so cameras to get effects like in The Matrix.
Or a time lapse camera.
Posts: 110
Joined: Mon Nov 21, 2011 8:19 pm
by toxibunny » Tue Nov 22, 2011 7:22 am
Quote from jamesh on November 2, 2011, 09:20
Quote from cadack on November 2, 2011, 09:06
Sorry, slightly off topic. With the r-pi handling 1080p@30fps, is it possible to reduce the resolution and increase the frame rate? Eg 540@120fps or does it just not work that way.

If it did, it would be great for motion capture at sport labs ect..


Yes.


instant tech boner
note: I may or may not know what I'm talking about...
Posts: 994
Joined: Thu Aug 18, 2011 9:21 pm