Camera Interface Specs

[GauVeldt]

Don't shaders need to work on textures in VRAM (ie: GPU's allocation)?
In which case leave the bayer in VRAM until the shaders have done whatever conversions (hopefully the debayer process can be coded as a GLSL/Cg shader depending on the fragment shader model that is supported). Don't move it to the ARM side until processing is required that cannot be done via shaders and has no public interface to perform via GPU directly.

I suppose that might be possible with a huge amount of Broadcom software effort. Which means it'll never happen since we already have something that works fine - all this does is contrive a convoluted route for data though the system to replace a nice easy route for data through the system. ie. replacing the dedicated devices specifically built for the purpose with something that isn't. And for what?

I managed to de-bayer my camera's image in GPU using these shaders: http://graphics.cs.williams.edu/papers/BayerJGT09/ (with very minor modifications). So this is not something that we have to wait for Broadcom to implement. Right now I am trying to figure out how to send this data directly to h.264 encoder, without copying it to CPU-s memory and back again. Copying a 1280x960 RGB buffer from VRAM to RAM takes about 40 ms.

Does the h.264 encoder API not support VRAM-sourced textures?

[jamesh]

Although I am sceptical that a 'universal video input' is even possible. HDMI in would be, but that wouldn't handle cameras. [...]

I know it's not on the agenda anytime soon, and maybe anytime ever. But many consumer video cameras, even consumer stills cameras with a video feature, now feature HDMI output, both live and in playback. I have two such myself (Panasonic, and Canon). So the ability to take HDMI in (unencrypted, obviously) would be quite a general-purpose thing and certainly open up a lot of possibilities.

I agree!

[jamesh]

I suppose that might be possible with a huge amount of Broadcom software effort. Which means it'll never happen since we already have something that works fine - all this does is contrive a convoluted route for data though the system to replace a nice easy route for data through the system. ie. replacing the dedicated devices specifically built for the purpose with something that isn't. And for what?

I managed to de-bayer my camera's image in GPU using these shaders: http://graphics.cs.williams.edu/papers/BayerJGT09/ (with very minor modifications). So this is not something that we have to wait for Broadcom to implement. Right now I am trying to figure out how to send this data directly to h.264 encoder, without copying it to CPU-s memory and back again. Copying a 1280x960 RGB buffer from VRAM to RAM takes about 40 ms.

Does the h.264 encoder API not support VRAM-sourced textures?

I think you can pass any GPU allocated memory to the encoder, certainly from what I remember of OpenMAX/mmal, the encoder just takes YUV420 frames from any source.

[Hardware_man]

JamesH wrote, “Not my call I’m afraid (universal video input)”

Hardware_man...

Then I request that you pass this information to the people who do make these decisions quickly. I agree I’m using a lot of speculation, but I suspect current geo-political conditions open the window right now for this to happen. And the window might not stay open for long. I realize this is primarily a technical forum, not a political forum. But much of the discussion in this thread has been about budgets and allocation of resources, technically we can resource wise we can’t. That’s why I mention current geo-political events as a possibility why the resources might be available right now but not for long.

[jamesh]

JamesH wrote, “Not my call I’m afraid (universal video input)”

Hardware_man...

Then I request that you pass this information to the people who do make these decisions quickly. I agree I’m using a lot of speculation, but I suspect current geo-political conditions open the window right now for this to happen. And the window might not stay open for long. I realize this is primarily a technical forum, not a political forum. But much of the discussion in this thread has been about budgets and allocation of resources, technically we can resource wise we can’t. That’s why I mention current geo-political events as a possibility why the resources might be available right now but not for long.

Although I can try (if I can find the right person - I'm a minion and have little say in anything like this), I think it highly unlikely that this would get the go ahead internally at Broadcom. There's just not the money in it to repay the investment. At this point I'm assuming a relatively simple HDMI input system. There's already an endeavour to sort out camera drivers to make them easier, but that's not for the Raspi, and will take some months and lots of cash.

Also, just because SS are having trouble (they won't for long) doesn't mean we have nothing to do! We are actually fairly busy at the moment.

[Hardware_man]

JamesH,

Any word from the decision makers?

Hardware_man

[Hardware_man]

JamesH,

Have you had an opportunity to talk to the right people yet?

Marty

[jamesh]

JamesH,

Have you had an opportunity to talk to the right people yet?

Marty

No, I'm away on business, in S. Korea.... I wouldn't hold you hopes up for a positive answer. We are all very busy at the moment, the hoped for lapse in work at end of project hasn't really arrived (since we haven't got to the end of the project)

[Hardware_man]

But you do have e-mail access. And I'm certain that there are people at the Foundation and / or Broadcom who would answer your e-mails who wouldn't answer mine. Just send them the link to this thread, ask them to read it.

Hardware_man

[jamesh]

But you do have e-mail access. And I'm certain that there are people at the Foundation and / or Broadcom who would answer your e-mails who wouldn't answer mine. Just send them the link to this thread, ask them to read it.

Hardware_man

Back in the UK now, can ask on Wednesday, but pretty sure just going to be laughed at; unless you can provide a market of 500k sales, this sort of this is just ignored. It simply isn't something Broadcom would be interested in doing since the entire MM team is busy of projects that would actually make money, rather than waste it. Foundation would be more interested, but limited dev time and other more important things to do first.

You the moment you should just assume this isn't going to happen in the near future.

[atto]

The missing of a reasonable fast interface into and out of the device may be the reason for its reduced success. I have found many people seeking for this information ... I myself have a serious project which I thought could be solved by Raspberry Pis (the target market is rather not so small) ... but it looks like I simply can't get the video data into the Pi ... so we'll end up using something else. I mean, really, that there's not simple parallel bus is a shame. And while I see the benefits of using the Broadcom chip I also feel that its proprietary character annoys lots of people (including myself). It's not fun to be boxed in where there's no real need for it.

[Neil]

The missing of a reasonable fast interface into and out of the device may be the reason for its reduced success.

You have an amusing definition of success....

Neil

[jamesh]

The missing of a reasonable fast interface into and out of the device may be the reason for its reduced success. I have found many people seeking for this information ... I myself have a serious project which I thought could be solved by Raspberry Pis (the target market is rather not so small) ... but it looks like I simply can't get the video data into the Pi ... so we'll end up using something else. I mean, really, that there's not simple parallel bus is a shame. And while I see the benefits of using the Broadcom chip I also feel that its proprietary character annoys lots of people (including myself). It's not fun to be boxed in where there's no real need for it.

Hmm. The Raspi costs $25/35. It simply cannot have everything that people may want at that price. To add video in (if you have not read the thread completely) would require extra hardware, and extra software. Both would add cost. Any increased the cost of the Raspi, would have reduced its 'success'.

I guess since you are now having to use something else, that device is a bit more expensive?

[B1234]

The missing of a reasonable fast interface into and out of the device may be the reason for its reduced success.

You have an amusing definition of success....

Neil

Id think such people also request that Ford make their basic compact cars do 0-60 in under 5s, and complain bitterly when Ford point them at other vehicles that are designed with such things in mind.

[jamesh]

But you do have e-mail access. And I'm certain that there are people at the Foundation and / or Broadcom who would answer your e-mails who wouldn't answer mine. Just send them the link to this thread, ask them to read it.

Hardware_man

Have spoken to the appropriate person, and Broadcom are simply not interested in this work. His suggestion was that it needs to be done by the Foundation themselves.

[Andreas]

Hi guys, I think there is a minimal solution where the foundation and community work together.

On the foundation-side of things you would add a "blind"-mode option to the driver that tells it to skip the camera probing and configuration.

Then you pick a resolution that the camera can be configured to transmit and is already supported by the driver and publish just enough information on the wire protocol for that specific resolution along with the ribbon cable pinout.

The community in return, will come up with a working hardware prototype in a few months. It wouldn't be cheap to buy but it solves this problem with minimal time investment from the foundation and reveals no proprietary information. (You can find the CSI-2 draft specs on the net anyway, so it is hardly secret.)

If and when there is working hardware, you could add support for more modes e.g common resolutions like 720p in the driver.

[jamesh]

Hi guys, I think there is a minimal solution where the foundation and community work together.

On the foundation-side of things you would add a "blind"-mode option to the driver that tells it to skip the camera probing and configuration.

Then you pick a resolution that the camera can be configured to transmit and is already supported by the driver and publish just enough information on the wire protocol for that specific resolution along with the ribbon cable pinout.

The community in return, will come up with a working hardware prototype in a few months. It wouldn't be cheap to buy but it solves this problem with minimal time investment from the foundation and reveals no proprietary information. (You can find the CSI-2 draft specs on the net anyway, so it is hardly secret.)

If and when there is working hardware, you could add support for more modes e.g common resolutions like 720p in the driver.

I don't think it's quite as simple (YMMV) as that.

I'd suggest Toshiba (IIRC) as a manufacturer of HDMI to CSI chips (Sony may also make them), but there is quite a bit of software work required on the GPU to plumb them in. A new driver, and improved plumbing upwards from there, since unlike a camera, where the GPU tells it what to do, in this case, the I/F chip is telling the GPU what to expect. Quite a different use case that isn't currently supported.

[Andreas]

But that's the thing: you need to do a lot of work on the driver in that case.

With a board that adapts the video to the Raspberry Pi instead, specifically some mode that are already supported, most of the costs and engineering falls on the community. The main difference is the FPGA that would sit between the HDMI deserializer and the Pi. HDMI negotiation over I2C could be handled by the ARM via the GPIO pins. It's a bit of a hack, but I'm sure nobody minds.

It is going to be more expensive to buy and take months to design, but that is volunteer work which can be done in the open. Besides, with this setup it is possible to make a high-speed gadget that isn't generating video, like a logic analyzer or oscilloscope. As long as the data is packed into the right format and not processed by the ISP, it should work the same.

[jamesh]

But that's the thing: you need to do a lot of work on the driver in that case.

You need to do a lot of work on the driver in ANY case.

[ghans]

@Andreas

If the thing you propose is actually possible , you could already
have started. The camera module has testers , remember ?

ghans

[Andreas]

Fair enough, you know the details of the driver and I don't.

Would you or someone else at the foundation be willing to make these driver changes and/or create documentation if people got together and raised money for it?

If modding the driver is out of the question due to time constraints, would it still be possible for us to pay you to write a bare-bones documentation of the wire protocol used when video is being transmitted in some well-chosen video resolution?

@ghans

Hah, are you asking me to put my money where my mouth is?
To be honest, I don't have the several hundred hours one needs to get it done. However, I'm willing to donate 100 euros to the foundation so it at least becomes a possibility for someone to get started.

Editing to add: 100 euros would only pay for an hour or two of work, so it won't be nearly enough, but hopefully I'm not the only one who is willing to donate.

[jamesh]

Fair enough, you know the details of the driver and I don't.

Would you or someone else at the foundation be willing to make these driver changes and/or create documentation if people got together and raised money for it?

If modding the driver is out of the question due to time constraints, would it still be possible for us to pay you to write a bare-bones documentation of the wire protocol used when video is being transmitted in some well-chosen video resolution?

@ghans

Hah, are you asking me to put my money where my mouth is?
To be honest, I don't have the several hundred hours one needs to get it done. However, I'm willing to donate 100 euros to the foundation so it at least becomes a possibility for someone to get started.

Editing to add: 100 euros would only pay for an hour or two of work, so it won't be nearly enough, but hopefully I'm not the only one who is willing to donate.

I'm not convinced what you suggest is actually possible. By far the best approach (so far - open to other suggestions) would be that suggested above. The HW would be much simpler, and the amount of driver work similar to what you suggest - maybe even less.

I'm not sure what you mean by wire protocol, but I assume you mean the CSI-2 spec, and the I2C spec for control? CSI-2 is a physical standard, but doesnt cover ththe format of the transmission, and there are all sorts of commands for different cameras.

[gsh]

Interesting that I haven't heard about this conversation since if anyone is the right person to ask its me!

Anyway concerning HDMI -> CSI

So there is the possibility of doing this, but it'll have to take a back seat behind and number of other projects. Of course it would be a very useful project for one of our summer interns and I might keep it in mind for then.

As to handling the software etc. Actually that's rather easy, I just get BRCM to add the CSI hardware to the 2835 peripheral spec (which isn't difficult at all) and then leave it up to a couple of interns to handle the software.

Will keep it in my notes for suitable intern projects

Gordon (Raspberry Pi - Director of Engineering)

[Hardware_man]

First positive thing that I have heard!

Hardware_man

[Andreas]

I'm not convinced what you suggest is actually possible. By far the best approach (so far - open to other suggestions) would be that suggested above. The HW would be much simpler, and the amount of driver work similar to what you suggest - maybe even less.

I agree it's cheaper and simpler, but you would have to spend a lot of time on it. On the other hand, if the FPGA option means even more driver work like you say, then that is no good either.

I'm not sure what you mean by wire protocol, but I assume you mean the CSI-2 spec, and the I2C spec for control? CSI-2 is a physical standard, but doesnt cover ththe format of the transmission, and there are all sorts of commands for different cameras.

Yes, the camera specific part, i.e the values one sees on the I2C bus when the Pi requests a frame from the camera or (ideally) the start/stop commands for a video stream. Also, the transmission format used on the the high-speed channel, for the mode the camera has been configured with.

Just one known-good mode is fine. Then it is easy to tell that any problem that crops up is not on the Pi-side.

Even if you can't do any driver development at all, this information is still useful for making a generic high-speed input.

@Gordon: Awesome, thank you!