HDMI to CSI-2 via TC358743 on kernel 4.1

[realies]

I have been trying to find a solution for capturing HD video from HDMI sources with the Raspberry Pi for quite some time now. I've seen some forum topics in which users and moderators are discussing if this is possible, Auvidea's products as well as development bridge boards from the Raspberry Pi Foundation.
As far as I'm aware, the foundation and Auvidea are using the same Toshiba TC358743XBG bridge chip which is now supported in the kernel 4.1-rc2.
My question here is, what would happen if I get a B101 HDMI to CSI-2 Bridge (15 pin FPC), connect it to the Pi, update kernel to 4.1 and start raspivid?
Would it be theoretically possible to capture 1080p30 trough the two CSI lanes available on the Pi?

[6by9]

(This isn't one of the normal forums I lurk on - Camera Board, or Interfacing CSI/DSI/I2C are more likely for me to notice)

The answer may be paritially, but not because you're using kernel 4.1. There is no kernel support for the CSI2 receiver peripheral at the moment (partly due to IP issues around control registers).
However the Foundation had implemented the driver solely within the GPU. They perceived that there wasn't a sufficient market to productise the board. It is a reasonable guess that the Auvidea board would be compatible and just be recognised as the same, but I don't know of anyone who has tried it.

There is now a wrapper around the CSI2 receiver that has demoed it receiving the raw Bayer data from the standard camera - viewtopic.php?f=43&t=109137
It should be possible to write a suitable kernel driver to wrap that, in the same way that the V4L2 camera driver wraps the full camera component. The bit of information I haven't managed to determine is how to hook everything together to make the TC358743 driver and the CSI-2 receiver peripheral work together and form a system. At one point it looked like the soc-camera framework was the way, but that appears to have been deprecated in favour of the media framework.

If anyone has any experience of hooking these things together to make a working system, please let me know and it would motivate me to invest a bit more time in it. If there isn't anyone lurking here, I may go direct to the driver writer to get some assistance.

NB This driver only produces UYVY or RGB888 data, neither of which is currently consumable by the video encoder. Conversion on the ARM would be a heavyweight operation.
And yes, 1080P30 can be carried on 2 lane CSI2 - each lane is 1Gb/s, and 1920x1080x24bppx30 = 1.493Gb/s. YUYV is slightly less than that as it is only 16bpp.

[realies]

I'm really interested into testing that, hopefully would get a reply from Auvidea about purchasing a B101 board and post results here.

[Orbital6]

Can confirm, calling raspivid with a b100 board starts showing whatever your hdmi feed is (without audio). can make it stream with raspvid -f 0

[realies]

Can confirm, calling raspivid with a b100 board starts showing whatever your hdmi feed is (without audio). can make it stream with raspvid -f 0

Looking forward to test the board myself asap. Did you use an adapter for the b100? It still a pity that the board costs 20 times the chip price...

[Orbital6]

Can confirm, calling raspivid with a b100 board starts showing whatever your hdmi feed is (without audio). can make it stream with raspvid -f 0

Looking forward to test the board myself asap. Did you use an adapter for the b100? It still a pity that the board costs 20 times the chip price...

That breakout board is worth it, if i was you and had time for DIY'ing then i'd buy the female receptacle, and make your own breakout board - the pins are not well silk screened ( i had to test what each pin was by cont testing with reference to the schematic) and found there to be 0 documentation. I don't know many others who got the b100 so keep me updated if you find out anything useful, currently trying to send useful i2c commands to it.

[auvidea]

Hi Orbital6, this is great news. I have been waiting a long time for driver support for the Raspberry Pi. It looks like, that things are starting to move now. We have designed the B101 specifically for the Raspberry Pi. It does not require an adapter board, as it features the same 15 pin FPC connector as the Pi.
http://www.auvidea.eu/index.php/theme-s ... 32-06/b101
1. First I need to get a working test setup on a Raspberry Pi. We at Auvidea are not Raspberry software specialists so we would need some assistance to get this up and running quickly. Can you provide some help? Then we can test the B101 module.
2. Next I would like to offer a special price on the B101 module including 15 pin FFC cable: it is EUR 69 net without VAT and duty. This is half the current list price (EUR 139 net). German and EU customers need to pay the 19% German VAT. So their price is EUR 82 incl. VAT and shipping. This offer lasts until Dec 31, 2015.
This price includes the Paypal charge and the international shipping charge by registered postal mail by the German Post (with tracking number to most countries). They claim a delivery time of 2-4 days to European countries and 2-5 days to the US. Other countries my take up to 12 days.
For the moment please send your order by email to the info address at auvidea.com. Please mention "B101 Pi special" on your order.
Regarding the build quality of the adapter board for the B100: the initial batch of these adapter boards have been build by hand, so the soldering quality was not that great. Now it is much better as we now use automatic manufacturing equipment.
Regards, Jurgen

[6by9]

@auvidea: Your website implies that you have the HDMI input board running with HummingBoard, Nitrogen6x, Jetson TK1, and other platforms. Is that actually using the Linux soc_camera or media sub device framework, or via some other mechanism?
We do have access to the raw CSI-2 receiver via MMAL (viewtopic.php?f=43&t=109137), but I haven't pieced together how the Linux kernel support actually fits together to make a usable video device.
If you can give me some guidance as to how to make it a usable system, then I may be able to wrap that in a V4L2 subdevice driver (the main camera V4L2 driver is also wrapping MMAL on the GPU, so we have a fair amount of useful framework already).

The new kernel TC358743 driver does not make for easy reading. It was a few weeks ago that I looked at it in any detail, and can't remember whether it tried to do anything with the audio or not. I was contemplating contacting the kernel driver's author to try to get some guidance, but haven't had the time to fully investigate. If you have some knowledge of the V4L2 architecture, then I can avoid hassling them.

PS Nothing has changed on the firmware - the drivers have been there pretty much since the demo. I guess no one had tried running raspivid with your board. It is NOT supported as a product though.
I started trying to sort out using the above CSI2 receiver module from userspace to grab frames - ran out of time to test it. The BGR888 format that the data tends to be received from the Toshiba chip in isn't widely supported on the GPU, so there's some magic required. The Foundation demo is using the camera ISP hardware as format conversion to get it into YUV that can be used by the rest of the system.

[auvidea]

It works!

Today we have tested both the B100 module with adapter board and the new B101 module, which is specifically designed for the Raspberry Pi. We have tested it with the Raspberry Pi 2 Model B. We have used a second Raspberry Pi with an output of 1080p30 to test the setup. It works like a beauty. I will post some pictures and a video later on.

The only issue is, that it only supports raspivid with an input resolution of 1080p30 at this time. Hopefully we get some support from the Raspberry Foundation to get other modes supported.

Special offer
Please order your B101 module with 15 pin FFC cable today. This offer lasts until Dec 31, 2015. It is more than 50% off the list price of EUR 139 net. Just send us an email with your order. I have modules in stock. So it could ship immediately (as long as supply lasts). When they are gone, I will get more produced.

Countries in the EU: EUR 83.06 incl. 19% German VAT and shipping by registered mail
Countries outside the EU: 69.80 incl. shipping by registered mail

Payment by Paypal (Paypal fee included).

Regards, Jurgen

[mikerr]

What format does it save the data in ?

[mikronauts]

Congratulations!

A few questions:

- does your B100/B101 report 1080p30 to the hdmi source?
- can it support 1080i60?
- do they support 1080p60 on the hummingboard?
- are there any other differences between the B100 & B101 other than the B101 not needing an adapter board?

I'd love 1080p60 so I hope the RPF gives you the support you need to make that happen!

It works!

Today we have tested both the B100 module with adapter board and the new B101 module, which is specifically designed for the Raspberry Pi. We have tested it with the Raspberry Pi 2 Model B. We have used a second Raspberry Pi with an output of 1080p30 to test the setup. It works like a beauty. I will post some pictures and a video later on.

The only issue is, that it only supports raspivid with an input resolution of 1080p30 at this time. Hopefully we get some support from the Raspberry Foundation to get other modes supported.

Special offer
Please order your B101 module with 15 pin FFC cable today. This offer lasts until Dec 31, 2015. It is more than 50% off the list price of EUR 139 net. Just send us an email with your order. I have modules in stock. So it could ship immediately (as long as supply lasts). When they are gone, I will get more produced.

Countries in the EU: EUR 83.06 incl. 19% German VAT and shipping by registered mail
Countries outside the EU: 69.80 incl. shipping by registered mail

Payment by Paypal (Paypal fee included).

Regards, Jurgen

[6by9]

The only issue is, that it only supports raspivid with an input resolution of 1080p30 at this time. Hopefully we get some support from the Raspberry Foundation to get other modes supported.

Er, hello! I'm probably the closest person to the Pi Foundation/Trading that might work with you on this. I'd posted after your last message, but you seem to have ignored it.

The support for that chip in the firmware is NOT officially maintained, as Pi Towers have never released it as a product. If it ever breaks, then don't expect much sympathy or a rapid response.

There is a route forward already open to you using components that will be supported (CSI-2 receiver peripheral, and you can use the ISP to resize/format convert now). I've already offered support on that within time constraints (I am not employed by Pi Towers). I am in the process of creating a userland app that creates the basic pipeline to receive the data, process it, and slap it on the screen. Video encode can follow.
I would like to sort it out to use the official kernel driver, but I am missing some information on how the pipeline gets set up. Please (anyone!) advise if you have any extra information. I'm happy to switch to private message or email for that, rather than having public discussions.

For those asking questions, the data is received from the Toshiba chip as RGB888, converted to YUV4:2:0 and potentially resized in the ISP, and that is fed to the encoder. So output is the same selection as for the normal camera - MJPEG or H264.
1080P60 is 2.8Gbit/s if in RGB888. Each CSI2 lane can carry 1Gbit/s of data, and the normal Pi only has 2 lanes exposed, so insufficient bandwidth there. The Compute Module does expose the 4 lane interface on the CAM1 interface. There is a YUV4:2:2 mode from the chip which would just squeeze into 2 lanes.
I'm not sure what you then want to do with 1080P60 though - the H264 encoder supports a max of 1080P30.

The fun side of this chip is that most pipelines are expecting cameras, so the app sets the resolution. With this chip if you program in multiple resolutions via the EDID, then the source can change the resolution on you. Getting a framework to handle that nicely is not trivial. That is why there is only one resolution programmed into the EDID. If we can use the official kernel driver then that problem becomes theirs, not mine!
Adding in audio support is also entertaining - the chip can do I2S, or it can send it over the CSI2 bus in an appropriately marked format. If done over CSI2 then you've suddenly got to manage an audio device as well as video, at which point my head starts hurting with the Linux subsystems.

[auvidea]

Hi 6by9,

I did not mean to ignore your message. I appreciate very much the support which you are offering. Also I like to thank you for the information, you have given.

I wonder, how we can go from here. It looks like there are plenty of action type video cameras out there, which support 1080p30 as an output format. So having only support for 1080p30 may mean, that it is already very usable. However, it would be great if there would be support for 1080i60 or 720p60. But if I read you right, it seems to be not easy at all to get this integrated.

I2S: I get from your mail that it may be easier to get audio support via the I2S bus. Now the I2S bus is on the 40 pin GPIO header. I guess I could redesign the B101 so It would connect to the the 15 pin camera connector for video and the 40 pin header for audio. What would this mean in software support? I guess the TC358743 registers have to be initialized so that the I2S bus is enabled. Can this be changed? Or do I need to put a micro controller onto my board to intercept the I2C bus?

I am happy to switch to private messages to discuss things further.

Regards, Jurgen

[auvidea]

A photo of the B101 module. It connects with a 15 pin FFC cable to the Raspberry Pi.

A photo of the test setup. It shows the playout of the 1080p30 video received with the B101.

A closeup of the B101 module which is mounted on top of a Raspberry Pi 2.

[6by9]

Responding to other questions via PM.

I wonder, how we can go from here. It looks like there are plenty of action type video cameras out there, which support 1080p30 as an output format. So having only support for 1080p30 may mean, that it is already very usable. However, it would be great if there would be support for 1080i60 or 720p60. But if I read you right, it seems to be not easy at all to get this integrated.

720P60 should be possible as it is within H264 level 4.0. I don't believe the encoder has any support for interlaced encoding.
Please look at the kernel driver that has been released (https://github.com/raspberrypi/linux/bl ... tc358743.c) - that is the driver that will be supported in future if we can, not the firmware one. We're probably 90% of the way there, but I need some help understanding the last few percent. There may be an interim stage where we just have everything in userspace, at which point almost anything goes.
Looking again at the kernel driver, it looks like they take in an EDID from the client via tc358743_s_edid and the ioctl VIDIOC_SUBDEV_S_EDID. Sounds like passing the buck to me!

I2S: I get from your mail that it may be easier to get audio support via the I2S bus. Now the I2S bus is on the 40 pin GPIO header. I guess I could redesign the B101 so It would connect to the the 15 pin camera connector for video and the 40 pin header for audio. What would this mean in software support? I guess the TC358743 registers have to be initialized so that the I2S bus is enabled. Can this be changed? Or do I need to put a micro controller onto my board to intercept the I2C bus?

All control in the kernel (or possibly userspace) means anything is potentially possible as you will have full source for the driver.
For the kernel option, it depends how well you understand the Linux kernel V4L2 framework. It looks like the current driver only supports I2S based on https://github.com/raspberrypi/linux/bl ... 743.c#L796
If you can work out how to get the data from the relevant part of V4L2 into the audio stack, then it could be done over CSI2. Via userspace, you've got to work out how to get data from your application back into the system so it looks like normal audio data. The Linux audio stack is not something I know at all, so I'm not the right person to ask.

Please let me restate: the GPU firmware driver for this does work at present. However it is not officially supported, and may be deprecated or removed in future if there is a working alternative.

[Orbital6]

I have the b100 board and have compiled a kernel module to decode the audio on the i2s bus but am having some sample overrun errors.

I2S: I get from your mail that it may be easier to get audio support via the I2S bus. Now the I2S bus is on the 40 pin GPIO header. I guess I could redesign the B101 so It would connect to the the 15 pin camera connector for video and the 40 pin header for audio. What would this mean in software support? I guess the TC358743 registers have to be initialized so that the I2S bus is enabled. Can this be changed? Or do I need to put a micro controller onto my board to intercept the I2C bus?

I'm not as good as 6by9, but in my experience i have gotten audio out of the i2s bus from the toshiba IC without the need of any other microcontroller (i've confirmed this with an oscilloscope) - you just need to set the i2s option in the 0x4 register and look at the datasheet on how to set the audio configuration, for example the 'standard sony config is SDO_MODE1.SDO_FMT=2’b00, ConfCtl.I2SDlyopt=1’b0, SDO_MODE0.LR_POL=1’b1'.

There may be timing issues without the intercepting microcontroller (edit: this is risky, read the next few posts), but in my experience i've had success setting the registers when the video was being output on raspivid.

I've sent you a couple of emails to ask for any drivers you may already have that i could work on.

a 40 pin header may be overkill given that the i2s bus only requires 3 pins, and another 2 for the i2c if you wanted to talk on bus 1.

[6by9]

I'm not as good as 6by9, but in my experience i have gotten audio out of the i2s bus from the toshiba IC without the need of any other microcontroller (i've confirmed this with an oscilloscope) - you just need to set the i2s option in the 0x4 register and look at the datasheet on how to set the audio configuration, for example the 'standard sony config is SDO_MODE1.SDO_FMT=2’b00, ConfCtl.I2SDlyopt=1’b0, SDO_MODE0.LR_POL=1’b1'.

There may be timing issues without the intercepting microcontroller, but in my experience i've had success setting the registers when the video was being output on raspivid.

As per our email discussions, you have directly connected I2C-1 (controlled by the ARM) to I2C-0 (controlled by the GPU) so that both could talk to the Toshiba chip. AFAIK there is no support for multiple masters in the 2835/6 I2C peripherals, so you may get collisions and find that things occasionally just don't work. The GPU does still poll the Toshiba chip whilst video is streaming, so it is not safe to assume the bus is quiet once raspivid has started.

[Orbital6]

Hi Orbital6, this is great news. I have been waiting a long time for driver support for the Raspberry Pi. It looks like, that things are starting to move now. We have designed the B101 specifically for the Raspberry Pi. It does not require an adapter board, as it features the same 15 pin FPC connector as the Pi.
http://www.auvidea.eu/index.php/theme-s ... 32-06/b101
1. First I need to get a working test setup on a Raspberry Pi. We at Auvidea are not Raspberry software specialists so we would need some assistance to get this up and running quickly. Can you provide some help? Then we can test the B101 module.
Regards, Jurgen

Hi Jurgen,
apologies for not seeing your post earlier. As i mentioned i have attempted to make some contact with you over email but if it's easier then we can keep it here.

I have the b100 board, can i confirm that the only difference with the b101 is the hardware layout that is better suited to the RPi on the b101? and can i confirm the order of the 6 i2s pins on the b100 rpi breakout board are in the following order (top to bottom): 3.3V, gnd, LRCK, MCLK, BCK, DATA?

To answer your first question, I would like to help and don't mind testing a b101 board, but have some points:

About the i2c (to get i2s audio out), 6by9 correctly states that manual commands sent on the i2c bus 1 (for example i2s audio) may require extra hardware for safe communications, i just wanted to make the point that i was able to get i2s audio out without any extra hardware but there absolutely is the risk of the video breaking if a manual command sent on bus 1 overrides or collides the raspivid i2c communication on bus 0. raspivid (when a video is playing) does (shown on a probe) poll the toshiba IC every ~1 second, so you could make a small system that connects both i2c from bus 0 and 1, queuing commands from both buses and prioritising the gpu i2c bus to avoid any timing issues that 6by9 correctly mentions. This should allow safe manual override of the video/audio registers when raspivid is playing, as i have noticed raspivid overrides existing configurations at execution. I am not sure if this sort of system is what your 'optional uC for i2c comms' covered for your B100 solution.

As per our email discussions, you have directly connected I2C-1 (controlled by the ARM) to I2C-0 (controlled by the GPU) so that both could talk to the Toshiba chip.

It's worth noting to others that this bridge exists on the b100 - i confirmed by cont testing the two i2s ports (from the csi2 i2c bus and two pins on the breakout board).

[Orbital6]

PS Nothing has changed on the firmware - the drivers have been there pretty much since the demo. I guess no one had tried running raspivid with your board. It is NOT supported as a product though.

I wonder if i could re-code some of the existing kernel drivers so that when viewing video from the toshiba IC via raspivid, raspivid sets up audio routing through the i2s bus instead? I could probably do this if someone points me to the right .c file to edit and re-compile.

[6by9]

I wonder if i could re-code some of the existing kernel drivers so that when viewing video from the toshiba IC via raspivid, raspivid sets up audio routing through the i2s bus instead? I could probably do this if someone points me to the right .c file to edit and re-compile.

Raspivid is solely using a GPU side driver for the TC, so there is no open source code for it.

The Pi Towers board was set up to receive the audio over CSI2 and route it into the GPU audio subsystem - that should still work, but doesn't fit in with the Linux side of things easily therefore Gordon directed you at I2S. In theory a userspace app can pick up on the same audio stream, I just haven't got to that point yet. The Linux kernel driver appears to set the audio to I2S, so that may not have a sensible option for receiving via CSI2.

[vade]

Hello

This is very exciting. Ive been wanting HDMI input to the Raspberry Pi for a long time. Congratulations to everyone who contributed!

I'm curious of the realistic capabilities of video processing on the RPI2 GPU for 1080p30 input is. Does this board enable direct to texture / zero copy or some sort of fast path to OpenGL ES texture?

Can OpenMax or whatever the current hardware accelerated video processing library on the RPI actually run operations and blit to the screen at full frame rate, one to one output?

Has anyone been able to actually process a 1080p frame at 30Hz on the RPI via a full screen GLSL shader or some such? I imagine GPU fill rate is a concern.

I have done some experimenting with the native RPI camera and a custom GLES app and was unable to get anywhere near optimal realtime performance. By no means am I stating its not possible, I am simply curious what one should expect. I'd love someone more knowledgable than I to lay out what could be theoretically possible.

Thank you!

[ritesh7]

hey i am using http://www.auvidea.eu/index.php/theme-s ... 32-06/b101 with raspberry pi 2.

1080p and 720p works fine.
But i am having issue with 1080i. In india DTH set-top are having output of 1080i.
So when i connect 1080i 50Hz. The "raspivid -t 0" shows cropped video feed and this feed also flickers periodically. And 1080i 60Hz shows cropped + messy video feed with flickering.
Changing settop box's output to 720p works fine.
So is there any solution for 1080i?

And also i am not getting audio output? So is there any workaround?

[6by9]

1080p and 720p works fine.
But i am having issue with 1080i. In india DTH set-top are having output of 1080i.
So when i connect 1080i 50Hz. The "raspivid -t 0" shows cropped video feed and this feed also flickers periodically. And 1080i 60Hz shows cropped + messy video feed with flickering.
Changing settop box's output to 720p works fine.
So is there any solution for 1080i?

You are using a third party product with unsupported code from the Pi Foundation - it was written for a demo only.

1080i is NOT supported. It shouldn't be included in the EDID advertised to the source device. The signalling of which field is which is not easy to pick up via the CSI2 interface.
I thought the EDID only advertised 720P, but may be wrong. I'm not going to investigate it.
Realise that the data is still going through the imaging pipeline, so the resolution being fed into the board from your source does NOT have to be the same as the output resolution from raspivid - it will upscale/downscale/crop quite merrily.

And also i am not getting audio output? So is there any workaround?

Code for a demo!
raspivid never does audio anyway. There is code in the firmware blob that should route the audio through the GPU audio system, but that isn't really used on the Pi so there is no example code for how to extract it.
(edit Remove reference to other people's work on this chip - voluntary support on anything is just that, so don't hassle people!)

I am working on getting the Pi Foundation demo board working solely from userspace, but I've not had much spare time to work on it recently. The Auvidea board is a good enough clone of that unreleased board that it works with that code.

[samir_sogay]

Hi,

I have a B101 and when I run EDID on the Pi (while raspivid -t 0) with HDMI Input connected to B101, it gives multiple resolutions of which 720p60 is preferred.
"HDMI:EDID preferred mode remained as CEA (4) 1280x720p @ 60 Hz with pixel clock 74 MHz"
I wonder how to set EDID on TC and shouldn't it already be handing out 1080p30 as it is currently supported?
The maximum working resolution I have till now is 1080p25. If I set 1080p30, the output is distorted.

@Auvidea: Is there anything specific you did on RPi Output to make it work with 1080p30? I have set hdmi_group=1 and hdmi_mode=34 .

Also, it seems recording HDMI video to RPi is flaky. Once raspivid -o test.h264 is run, it will record the video and the same command cannot be rerun after that as it will give the "mmal: No data received from sensor. Check all connections, including the Sunny one on the camera board". Even rebooting the Pi does not solve the issue. Only shutting down and restarting works but after that also it will work only once. It seems that something is not exiting cleanly.

[Orbital6]

Hi,

I have a B101 and when I run EDID on the Pi (while raspivid -t 0) with HDMI Input connected to B101, it gives multiple resolutions of which 720p60 is preferred.
"HDMI:EDID preferred mode remained as CEA (4) 1280x720p @ 60 Hz with pixel clock 74 MHz"
I wonder how to set EDID on TC and shouldn't it already be handing out 1080p30 as it is currently supported?
The maximum working resolution I have till now is 1080p25. If I set 1080p30, the output is distorted.

@Auvidea: Is there anything specific you did on RPi Output to make it work with 1080p30? I have set hdmi_group=1 and hdmi_mode=34 .

Also, it seems recording HDMI video to RPi is flaky. Once raspivid -o test.h264 is run, it will record the video and the same command cannot be rerun after that as it will give the "mmal: No data received from sensor. Check all connections, including the Sunny one on the camera board". Even rebooting the Pi does not solve the issue. Only shutting down and restarting works but after that also it will work only once. It seems that something is not exiting cleanly.

Please read the entire thread. The board is not supported, you're using an old driver developed for a prototype board. And it's not flakey, you need to take control of the toshiba registers through i2c to allow repeat plays.