Schnoogle
Posts: 35
Joined: Sun Feb 11, 2018 4:47 pm

[Partly-Solved] VCHIQ/MMAL Camera access

Fri May 11, 2018 5:15 pm

Hi there

I'm implementing a bare metal environment that allows me to access the camera of the PI. I know it's not that easy, however I'm quite close to get it to work. I've implemented a - let's call it lean - VCHIQ that does not use the IOCTL stuff as linux does. Also I'm not intending to differ between user space and kernel space memory as such. It will be a boxed environment ...

I already get the red light of the camera "glowing" indicating that the camera is active and able to send the image data to my application.
As far as I understand the MMAL implementation for the camera I need to send a buffer to the camera where the image data should be stored. This is done in VCHIQ using a "BULK" transfer/recieve mechanism. There I'm struggeling as my implementation of the VCHIQ does not seem to recieve any valid data in the buffer I'm passing.
As far as I've seen in the Circle and Ultibo source code you guys do have your implementation matching more or less the linux features there. However, one is using the "CREATE_PAGELIST" to pass the buffer the bulk recieve should store the data and one does not. I've no clue what this function does and whether one of you ever had successfully used the features of your VCHIQ implementations to recieve bulk data from a VC component.

I know it's hard to support without any kind of source code but I would like to know if someone got this to work already and might want to share your experience here...

Roughly speeking I'm already able to do this on pseudo code:

Code: Select all

camera_component_create
camera_port_parameter_set(width=64, height=64)
camera_encoding_set(encoding = ENCODING_RGB16, variant = ENCODING_RGB16)
camera_enable
camera_port_enable
camera_send_buffer --> this triggers the video callback....

...

video_buffer_callback(BUFFER* buffer) => the buffer that has been passed to the camera could be accessed here
The recieved buffer does seem to contain valid data but no image frame data.
The recieved flag in the buffer is set to MMAL_BUFFER_HEADER_FLAG_FRAME_END | MMAL_BUFFER_HEADER_FLAG_TRANSMISSION_FAILED but I don't know what is the reason for the failing transmission. Is the any trace/logging I could activate on VC/GPU side to track down the root cause??

I've also seen that the flag in the buffer is set to MMAL_BUFFER_HEADER_FLAG_FRAME_END only, but still no image data.

I'd really appreciate if someone has also tried this and may have some experience to share ...
Looking forward to get the hopefully last small pieces together to access the camera and video streams from my baremetal environment ;)

Thanks in advance
BR Schnoogle
Last edited by Schnoogle on Sat Jun 02, 2018 4:35 pm, edited 2 times in total.

User avatar
Ultibo
Posts: 130
Joined: Wed Sep 30, 2015 10:29 am
Location: Australia
Contact: Website

Re: VCHIQ/MMAL Camera access

Sat May 12, 2018 1:46 am

Hi BR Schnoogle,

Sounds like you are close to getting it working if you have got that far.

Without any other information it is hard to offer much advice but I can tell you that both bulk transmit and receive work in the Ultibo port of VCHIQ, data is able to be received from the camera and we have working ports of both Raspistill and Raspivid.

The handling of the page lists for the bulk transfer is quite a complex part of the protocol considering you have to deal with whole pages, leftover fragments and managing cache coherence, my first thought would be that you just have something slightly wrong in your code but finding it might take some very detailed debugging.
Ultibo.org | Make something amazing
https://ultibo.org

Threads, multi-core, OpenGL, Camera, FAT, NTFS, TCP/IP, USB and more in 3MB with 2 second boot!

Schnoogle
Posts: 35
Joined: Sun Feb 11, 2018 4:47 pm

Re: VCHIQ/MMAL Camera access

Sat May 12, 2018 7:38 pm

Hi Ultibo,

thx for your quick response and your believe that I'm very close ;) I believe this too :D
It's good to know that the Ultibo port of VCHIQ is working fine so I have at least some sort of "reference"...

I'm developing under windows so UART console is my friend as QEMU does not work on Windows for real debugging as far as I know and I guess VCHIQ might not be properly emulated there as well....

However, could you give some insights what this page list is about to provide from a feature point of view ?
As far as I understood, instead of just passing the start address of the buffer that should be filled it passes a list of pointers where each one points to the exact start address of an x sized memory page. Further more there is some offset provided where the real buffer starts based on the first page starting address. Is my understanding correct? May be I've overlooked the point, but is there any other info provided than that? This kind of design looks a bit weird to me not understanding the full impact and advantages of this approach over the one to pass just the buffer start address and a size ;)

Nevertheless, in case you could shed some light on the page list stuff this would be very great. I'll try to double and tripple check my code and see whether I find some obvious or not that obvious errors ;)

Thanks.

BR Schnoogle

User avatar
Ultibo
Posts: 130
Joined: Wed Sep 30, 2015 10:29 am
Location: Australia
Contact: Website

Re: VCHIQ/MMAL Camera access

Sun May 13, 2018 1:14 am

Schnoogle wrote:
Sat May 12, 2018 7:38 pm
I'm developing under windows so UART console is my friend as QEMU does not work on Windows for real debugging as far as I know and I guess VCHIQ might not be properly emulated there as well....
When doing bare metal the UART is always your friend ;) I think it could be a very long time before there is any chance of emulating the VCHIQ.
Schnoogle wrote:
Sat May 12, 2018 7:38 pm
As far as I understood, instead of just passing the start address of the buffer that should be filled it passes a list of pointers where each one points to the exact start address of an x sized memory page. Further more there is some offset provided where the real buffer starts based on the first page starting address. Is my understanding correct? May be I've overlooked the point, but is there any other info provided than that? This kind of design looks a bit weird to me not understanding the full impact and advantages of this approach over the one to pass just the buffer start address and a size ;)
That's the basis of the page list but maybe you are missing some of the details which might be hurting your implementation.

I'll refer to the Linux code to point out some key items, people seem to think we are only Pascal but we can do C/C++ as well.

The PAGELIST_T struct that is passed as part of a bulk transfer provides multiple pieces of information to the GPU.

First it says whether this is a read or a write in the type field (also indicated by the request as well), then it includes the number of full pages in the length field.

A page in the Pi is always 4096 bytes and always begins on a 4096 byte boundary, this is important for the next part of the page list.

The addrs field provides an array containing the start address of each page to be read or written, note that the least significant 12 bits of each address is used to indicate the number of consecutive pages at that address in order to cut down the number of entries in the list.

Now clearly the reason why the VCHIQ uses this page list structure instead of a simple address and length is to allow for mapping and unmapping virtual memory pages, in a bare metal implementation you may just use one to one mapping of physical to virtual memory and so you can just pass the start address of each page but Linux has to map the user process virtual pages to physical pages for passing to the GPU. Even in Ultibo we use a virtual memory space for supporting the contiguous memory block behind the malloc implementation used in the C library (Newlib).

The final bit of the page list structure is the offset field which says how many bytes offset into the first page before the start of the actual data.

But sadly that's not all, there is also the fragment list which is needed if any part of the head or tail of the data does not fill a complete cache line (on VCHIQ the cache line size seems to be hard coded to 32 bytes which actually may not be correct on later CPUs).

Because the process of transferring the data to or from the GPU using the page list requires an invalidate of any cached pages on a bulk read then data which falls outside of a cache line sized block must be copied to and from the fragment buffers in order to prevent accidental damage to other (unrelated) data.

The fragment list is actually created during vchiq_platform_init so if you were working your way through the driver you might easily overlook it until finally reaching the bulk transmit and receive functions where it is used.

I'll leave it there for the moment, this is a very complex interaction between multiple elements so you need to be able to absorb it and clearly picture the process in your mind. Rather than me writing pages of detail trying to explain it maybe you can just ask about any bits that don't make sense.

Good luck.
Ultibo.org | Make something amazing
https://ultibo.org

Threads, multi-core, OpenGL, Camera, FAT, NTFS, TCP/IP, USB and more in 3MB with 2 second boot!

Schnoogle
Posts: 35
Joined: Sun Feb 11, 2018 4:47 pm

Re: VCHIQ/MMAL Camera access

Sun May 13, 2018 10:22 am

Hey Ultibo,

I really ow you more than just a cold beer for your excellent answer and details. I guess I now have a very good starting point to draw the right picture of the process in my mind and I already saw some points I was not aware of and need to adjust in my code ;)

I’ll check my implementation and will let you know in case I come across detailed questions...
For now I’m more than good to go ... thanks a ton!!!

BR,
Schnoogle

User avatar
Ultibo
Posts: 130
Joined: Wed Sep 30, 2015 10:29 am
Location: Australia
Contact: Website

Re: VCHIQ/MMAL Camera access

Sun May 13, 2018 11:21 am

You are welcome Schnoogle, please let us know how you go.

PS. We are from Australia so cold beer is good!!
Ultibo.org | Make something amazing
https://ultibo.org

Threads, multi-core, OpenGL, Camera, FAT, NTFS, TCP/IP, USB and more in 3MB with 2 second boot!

Schnoogle
Posts: 35
Joined: Sun Feb 11, 2018 4:47 pm

Re: VCHIQ/MMAL Camera access

Sun May 27, 2018 6:28 pm

Hey Ultibo,

thanks again for your explanation. After some time off and vacation I was able to continue on this topic ;). With your details I was able to get it working. I'm now able to get the data with the bulk request from the camera :)

Thanks a lot.... I'll send at least a virtual beer from Germany to Australia :)

BR Schnoogle

Schnoogle
Posts: 35
Joined: Sun Feb 11, 2018 4:47 pm

Re: [Partly-Solved] VCHIQ/MMAL Camera access

Sat Jun 02, 2018 4:46 pm

Hi there,

unfortunatley the luck was not long on my side - in case it has anything to do with luck ;)
The camera access and capturing as well as displaying on the screen runs quite well as far as I can tell.
BUT: Only for small capture sizes. I started experimenting with RGB16 and sizes of 32x32 and 64x64. As everything runs very smooth I tried to get larger resolutions but once I reach 128x128 it immedietly stops from working.

What I was able to track down is, that setting the MMAL port format provides the buffer size that need to be allocated for this port to proper work. As soon as this size gets larger than 32768 the function to send this buffer to VCHIQ does not "return" nor does it kick of the follow-on processing of this buffer. As I'm not abel to activate any logs/traces on VC side I've no clue what might be the root cause for this. No error is thrown from VCHIQ. The message is queued and thats it, but VC does not seem to answer to this message....

What I recogniced and found very weird so far, is that for any reason the red LED which usualy indicates the Pi is running starts blinking and than turns off. The PI is still running I guess as another LED conected to a GPIO is still bright.
I've seen the green LED blinking during my arky tries to communicate with the VC which eem to indicate that the VC complains on my message send, but the red LED ??? I'm some how lost here... Anyone out there who have an Idea what the root cause might be ?

I'm using startup_x.elf and fixup_x.dat on my sd card. If set gpu_mem to 256 in config.txt, however the mailbox reports GPU does only hav 128MB assigned but I guess this should be enough to capture 256x256 image from the camera ;)

BR,
Schnoogle

LdB
Posts: 751
Joined: Wed Dec 07, 2016 2:29 pm

Re: [Partly-Solved] VCHIQ/MMAL Camera access

Sat Jun 02, 2018 5:19 pm

Not sure but that sounds like the same problem I ran into when bashing it baremetal.

The GPU defaults to only a small memory of 64Mb and I kept running out of memory on large images.
It was only later I found on linux they tell you that you have to run gpu_mem=128 when using the camera I started making ground.

It was then I ran into the next problem the L2 cache on the GPU can only be on if you have the L2 cache
on the ARM running. The GPU cache L2 starts on by default you turn it off with "disable_l2cache".
As I didn't initially have the L2_cache up on the ARM the transfer was crashing and so I had to turn it off on the GPU.
Only later when I got the L2 cache up on the ARM was I able to leave it enabled on the GPU.

The weird part was I didn't see the L2 cache issue on small images only when they got bigger and the GPU would randomly crash.

Schnoogle
Posts: 35
Joined: Sun Feb 11, 2018 4:47 pm

Re: [Partly-Solved] VCHIQ/MMAL Camera access

Sat Jun 02, 2018 8:09 pm

Hi Ldb,

thx for your quick response. Unfortunately this does not solve my issue.
My config file now looks like this:

Code: Select all

start_file=start_x.elf
start_fixup=fixup_x.dat
disable_l2cache=1
gpu_mem=256
Is there any VC trace option one could activiate in the config.txt file ?

BR,
Schnoogle

User avatar
Ultibo
Posts: 130
Joined: Wed Sep 30, 2015 10:29 am
Location: Australia
Contact: Website

Re: [Partly-Solved] VCHIQ/MMAL Camera access

Sun Jun 03, 2018 1:07 am

Hi again Schnoogle,

So close and yet so far!

I just retested the values with our implementation, we can happily capture a 3280 x 2464 image from the camera so your 128 x 128 should be easily within limits.

I can also confirm that to use the camera you must have start_x.elf (and fixup_x.dat) with the settings start_x=1 and gpu_mem=128 or higher in config.txt.

As a side note if you are setting gpu_mem=256 and not seeing that reported by the mailbox functions you might want to sort that out first, it could be a symptom of a larger issue.
Schnoogle wrote:
Sat Jun 02, 2018 4:46 pm
What I recogniced and found very weird so far, is that for any reason the red LED which usualy indicates the Pi is running starts blinking and than turns off. The PI is still running I guess as another LED conected to a GPIO is still bright.
I've seen the green LED blinking during my arky tries to communicate with the VC which eem to indicate that the VC complains on my message send, but the red LED ??? I'm some how lost here... Anyone out there who have an Idea what the root cause might be ?
I don't think I have ever seen the red LED blink in the way you describe but it is possible you are crashing the VCHIQ protocol and the VC4 is reporting that via the blinking LED.

I'm assuming (although you never really said) that you have a reasonably full functioning environment to work with, eg you have the MMU and caches enabled, you have some form of tasking and have constructs like semaphores and mutexes available to support the needs of MMAL and the VCHIQ.

As far as debug, the only bit of the entire Userland that has not been released seems to be the source for the vcdbg tool, you can make the relevant requests to the VC4 but there is no way to decode the results. It would appear that the code might give away internal information about the VC4 that is not intended to be public.
Ultibo.org | Make something amazing
https://ultibo.org

Threads, multi-core, OpenGL, Camera, FAT, NTFS, TCP/IP, USB and more in 3MB with 2 second boot!

LdB
Posts: 751
Joined: Wed Dec 07, 2016 2:29 pm

Re: [Partly-Solved] VCHIQ/MMAL Camera access

Sun Jun 03, 2018 3:15 am

Agree with above first work out why your memory isn't reporting properly.

If you still have problems the you need to look at registers at 0x3F801000 and dump them
If you don't have the registers defined yet download the document https://docs.broadcom.com/docs/12358546
In brcm_usrlib/dag/vmcsx/vcinclude/hardware_vc4.h you will find lots of useful information

That is the closest you will be able to get to see what is going on.
Anyhow this is the register list

Code: Select all

#if defined(__BCM2708A0__)
   #define UNICAM_CTRL( x )    UNICAM_REG( x, 0x000 )
   #define UNICAM_STA( x )     UNICAM_REG( x, 0x004 )
   #define UNICAM_ANA( x )     UNICAM_REG( x, 0x008 )
   #define UNICAM_PRI( x )     UNICAM_REG( x, 0x00c )
   #define UNICAM_CLK( x )     UNICAM_REG( x, 0x010 )
   #define UNICAM_DAT0( x )    UNICAM_REG( x, 0x014 )
   #define UNICAM_DAT1( x )    UNICAM_REG( x, 0x018 )
   #define UNICAM_DAT2( x )    UNICAM_REG( x, 0x01c )
   #define UNICAM_DAT3( x )    UNICAM_REG( x, 0x020 )
   #define UNICAM_CMP0( x )    UNICAM_REG( x, 0x024 )
   #define UNICAM_CMP1( x )    UNICAM_REG( x, 0x028 )
   #define UNICAM_CAP0( x )    UNICAM_REG( x, 0x02c )
   #define UNICAM_CAP1( x )    UNICAM_REG( x, 0x030 )
   #define UNICAM_DBG0( x )    UNICAM_REG( x, 0x0f0 )
   #define UNICAM_DBG1( x )    UNICAM_REG( x, 0x0f4 )
   #define UNICAM_DBG2( x )    UNICAM_REG( x, 0x0f8 )
   #define UNICAM_ICTL( x )    UNICAM_REG( x, 0x100 )
   #define UNICAM_ISTA( x )    UNICAM_REG( x, 0x104 )
   #define UNICAM_IDI( x )     UNICAM_REG( x, 0x108 )
   #define UNICAM_IPIPE( x )   UNICAM_REG( x, 0x10c )
   #define UNICAM_IBSA( x )    UNICAM_REG( x, 0x110 )
   #define UNICAM_IBEA( x )    UNICAM_REG( x, 0x114 )
   #define UNICAM_IBLS( x )    UNICAM_REG( x, 0x118 )
   #define UNICAM_IBWP( x )    UNICAM_REG( x, 0x11c )
   #define UNICAM_IHWIN( x )   UNICAM_REG( x, 0x120 )
   #define UNICAM_IHSTA( x )   UNICAM_REG( x, 0x124 )
   #define UNICAM_IVWIN( x )   UNICAM_REG( x, 0x128 )
   #define UNICAM_IVSTA( x )   UNICAM_REG( x, 0x12c )
   #define UNICAM_DCS( x )     UNICAM_REG( x, 0x200 )
   #define UNICAM_DBSA( x )    UNICAM_REG( x, 0x204 )
   #define UNICAM_DBEA( x )    UNICAM_REG( x, 0x208 )
   #define UNICAM_DBWP( x )    UNICAM_REG( x, 0x20c )
#else
   #define UNICAM_CTRL( x )    UNICAM_REG( x, 0x000 )
   #define UNICAM_STA( x )     UNICAM_REG( x, 0x004 )
   #define UNICAM_ANA( x )     UNICAM_REG( x, 0x008 )
   #define UNICAM_PRI( x )     UNICAM_REG( x, 0x00c )
   #define UNICAM_CLK( x )     UNICAM_REG( x, 0x010 )
   #define UNICAM_CLT( x )     UNICAM_REG( x, 0x014 )
   #define UNICAM_DAT0( x )    UNICAM_REG( x, 0x018 )
   #define UNICAM_DAT1( x )    UNICAM_REG( x, 0x01c )
   #define UNICAM_DAT2( x )    UNICAM_REG( x, 0x020 )
   #define UNICAM_DAT3( x )    UNICAM_REG( x, 0x024 )
   #define UNICAM_DLT( x )     UNICAM_REG( x, 0x028 )
   #define UNICAM_CMP0( x )    UNICAM_REG( x, 0x02c )
   #define UNICAM_CMP1( x )    UNICAM_REG( x, 0x030 )
   #define UNICAM_CAP0( x )    UNICAM_REG( x, 0x034 )
   #define UNICAM_CAP1( x )    UNICAM_REG( x, 0x038 )
   #define UNICAM_ICTL( x )    UNICAM_REG( x, 0x100 )
   #define UNICAM_ISTA( x )    UNICAM_REG( x, 0x104 )
   #define UNICAM_IDI0( x )    UNICAM_REG( x, 0x108 )
   #define UNICAM_IPIPE( x )   UNICAM_REG( x, 0x10c )
   #define UNICAM_IBSA0( x )   UNICAM_REG( x, 0x110 )
   #define UNICAM_IBEA0( x )   UNICAM_REG( x, 0x114 )
   #define UNICAM_IBLS( x )    UNICAM_REG( x, 0x118 )
   #define UNICAM_IBWP( x )    UNICAM_REG( x, 0x11c )
   #define UNICAM_IHWIN( x )   UNICAM_REG( x, 0x120 )
   #define UNICAM_IHSTA( x )   UNICAM_REG( x, 0x124 )
   #define UNICAM_IVWIN( x )   UNICAM_REG( x, 0x128 )
   #define UNICAM_IVSTA( x )   UNICAM_REG( x, 0x12c )
   #define UNICAM_ICC( x )     UNICAM_REG( x, 0x130 )
   #define UNICAM_ICS( x )     UNICAM_REG( x, 0x134 )
   #define UNICAM_IDC( x )     UNICAM_REG( x, 0x138 )
   #define UNICAM_IDPO( x )    UNICAM_REG( x, 0x13c )
   #define UNICAM_IDCA( x )    UNICAM_REG( x, 0x140 )
   #define UNICAM_IDCD( x )    UNICAM_REG( x, 0x144 )
   #define UNICAM_IDS( x )     UNICAM_REG( x, 0x148 )
   #define UNICAM_DCS( x )     UNICAM_REG( x, 0x200 )
   #define UNICAM_DBSA0( x )   UNICAM_REG( x, 0x204 )
   #define UNICAM_DBEA0( x )   UNICAM_REG( x, 0x208 )
   #define UNICAM_DBWP( x )    UNICAM_REG( x, 0x20c )
   #define UNICAM_DBCTL( x )   UNICAM_REG( x, 0x300 )
   #define UNICAM_IBSA1( x )   UNICAM_REG( x, 0x304 )
   #define UNICAM_IBEA1( x )   UNICAM_REG( x, 0x308 )
   #define UNICAM_IDI1( x )    UNICAM_REG( x, 0x30c )
   #define UNICAM_DBSA1( x )   UNICAM_REG( x, 0x310 )
   #define UNICAM_DBEA1( x )   UNICAM_REG( x, 0x314 )
   #define UNICAM_MISC( x )    UNICAM_REG( x, 0x400 )
#endif

Schnoogle
Posts: 35
Joined: Sun Feb 11, 2018 4:47 pm

Re: [Partly-Solved] VCHIQ/MMAL Camera access

Sun Jun 03, 2018 9:34 am

Thanks ultibo and ldb,

fortunately I'v now the GPU ram reported correctly. :)
The config.txt looks like this now:

Code: Select all

start_x=1
gpu_mem=128
disable_l2cache=1
enable_l2cache=0
But still the same issue.
I've the MMU setup with L1 caching active. I've no clue how to get L2 caches setup but the disable_l2cache switch in the confog.txt should deactivate it in GPU anyway if I understood it correct. I also have spin-locks and semaphores implemented to ensure cross core communication and locking. My VCHIQ implementation spreads his tasks accross 2 cores, each one running a specific single threaded process and handling parts of the protocoll which in the original are spread accross several threads. As this is working at least for small capture sizes quite stable I assume there nothing much wrong mith my implementation. Also I does not get stuck in my code, it's really the VC that does not respond to the message that I'm passing.

I haven't seen the red led blinking any longer - hm. may be related to insufficient battery my berry is usually running. Since I'm using connected power line the blinking is gone...

I'l check the registers you have provided Ldb and see if this points into any direction.

Thx for your support :)
BR,
Schnoogle

Schnoogle
Posts: 35
Joined: Sun Feb 11, 2018 4:47 pm

Re: [Partly-Solved] VCHIQ/MMAL Camera access

Sun Jun 03, 2018 10:56 am

Hey Ldb,

so I dumped the UNICAM registers during the camera setup process everytime a VCHIQ message has been queued - befor the VC responded...
As I can't ell much on the contents of the registers I'm not sure what they meen.
However, I've comared them when setting up camera capture with 128x100 (which is working) and 128x128(which is not working)....
As soon as port parameter MMAL_PARAMETER_CAPTURE is about to be set the values of those registers start to differ, but only a few of them.

PORT_PARAMETER_SET(MMAL_PARAMETER_CAPTURE)

Code: Select all

WORKING:
c:0-CAM_STA=0x3
c:0-CAM_DAT0=0xC0000005
c:0-CAM_DAT1=0xC0000005
c:0-CAM_IBWP=0xFE9F7F80

NOT-WORKING:
c:0-CAM_STA=0x2
c:0-CAM_DAT0=0x6000005
c:0-CAM_DAT1=0x6000005
c:0-CAM_IBWP=0xFEA656E0
ENABLE_PORT

Code: Select all

WORKING:
c:0-CAM_DAT1=0xC0000005
c:0-CAM_IBWP=0xFEADD480

NOT-WORKING:
c:0-CAM_DAT1=0x6000005
c:0-CAM_IBWP=0xFE9B02B0
SEND_BUFFER

Code: Select all

WORKING(buffer-size=28672):
c:0-CAM_DAT1=0xC0000005
c:0-CAM_IBSA0=0xFE546000
c:0-CAM_IBEA0=0xFE6C9480
c:0-CAM_IBWP=0xFEADD480

NOT-WORKING(buffer-size=32768)
c:0-CAM_DAT1=0x6000005
c:0-CAM_IBSA0=0xFE95A000
c:0-CAM_IBEA0=0xFEADD480
c:0-CAM_IBWP=0xFEA0D570
HOwever, those values are somehow a bit "unstable" and does change slightly on each call on the working as well as on the not-working side, but usually those are the registers that differ for what ever reason...

Ayn further hint is much appreciated...
Thx in advance
BR,
Schnoogle

LdB
Posts: 751
Joined: Wed Dec 07, 2016 2:29 pm

Re: [Partly-Solved] VCHIQ/MMAL Camera access

Sun Jun 03, 2018 12:42 pm

CAM_IBSA0 = CAM IMAGE BUFFER 0 START ADDRESS
CAM_IBEA0 = CAM IMAGE BUFFER 0 END ADDRESS
CAM_IBLS0 = CAM IMAGE BUFFER 0 LINE SIZE
CAM-_IBWP0 = CAM IMAGE BUFFER 0 WRITE POINTER

Double check your image buffer memory size allocation there is something weird going on with the image buffer size (hint subtract the start from the end in both results .. did you forget to adjust the image buffer size????) and make dam sure it is align 16.

Schnoogle
Posts: 35
Joined: Sun Feb 11, 2018 4:47 pm

Re: [Partly-Solved] VCHIQ/MMAL Camera access

Sun Jun 03, 2018 9:12 pm

Hi LdB,

thx for your tips. However, looking at the working and not working examples the difference between CAM_ISBE0 and CAM_ISBA1 is 1586304 :O :O
When switching the image capture size I'll completely re-boot the Pi so there should not be any configuration from a previous run left...

I'm not sure about the part regarding image buffer size calculation. Well I do send a mmal_port_format_commit with the desired video capture width and height to the camera output port. This is passing a MMAL_WORKER_PORT_INFO_SET message to the VC.
The reply of the VC is giving the minimum and recommended buffer size to be used with this port:
amera output buffer:

Code: Select all

num=1, size=32768, recom-num=1, recom-size=32768, min-num=1, min-size=32768, align-min=0
I take this buffer size which is for an 128x100 capture size given as 28672 and for the 128x128 capture size as 32768. With this size I'm creating the payload buffer (inside a port_pool) which is allocated at a 16bit aligned address (in this case at 0x208FB0).

However, without passing this buffer to the camera yet, just setting the parameter to enable capturing with message MMAL_WORKER_PORT_PARAMETER_SET the CAM_IBSA0 and CAM_IBEA0 are getting this weird values and are not changing when passing the buffer to the VC with the message MMAL_WORKER_BUFFER_FROM_HOST.
Also changing the order by first passing a buffer to the output port and then enabling the capture as well as the port does not change anything.
It looks like somthing is missing or overflowing with 32768 as buffer size which is a boundry of a "short int" value... so well may be I overlooked something? I do not have the feeling that I'm influencing the CAM_IBSA0 and CAM_IBEA0 values as theire difference is in both cases the same :/

BR,
Schnoogle

User avatar
Ultibo
Posts: 130
Joined: Wed Sep 30, 2015 10:29 am
Location: Australia
Contact: Website

Re: [Partly-Solved] VCHIQ/MMAL Camera access

Mon Jun 04, 2018 2:27 am

Hey Schnoogle,
Schnoogle wrote:
Sun Jun 03, 2018 9:34 am
I've the MMU setup with L1 caching active. I've no clue how to get L2 caches setup but the disable_l2cache switch in the confog.txt should deactivate it in GPU anyway if I understood it correct.
Based on that I'd suggest that the disable_l2cache/enable_l2cache setting is at best irrelevant and at worst possibly detrimental.

The original Pi (A/B/A+/B+/Zero) shares the L2 cache between the CPU and the GPU, on the Pi 2 and 3 the 4 CPUs have their own L2 cache that they share between them but it is not shared with the GPU anymore. That is why there are differences between the Pi 1 and Pi 2/3 about the way cached memory is handled when communicating with the GPU.

If you just do your cache clean and/or invalidate operations at the appropriate places and supply correct bus addresses to the GPU then everything should all work correctly without trying to mess with cache settings.
Schnoogle wrote:
Sun Jun 03, 2018 9:34 am
As this is working at least for small capture sizes quite stable I assume there nothing much wrong mith my implementation. Also I does not get stuck in my code, it's really the VC that does not respond to the message that I'm passing.
I'd probably look at it the other way, since the VC4 implementation of the VCHIQ protocol is known to work then if it doesn't respond to your request it is more likely to be because you have something (possibly subtle) that is wrong on your side.

The trickiest part is to break it down to the point where you can see the problem.
Ultibo.org | Make something amazing
https://ultibo.org

Threads, multi-core, OpenGL, Camera, FAT, NTFS, TCP/IP, USB and more in 3MB with 2 second boot!

LdB
Posts: 751
Joined: Wed Dec 07, 2016 2:29 pm

Re: [Partly-Solved] VCHIQ/MMAL Camera access

Mon Jun 04, 2018 3:57 am

I am going to add a caveat to all that comes next that I am assuming the VCHIQ/MMAL image buffer passing works the same as the baremetal interface is and it doesn't internally buffer. I only pulled VCHIQ/MMAL code apart enough to find the register addresses and where the status and semaphore flags were.

That in mind when doing this baremetal the image buffer I required was always (width + 480) * height which is whatever colour mode it defaults to (I never bothered to change it or even investigate changing it as I was never that interested).

So 128*100 I would need 60,800 bytes and 128x128 I would require 77,824 bytes

To be honest when I first started out I just allocated a 512Kb or 1MB chunk as I didn't know the formula, I worked the formula at by looking at the buffer register data. I have no idea why that is the formula it simply comes from looking at the buffer use. You might want to consider doing something along that line if it's possible :-)

The interesting number for me for your colour mode would be the line size from the register
CAM_IBLS0 = CAM IMAGE BUFFER 0 LINE SIZE

Now I could be way off track here and I don't want to mislead you because this is uncharted territory for me but can you tell me the mode and what your formula for memory size is?

I can also suggest one other thing which is use the GPU memory allocate function (mailbox tag: 0x00040001) for your image buffer or even pass it a raw framebuffer for your image buffer (that is the image will go direct to screen). The reason being that the GPU owns that memory not the ARM and it will have sorted all the cache out to it's requirements (AKA the l2cache disable thing). If you do the mailbox allocate function make sure you ask for more than you think you need just to be safe.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5128
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: [Partly-Solved] VCHIQ/MMAL Camera access

Mon Jun 04, 2018 10:49 am

Don't get confused between the resolution off the sensor (as you'll see reported via the CAM1_xxx registers) and the output resolution.
https://picamera.readthedocs.io/en/latest/fov.html is a good description of how the processing pipe is working. The ISP hardware block will be cropping and resizing a fixed set of resolutions off the sensor to whatever the output resolution/format is that is requested.

If you really want to dig into the Unicam registers (CAM1_xx), then have a look at https://github.com/raspberrypi/linux/pull/2513, but that removes the ISP from the pipeline so you have no conversion from Bayer to YUV/RGB, nor AE/AGC/AWB control loops.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

LdB
Posts: 751
Joined: Wed Dec 07, 2016 2:29 pm

Re: [Partly-Solved] VCHIQ/MMAL Camera access

Mon Jun 04, 2018 3:25 pm

Oh that is great documentation 6by9 now, it is so much improved. However it is probably more interesting to me than him because it explains some things I couldn't work out in Baremetal, where much of that happens behind the scenes via the VHCIQ. Yes I am seeing a different part of the pipeline flow than he is and that makes sense now. I might dust off my project code again if I get time now you have shown me that because I can probably go a lot further :-)

It's amazing how just seeing how the pipeline actually flows makes it so much easier to program and the synchronization requirements become obvious and not just some messy piece of code you are trying to reverse engineer and understand.

Schnoogle
Posts: 35
Joined: Sun Feb 11, 2018 4:47 pm

Re: [Partly-Solved] VCHIQ/MMAL Camera access

Mon Jun 04, 2018 5:25 pm

Hi 6by9,

thanls for those links, they give me also much of an insight into what's going on behind the scenes here.

Hey LdB,

well I do hope that there was no miss-understanding but I assume that you are talking about a more "real-baremetal" approach to accessing the camera? I do have an own VCHIQ/MMAL implementation running which is lightweighted in a way that it does not have the IOCTL stuff in the middle as I'm not having any user or kernel mode in my baremetal implementation....

So I'm dealing with VCHIQ messages send to communicate with the camera (as in the doc shared by 6by9)...
When doing so I'm using VCHIQ messages to set the output size of the camera component. Taking image 6.1.7 of the refernced doku (https://picamera.readthedocs.io/en/latest/fov.html) this I guess would be setting up what the ISP is doing as it's result (step 3).

When I pass an image buffer via VCHIQ to the camera I guess the step (4) is initiated and the GPU copies the image data from GPU memory space to ARM memory space.

As I do not use a splitter component after the video-output port I'm setting the Video-Output-Port to MMAL_ENCODING_RGB16, width=128, height=128, frame_rate.num = 0, frame_rate.denum=1.

From my point of view everything seem to be fine as setting the portformat returns SUCCESS from MMAL/VCHIQ and the port parameters are updated accordingly. Once the format has been set on the port thi gives us the desired buffer_number and buffer_size this port is expecting to be needed...

I'm finally passing a payloadbuffer to this port of exactly this size.
If I change the buffer size to not being the one given by the port format call response and set it to a fixed high number, eg. 131072 (128kB), it does not effect the behavior. When I set the format to 128x100 captures it works, and if I set 128x128 capture it does not work :(

The assumtion is, that this is not only the buffer size passed to the camera/VC using a VCHIQ message....

So either I'm dump and not understanding the obvious stuff here or it is this tiny bit/piece that does not harm until the size of the capture messes up the tiny thing and it becomes a big one that the VC cannot longer handle :/

BR,
Schnoogle

LdB
Posts: 751
Joined: Wed Dec 07, 2016 2:29 pm

Re: [Partly-Solved] VCHIQ/MMAL Camera access

Mon Jun 04, 2018 6:36 pm

Yes from the pipeline flow I would conclude the same thing. I only looked briefly at the linux files but there does seem to be some internal size stuff done by the VHCIQ system so I think you are right on that regard.

So you are down to a subtle implementation issue or still a memory issue.

I still think to quickly eliminate the memory issue allocate the buffer in GPU memory its a two second change as you must have mailbox up and running.

It's just tag 0x0003000c with 3 values .1.) memory Size you want, 2.) alignment is 0x1000 and 3.) flags MEM_FLAG_COHERENT | MEM_FLAG_ZERO (the values for flags is on the document). It returns a handle you hold that for next call.

So use the handle to lock the memory (tag 0x0003000d) you provide the handle it gives you back the locked GPU bus address that is the value you pass via VCHIQ as your buffer. You convert the value back to an ARM address the same as for a frame buffer address (AND NOT 0xC0000000) or whatever code you have that does it if you want to read the image. Although you can tell if it works or not by simply checking your status in your case. So really all you end up with is the buffer with a strange high address like the FrameBuffer rather than your memory pool address down with an ARM address.

Again why I am worried is that was the only way I could push the VC4 around with GL pipeline exchanges and camera when I had the MMU down. If I tried to do it with a buffer in the arm address range she would blow up on me and crash and burn. From memory I think I narrowed it down to any call that did a DMA exchange from the GPU address range to the ARM address range.

If that doesn't work your VCHIQ shim is bugged because the GPU has no issues with it's memory it owns.

If it helps here are my GPU alloc and lock code they are trivial

Code: Select all

uint32_t V3D_mem_alloc (uint32_t size, uint32_t align, uint32_t flags)
{
	uint32_t buffer[6];
	if (mailbox_tag_message(&buffer[0], 6,
		MAILBOX_TAG_ALLOCATE_MEMORY, 12, 12, size, align, flags)) {
		return buffer[3];
	}
	return 0;
}

uint32_t V3D_mem_lock (uint32_t handle)
{
	uint32_t buffer[4];
	if (mailbox_tag_message(&buffer[0], 4,
		MAILBOX_TAG_LOCK_MEMORY, 4, 4, handle)) {
		return buffer[3];
	}
	return 0;
}

Schnoogle
Posts: 35
Joined: Sun Feb 11, 2018 4:47 pm

Re: [Partly-Solved] VCHIQ/MMAL Camera access

Mon Jun 04, 2018 7:52 pm

Hi LdB,

thx for constantly supporting me here :)...

I tried your recommendation. The memory for the payload is now allocated on GPU side. The result is still the same. For 128x100 capture it works, for 128x128 it doesn't. The payload buffer is allocated at 0x3E3CD000 and the GPU memory is reported to start at 0x38000000. Is this an indication that something is buggy in the firmware/VCHIQ on GPU side?

Thx.

BR,
Schnoogle

LdB
Posts: 751
Joined: Wed Dec 07, 2016 2:29 pm

Re: [Partly-Solved] VCHIQ/MMAL Camera access

Tue Jun 05, 2018 1:15 am

Unfortunately I think it means there is something wrong with your VCHIQ bulk transfer implementation. I suspect you will need to do some careful debugging and dumps of values at points along the code. It must be subtle because it's working in one case and not another but as you were told there are some tricky parts you need to check.

I strongly doubt it but I guess if you really want to check for a GPU bug then load up a linux SD card then write a quick test program to test it. I think I saw a simple test program on 6by9's github or just search for a simple one. Alternatively use the one in Pi userland but that is a bit over the top.

Using the information from 6by9 I just started trying to set up real baremetal example, but work is busy and so it's matter of finding time.

Schnoogle
Posts: 35
Joined: Sun Feb 11, 2018 4:47 pm

Re: [Partly-Solved] VCHIQ/MMAL Camera access

Tue Jun 05, 2018 6:03 pm

Hey LdB,

thanks for your efforts. I'll try to see if I can find the root cause. The thing is, that the VCHIQ bulk transfer isn't even initiated. It stopps already when calling the VCHIQ interface with the message: MMAL_WORKER_BUFFER_FROM_HOST.
This would usually trigger the VCHIQ callback which is forwarded into MMAL with reason VCHIQ_MESSAGE_AVAILABLE that does check the returned messageID to be MMAL_WORKER_BUFFER_TO_HOST and than triggers the VCHIQ_BULK_RECIEVE. But this initial callback from VCHIQ, once the buffer has been passed is not triggered. So I would assume this must be before any bulk recieve is triggered or any tricky parts of the bulk recieve come into play, right?

BR,
Schnoogle

Return to “Bare metal, Assembly language”

Who is online

Users browsing this forum: No registered users and 4 guests