User avatar
dozencrows
Posts: 172
Joined: Sat Aug 04, 2012 6:02 pm

mmal port callbacks on JPEG component issue

Mon Jun 30, 2014 4:01 pm

Some users of my app (mmal fork of motion) have reported a problem when using the latest Raspbian+firmware (see http://www.raspberrypi.org/forums/viewt ... 31#p570931 and http://www.raspberrypi.org/forums/viewt ... 87#p567587).

I'm a little stumped at the moment, so was wondering if anyone else with mmal programming experience could comment on the following? Or if anyone knows about recent mmal, kernel or Pi firmware changes that might affect my app? Any info would be most gratefully received.

The mmal configuration that shows the problem is this:

Code: Select all

camera: [video port output] -> splitter: [output 0] -> JPEG encoder: [output] -> callback
                                         [output 1] -> resize:       [output] -> callback 
Each port that connects to a callback is associated with a buffer pool of the size recommended by the port, and an associated queue. All the callbacks do when called is to put the indicated buffer into the associated queue. The application polls these queues to retrieve each buffer, extracts the contents, releases the buffer and sends it back to the port in the usual fashion.

What I see in the application are callbacks for both the resize and JPEG encoder happening, but after a few iterations the app stops getting callbacks for the JPEG encoder port. Because of the way the app is written, it then hangs waiting for buffers to appear in that queue (there's a separate timeout watchdog running elsewhere that eventually wakes up to make the app fail gracefully). According to the logging in my code, buffer handling has proceeded as expected (i.e. all buffers returned to ports, no leaks).

If the app is shutdown after this point, it runs through code to shut down mmal (disable ports, disable and destroy components, etc) and at this point I suddenly get callbacks for buffers from the JPEG encoder - but it's too late by then.

On earlier versions of Raspbian/Pi firmware, this model works absolutely fine (e.g. Linux version 3.6.11+ #557 PREEMPT Wed Oct 2 18:49:09 BST 2013).

The problem seems to be related to the JPEG encoder component - if the mmal configuration is modified to remove it and just attach a callback to output 0 of the splitter, it works fine on the latest Raspbian & firmware.

BoehserWolf
Posts: 44
Joined: Thu Jun 12, 2014 2:53 pm

Re: mmal port callbacks on JPEG component issue

Tue Jul 01, 2014 5:08 pm

Sounds pretty much the same than the problem I'm facing:
http://www.raspberrypi.org/forums/viewt ... 76#p569876

Regards
Ralf

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 26699
Joined: Sat Jul 30, 2011 7:41 pm

Re: mmal port callbacks on JPEG component issue

Tue Jul 01, 2014 9:44 pm

If I have time and remember I'll take a look to see what has changed in the component - I wasn't aware of any changes though.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

User avatar
dozencrows
Posts: 172
Joined: Sat Aug 04, 2012 6:02 pm

Re: mmal port callbacks on JPEG component issue

Tue Jul 08, 2014 6:54 pm

lowflyer has done some detective work, and isolated the appearance of the issue to a firmware update where the image encoder component was changed - see http://www.raspberrypi.org/forums/viewt ... 06#p575006

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9012
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: mmal port callbacks on JPEG component issue

Wed Jul 09, 2014 12:09 pm

Thanks for the detective work.

A couple of questions then:
- What MMAL_ENCODING_xxx are you using between your splitter and image_encode? I'm guessing I420 as OPAQUE is unlikely to work.
- What resolution is your encoding, and do all resolutions fail, or just some?

The change was to do the correct things when the width or height weren't multiples of the macroblock size (16), and in particular slightly changed the behaviour for height.
There should be no difference if the height is a multiple of 16.

Any chance of a simple test case? Things are rather frantic here at the moment, so I'm unlikely to find the time to write a test case but could spend half an hour on debugging with an existing one.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
dozencrows
Posts: 172
Joined: Sat Aug 04, 2012 6:02 pm

Re: mmal port callbacks on JPEG component issue

Wed Jul 09, 2014 1:42 pm

Very many thanks, 6by9!

I can confirm the encoding used throughout the MMAL component path is MMAL_ENCODING_I420.

I don't have info on the variety of resolutions which cause the problem, but it definitely occurred on 1024 x 576. My camera setup code always rounds up input resolution values to multiples of 16.

I can create a simple test case, but it may take some days before I can deliver that - for similar reasons to yourself ;) .

Alternatively, I can provide info on how to build the actual app itself (not at all tricky or time consuming) plus a configuration file that reproduces the issue. The source and build instructions are available here: https://github.com/dozencrows/motion/tree/mmal-test

Let me know which you'd prefer.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9012
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: mmal port callbacks on JPEG component issue

Wed Jul 09, 2014 3:59 pm

@dozencrows: I'll download and build your mmal-test tree if you could provide a configuration file that triggers the failure.
1024x576 seems most innocuous, so I'm still quite surprised by this.

If I remember the history correctly on this, I originally fixed an issue with replicating the border pixels across when the width/height wasn't a multiple of the macroblock size to avoid odd encoding artifacts. That broke XBMC encoding thumbnails IIRC, and Dom fixed that up but in a way that broke MMAL opaque buffers. The last commit was to hopefully fix both, but that is the one you're saying breaks your use case! Hopefully running the code will make it obvious.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

lowflyer
Posts: 78
Joined: Sat Jun 01, 2013 2:27 pm

Re: mmal port callbacks on JPEG component issue

Wed Jul 09, 2014 4:23 pm

@dozencrows and @6by9

Here is the config file I used in my tests. I experimented with a few widths and heights and never found any that worked.

https://www.dropbox.com/s/b6fkapen0wqci ... alcam.conf

So in my setup I ran

Code: Select all

/home/pi/motion/motion -c /home/pi/motion/lowf-motion-mmalcam.conf
Hope this helps!

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9012
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: mmal port callbacks on JPEG component issue

Wed Jul 09, 2014 4:48 pm

Sorry, our IT department in their infinite wisdom block access to dropbox. I can try to remember to pick it up from home later, or could you post it elsewhere?

If necessary, email it direct to me (edit: remove email details).
Thanks.
Last edited by 6by9 on Thu Jul 10, 2014 7:52 am, edited 1 time in total.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

lowflyer
Posts: 78
Joined: Sat Jun 01, 2013 2:27 pm

Re: mmal port callbacks on JPEG component issue

Thu Jul 10, 2014 5:49 am

@6by9 Direct mail to you last night as requested.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9012
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: mmal port callbacks on JPEG component issue

Thu Jul 10, 2014 5:25 pm

Thanks both lowflyer and dozencrows for your detailed information.

Having run with lowflyer's config, I can see the lockup and the reason was fairly obvious with the debugger (a line count was being reset too late, so we got incorrect image buffer heights being processed).

I've made what I hope is the appropriate change and pushed a test start.elf to my github account at https://github.com/6by9/RPiTest/blob/ma ... /start.elf. It is misnamed and should really be start_x.elf as it includes all the extra components for camera etc (on my setup I always just push the full thing to both start.elf and start_x.elf as it removes the unknown!).

I've had that running for about 10 minutes now, whereas it was stalling within about 30secs before. Do you want to give it a quick test before I declare this as the correct fix and push it for official release?
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

lowflyer
Posts: 78
Joined: Sat Jun 01, 2013 2:27 pm

Re: mmal port callbacks on JPEG component issue

Thu Jul 10, 2014 7:03 pm

@6by9 WOW!!! You are indeed a very wonderful human being!

Replacing start_x.elf with your updated version works for me with both the first broken rpi_update and also the most up to date rpi-update.

A very big THANK YOU :)

User avatar
dozencrows
Posts: 172
Joined: Sat Aug 04, 2012 6:02 pm

Re: mmal port callbacks on JPEG component issue

Thu Jul 10, 2014 9:35 pm

Fantastic - thank you very much, 6by9! And thank you @lowflyer for the detective legwork!

I'm going to give it a test, but won't get the chance to do so until Sunday. So if you're happy with lowflyer's results...

User avatar
jbeale
Posts: 3683
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: mmal port callbacks on JPEG component issue

Fri Jul 11, 2014 12:26 am

For what it's worth- after having done:

Code: Select all

cd /boot
sudo mv start_x.elf OLD_start_x.elf_OLD
sudo wget https://github.com/6by9/RPiTest/blob/master/motion/start.elf?raw=true -O start_x.elf
sudo reboot
I can confirm that dozencrows' updated motion-mmal now continues to work, even when 'mmalcam_secondary_buffer_jpeg' is not set to zero in the motion config file, which bug had stopped me up until now. That is the bug mentioned here:
http://www.raspberrypi.org/forums/viewt ... 25#p561020

so the start_x.elf fix looks good here!

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9012
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: mmal port callbacks on JPEG component issue

Fri Jul 11, 2014 8:09 am

Thanks lowflyer and jbeale for the testing. Glad it's working for you.

I've just looked again at the code before pushing and there's now a slight nasty when using opaque mode that needs to be fixed before I can submit. It should work under most circumstances on Pi, but would be better to have a clean solution. Probably won't get done today, but will try to get to it on Monday and it'll make it's way into the official release at some point.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

lowflyer
Posts: 78
Joined: Sat Jun 01, 2013 2:27 pm

Re: mmal port callbacks on JPEG component issue

Fri Jul 11, 2014 8:36 am

@6by9 Just one strange thing...

The new (test) start_x.elf doesn't see the full 512MByte RAM in my model B. With the latest rpi_update I get 382868KBytes showing in top but with the new start_x.elf, I only get 123072KBytes. I have gpu_mem=128 in my /boot/config.txt.

Did I miss something, or does the new start_x.elf really want to be in a model A?

Thanks again.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9012
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: mmal port callbacks on JPEG component issue

Fri Jul 11, 2014 9:07 am

lowflyer wrote:@6by9 Just one strange thing...

The new (test) start_x.elf doesn't see the full 512MByte RAM in my model B. With the latest rpi_update I get 382868KBytes showing in top but with the new start_x.elf, I only get 123072KBytes. I have gpu_mem=128 in my /boot/config.txt.

Did I miss something, or does the new start_x.elf really want to be in a model A?
There's some magic that I'm missing with my builds to make it look at the boot/config.txt for gpu_mem split. It's a fixed split with that ELF. Dom knows the magic and it gets done on official releases.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

lowflyer
Posts: 78
Joined: Sat Jun 01, 2013 2:27 pm

Re: mmal port callbacks on JPEG component issue

Wed Jul 16, 2014 7:50 am

@6by9 Thanks for that clarification.

I just tried mmal-motion after yesterday's commit on https://github.com/raspberrypi/firmware/commits/master (09c0f5fc8cd9701f01e8e403c398699e1b1a1eb6 dated 15 July) and it still fails. I guess that your patches to start_x.elf haven't yet made it to the official release. Fingers crossed for next time!

It looks as if you have been busy with the Model B+. Very good!! Thanks again!

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9012
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: mmal port callbacks on JPEG component issue

Wed Jul 16, 2014 10:48 am

Model B+ was actually a surprise to me - I'm a Broadcom person, not Foundation. But, yes, been rather busy working on sorting software for a different chip.

I just need to do a little bit more testing with my updated fix (I'll push it to my github same as before) to ensure it still works with raspistill et al without issue.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

lowflyer
Posts: 78
Joined: Sat Jun 01, 2013 2:27 pm

Re: mmal port callbacks on JPEG component issue

Sun Jul 20, 2014 7:27 pm

Hooray!!!

Code: Select all

sudo rpi-update
now works for me.

Many thanks to @6by9 and @popcornmix.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 9012
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: mmal port callbacks on JPEG component issue

Mon Jul 21, 2014 10:09 am

lowflyer wrote:Hooray!!!

Code: Select all

sudo rpi-update
now works for me.

Many thanks to @6by9 and @popcornmix.
Hmm, my fix didn't get pushed yet - I sent it to popcornmix on Thursday for him to test with the builds he cares about. It may be that the issue reoccurs at some later point - sorry to burst your bubble.
Hopefully my fix should be pushed soon.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

andies
Posts: 141
Joined: Mon Nov 11, 2013 8:12 pm
Location: Berlin

Re: mmal port callbacks on JPEG component issue

Mon Jul 21, 2014 11:29 am

lowflyer wrote: sudo rpi-update now works for me.
Same for me!
raspberry B, Noir camera, Mac Book Air, iPhone, Bezzera

GrubbyHalo
Posts: 29
Joined: Thu Jul 10, 2014 9:33 am

Re: mmal port callbacks on JPEG component issue

Mon Aug 11, 2014 11:11 pm

I am having the exact same issue. I have the same setup except that i have a h264 encoder as opposed to the jpeg encoder. Everything works for about 30 seconds and then i stop getting callbacks. I have the lastest firmware.

andies
Posts: 141
Joined: Mon Nov 11, 2013 8:12 pm
Location: Berlin

Re: mmal port callbacks on JPEG component issue

Tue Aug 12, 2014 6:58 am

GrubbyHalo wrote:I am having the exact same issue.
I am surprised. I had the problem and it disappeared after the firmware update. Now it seems to appear with the new firmware...
raspberry B, Noir camera, Mac Book Air, iPhone, Bezzera

GrubbyHalo
Posts: 29
Joined: Thu Jul 10, 2014 9:33 am

Re: mmal port callbacks on JPEG component issue

Tue Aug 12, 2014 7:35 am

andies wrote:
GrubbyHalo wrote:I am having the exact same issue.
I am surprised. I had the problem and it disappeared after the firmware update. Now it seems to appear with the new firmware...
I can see the frame rate dropping slightly throughout the 30 odd seconds it takes and then suddenly nothing.

Return to “Camera board”