devmonkey
Posts: 8
Joined: Tue Jul 05, 2016 7:38 pm

JPEG encoder restart markers

Tue Jul 05, 2016 7:56 pm

Hi,

The scan data in a JPEG is encoded as a stream of MCUs, each MCU contains some number of 8x8 blocks, one block per component (usually a luminance and two chroma components).

The JPEGs produced by the PI encoder use chroma sub-sampling to produce an MCU containing 4x8x8 luminance blocks and 2x8x8 sub sampled chroma blocks, i.e. the entire MCU covers 16x16 pixels. All the JPEG data scan data produced by the PI encoder is run-length-encoded meaning MCUs do not break on byte boundaries.

This is all normal however the JPEG spec also allows an encoder to use a 'restart-interval', meaning that restart-interval MCUs are encoded, followed by a restart marker, followed the next restart-interval MCUs starting byte aligned in the byte following the restart marker.

This is to allow some redundancy when transmitting the JPEG data over a lossy transport as i'm doing in my project. Entire sets of MCUs can be dropped and but those received intact will still display correctly.

My question is, does anyone know if it is possible to instruct the PI JPEG encoder (i'm using MMAL) to use a restart-interval and and insert the restart marker? FWIW, the decoder must be capable of handling them as this is part of the JPEG spec. I can't find any JPEG encoder settings exposed other than MMAL_PARAMETER_JPEG_Q_FACTOR (and the various EXIF/meta-data insertions).

I'm halfway through just re-framing the jpeg data in software to add the restart markers and bit shift each segment back to byte alignment but it would be much easier should the HW encoder be capable of this...

Thanks, Joe

Thanks, Joe

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7409
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: JPEG encoder restart markers

Wed Jul 06, 2016 8:40 am

I'll have a look at the firmware source later (replying puts this thread into my normal forum search as a reminder).
I don't recall seeing anything in the low level api, but that's not to say it isn't there.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7409
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: JPEG encoder restart markers

Wed Jul 06, 2016 9:58 am

Good news for you - there does appear to be a property on the codec itself to set the restart interval, so it's just a matter of plumbing that out to the outside world.
Hopefully that should be a relatively easy job, so I'll try to fit it in before the weekend and pass it to Pi Towers to release. I'll apologise in advance if those timescales slip - Pi firmware support isn't my day job, evenings often go haywire, and my memory is getting worse remembering which jobs I've said I'll do!
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

devmonkey
Posts: 8
Joined: Tue Jul 05, 2016 7:38 pm

Re: JPEG encoder restart markers

Wed Jul 06, 2016 12:15 pm

6by9 wrote:Good news for you - there does appear to be a property on the codec itself to set the restart interval, so it's just a matter of plumbing that out to the outside world.
Hopefully that should be a relatively easy job, so I'll try to fit it in before the weekend and pass it to Pi Towers to release. I'll apologise in advance if those timescales slip - Pi firmware support isn't my day job, evenings often go haywire, and my memory is getting worse remembering which jobs I've said I'll do!
Wow, that would be amazing, thanks very much!

If you are going to do this it equally applies to MJPEG so if that can be exposed at the same time even better.

Thanks, Joe

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7409
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: JPEG encoder restart markers

Wed Jul 06, 2016 12:28 pm

devmonkey wrote:Wow, that would be amazing, thanks very much!

If you are going to do this it equally applies to MJPEG so if that can be exposed at the same time even better.
I got bored over lunch, so code changes are already made and build, but I can't test it until I can grab a Pi.

MJPEG is a bit more of an issue as it is a very unloved codec, but I'll see what is possible.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

devmonkey
Posts: 8
Joined: Tue Jul 05, 2016 7:38 pm

Re: JPEG encoder restart markers

Wed Jul 06, 2016 12:42 pm

If you want to test the markers you should see in the JPEG byte stream are:

Code: Select all

0xFFDD - restartInterval (followed by two bytes defining the actual interval in MCUs)
0xFFD0 -> 0xFFD7 restart markers between restartInterval runs of MCUs (first will be D0, second D1, after D7 it will return to D0).
If restarts are not configured none of these markers will exist.

How do I go about getting your patch?

Cheers, Joe

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7409
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: JPEG encoder restart markers

Wed Jul 06, 2016 1:14 pm

devmonkey wrote:If you want to test the markers you should see in the JPEG byte stream are:

Code: Select all

0xFFDD - restartInterval (followed by two bytes defining the actual interval in MCUs)
0xFFD0 -> 0xFFD7 restart markers between restartInterval runs of MCUs (first will be D0, second D1, after D7 it will return to D0).
If restarts are not configured none of these markers will exist.
Ta. Saves me some digging into the spec for what difference it makes.
How do I go about getting your patch?
You wait until it is pushed by Pi Towers, and then do a "sudo rpi-update". It'll be an update to the firmware mainly, and then userland too to define the new MMAL and IL enums. If you're very lucky I'll update raspistill too to add a new command line option for it.
The bump to the raspbian stable firmware happens every so often when either a significant feature is added or big fix made.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

devmonkey
Posts: 8
Joined: Tue Jul 05, 2016 7:38 pm

Re: JPEG encoder restart markers

Wed Jul 06, 2016 1:33 pm

I'm passing MMAL buffers directly to a wifi card via pcap injection so don't personally need it added to raspi* but others may find it useful.

Thanks again, Joe

User avatar
scruss
Posts: 2542
Joined: Sat Jun 09, 2012 12:25 pm
Location: Toronto, ON
Contact: Website

Re: JPEG encoder restart markers

Wed Jul 06, 2016 1:52 pm

devmonkey wrote:I'm halfway through just re-framing the jpeg data in software …
Unless I'm misunderstanding what you're doing (always a possibility with me), jpegtran (in the libjpeg-progs package) will insert restart markers. It's a handy tool to have around for manipulating JPEGs without data loss.
‘Remember the Golden Rule of Selling: “Do not resort to violence.”’ — McGlashan.

devmonkey
Posts: 8
Joined: Tue Jul 05, 2016 7:38 pm

Re: JPEG encoder restart markers

Wed Jul 06, 2016 2:23 pm

scruss wrote:
devmonkey wrote:I'm halfway through just re-framing the jpeg data in software …
Unless I'm misunderstanding what you're doing (always a possibility with me), jpegtran (in the libjpeg-progs package) will insert restart markers. It's a handy tool to have around for manipulating JPEGs without data loss.
Thanks i've looked at jpegtran, I also need to index the mcu bitstreams at the sametime so I believe I would have had to alter jpegtrans quite a bit, I don't need to decompress and store the DCT co-efficients other than to find out how many bits long each one is to further determine the MCU boundaries. I have this working on a PC and need to recode it on the PI, but it will hopefully be substantially more efficient if done HW encoder with 6by9's patch.

Basically i'm doing low latency video over injection mode wifi. Currently i'm streaming JPEGs at [email protected] ~30-50ms glass-glass latency however I get nasty artifacts with packet loss which is considerable with injection mode and don't really want to use the CPU and additional latency for high redundancy FEC.

What I have now can be made much more robust to packet loss if I can rebuild the next frame from parts of the previous frame and preserve all MCUs that do arrive intact. The RLE in the JPEG bitstream means that usually after the first corruption/missing packet the rest of the frame garbage, restarts fix this.

I'm also planning to vary the transmission rate within the frame, to produce more redundancy/higher fps around the ROI. Basically i'm trying to work around lots of the problems you get with H.264 over very lossy transports where my most important requirement is instantaneously rendering new information, by using JPEGs instead and some very specific optimisations I can make for this particular usecase, which wouldn't for example be acceptable for normal consumer video broadcast.

Cheers, Joe

User avatar
RaTTuS
Posts: 10484
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK

Re: JPEG encoder restart markers

Wed Jul 06, 2016 2:37 pm

How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

devmonkey
Posts: 8
Joined: Tue Jul 05, 2016 7:38 pm

Re: JPEG encoder restart markers

Wed Jul 06, 2016 2:45 pm

Yes, my injection code is based on befi's injection code. My project is really about reducing the latency of the wifibroadcast project to the absolute minimum, I believe he is already there for h.264 but i'm hoping to cut this in half using these various JPEG hacks. The PI JPEG HW is very very fast.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7409
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: JPEG encoder restart markers

Wed Jul 06, 2016 10:36 pm

I've passed the firmware patch to Pi Towers to be reviewed and released.
0xFFDD - restartInterval (followed by two bytes defining the actual interval in MCUs)
It appears that's not quite right based on what the codec is producing. It's "FF DD 00 04 len len" as you need the length of the block encoded in there.

There also appear to be a couple of constraints on valid values for the restart interval, although I don't know if that is the JPEG standard or the hardware block.
The comments say

Code: Select all

   /*  The restart interval must obey some restrictions, to ensure
       that the first pixels after each restart marker are always
       aligned.  */
And the code is then

Code: Select all

   if (interval % mcu_width != 0) { /* Multiples of rows are always OK.  */
      if (mcu_width & 1) {
         /* Force to an integer multiple of MCU rows.  */
         interval += mcu_width-1;
         interval /= mcu_width;
         interval *= mcu_width;
      } else {
         /* Make even.  */
         interval += interval & 1;
      }
   }
So any multiple of the number of MCUs per row is fine.
Otherwise, if there are an odd number of MCUs per row, then align it up to an integer number of rows. If an even number of MCUs per row, then just make sure the interval is even (rounding up).
That's the code and I'm not going fiddling in there. I'll trust it's correct.

Keep an eye on https://github.com/Hexxeh/rpi-firmware for the patch being released, and then use "sudo rpi-update" to get it (you may need to "sudo apt-get install rpi-update" first).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

devmonkey
Posts: 8
Joined: Tue Jul 05, 2016 7:38 pm

Re: JPEG encoder restart markers

Thu Jul 07, 2016 9:16 am

Thanks so much. Is there a commit tag/id I should look out for?

What is the new MMAL parameter?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7409
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: JPEG encoder restart markers

Thu Jul 07, 2016 9:33 am

devmonkey wrote:Thanks so much. Is there a commit tag/id I should look out for?
The commit text I used:

Code: Select all

image_encode: Add parameter to set the restart interval.
    
    Requested on
    https://www.raspberrypi.org/forums/viewtopic.php?f=38&t=153553
At least the first line of that is normally copied along with any github issue or forum reference, typically prefixed with "firmware:".
devmonkey wrote:What is the new MMAL parameter?
The change to mmal_parameters_camera.h is

Code: Select all

diff --git a/interface/mmal/mmal_parameters_camera.h b/interface/mmal/mmal_parameters_camera.h
index 76a7715..4c628ce 100644
--- a/interface/mmal/mmal_parameters_camera.h
+++ b/interface/mmal/mmal_parameters_camera.h
@@ -110,6 +110,7 @@ enum {
    MMAL_PARAMETER_CAMERA_RX_CONFIG,          /**< Takes a @ref MMAL_PARAMETER_CAMERA_RX_CONFIG_T */
    MMAL_PARAMETER_CAMERA_RX_TIMING,          /**< Takes a @ref MMAL_PARAMETER_CAMERA_RX_TIMING_T */
    MMAL_PARAMETER_DPF_CONFIG,                /**< Takes a @ref MMAL_PARAMETER_UINT32_T */
+   MMAL_PARAMETER_JPEG_RESTART_INTERVAL,     /**< Takes a @ref MMAL_PARAMETER_UINT32_T */
 };
 
 /** Thumbnail configuration parameter type */
That'll get updated in the userland repo too. A simple

Code: Select all

mmal_port_parameter_set_uint32(image_encode->output[0], MMAL_PARAMETER_JPEG_RESTART_INTERVAL, restart_interval);
should do the trick for you (check the return code if you want).
(Internally the value gets used as a signed int, but that'll only affect you when you encode a 549TPix image, so not really one to worry over).

I did do a quick patch to raspistill too, so that should be released via userland too. New parameter there is "-rs N" or "--restart N".

I didn't get a chance to look into the MJPEG codec as yet - I'll add it to the list of jobs.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7409
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: JPEG encoder restart markers

Fri Jul 08, 2016 2:35 pm

Firmware update done - https://github.com/Hexxeh/rpi-firmware/ ... d0df7c8d38
"sudo rpi-update" to get it (you may need "sudo apt-get install rpi-update" first). It doesn't look like userland has been updated as yet - you've got the relevant changes anyway, and I'll nudge gently if it doesn't happen in the next day or so.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

devmonkey
Posts: 8
Joined: Tue Jul 05, 2016 7:38 pm

Re: JPEG encoder restart markers

Fri Jul 08, 2016 3:33 pm

I've just updated to get your patch and it works perfectly, Thanks.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7409
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: JPEG encoder restart markers

Fri Jul 08, 2016 4:05 pm

Glad to have another happy customer :D
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Return to “Graphics, sound and multimedia”