MPEG-2 Decoding

[portets]

Are those benchmarks from an overclocked pi?

[jacksonliam]

Hi Guys,

I am really impressed what you are doing here. MPEG2 decoding is for me (and a lot of others) the last missing part of PI functionality to make it the greatest piece of hardware I have
Unfortunately I am not bale to help you by doing some coding or so... But at least I try to find some information that can help you a bit...
Maybe you have already consider it, but it seems interesting so I let you know about it:
http://www.memetic.org/raspbian-benchma ... -vs-armhf/
http://forum.stmlabs.com/showthread.php?tid=1237

Seems the guys could get some more performance from the PI.

That's some worrying audio decoding figures (the 2.0 mp3 ones are ok).
I will change to the new raspbian based image and do some new benchmarks, but as discussed earlier the floating point stuff doesnt really help, the ARMv6 stuff and the memcpy stuff might though!
I would like to write the video and audio output routines soon though and do a demo playing some live DVB-T (or at least a .ts capture).

Are those benchmarks from an overclocked pi?

None of mine are, Since members of the foundation run theirs overclocked I might try a 800mhz Pi, I think 25fps with 2.0 audio at 700mhz will be doable - Without the openGL code from MartenR (which should help out a lot!) - However 800-1000Mhz would give a bit more headroom for higher bitrates and 5.1 audio (like PAL DVDs)

[JonathanGraham]

Some updates on my progress using mplayer with data.
Hardware:Stock Raspberry Pi B
Distro: Raspbian
Kernel: Custom 3.2.21
Compile flags: default (In raspbian this usually means VFP but see my note below about compiler flags)

Application: mplayer
Version: SVN-r35030-4.6
Flags: -vo rpi -fs -quiet -nosound -benchmark -frames 300 -dr -vc ffmpeg2 -cache-min 99

Code: Select all

Media: http://hubblesource.stsci.edu/sources/video/clips/details/images/hst_2.mpg
Details: 720x480, 6000 kbps
BENCHMARKs: VC:  10.325s VO:   0.629s A:   0.000s Sys:   0.359s =   11.313s
26.5 fps

Code: Select all

Media: Disney Opening
Details: 720x480, 9800 kbps
BENCHMARKs: VC:  10.932s VO:   0.636s A:   0.000s Sys:   0.427s =   11.995s
25.2 fps

Procedure:
Each command was run three times. The posted benchmarks are from the third run. The fps value was calculated manually from the total run time divided by the total number of frames (always 300)
Pros:

• Clips both involve heavy scrolling which means fewest P and B frames thus should create a near worst case scenario.
  If ~25 fps can be maintained then it's within reach of HD 9000kbps video using framedrop.
  Overclocking may close the gap.

Cons:

• No sound yet. Using SDL or ALSA uses 10% of CPU
  ffmpeg appears to be heavily optimized with ARM assembly may not be much room for micro optimizations

Compiler flags: Raspbian's kernel and packages should be compiled with -vfp and each program compiled under it should take that flag by default. However given that configure scripts are complicated it's not guaranteed. Likewise this is a custom kernel and I can't swear that the opts I used when building were optimal. This will be shored up in my next set of tests. As I don't yet have a cross-compile setup for mplayer it takes a good few hours to compile.

Next Steps: Stop twiddling around with the VO and write the AO module. On the upside based on my prior posting it seems my messing about dropped the CPU utilization from 1.2s to 0.6.

[jacksonliam]

Nice benchmarks, I'm not sure how vo libs work, but is it threaded? The sync call blocks until vsync so you might get a few fps by writing it threaded!

If so, could I see your vo code?

[JonathanGraham]

Nice benchmarks, I'm not sure how vo libs work, but is it threaded? The sync call blocks until vsync so you might get a few fps by writing it threaded!

I drop dispmanx_update_submit_sync into a pthread. I've tried threading out the call to vc_dispmanx_resource_write_data but it seems to make things choppy. I may give it another go.

If so, could I see your vo code?

Sure but there's a lot of mplayer specific code in it (because mplayer is a ratsnest of features!)
It might be better if I explain it.

libvo exposes some functions.

draw_slice - which takes a x,y,w,h a few buffers and stride values and expects you to draw the contents of the buffers into a buffer that you later blit to video.
flip_page - updates the screen
get_image - which can be implemented optionally so that you can provide buffers to the codec and it can draw into them. This is called direct-rendering and if done correctly then draw_slice isn't needed.

So my draw_slice is essentially some memcpy's into the var->image structure in the example. Sadly all this does is get called for each row of MPEG2 macroblocks (width x 2 - ie. 16 lines) no attempt is made (with ffmpeg anyway) to only update the parts of the screen that were changed in the last decode.

flip_page - calls vc_dispmanx_update_start(0), then vc_dispmanx_resource_write_data() blitting the whole buffer into vmemory then creates a thread for vc_dispmanx_update_submit_sync

get_image - is mplayer weirdness but essentially it tells mplayer to write into the var->image buffer. It also manages multiple buffers for the I,P and B frames. This gets us a little advantage since while we have to blit the whole frame into video memory. ffmpeg doesn't need to decode the whole thing. If I could figure out a way to get ffmpeg to signal the portion of the frame that needs to be blitted I could probably do a little better. However as you can tell I can only save at most 0.6s which is only about 1.5 fps.

Pretty much everything else in the dispmanx example code I do once during initialization.

[jacksonliam]

Thanks for that, Im trying to figure out how to use a pThread on the sync call now, did you see an example?

I seem to be getting mixed results with the raspbian image, especially with the high bitrate hubble sample, one moment im getting 28-32fps the next time I run it its hanging around 17fps!

However, my BBC streams seem to be going at about 29FPS in libavcodec with my horribly in-efficient (around 4fps hit) video output routine! With my modded libmpeg2 I'm getting around 30-40fps with no video output routine!

After overclocking to 800 with sdram of 500, I've gained around 2-4fps in the top end, but I think the low values are much better!

[JonathanGraham]

Thanks for that, Im trying to figure out how to use a pThread on the sync call now, did you see an example?

I happen to have one right in this post!

#include <pthread.h>
pthread_t thread1;

//It's legal to join up threads even if one hasn't spawned yet.
pthread_join(thread1,NULL);
//Create a thread out of the function synch called with the parameter vars->update
pthread_create(&thread1,NULL,synch,vars->update);
// Here's our function which becomes our thread.
void *synch(void *ptr)
{
return vc_dispmanx_update_submit_sync( ptr );
}

We can loop around the join/create calls each time we draw the screen. i.e. join thread, start update, write data, launch thread (which synchs the update).

I seem to be getting mixed results with the raspbian image, especially with the high bitrate hubble sample, one moment im getting 28-32fps the next time I run it its hanging around 17fps!

I just recompiled the kernel with flags and will be posting some benchmarks later using raspbian+kernel with optimal flags+mplayer with optimal flags.

While I was in the kernel config I also made a few changes to make it a bit more focused on single tasks. I changed the elevator and scheduler options. I will post my config when do my next official benchmark posting.

However, my BBC streams seem to be going at about 29FPS in libavcodec with my horribly in-efficient (around 4fps hit) video output routine! With my modded libmpeg2 I'm getting around 30-40fps with no video output routine!

Good job!

After overclocking to 800 with sdram of 500, I've gained around 2-4fps in the top end, but I think the low values are much better!

I'm saving that one for afters. After this compile I'm going to start profiling the code. I might also add a 1080i clip to the test bench - apparently some broadcast video is done in this format?

[AndrewS]

On the subject of performance tweaks - can't remember if it's been mentioned in this thread or not yet, but you can also get an extra boost by (temporarily) disabling the USB driver www.raspberrypi.org/phpBB3/viewtopic.php?f=28&t=7866

[JonathanGraham]

On the subject of performance tweaks - can't remember if it's been mentioned in this thread or not yet, but you can also get an extra boost by (temporarily) disabling the USB driver http://www.raspberrypi.org/phpBB3/viewt ... =28&t=7866

Good catch, there's apparently a patch too (not without some problems but perhaps better than turning off networking altogether)

[JonathanGraham]

It took me forever to compile mplayer with opmtimized flags - I had the option on for it to grab the latest ffmpeg using git and about three times on Friday different people checked in something inanely stupid into the source tree. Which meant my 4 hr compile turned into multiple 4 hr compiles across several days. I also noticed that there was simply too much variance with the flags I was using in mplayer. So I replaced -cache-min 99 with -nocache

Anyway right now the hubble clip is sitting at 28.1 fps and the Disney clip is still at 25.2 fps

Apparently there's a new FIR patch for the 8000 interrupt problem going in soon which may give us more CPU. Now I'm off to do some profiling.

[JonathanGraham]

So my theory this morning was...

mplayer's libvo doesn't actually have a callback or any other mechanism to insure that data is written to the display before the next frame goes. In writing code for dispmanx we've been attempting to force this behavior using the vc_dspmanx_update_sumbit_sync call. Then mitigate the performance hit by using threads. However that led me to a couple of interesting inferences.

1. Except in -benchmark mode mplayer is never going to update the display faster than the frame rate of the clip.
2. Even then it would only matter if the rest of the decode part of the cycle was faster than the screen update.
3. This is clearly never the case.
4. We can then replace the blocking call to vc_dspmanx_update_sumbit_sync() with vc_dspmanx_update_sumbit() and forget about threading.

vc_dspmanx_update_sumbit() is designed to use a callback function and parameters. However setting those to NULL seems to work fine. This appears to provide a gain of ~0.7 fps ( this is because of iii. ). This isn't a huge gain but it does make the code simpler.

Right now the VO portion of the system uses about 4.8% of the overall CPU which might be as lean as we can get it. Unless I can start selectively blitting.

In other news I did a common macro-optimization: replacing memcpy/memset. There's a thread on it here:

I also tried messing with fbset which allows you to mess with the framebuffer this will screw up your text console (but serial and ssh are maintained). Details are here.

Code: Select all

fbset -xres 8 -yres 8 -vxres 8 -vyres 8 -depth 8

Considering I'm trying to get a STB type experience the FB console isn't necessary.

With all of these system optimizations in place I'm hitting about: 27fps in the Disney clip. I've also had to lengthen my sample time from 300 frames to1200 frames in order to gauge the value of some of these techniques so that might be biasing my results slightly too.

Profiling has shown a few places where I can attempt to optimize audio output. So I'll get onto those next - if I can drop the CPU usage of audio then OC can easily give us 30fps.

[jacksonliam]

Nice work, so Mplayer with a dispmanx output really is fast enough! Nice thing about Mplayer I guess is that your work benefits all non-GPU-accelerated video file formats, not just Mpeg-2 - Right?

As for audio, what about the OpenMax audio renderer that omxplayer uses? That bypasses alsa IIRC so might be beneficial?

[JonathanGraham]

Nice work, so Mplayer with a dispmanx output really is fast enough! Nice thing about Mplayer I guess is that your work benefits all non-GPU-accelerated video file formats, not just Mpeg-2 - Right?

Yeppers. I watched a bunch of my old divx files - even streaming over the network seemed to work. Stretching them to full screen actually works pretty well. I also need to put some time in to get soft-subtitles though.

As for audio, what about the OpenMax audio renderer that omxplayer uses? That bypasses alsa IIRC so might be beneficial?

This was my thinking too and I may still do that. The OpenMax IL API is pretty straightforward and it's structure mimics the libao api to an extent. Interesting fact that the profiler did not show ALSA as being the main use of CPU. So I may fall back the the OpenMax but first I'm probably going to try a few spot optimizations.

I think we'll also get a boost once the interrupt situation is patched.

[AndrewS]

1. Except in -benchmark mode mplayer is never going to update the display faster than the frame rate of the clip.
2. Even then it would only matter if the rest of the decode part of the cycle was faster than the screen update.
3. This is clearly never the case.

Please excuse my stupid question (most of what you're doing is over my head, but I'm still having fun watching progress) but could it be the case if playing back low-res uncompressed video?

[AndrewS]

I also need to put some time in to get soft-subtitles though.

Dunno if http://www.raspberrypi.org/phpBB3/viewt ... 04#p131004 helps with that?

[jacksonliam]

1. Except in -benchmark mode mplayer is never going to update the display faster than the frame rate of the clip.
2. Even then it would only matter if the rest of the decode part of the cycle was faster than the screen update.
3. This is clearly never the case.

Please excuse my stupid question (most of what you're doing is over my head, but I'm still having fun watching progress) but could it be the case if playing back low-res uncompressed video?

Only if the frame rate of the video exceeds that of the display, as the frame rate is set by the video and mplayer will not go faster than that except for in benchmarks!

A 60fps video on a 50hz monitor could potentially cause a problem, I'm not too sure! It might just mean some frames are not displayed or it may cause audio and video sync issue on such a video sample. It would have to be low res for the Pi to be able to keep up with 60fps so I'm not sure what the point of such a sample would be? :-\

[JonathanGraham]

I also need to put some time in to get soft-subtitles though.

Dunno if http://www.raspberrypi.org/phpBB3/viewt ... 04#p131004 helps with that?

Actually in mplayer I believe most codecs use the libvo function display_osd to accomplish soft/DVD subtitles. It looks trivial to implement - I just haven't had much time.

Once I sort out MPEG2 it would make sense to get hardware decode working I would have to implement the guts of OMXplayer into a video codec and either reuse the dispmanx vo or write another one. This would allow hardware accelerated h264 with subtitles. This is, coincidentally my "gold standard" for media player stress testing. H264 with three sets of stiles subtitles and AC3 audio (There's an episode of Macross Frontier which has this going on).

I corrupted my raspbian install yesterday so I've been set back a couple of days on my sound work. I really need to get a reliable cross-compiling environment for mplayer working - it just has so many dependencies.

[JonathanGraham]

A 60fps video on a 50hz monitor could potentially cause a problem, I'm not too sure! It might just mean some frames are not displayed or it may cause audio and video sync issue on such a video sample. It would have to be low res for the Pi to be able to keep up with 60fps so I'm not sure what the point of such a sample would be? :-\

mplayer hasn't the slightest idea what rate your hardware scans the framebuffer at. So all that would happen is the framebuffer, which would normally get updated approximately once per scan would, ten times each second get updated twice. Which would probably result in some aliasing and a waste of CPU (for rendering a frame you will never see). The later effect can be compensated for by mplayer's -fps flag.

[jacksonliam]

A 60fps video on a 50hz monitor could potentially cause a problem, I'm not too sure! It might just mean some frames are not displayed or it may cause audio and video sync issue on such a video sample. It would have to be low res for the Pi to be able to keep up with 60fps so I'm not sure what the point of such a sample would be? :-\

mplayer hasn't the slightest idea what rate your hardware scans the framebuffer at. So all that would happen is the framebuffer, which would normally get updated approximately once per scan would, ten times each second get updated twice. Which would probably result in some aliasing and a waste of CPU (for rendering a frame you will never see). The later effect can be compensated for by mplayer's -fps flag.

I thought that would be about the case! The tip about the vc_dspmanx_update_sumbit() call really helped, Im hitting some really impressive FPS numbers, in the high 40's and peaking at 63 for a DVB-T sample - Thats with display on screen!
I've also disabled the USB and am using the Pi via its UART console for now.

USB on:

Code: Select all

  54 frames rendered in 2.0199 seconds -> FPS=26.7338
  67 frames rendered in 2.0056 seconds -> FPS=33.4072
  66 frames rendered in 2.0210 seconds -> FPS=32.6572
  71 frames rendered in 2.0166 seconds -> FPS=35.2085
  91 frames rendered in 2.0191 seconds -> FPS=45.0691
  92 frames rendered in 2.0055 seconds -> FPS=45.8728
  89 frames rendered in 2.0084 seconds -> FPS=44.3135
  86 frames rendered in 2.0020 seconds -> FPS=42.9572
  85 frames rendered in 2.0015 seconds -> FPS=42.4688
  87 frames rendered in 2.0011 seconds -> FPS=43.4757
  82 frames rendered in 2.0098 seconds -> FPS=40.7991
  96 frames rendered in 2.0151 seconds -> FPS=47.6400
  87 frames rendered in 2.0184 seconds -> FPS=43.1044
  74 frames rendered in 2.0169 seconds -> FPS=36.6907
  61 frames rendered in 2.0046 seconds -> FPS=30.4301
  82 frames rendered in 2.0239 seconds -> FPS=40.5149

USB off:

Code: Select all

  61 frames rendered in 2.0060 seconds -> FPS=30.4094
  75 frames rendered in 2.0215 seconds -> FPS=37.1003
  71 frames rendered in 2.0117 seconds -> FPS=35.2938
  87 frames rendered in 2.0026 seconds -> FPS=43.4432
 104 frames rendered in 2.0226 seconds -> FPS=51.4191
  96 frames rendered in 2.0035 seconds -> FPS=47.9165
  96 frames rendered in 2.0015 seconds -> FPS=47.9634
  95 frames rendered in 2.0215 seconds -> FPS=46.9942
  93 frames rendered in 2.0000 seconds -> FPS=46.4994
  92 frames rendered in 2.0206 seconds -> FPS=45.5319
 108 frames rendered in 2.0108 seconds -> FPS=53.7099
  89 frames rendered in 2.0242 seconds -> FPS=43.9681
  83 frames rendered in 2.0122 seconds -> FPS=41.2491
  87 frames rendered in 2.0198 seconds -> FPS=43.0733
  80 frames rendered in 2.0205 seconds -> FPS=39.5935
  79 frames rendered in 2.0011 seconds -> FPS=39.4778
  87 frames rendered in 2.0170 seconds -> FPS=43.1339

About 10% better...

[JonathanGraham]

The tip about the vc_dspmanx_update_sumbit() call really helped, Im hitting some really impressive FPS numbers, in the high 40's and peaking at 63 for a DVB-T sample - Thats with display on screen!
I've also disabled the USB and am using the Pi via its UART console for now.

Good job. I think we're close. We should both post our test clips on some online service somewhere. That way we can more accurately compare numbers. I can always incorporate your code into mplayer if it turns out to be the winner (mplayer can optionally use libmpeg instead of ffmpeg).

Are your times above with stock clocks or are you still OC'd?

Edit: Did you also make the changes to the framebuffer? Since you're using the console anyway...
One thing I noticed that the fbset commands can easily crash the Pi. So I've just compiled a kernel without framebuffer support. Should be interesting.

[jacksonliam]

Well that's a simple libavcodec loop, I'm sorry I know I keep changing between the two

I haven't been able to get libmpeg2 to output to the dispmanx yet, I think I need to do my own buffer management...

I will post a BBC clip, I'd like to try your Disney clip if possible, but if it's copyright I don't think we can post links here! I think I'm going to include a vob file from a dvd in my tests too.

Nope I didn't modify framebuffer, are you using the new raspbian based wheezy from the foundation?

[JonathanGraham]

Well that's a simple libavcodec loop, I'm sorry I know I keep changing between the two

I haven't been able to get libmpeg2 to output to the dispmanx yet, I think I need to do my own buffer management...

Ahh ok. Yeah, in mplayer it's even worse. There are three separate buffers and a semi-complex (but almost entirely undocumented) system for switching between them depending on if you're decoding an I,P or B frame...and that doesn't even provide what standard "double buffering" does (i.e. no tearing). For that I'll probably have to figure something else out. On the other hand the codec draws directly to the buffer that write_data sends to the GPU so it saves me a blit. It would be nice to be able to write directly to the GPU texture as write_data is the last part of the process that takes any appreciable
time

I will post a BBC clip, I'd like to try your Disney clip if possible, but if it's copyright I don't think we can post links here! I think I'm going to include a vob file from a dvd in my tests too.

I'll get it up somewhere and I'll assert that what we're doing is fair-use (research purposes) if our hosts complain I'll take it down.

Nope I didn't modify framebuffer, are you using the new raspbian based wheezy from the foundation?

No I'm still using the old pisces one but now that have an idea of what I'm doing I might switch. I'll just keep my custom kernel.

[JonathanGraham]

Just a note with framebuffer alterations and USB off

I hit 31fps on the disney clip without sound and 24 fps with...

[Hexagon]

Keep up the good work guys! I'm waiting patiently for progress over here

[linuxstb]

I've finally got my Pi and have started looking at libmpeg2. My approach has been to just copy the .c/.h/.S files from the libmpeg2 distribution and, along with the "simple1.c" example program, create my own test player.

So I've created my own simple Makefile instead of using the libmpeg one, and have removed all unused files to make it easier to work with.

I've enabled the official ARM optimisation (motion_comp_arm_s.S) included with libmpeg2 but haven't yet managed to get the ARM-optimised idct from Rockbox working - it just segfaults...

My simple decoder is managing faster than real-time for all my MPEG-2 clips, but I don't think this is news. My main use will be to player UK DVB-T broadcasts, and one 30-second test sample (from the channel "Dave") decodes in about 13 seconds. This is quite a low bitrate (I think around 1.5Mbits/s-2Mbits/s). A second 30 second sample (BBC1 Scotland from DVB-S) is much higher bitrate (around 5Mbits/s I think), and that takes close to 30 seconds to decode.

So this shows how dependent on bitrate the decoding speed is. Fortunately for this purpose, the UK's low bitrate DVB-T is a good thing

I intend to change my test program so that it reads the entire file into RAM before decoding. I'm expecting this to provide more consistent benchmark figures, and when I do that, I'll post some numbers to compare to others.

But now to my question - does anyone have any YUV display code they're willing to share that I can try to integrate with libmpeg2?

Thanks.