lancid
Posts: 9
Joined: Mon Apr 22, 2019 1:41 am

Stuttering video at 720p. Works great in window at 1080p - Perl + SDL

Sun May 05, 2019 12:08 am

Hi Guys,

I'm feeling really frustrated and panicked... I've been writing a multimedia program (for free) for a friend's business. The program is written in Perl and uses SDL running on a RPi 3 B+. The program does lots of video (video frames are separate images that I read into variables and then display sequentially - I did it this way because I have to check for button presses, setting changes, etc. between frames). I have some audio and static images too. I have a ton of images stored in memory/variables. I'm pushing the Pi fairly hard.

The program itself is designed to display on a 720p flatscreen TV. All static images and video image frames are 1280 px x 720 px.

For months, I've been writing the program with the RPI in 1080p mode (CEA hdmi_mode 16 - 1080p 60Hz) where I run the program in a GUI window while I have a CLI window open in the background that displays variable values, timing, events, etc... The program works great. It works great if I run the program in 'fullscreen' mode @ 1080p. Of course, the program doesn't use the full screen since it's targeted at 720p resolution displaying on a 1080p RPi resolution (it has a 5" or 6" black border around it on my 50" TV).

Last night I decided to look at the nearly-finished program in 720p mode (CEA hdmi_mode 4 - 720p 60Hz) and was horrified to find that the video sequences were often stuttering and sometimes halting for half a second at a time.

I thought if the program ran fine in a window @ 1080p, it would run just as well or better fullscreen @ 720p (The Pi would actually be controlling fewer pixels since there was no GUI desktop or my CLI window). I guess I was wrong. It's a 4k TV. Could it be a TV issue having to scale the image?

I've tried everything I can think of. I've tried turning 'overscan' on and off on the RPi (seems to run slightly better when overscan is on - not what I expected). I overclocked the Pi, hoping it would fix the problem (no such luck).

My friend is coming over in the next few days to see the final product and I'm embarrassed to show him the program running fullscreen at 720p (as it was designed) because of stuttering. I can't have him run it in 1080p fullscreen mode because his first question would be "Why doesn't the program fill the screen?"

Any help or suggestions would be greatly appreciated.

Thanks!

stuartiannaylor

Re: Stuttering video at 720p. Works great in window at 1080p - Perl + SDL

Sun May 05, 2019 6:20 pm

I think its got to do with the proprietory VC4 drivers and accelerated video is allocated as a block of the screen size.
Noticed exactly the same with VLC that only full screen works and windowed doesn't.

You can have a read through this topic as my trials and tribulations are listed there, but scroll down a bit.
viewtopic.php?f=28&t=239023
Even with wpe-webkit they only provide full screen the whole VC4 is a cul-de-sac to one guy who froze the vc4 repo and opened another with no licencing and all rights remaining and it would seem that everything VC4 wise lies in his hands.

lancid
Posts: 9
Joined: Mon Apr 22, 2019 1:41 am

Re: Stuttering video at 720p. Works great in window at 1080p - Perl + SDL

Tue May 07, 2019 10:46 pm

As a follow-up, I've had a small amount of success by reducing the amount of memory my program uses. I now rotate image frames in and out of memory as they are needed (instead of storing everything in ram the entire time). I've also added some micro sleep (Time:HiRes -> usleep) in the loops that create animation from image frames. This has freed the CPU up a lot for background processes. I still have periodic stutters and freezes, but it's better (still runs perfect in a window @ 1080p).

One thing I just realized is that when my wave files play audio at 720p, there is static in the output (almost sounds like a blown speaker) but when the exact same wav files play at 1080p, they sound perfectly 'clean'. Maybe some/most of my issue is a driver issue SDL has at 720p.

stuartiannaylor

Re: Stuttering video at 720p. Works great in window at 1080p - Perl + SDL

Tue May 07, 2019 11:44 pm

As far as I can gather VC4 is right at the very edge of its abilities with 1080p. It creates a direct frame buffer matching 1080p and manages but with no further room for scaling.

But guessing you have hardcoded config.txt for hdmi res and its actually not dying on 4k?

As for a TV having scaling issues its extremely doubtful unless its just maybe mode incompatibility, but 720p is so mainstream and media chips in modern tvs make vc4 look antiquated.

720p though I am unsure as you prob have tested generally with vlc the pi doesn't have much of a problem be it vlc, chromium or whatever app you wish to use.
Its 1080p source that saps the pi where it just manages a direct framebuffer copy but falls flat when trying to downscale 1080p to anything less than 1080p fullscreen.

Code: Select all

GPU_MEM_512 = "196"
GPU_MEM_1024 = "396"
With the non KMS original driver but maybe try the full and fake KMS ones too.
Are good settings for complex media operations.

I would have a read of https://github.com/Igalia/meta-webkit/wiki/RPi

As for playing wav files @ 1080p/720p? That one doesn't really compute but guess you mean the resultant video format.
Also what output encoding are you using as it needs to be H.264 to utilise hardware acceleration.

I know nothing really of SDL 720p should work but maybe you need to pick another framework.
Also install ffmpeg and see what comparitive results you get http://jollejolles.com/installing-ffmpe ... pberry-pi/
Adapt for pi or sudo apt-get install libav-tools
https://stackoverflow.com/questions/249 ... ith-ffmpeg
https://johnvansickle.com/ffmpeg/

Got a feeling that SDL doesn't use H-264 so no hardware acceleration though.

gkreidl
Posts: 6040
Joined: Thu Jan 26, 2012 1:07 pm
Location: Germany

Re: Stuttering video at 720p. Works great in window at 1080p - Perl + SDL

Wed May 08, 2019 7:04 am

stuartiannaylor wrote:
Tue May 07, 2019 11:44 pm
As far as I can gather VC4 is right at the very edge of its abilities with 1080p. It creates a direct frame buffer matching 1080p and manages but with no further room for scaling.
I've noticed that you are spilling the same kind of wrong information and misunderstandings regarding the VC4 and video decoding/playing in a number of forum threads.

1) The VC4 H264 decoder is specified for up to 1080p30. But it can easily decode 1080p60 if the H264 Decoder/Encoder (or the complete GPU) is slightly overclocked (300MHz). If I remenber correctly, omxplayer does this automatically, if it detects higher frame rates. I have been safely using a GPU clock of 500MHz on all my Rpis (starting from the Rpi B to the RPi 3B+).

2) The output of the video decoder is usually some kind of YUV format. The VC4 can send this directly to the video output using an overlay. This method is used by omxplayer, for example.
If you weant to use it inside a XORG-Window-Application, it has to be copied in real time into the frambuffer first. This is not just a simple copy but includes a rather costly color conversion from YUV to RGB(A). This method is used (added by the foundation developers) in chromium and VLC (and also in Epiphany up to Jessie).
The copying/conversion process has never worked perfectly in full screen resolution (at least not on 1920x1080 screen resolutions). There are always a number of lost frames. For this reason, the Foundation developers have added a special module to VLC (MMAL X11 Splitter). It switches to overlay method if you switch from window mode to full screen. No lost frames any more!

3) Video scaling is not a problem at all for the GPU. I'm using it all the time in my omxplayerGUI. This GUI wrapps a window around the (scaled) video output of omxplayer, but the video content is still running in an overlay.

4) 96 to 128 MB GPU memory are always enough for video in any resolution. More GPU memory is only needed for special applications which also use other parts of the GPU (Kodi, for example, needs a minimum of 160 MB). Another example is my Real Time Transcoder for DVB streams. Using more GPU memory will remove valuable memory from the ARM processor and in fact slow down the system (because it needs more swapping).

5) The DRM (full or fake KMS) drivers have nothing to do with video decoding acceleration at all. In fact, they may keep some things from working. They are still in alpha state and should only be used, if you want to accelerate 3D (and also some 2D) graphics.
Minimal Kiosk Browser (kweb)
Slim, fast webkit browser with support for audio+video+playlists+youtube+pdf+download
Optional fullscreen kiosk mode and command interface for embedded applications
Includes omxplayerGUI, an X front end for omxplayer

stuartiannaylor

Re: Stuttering video at 720p. Works great in window at 1080p - Perl + SDL

Wed May 08, 2019 8:04 am

gkreidl wrote:
Wed May 08, 2019 7:04 am
stuartiannaylor wrote:
Tue May 07, 2019 11:44 pm
As far as I can gather VC4 is right at the very edge of its abilities with 1080p. It creates a direct frame buffer matching 1080p and manages but with no further room for scaling.
I've noticed that you are spilling the same kind of wrong information and misunderstandings regarding the VC4 and video decoding/playing in a number of forum threads.
Try them with the current release, try them with frameworks, try them at full screen then windowed.
Then maybe post some findings than just quoting what its supposedly supposed to do, which often they are not.
Also read the posts as SDL does not output its stream to h264 so its likely no hardware acceleration is involved.
What its not doing is working and maybe suggest something that he could try.
Set the video to the levels suggested as it deliberately more than enough and that can be taken out of the problem equation.
The video drivers are Kernel Mode Setting rather than user space and something under development and have a far wider scope than just 3D
Its more dependent on the libs SDL relies on but on an off chance give them a go.
I am really not sure of the state of play and that even the V4L2 driver can be used and is also undergoing development.
I got my first Pi3 in 2016 and there have been various 'fixes' that always seem just to produce more fixes needed and that has never changed.
The majority works but not all and what is currently under development focus I am not really sure to be honest.
The one I forgot which is prob the most important is apt-get dist-upgrade as things do seem to change.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7124
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Stuttering video at 720p. Works great in window at 1080p - Perl + SDL

Wed May 08, 2019 4:07 pm

@stuartiannaylor On all low power SBCs you will need to look at the supported APIs for hardware acceleration if you want performance. Many of the generic frameworks do not consider optimisation in any shape or form and assume compute power is unlimited. That works fine on a nice beefy i7 with big graphics card and lots of SDRAM bandwidth, but not on ARM SBCs.
Most ARM SBCs (Pi included) will have some pretty powerful composition hardware that will render any layers that you care to throw at it. x86 boards don't bother and throw a huge amount of processing grunt at the problem instead.

I'd be more concerned if things weren't being developed as it would mean you're using a dead platform.

X11 is a right pain in the neck for efficient composition as it doesn't support overlays. All image elements either have to be blitted into a flat frame buffer, or composed using OpenGL which often requires format conversions.
The Raspbian versions of both Chromium and VLC have additional code in them to use the hardware blocks to convert YUV video frames to RGBA, and resize to exactly the right size for the window. X then has to do a "simple" blit of the image data into the frame buffer. Even doing that you're copying 8MB per frame (1080P @ 32bpp), 60fps, and that starts taxing SDRAM bandwidth. X doesn't double-buffer the frame buffer so you often see tearing too.
With VLC when you go full screen it takes a different path again which bypasses X totally and passes the YUV frames direct to the composition hardware, that's why full screen is smoother again than windowed mode. However you do lose the control bar as it is sitting on the X desktop that is hidden by the video.
Chromium doesn't have such a path available, hence full screen can still be jerky.

Reading lancid's original posts, if you really are storing all your image resources in RAM then I'd suspect something is triggering swapping.
You'll have to provide some example of exactly what you're doing if you want someone to investigate it further and comment on.
How are you presenting your image buffers for SDL to render? Something that can be run will save a lot of time and increase the likelihood of someone in the know looking at it.
Are you using vc4-kms-v3d, vc4-fkms-v3d, or the legacy graphics stack. They do vary significantly in what functionality is available.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

lancid
Posts: 9
Joined: Mon Apr 22, 2019 1:41 am

Re: Stuttering video at 720p. Works great in window at 1080p - Perl + SDL

Wed May 08, 2019 6:31 pm

Thanks for all the thoughts and suggestions. I didn't find an answer, but I think I found a solution.

I spent a considerable amount of time using 'top' and 'vmstat' to watch memory and cpu usage. I made changes to ensure the Pi wasn't having to swap or use excessive CPU or anything else that I could detect that would cause jittery video and pauses. I still had no luck getting smooth playback when the Pi was in 720p resolution - CEA hdmi_mode 4 - 720p 60Hz. (Again, I'm using SDL v 1.2 + Perl. I'm creating video by loading each video frame as a separate image into an array in RAM and then using SDL to blit each image onto the display screen.). And, odd as it seems, my wav files have added static (sounds like a blown speaker) when playing in 720p resolution (I don't understand the correlation between audio playback and the video resolution mode, but there's no question the audio has issues at 720p).

Since everything sounds and looks great when played in a window when the Pi is in 1080p resolution - CEA hdmi_mode 16. I decided to scale up all my images and 'video' for 1080p (1920 x 1080) and try running fullscreen 1080p to see what would happen. To my surprise, it looks and runs awesome. No stutters, pauses or hiccups. It plays smooth and looks great. The wav files have no static.

The only conclusion I can come to is that it must be an issue with SDL/drivers at the 720p resolution or maybe my TV is adding extra scaling (it's a TCL Roku 4k TV).

The only drawback is that now that I'm using 1080p resolution, my data is 2+x larger and I'm having to take extra steps to make everything work without swapping.

Thanks again for your help!

stuartiannaylor

Re: Stuttering video at 720p. Works great in window at 1080p - Perl + SDL

Wed May 08, 2019 11:13 pm

https://github.com/StuartIanNaylor/zram-config

Code: Select all

sudo apt-get install git
git clone https://github.com/StuartIanNaylor/zram-config
cd zram-config
sudo sh install.sh
Edit /etc/ztab and set the zram memory swap size.
Because it is memory based the defaults completely change and the default page_cache is 0 and swappiness 90.
If you are on a 1gb Pi make the mem_limit approx 100-300mb and make a disk_size of approx x3 of that.

Code: Select all

mem_limit                  disk_size                          page_cache                        swapiness
300                        900                                0                                 100
The minimum avg compression of lz4 never seems to dip below 300% and I err on the low side.
If its a 512mb version just half the above memory sizes as its not all allocated until used unlike the gpu.

If you have a working dir with a lot of writes you could also try adding a dir to ztab as only the upper is RW whilst the lower can be large directories but its common that writes are to a select few files so often tiny ammounts of mem can cope.
You also get a really speedy directory for the active files, whilst minimising block wear and poweroff corruption .
Last edited by stuartiannaylor on Thu May 09, 2019 12:28 am, edited 8 times in total.

cjan
Posts: 718
Joined: Sun May 06, 2012 12:00 am

Re: Stuttering video at 720p. Works great in window at 1080p - Perl + SDL

Thu May 09, 2019 12:14 am

archlinuxarm had a sdl2-rpi patch build, might help?

Return to “Advanced users”