PandaSupreme
Posts: 3
Joined: Sun Jan 19, 2020 4:20 pm

Display a video on a LED matrix

Sun Jan 19, 2020 4:44 pm

Hello fellow raspberry users !

I have a 40x30 SK6812 LED matrix I made from LED strips, connected to my raspberry pi by one pin, and would like to display videos on it. I have successfully displayed images and gifs by converting them to a simple binary object of unsigned int of 8 bits (enough to store RBG values), where each object is a pixel represented by its RBG values. When it comes to videos, I would like to display them in real time on the matrix.

At the moment I resize the videos to 40x30, convert each frame to PNG, loop through each frame and display it as I would regularly do with images.

This ails me for a couple reasons :

- I have to preprocess each video (resize -> convert to PNG -> convert to binary array with RBG values).
- It's not very optimized as I could avoid the resizing step by "jumping" between the pixels : If my video is in 1280x720, I could read the RGB value of every 1280/40 pixel (as my matrix is 40 pixels in width), and obtain a result that's good enough.

So here's my question : do you guys have any idea of how to read the RGB value of each pixel of a video in real-time, or more generally speaking, how to display a video on a LED matrix, if possible using RGB values ? :D

Have a very nice day.

njh
Posts: 69
Joined: Wed Aug 09, 2017 2:07 pm

Re: Display a video on a LED matrix

Mon Jan 20, 2020 1:33 pm

Because of the way videos are compressed, there's no easy way to access individual pixels. Some conversions are needed.

For "offline" preparation of files, what you are doing already is fine. You can do it all in one go using ffmpeg:

Code: Select all

ffmpeg -i source_file -s 40x30 tmp%03d.png
This will generate a series of images called tmp001.png, tmp002.png etc.

If you need to stream the files directly, it's possible to pipe the data through ffmpeg, like this:

Code: Select all

ffmpeg -i source_file -s 40x30 -f rawvideo -pix_fmt rgb24 - | your_program
You'll need to write a program that reads its standard input as raw RGB triplets. You could also use a popen() call to do this. Your program may also have to include delays, for real-time display at the correct speed.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 8148
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Display a video on a LED matrix

Mon Jan 20, 2020 1:52 pm

Video files are generally stored as YUV 4:2:0, not RGB. You'll need a colourspace conversion to convert that to RGB.
Within MMAL you have an ISP component that supports resize and format conversion using the hardware block that is normally used for the camera. That's the easiest way to resize and convert your video to 40x30 RGB.

There's a simple MMAL video decode example at https://github.com/6by9/userland/blob/h ... eo/video.c. Connecting in a "vc.ril.isp" component isn't trivial but also isn't too complex.

Alternatively you can use the V4L2 devices which are normally under /dev/video10 (video_decode) and /dev/video12 (isp) with something like GStreamer as an alternate pipeline manager. You should be able to use fdsink to stream the data to stdout, and then pipe that into some other application to send it to your display.

Out of interest, what refresh rate can you achieve on your display? That may be your limiting factor here, or you'll need to arrange to drop frames.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 8148
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Display a video on a LED matrix

Mon Jan 20, 2020 2:03 pm

Running a quick test based on njh's ffmpeg line:

Code: Select all

ffmpeg -i foo.mp4 -s 40x30 -f rawvideo -pix_fmt rgb24 foo.rgb
On a Pi4 it's decoding and resizing at around 85fps with a 1080p source, and that's in pure software. Trying to use the hardware is actually running slower, which implies something is misconfigured.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

PandaSupreme
Posts: 3
Joined: Sun Jan 19, 2020 4:20 pm

Re: Display a video on a LED matrix

Mon Jan 20, 2020 6:42 pm

njh wrote: Because of the way videos are compressed, there's no easy way to access individual pixels. Some conversions are needed.

For "offline" preparation of files, what you are doing already is fine. You can do it all in one go using ffmpeg:

Code: Select all

ffmpeg -i source_file -s 40x30 tmp%03d.png
This will generate a series of images called tmp001.png, tmp002.png etc.
That's exactly what I've been doing yet :D That confirmed my inuition thanks ! I built a node server and a GUI that allows to upload a video, convert it to PNG with ffmpeg, and then convert it to a simple binary format of my own. Where all pixel values are stored as 8bits unsigned int.
6by9 wrote: Running a quick test based on njh's ffmpeg line:

Code: Select all

ffmpeg -i foo.mp4 -s 40x30 -f rawvideo -pix_fmt rgb24 foo.rgb
On a Pi4 it's decoding and resizing at around 85fps with a 1080p source, and that's in pure software. Trying to use the hardware is actually running slower, which implies something is misconfigured.
Niiiice, 85fps is more than enough !

I'll let you know about the framerate, but from the tests I have currently been running I easily reached 24fps, will try will 60fps soon !
Next step will be to resize + display the video in real-time using the pipe as njh proposed. I'll let you guys know how that goes :D

deepo
Posts: 457
Joined: Sun Dec 30, 2018 8:36 pm
Location: Denmark

Re: Display a video on a LED matrix

Mon Jan 20, 2020 7:40 pm

Is 40 x 30 pixels even useful, or is it all just for fun?

X-Gen
Posts: 87
Joined: Fri Dec 06, 2019 12:08 pm

Re: Display a video on a LED matrix

Tue Jan 21, 2020 3:50 am

Sounds like you've got more of a 'icon' resolution than an actual video resolution.
Each color pixel you need to extract, is actually a block of 40x30 pixels or so... I doubt RGB values really matter there.
I don't know how fast the GPIO pins will transfer data, but you could write your own .avi to .png tunnel, with a display out to the pins...
Sounds pretty complex.
I've once tried watching a HD movie on a 144 pixel screen, and it was surprisingly visible. Not detailed, but I could still understand what was happening on the screen, without reading subtitles (they're just a garble at 144Pix).

I would probably increase the resolution...

PandaSupreme
Posts: 3
Joined: Sun Jan 19, 2020 4:20 pm

Re: Display a video on a LED matrix

Tue Jan 21, 2020 10:15 pm

deepo wrote:Is 40 x 30 pixels even useful, or is it all just for fun?
This was never meant to be 'useful' :) It's actually for an art project and will only display pixel art animations. I actually also built an emulator that allows me to read the custom binary format I made and display the converted/resized videos, with a 40x30 screen, it's in fact hard to see anything when you are 50cm away from your screen, but from a 1 or 2m distance, it seems the brains fills in the details, and the result is quite satisfactory ! :D
X-Gen wrote: Sounds like you've got more of a 'icon' resolution than an actual video resolution.
Each color pixel you need to extract, is actually a block of 40x30 pixels or so... I doubt RGB values really matter there.
I don't know how fast the GPIO pins will transfer data, but you could write your own .avi to .png tunnel, with a display out to the pins...
Sounds pretty complex.
I've once tried watching a HD movie on a 144 pixel screen, and it was surprisingly visible. Not detailed, but I could still understand what was happening on the screen, without reading subtitles (they're just a garble at 144Pix).

I would probably increase the resolution...
The display will act as a screen, but definitly not meant to be HD. I have actually already built the mp4 -> png -> LED tunnel, it works fine with a 12 * 12 matrix, 60fps with no issue, I now have to test if the Pi can address the 1200 LEDs fast enough.

Return to “General discussion”