User avatar
perfo
Posts: 92
Joined: Mon Jan 30, 2012 2:42 am

Read Pixel colour from screen.

Thu Mar 14, 2019 8:44 am

I've put this in the general programming as I'm open to suggestions whether in C or python. I would prefer python as I understand it a lot more but maybe able to work with C if it isn't too complex.
So As I stated I want to pick a pixel at (X,Y) being displayed on screen (HDMI output).
is the Frame buffer the way to go ? Does the frame buffer contain whatever is currently being shown on the screen ? I ask as I found a bit of code in C that lists pixel data from the frame buffer but it seems to give me the same results no matter if the screen has a white back ground or a black one so I'm thinking there may be more to it...

Would any one have suggestions on the simplest way to do this ?
It seem fairly common to do this to the camera input (capture scan) or a jpeg but I can't find much about doing it to the screen.

Thanks

Andyroo
Posts: 3378
Joined: Sat Jun 16, 2018 12:49 am
Location: Lincs U.K.

Re: Read Pixel colour from screen.

Thu Mar 14, 2019 8:53 am

If no one comes up with an answer you could always use scrot to capture the screen and then look at the resulting file :lol:
Need Pi spray - these things are breeding in my house...

User avatar
perfo
Posts: 92
Joined: Mon Jan 30, 2012 2:42 am

Re: Read Pixel colour from screen.

Thu Mar 14, 2019 10:10 am

Thanks Andyroo.

The thing is I need to do this live so whatever method I use has to be able to process it at least as fast as the frame rate of the video (ie 60 FPS)

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 22747
Joined: Sat Jul 30, 2011 7:41 pm

Re: Read Pixel colour from screen.

Thu Mar 14, 2019 11:19 am

perfo wrote:
Thu Mar 14, 2019 10:10 am
Thanks Andyroo.

The thing is I need to do this live so whatever method I use has to be able to process it at least as fast as the frame rate of the video (ie 60 FPS)
That sort of speed seems unlikely, depending on requrerements, but you can work it out. How many pixels do you want to read per frame?

The frame buffer approach seems appopriate. You can open the /dev/fb0 file, mmap to memory and read from that perhaps? https://gist.github.com/Darfk/5790622 seems to do this.

Note that this will not work with camera output or video display as they are not presented to the framebuffer. You might be able to use dispmanx_snapshot in those cases?
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

User avatar
perfo
Posts: 92
Joined: Mon Jan 30, 2012 2:42 am

Re: Read Pixel colour from screen.

Thu Mar 14, 2019 12:48 pm

Hello Jamesh
I only want to read one pixel per frame no matter what the resolution is. I was hoping with only reading one it may enable it to be speedy enough,

I'm not interested in the camera for this application so that's not a problem but what do you mean by the video display is not presented to the frame buffer. It's the HDMI output I want to sniff pixels from is this the video display ? or do you mean the composite video output on the other port ?

I'm having a fiddle with the file you gave me a link to ...... I'm on 32bpp and it's set up for 8 but I'll play and see if I can figure anything out...

thanks...
Last edited by perfo on Thu Mar 14, 2019 1:07 pm, edited 1 time in total.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 22747
Joined: Sat Jul 30, 2011 7:41 pm

Re: Read Pixel colour from screen.

Thu Mar 14, 2019 1:01 pm

perfo wrote:
Thu Mar 14, 2019 12:48 pm
Hello Jamesh
I only want to read one pixel per frame no matter what the resolution is. I was hoping with only reading one it may enable it to be speedy enough,

I'm not interested in the camera for this application so that's not a problem but what do you mean by the video display is not presented to the frame buffer. It's the HDMI output I want to sniff pixels from is this the video display ? or do you mean the composite video output on the other port ?
thanks...
If you running some sort of video output via the GPU (e.g. using the HW aceleration) then the framebuffer never sees that information - its is composited over the top of the frame buffer in HW straight to HDMI. So reading a pixel will not get the video but the underlying framebuffer.

So ifyou want to get a pxiel from video, you need to use dispmanx_snapshot (or whatever its called), which will grab the entire image, which you would then grab the pixel from. However, that will be a LOT slower.

If I think of another way of doing it will let you know!
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

User avatar
perfo
Posts: 92
Joined: Mon Jan 30, 2012 2:42 am

Re: Read Pixel colour from screen.

Thu Mar 14, 2019 4:56 pm

Thanks for the tips James...
I'm using omxplayer to play a video and trying to grab a pixel from that and so far I wasn't getting much sense. Maybe you've just explained why that is...

I have other posts on this forum asking questions as I've tried different ways to achieve my project. My project is very simple in principle. I want an output (gpio) pin to go high when the video shows a particular frame.

I've tried using the vsync from dispmanx but that doesn't appear to be in sync with the frames. So that's why I'm thinking if I can hard code a particular pixel in the video to be a particular colour for pin on and another for pin off then if I can grab that pixel in code and control the GPIO I will always be in p[perfect sync with the frame regardless of refresh rates or fps of video etc..

bzt
Posts: 374
Joined: Sat Oct 14, 2017 9:57 pm

Re: Read Pixel colour from screen.

Tue Mar 26, 2019 2:24 pm

Hi,

To access the framebuffer read this. It's taking about plotting a pixel, all you need to do is to reverse that and read instead.
There are many example C codes out there, for example this one has an fb_getpixel32() routine.

But framebuffer is not the only way. If your video player is using X11, then you can get the info from it's window. That should be simpler, more portable and faster (because it's using RAM only, no MMIO). See X protocol and the Xlib manual. If your video player is fullscreen, then you'll probably need to use the XRootWindow(), otherwise I suggest a tool like xprop or xdotool to find the window id and use that. Example code here, grabc has interactive mode but you can also specify window id and x,y offset.

Those are all C sources, but there's a library to access Xlib from python (not sure how up-to-date though).

Cheers,
bzt

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 22747
Joined: Sat Jul 30, 2011 7:41 pm

Re: Read Pixel colour from screen.

Tue Mar 26, 2019 2:44 pm

bzt wrote:
Tue Mar 26, 2019 2:24 pm
Hi,

To access the framebuffer read this. It's taking about plotting a pixel, all you need to do is to reverse that and read instead.
There are many example C codes out there, for example this one has an fb_getpixel32() routine.

But framebuffer is not the only way. If your video player is using X11, then you can get the info from it's window. That should be simpler, more portable and faster (because it's using RAM only, no MMIO). See X protocol and the Xlib manual. If your video player is fullscreen, then you'll probably need to use the XRootWindow(), otherwise I suggest a tool like xprop or xdotool to find the window id and use that. Example code here, grabc has interactive mode but you can also specify window id and x,y offset.

Those are all C sources, but there's a library to access Xlib from python (not sure how up-to-date though).

Cheers,
bzt
As mentioned above, the OP is playing a video using OMXplayer, and that data never enters the framebuffer, so FB operations will not work.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

User avatar
perfo
Posts: 92
Joined: Mon Jan 30, 2012 2:42 am

Re: Read Pixel colour from screen.

Wed Mar 27, 2019 9:32 am

Thanks for the detailed answer you put some effort I to it and that is appreciated.
I did manage to get pixel stuff from the FB but as JamesH points out that was before I found out OmxPlayer didn't use the frame buffer and thus I couldn't make head nor tail out of the info I was managing to get. I believe I can play OMXplayer in an X11 window with the -b switch. So I may give that a go and see if it works any better...

I guess another question would be, would there be a fast video player that does use the FB ?
Thanks again...

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 22747
Joined: Sat Jul 30, 2011 7:41 pm

Re: Read Pixel colour from screen.

Wed Mar 27, 2019 9:44 am

perfo wrote:
Wed Mar 27, 2019 9:32 am
Thanks for the detailed answer you put some effort I to it and that is appreciated.
I did manage to get pixel stuff from the FB but as JamesH points out that was before I found out OmxPlayer didn't use the frame buffer and thus I couldn't make head nor tail out of the info I was managing to get. I believe I can play OMXplayer in an X11 window with the -b switch. So I may give that a go and see if it works any better...

I guess another question would be, would there be a fast video player that does use the FB ?
Thanks again...
No. Moving the data from the Videocore to the ARM then into the frame buffer is a lot of operations, so will always be slower. It might be quick enough, but will definitely be slower.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

User avatar
perfo
Posts: 92
Joined: Mon Jan 30, 2012 2:42 am

Re: Read Pixel colour from screen.

Wed Mar 27, 2019 11:30 am

Do you know of any players I could try ?

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 22747
Joined: Sat Jul 30, 2011 7:41 pm

Re: Read Pixel colour from screen.

Wed Mar 27, 2019 12:08 pm

VLC maybe?
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

User avatar
perfo
Posts: 92
Joined: Mon Jan 30, 2012 2:42 am

Re: Read Pixel colour from screen.

Wed Mar 27, 2019 1:01 pm

Ah the ol reliable VLC... I'll give it a bash and see what happens...
Thanks

bzt
Posts: 374
Joined: Sat Oct 14, 2017 9:57 pm

Re: Read Pixel colour from screen.

Thu Mar 28, 2019 9:13 am

Hi,

What about mplayer? It's a standard raspbian package. It has the capability to play in a X11 window, but you can also use it on a Linux console framebuffer without any Xserver at all.

It has a companion mencoder command which could be useful. With that you can convert each frame into a picture (like bmp or png) in real tme and pass that to your application to parse without the need of displaying the actual video on screen.

Cheers,
bzt

User avatar
perfo
Posts: 92
Joined: Mon Jan 30, 2012 2:42 am

Re: Read Pixel colour from screen.

Thu Mar 28, 2019 11:00 am

Thanks for the suggestion. I've got a few things to have a bash at this weekend....

User avatar
perfo
Posts: 92
Joined: Mon Jan 30, 2012 2:42 am

Re: Read Pixel colour from screen.

Thu Mar 28, 2019 11:01 am

out of interest , would you know of a player that does the opposite of Mencoder ? That's is I give it jpegs and it plays them as a video ?

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 22747
Joined: Sat Jul 30, 2011 7:41 pm

Re: Read Pixel colour from screen.

Thu Mar 28, 2019 11:06 am

Vooya?
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

User avatar
perfo
Posts: 92
Joined: Mon Jan 30, 2012 2:42 am

Re: Read Pixel colour from screen.

Thu Mar 28, 2019 11:48 am

Great thanks. another avenue to explore this weekend.

bzt
Posts: 374
Joined: Sat Oct 14, 2017 9:57 pm

Re: Read Pixel colour from screen.

Thu Mar 28, 2019 10:55 pm

perfo wrote:
Thu Mar 28, 2019 11:01 am
out of interest , would you know of a player that does the opposite of Mencoder ? That's is I give it jpegs and it plays them as a video ?
Mplayer can do that. Both mencoder and mplayer use the same codecs and filters, all can be used as input and for output as well, the only difference is that mencoder saves the result in a file (or streams to network) while mplayer displays it on screen.

Code: Select all

mplayer mf://*.jpg
Cheers,
bzt

User avatar
perfo
Posts: 92
Joined: Mon Jan 30, 2012 2:42 am

Re: Read Pixel colour from screen.

Fri Mar 29, 2019 11:55 am

So could I feed it jpegs with a C script ?
So I'd feed it jpeg 1 then switch my GPIO on then feed it Jpeg 2 then GPIO off etc ?

Return to “General programming discussion”