Dispmanx VNC Server

[hanzelpeter]

Hello.

I have been playing with dispmanx_snashot and using LibVNCServer and created a simple VNC server.
It takes snapshots from screen via dispmanx and sends them via VNC.
It can also view the 3D demos from hello_pi.

The code is here:
https://docs.google.com/file/d/0B0ImsXF ... sp=sharing
The video is here (55MB):
https://docs.google.com/file/d/0B0ImsXF ... sp=sharing


The keyboard is working, mouse is missing.
I set the FPS to 15 FPS and when nothing redraws on screen the CPU Usage it about 22%. When top in console is showed the CPU goes to 30%.

If someone want to try and any comments are welcome.
How can I achieve less CPU usage and so on.

[hanzelpeter]

And one comment. Tested only headless with TV out as display.

[AndyD]

This looks useful. If anyone is interested in building it a quick note you need to install libvncserver-dev

Code: Select all

sudo apt-get install libvncserver-dev

The code looks like it only works for 16 bits per pixel. Is that correct?

[hanzelpeter]

Should work with all resolutions. 16bit is only transport layer.
Dispmanx is saving snapshots to 16bit RGB (RGB565) and then VNC is setup to use RGB555. So only shifts in VNC server can be problem.

[AndyD]

Yes, sorry ... I understand now. You are creating a resource of type VC_IMAGE_RGB565 to do the snapshot, then removing the least significant blue bit. Thanks!

[AndyD]

Hi, there appears to be an alignment problem. I tried your VNC server with my HDMI screen (well HDMI->VGA adapter 1366x768) plugged in. The screen had an offset problem (the image was skewed across the screen). If you look at the hello_dispmanx example, the rows of pixels are 32 bit aligned. I am pretty sure that is what you need to do. I am working on a snapshot utility that saves PNG file and my code had the same problem. Once I changed my code so that the rows of the buffer are 32 bit aligned my snapshot came out correct.

[elektrknight]

hanzelpeter very nice and useful utility!
Are you planning on creating a project/repository for the dispmanx_vnc?

[danversj]

Looks very interesting!

I've been looking for a method to access the XBMC GUI (and video playback) remotely. Most raspberry pi implementations write directly to GLES, bypassing X11 and /dev/fb0 and so the GUI can't be accessed this way. Can dispmanx access what's being written to GLES?

Ultimately I'd also like to be able to pipe the XBMC GUI out as an h264 network stream (slingbox-type behaviour), but with my attempts so far, the colorspace conversion from RGB565 to YUV is done in the CPU, and basically is too slow to to a live stream.

[hanzelpeter]

Hello.

to AndyD: Yes, that's a bug and once I upload it to github.com or somewhere I will try to fix.
to danversj: Yes, that's also my idea to transfer screens as video. I have been looking into encode.c and found also OMX_COLOR_Format16bitBGR565 in definitions for color format. Does this works for video also? I have tried yet. Maybe you can try and when I will have time I will also try. (maybe this constant is not valid for video format....)

[danversj]

hanzelpeter, I've found it a bit hard to work out what will work using hello_encode because all it does is take an internally generated yuv test pattern and encodes it to a raw h264 file. I've changed the constant to def.format.video.eColorFormat = OMX_COLOR_Format16bitRGB565; and the output is just corrupted video - because what's being fed in is YUV. I don't know how to feed it real video.

I think we'd have more luck with either omxtx or omxtranscode. Unfortunately I can't get either to compile, I'm pretty clueless about compiling unless everything is automatically is in the right place and I can just type "make".

Looking at OMX.broadcom.video_encode I see that it does allow OMX_COLOR_Format16bitRGB565 as an input - if we could get omxtx or omxtranscode working we could play with this to see if it does the colorspace conversion when encoding AVC (h264) - h264 doesn't work with RGB AFAIK. If it doesn't work there's another option - OMX.broadcom.resize which can do the colorspace conversion in the GPU. I know omxtx has resize working, and it appears to have some support for OMX_IndexParamImagePortFormat.

[danversj]

Just an update - I've found that if I stop lightdm and run xbmc in standalone mode, I do indeed get XBMC menus and video playback via VNC! Excellent! Of course it's laggy and choppy, but it's a massive step forward. Now if we can feed Dispmanx_snaphot into omx.broadcom.encode we should have something very cool!

If only there was a VNC client that could decode h264.

Edit: and it looks like someone has done just that! http://lists.gnu.org/archive/html/qemu- ... 01388.html

[AndyD]

Yes, that's a bug and once I upload it to github.com or somewhere I will try to fix.

Thanks. I have a working version here. I have spent a while modifying your code, just because it is fun. I am now trying to get the mouse working. One thing that may cause problems for the mouse is the distortion in the Y direction (when headless or using composite output). As you reported and I added to here.

I've found it a bit hard to work out what will work using hello_encode because all it does is take an internally generated yuv test pattern and encodes it to a raw h264 file. I've changed the constant to def.format.video.eColorFormat = OMX_COLOR_Format16bitRGB565; and the output is just corrupted video - because what's being fed in is YUV. I don't know how to feed it real video.

This sounds interesting. Playing with the dispmanx vnc server it is clear the performance suffers when the whole screen is changing (reported fps goes down to about 4). The H264 video idea sounds interesting. When I get some time I will try and change the encode example to use vc_dispmanx_snapshot() to fill the buffer. I may just be able to get a VC_IMAGE_YUV420 snapshot. The one concern I have about this is that by using the snapshot function and then encoding, the data is being pulled from the GPU to user space and then pushed back to do the encoding, which may impact on performance. I wonder if there is a way to do this without touching user space at all (except of course to read the encoded video stream).

[hanzelpeter]

Hello.

I have created github project on it.
Check this: https://github.com/hanzelpeter/dispmanx_vnc
We can start on collaborate. In this version also first mouse draft is done.

[AndyD]

Thanks, sent you a pull request.

Quick question about the mouse code. I see that you are using EV_ABS for the x and y coordinates. That was what I tried too, as LibVncServer sends absolute coordinates to the callback. I couldn't get it to work though (with say the hello_triangle2 example). So, I changed to EV_REL and kept copies of the previous x and y values and worked out the deltas. This allowed the hello_triangle2 example to work with the mouse.

[jamesh]

Although you could use H264, you might get better performance than currently with a much simpler compress of individual frames - you could either go to JPEG or PNG. But the problem there is that would be done on the CPU whereas H264 would be done on the GPU. Or do you do that already? I pressume you already analyse the frame for area changes and only send the changes?

[AndyD]

Although you could use H264, you might get better performance than currently with a much simpler compress of individual frames - you could either go to JPEG or PNG. But the problem there is that would be done on the CPU whereas H264 would be done on the GPU. Or do you do that already? I pressume you already analyse the frame for area changes and only send the changes?

Hi James,

I am not the original author, but I am now pretty familiar with the code. The library being used (LibVncServer) does all the negotiation between the client and the server in term of the encoding being used (It appears to use JPEG encoding when I have used it).

The code compares the current frame with the previous frame. It finds the smallest rectangle (only one) that fits all the changes that have been made since the previous frame. The code walks from each edge of the frame until it finds a difference. This (dirty) rectangle is then used by the library to work out what to send. It works well enough, but code like hello_triangle kill the performance as most of the screen is marked as dirty.

I am assuming that to take snapshots and encode to H264 we need to pull the snapshot into user space with vc_dispmanx_snapshot/vc_dispmanx_resource_read_data and then push it back to the GPU to encode. Is that correct?

[dom]

I am assuming that to take snapshots and encode to H264 we need to pull the snapshot into user space with vc_dispmanx_snapshot/vc_dispmanx_resource_read_data and then push it back to the GPU to encode. Is that correct?

I believe that will be necessary. Possibly it won't make too much difference, as the transfer is done by dma, and it's probably dwarfed by network costs.
If you implement it that way, and profile how long each of the stages takes, I could look into a way of bypassing the copy (but only if the profile says it's a bottleneck).

[hanzelpeter]

Hello.

As you can see on github I merged the pull request.
With the mouse it will be fun. I tried only with X window and due to headless TV out and overscan framebuffer I get the mouse working but VNC cursor if above/left to X windows pointer. I then tried to run minecraft and the mouse was woking there, but moving too fast in 3D, but in menu of minecraft it was exact movement, I did not tried with hello_triangle2.

Maybe the REL mouse is the better way. I will try.

[snapso]

this is very impressive.

I've just compiled this on my RaspBMC PI and the output of the screen works beautifully.

Unfortunately for me the keyboard input doesn't work, I suspect this is due to the fact raspbmc doesn't have bmc_host.h included and so I downloaded some of the latest firmware files into /opt/vc/ to enable this to compile.

I can use my android phone as an XBMC remote and see the screen updates/refreshes working so might attempt a full firmware update and re-compile to see if the keyboard works.

thank you for this.

[hanzelpeter]

Hello.

The keyboard and mouse can be tricky. I use "kernel uinput system". So it is like adding new virtual keyboard to system.
And the application must be capable of handling this. For example if I want to use DispmanX_VNC with X window system I need to modprobe evdev. (it is not loaded automatically by boot of RPi). Then it creates evdev input devices which X can handle.

So I don't know if there is a way to check this module. Or I can check X11Vnc how to handle keyboard another way.

[danversj]

I got this to work on Openelec on the Pi. Performance was pretty bad with 1920x1080, as expected . But not too bad with 720x480 (composite video monitor connected locally). You get XBMC menus and video playback, and mouse and keyboard work. dispmanx_vncserver uses about 20% CPU when running, and gets about 6-10fps. With the vnc server running, when playing video (via omxplayer) you do get frequent black flashes in the video on the locally-connected monitor.

I compiled dispman_vncserver on Raspbian then copied these files from Raspian:

Code: Select all

~/dispman_vnc/dispman_vncserver
/usr/lib/arm-linux-gnueabihf/libvncserver.so.0
/usr/lib/arm-linux-gnueabihf/libjpeg.so.8
/usr/lib/arm-linux-gnueabihf/libgnutls.so.26
/usr/lib/arm-linux-gnueabihf/libtasn1.so.3
/usr/lib/arm-linux-gnueabihf/libp11-kit.so.0

over to the /storage/vncserver directory on Openelec. I then added the /storage/vncserver directory to $LD_LIBRARY_PATH.

[snapso]

Update:

it appears the keyboard interface works fine if I disable the Raspbmc xbmc auto-start. So there must be something happening there to disable the uinputs.

thanks again for this great little vnc server, I may tinker with some tweaks myself but because I don't have much faith in my c++ skills I'm not expecting to get very far.

[dom]

With the vnc server running, when playing video (via omxplayer) you do get frequent black flashes in the video on the locally-connected monitor.

This should be fixed in latest firmware. Run rpi-update.

[AndyD]

... I may just be able to get a VC_IMAGE_YUV420 snapshot. ...

Sorry for the delay. I have tried calling vc_dispmanx_snapshot() using a resource handle created with type VC_IMAGE_YUV420. I can run the code once, and vc_dispmanx_snapshot() returns -1. If I try and run it again the code hangs in vc_dispmanx_snapshot() and requires a reboot.

[hanzelpeter]

That's sad to here. I will try this. But I was confused with searching for VNCViewer that supports H264 encoding.
I had a look in libvncclient and there is a condition compilation and so on. So I don't know if for example RealVNC viewer for windows supports H264 encoding.

Does someone knows?