robypez
Posts: 3
Joined: Tue Nov 19, 2019 2:36 pm

Raspberry Pi4 to measure HDMI input lag

Tue Nov 19, 2019 2:46 pm

Hello everybody, I want to use a Raspberry Pi4 to measure the input lag of a TV.

Basically I need to generate a small white square on the screen using the HDMI (millisecond = 0) and I need to read a value from a photosensor. When the photosensor capture the brightness change it stops the timer, and I have the time, in millisecond, the video signal need for visualization.
I will continue the measure in loop, so I can have an average.

I'm a total raspberry pi beginner, which OS do you suggest? Can I do this with Python?

Heater
Posts: 13885
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspberry Pi4 to measure HDMI input lag

Tue Nov 19, 2019 4:01 pm

I suggest using the standard Raspbian Linux operating system as supplied by the Pi Foundation. https://www.raspberrypi.org/downloads/

It is certainly capable of doing what you want and is the best supported with the most help available on this forum.

I would be wary of using Python for this due to it's performance and non-deterministic timing. Personally I would use a compiled language that does not require garbage collection, Pascal, C. C++, Rust, perhaps others I have not thought of.

Be aware that you will have to take care using any Linux for any such time measurements in that range. Linux is a multi-users, multi-tasking operating system so your timing can be interrupted by having your program rescheduled by the kernel at any time. Thus introducing a lot of jitter into your measurements.

However that are some fairly simple steps one can take to reduce such jitters down to sub ms time scales and averaging a lot of measurements will make your result more accurate.

We can discuss those steps when you have your OS and hardware set up.

Edit: Of course you will not just be measuring the lag of the TV you are testing. You will also be measuring the lag of getting that white square from your program to the HDMI through the Pi graphics hardware. I suspect that will already be 16ms for the 60fps of the Pi's VPU, or whatever it is.
Memory in C++ is a leaky abstraction .

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7533
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raspberry Pi4 to measure HDMI input lag

Tue Nov 19, 2019 5:02 pm

This is the question that I've never fully investigated with HDMI and displays.

It's a serial link, so there's a temporal difference between when the first and last pixel are presented over the link. Do displays work in the same way as rolling shutter cameras and update as they are fed the data, or do they have a backing memory that is updated, and at vsync they switch all pixels at once?
What do the response time values that displays always quote actually measure? I believe it's just the LCD reaction time, not the time taken to get a suitable update in from a source to refresh the specific pixel. I'd even query how you'd manage to update a frame within the few milliseconds that the monitor quotes.

The Pi can provide you with a callback when it transmits the vsync to the display, but the first line of the display update will have been 16ms before that for a 60Hz mode.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24133
Joined: Sat Jul 30, 2011 7:41 pm

Re: Raspberry Pi4 to measure HDMI input lag

Tue Nov 19, 2019 5:16 pm

At 1:35 in to this video you can see a beat frequency between the camera and the LCD monitors. I wonder if that means the displays are rolling shutter?

https://www.youtube.com/watch?v=8GXtwR71huE


TBH, just an excuse to show the video.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7533
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raspberry Pi4 to measure HDMI input lag

Tue Nov 19, 2019 5:24 pm

Yes, I think they are rolling updates, same as old CRTs used to be.
So that does mean that the time to update relative to vsync will vary depending on which line you're looking at.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

robypez
Posts: 3
Joined: Tue Nov 19, 2019 2:36 pm

Re: Raspberry Pi4 to measure HDMI input lag

Tue Nov 19, 2019 8:44 pm

Mmm… usually the input lag on a good display is 13 to 15 ms. Maybe the raspberry is not the right device to build something to measure it...

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7533
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raspberry Pi4 to measure HDMI input lag

Tue Nov 19, 2019 10:02 pm

I think the message should be to think very carefully about what exactly you are trying to measure.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

noggin
Posts: 99
Joined: Sun Feb 21, 2016 1:55 pm

Re: Raspberry Pi4 to measure HDMI input lag

Wed Nov 20, 2019 9:29 am

6by9 wrote:
Tue Nov 19, 2019 5:02 pm
This is the question that I've never fully investigated with HDMI and displays.

It's a serial link, so there's a temporal difference between when the first and last pixel are presented over the link. Do displays work in the same way as rolling shutter cameras and update as they are fed the data, or do they have a backing memory that is updated, and at vsync they switch all pixels at once?
What do the response time values that displays always quote actually measure? I believe it's just the LCD reaction time, not the time taken to get a suitable update in from a source to refresh the specific pixel. I'd even query how you'd manage to update a frame within the few milliseconds that the monitor quotes.

The Pi can provide you with a callback when it transmits the vsync to the display, but the first line of the display update will have been 16ms before that for a 60Hz mode.
Couple of thoughts on this from someone who has used HDMI displays in reasonably time-critical display scenarios (think lip sync in broadcast control room monitoring)

1. Scaling of non-native resolution signals (or over scanning at native resolution) usually requires a degree of delay. Whilst you can scale at line rate (introducing just a few lines of delay in the scaling process, I think a lot of systems use a frame store approach to store the output frame)
2. Deinterlacing of native interlaced signals usually requires a degree of delay - as for anything other than Bob they need both fields to generate a frame. (Some high-end broadcast monitors let you go for a lower quality deinterlace to reduce latency, but you still need to scale fields to frames)
3. Processing of video signals (domestic TVs offer noise reduction, contrast enhancement, live colour)
4. Temporal interpolation of video signals (Natural Motion, Motion Flow etc.) require motion analysis across multiple frames prior to display.

Computer monitors with reduced processing options, and cheaper displays, will have lower latency, but I'd expect most to have a frame store approach that is likely to be introduce a frame (or deinterlaced frame/field) delay at a minimum (so 16ms at 60Hz, 20ms at 50Hz) It is possible that some monitors specifically optimise for minimal latency and do as much as possible at line rate, and VGA and HDMI still include line and field blanking periods that allow for this. (CRT monitors fed HDMI->SDI can be very low latency)

One the other hand I've seen domestic TVs with 160+ms delays with all the processing enabled...

In terms of rolling shutter type display, I don't think that happens significantly with LCD, the flicker that you see on camera in some situations is more a function of backlight modulation (some displays pulse the backlight to insert black frames to improve motion rendition for instance), and you don't see shutter bar effects like you used to with off-frequency CRTs.

ISTR that one of the areas of research that was carried out by TV researchers was how the difference between tubed cameras and tubed displays (which were 'in sync rolling' displays) compared with modern CCD + Flat Panel displays - where the whole frame was captured and displayed as 'whole frame' snapshots (i.e. top left and bottom right samples were taken and displayed for identical exposure times AND displayed simultaneously) The move from smoothly continous sampling and display to a more 'snapshot' based approach had a difference in eye-brain perception when it came to motion rendition.

It's something that the broadcast industry is having to think about again - as whilst CCD cameras were pretty universally global capture devices (whole frame captured simultaneously), CMOS cameras can be rolling shutter. Rolling shutter CMOS cameras are definitely avoided where possible in broadcast (the first gen UHD 4K cameras used for broadcast had them - but Sony's latest range have global shutters to avoid this. Similarly for their electronic 'film' cameras - the cheaper ones roll, the more expensive ones don't)
Last edited by noggin on Wed Nov 20, 2019 9:38 am, edited 1 time in total.

robypez
Posts: 3
Joined: Tue Nov 19, 2019 2:36 pm

Re: Raspberry Pi4 to measure HDMI input lag

Wed Nov 20, 2019 9:38 am

I'm the CTO of a magazine and we test TV.
Everyone who review TV use this device

https://www.leobodnar.com/shop/?main_pa ... cts_id=212

This is 1080p, and of course no 4K no HDR.
Input lag is something very important for gamers, they want to but the TV with lowest input lag. I want to add a measure for several resolution, so not only 1080p

The device we use now has an asic + a hdmi controller, and it's very easy: generate a white square on the tv and with a photoresistor measure the time it takes for the signal to arrive on the screen. I know that there are some errors, but this is the standard now. And is 1080.
I think to create something similar with raspberry, but maybe this is not the right dev board.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7533
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raspberry Pi4 to measure HDMI input lag

Wed Nov 20, 2019 10:17 am

noggin wrote:
Wed Nov 20, 2019 9:29 am
Couple of thoughts on this from someone who has used HDMI displays in reasonably time-critical display scenarios (think lip sync in broadcast control room monitoring)
Thank you! I like learning new things from those in the know.
noggin wrote:1. Scaling of non-native resolution signals (or over scanning at native resolution) usually requires a degree of delay. Whilst you can scale at line rate (introducing just a few lines of delay in the scaling process, I think a lot of systems use a frame store approach to store the output frame)
2. Deinterlacing of native interlaced signals usually requires a degree of delay - as for anything other than Bob they need both fields to generate a frame. (Some high-end broadcast monitors let you go for a lower quality deinterlace to reduce latency, but you still need to scale fields to frames)
3. Processing of video signals (domestic TVs offer noise reduction, contrast enhancement, live colour)
4. Temporal interpolation of video signals (Natural Motion, Motion Flow etc.) require motion analysis across multiple frames prior to display.

Computer monitors with reduced processing options, and cheaper displays, will have lower latency, but I'd expect most to have a frame store approach that is likely to be introduce a frame (or deinterlaced frame/field) delay at a minimum (so 16ms at 60Hz, 20ms at 50Hz) It is possible that some monitors specifically optimise for minimal latency and do as much as possible at line rate, and VGA and HDMI still include line and field blanking periods that allow for this. (CRT monitors fed HDMI->SDI can be very low latency)

One the other hand I've seen domestic TVs with 160+ms delays with all the processing enabled...
I hadn't thought about the amount of potential processing for scaling etc in the monitor, only the source.
Yes, scaling from non-native resolution will require at least a line or two of delay, but frame based is often easier.
Interlacing I like to ignore these days. Interlacing with scaling probably causes even more grief, unless you do the trivial thing merging the two fields and then scaling the result.
Processing opens another can of worms, and probably one that you don't want in the broadcast industry, at least on the production side.
noggin wrote:In terms of rolling shutter type display, I don't think that happens significantly with LCD, the flicker that you see on camera in some situations is more a function of backlight modulation (some displays pulse the backlight to insert black frames to improve motion rendition for instance), and you don't see shutter bar effects like you used to with off-frequency CRTs.

ISTR that one of the areas of research that was carried out by TV researchers was how the difference between tubed cameras and tubed displays (which were 'in sync rolling' displays) compared with modern CCD + Flat Panel displays - where the whole frame was captured and displayed as 'whole frame' snapshots (i.e. top left and bottom right samples were taken and displayed for identical exposure times AND displayed simultaneously) The move from smoothly continous sampling and display to a more 'snapshot' based approach had a difference in eye-brain perception when it came to motion rendition.

It's something that the broadcast industry is having to think about again - as whilst CCD cameras were pretty universally global capture devices (whole frame captured simultaneously), CMOS cameras can be rolling shutter. Rolling shutter CMOS cameras are definitely avoided where possible in broadcast (the first gen UHD 4K cameras used for broadcast had them - but Sony's latest range have global shutters to avoid this. Similarly for their electronic 'film' cameras - the cheaper ones roll, the more expensive ones don't)
4k global shutter sensors :o those are going to be pricey! The most you tend to see are around the 2MPix mark, and those aren't cheap.
Perhaps I need to speak to our friends at Sony about them :D No guarantee that they're CSI2 sensors anyway.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7533
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Raspberry Pi4 to measure HDMI input lag

Wed Nov 20, 2019 10:35 am

robypez wrote:
Wed Nov 20, 2019 9:38 am
I'm the CTO of a magazine and we test TV.
Everyone who review TV use this device

https://www.leobodnar.com/shop/?main_pa ... cts_id=212

This is 1080p, and of course no 4K no HDR.
Input lag is something very important for gamers, they want to but the TV with lowest input lag. I want to add a measure for several resolution, so not only 1080p

The device we use now has an asic + a hdmi controller, and it's very easy: generate a white square on the tv and with a photoresistor measure the time it takes for the signal to arrive on the screen. I know that there are some errors, but this is the standard now. And is 1080.
I think to create something similar with raspberry, but maybe this is not the right dev board.
Thanks for clarifying your actual use case - it's impossible to tell from many posts whether people are hobbyists playing around and thinking something might be a good idea, or people who actually know what they're doing with a specific use case in mind.

You can arrange for the Pi to update the screen from eg black to white in sync on a VSYNC, and then time until the photoresistor detects a change. You probably want to be looking at either the libDRM API, or DispmanX. Both can provide you with a vsync callback or a flip/update confirmation callback from which you can start your timer, and then you can monitor a GPIO for the the output of your photoresistor.

Note that HDR is still to be implemented on the Pi4, and support through DRM has only been added in about the 5.3 Linux kernel release (we're currently still on 4.19, but bringing up 5.4 is well underway).
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

Heater
Posts: 13885
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspberry Pi4 to measure HDMI input lag

Wed Nov 20, 2019 10:56 am

Hmm... 150ms delay, just in the TV.

It's kind of sobering to recall that back in the day pixels came in shades of grey and the lit up the phosphor on your TV screen a handful of microseconds after the camera detected the photos in the scene it was looking at. The signal having traveled through the entire broadcast system!

We don't have live action anymore.
Memory in C++ is a leaky abstraction .

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 24133
Joined: Sat Jul 30, 2011 7:41 pm

Re: Raspberry Pi4 to measure HDMI input lag

Wed Nov 20, 2019 11:08 am

Heater wrote:
Wed Nov 20, 2019 10:56 am
Hmm... 150ms delay, just in the TV.

It's kind of sobering to recall that back in the day pixels came in shades of grey and the lit up the phosphor on your TV screen a handful of microseconds after the camera detected the photos in the scene it was looking at. The signal having traveled through the entire broadcast system!

We don't have live action anymore.
https://www.youtube.com/watch?v=ue7wM0QC5LE
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I think it’s wrong that only one company makes the game Monopoly.” – Steven Wright

Heater
Posts: 13885
Joined: Tue Jul 17, 2012 3:02 pm

Re: Raspberry Pi4 to measure HDMI input lag

Wed Nov 20, 2019 12:47 pm

jamesh,

Aye lad, we had it tough.

Least them ruddy TV's n' Radios didn't try to spy on us all the time.
Memory in C++ is a leaky abstraction .

User avatar
HermannSW
Posts: 1655
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Raspberry Pi4 to measure HDMI input lag

Mon Nov 25, 2019 5:24 pm

Heater wrote:
Wed Nov 20, 2019 10:56 am
Hmm... 150ms delay, just in the TV.
The article pointed to in the product link says between 16.9ms and 129.8ms -- 11 are below 50ms, 18 below 100ms:
https://www.cnet.com/news/game-mode-on- ... input-lag/


Today I realized that uv4l does not work under Buster. I found this thread, but the method described did not work:
https://www.raspberrypi.org/forums/view ... ?p=1510262

At least streaming mjpeg did work, and I wanted to measure lag of mjpeg stream. Before I used online stopwatch for that, but the milliseconds display in a very small font.

So I wrote this small program that displays seconds modulo 100 and milliseconds, always at same place. Minimizing size of terminal with increasing fontsize gives a good millisecond precision watch for lag control:

Code: Select all

// gcc -O6 -Wall -Wextra -pedantic millis.c -o millis
//
#include <stdio.h>
#include <time.h>

int main()
{
  struct timespec time1;

  while (1)
  {
    clock_gettime(CLOCK_REALTIME, &time1);
    fprintf(stderr, "\r%02ld.%03ld", time1.tv_sec%100, time1.tv_nsec/1000000);
  }
  return 0;
}

I did start mjpeg stream with this command (from uv4l manpage with width/height/framerate added):

Code: Select all

$ uv4l --auto-video_nr --sched-rr --driver raspicam --encoding mjpeg --width=640 --height=480 --framerate=10 --server-option '--port=9001' --server-option '--admin-password=myp4ssw0rd!'
Image


I did test with Pi4B Buster wirelessly connected to home AP, as well as old Ubuntu 18.04.3 LTS laptop wirelessly connected to AP.

Initially lag is fine, 659-283=376ms, but this increases gradually and after 2 minutes lag is 2877-2363=514ms. Later the stream becomes unusable. I will need to figure out how to get uv4l running under Buster, works perfectly under Stretch on Pi2B.

P.S:
Since the two v1 cameras on Arducam stereo hat are hardware synchronized, left and right camera part of mpjep stream show always identical timer values.

P.P.S:
I wrote a test program that did execute the inner loop exactly 1,000,000 times and did measure time before and after the loop. The average time taken per loop execution is 5927.26us, which means the output gets generated 168 times per second, much higher than display refresh rate.
⇨https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264
https://github.com/Hermann-SW/fork-raspiraw
https://twitter.com/HermannSW

User avatar
HermannSW
Posts: 1655
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Raspberry Pi4 to measure HDMI input lag

Tue Nov 26, 2019 7:05 pm

Today's posting might shed some light on HDMI input lag, find details there:
https://www.raspberrypi.org/forums/view ... 3#p1572013

Whenever a uv4l mjpeg stream starts, it displays camera (preview?) on HDMI monitor as well.
HDMI lag is 260ms(!), browser lag is 676ms:
Image
⇨https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264
https://github.com/Hermann-SW/fork-raspiraw
https://twitter.com/HermannSW

User avatar
HermannSW
Posts: 1655
Joined: Fri Jul 22, 2016 9:09 pm
Location: Eberbach, Germany
Contact: Website Twitter YouTube

Re: Raspberry Pi4 to measure HDMI input lag

Thu Nov 28, 2019 10:55 am

Further correction, it was not a good idea to run millis under X11.
Did measurements again running millis without X11 in console, for v1 camera and v1 stereo camera.
Both have HDMI lag <100ms!
https://www.raspberrypi.org/forums/view ... 5#p1572894
Image
⇨https://stamm-wilbrandt.de/en/Raspberry_camera.html

https://github.com/Hermann-SW/Raspberry_v1_camera_global_external_shutter
https://stamm-wilbrandt.de/github_repo_i420toh264
https://github.com/Hermann-SW/fork-raspiraw
https://twitter.com/HermannSW

Return to “General discussion”