IR strobe for piNoir in sync with raspivid

[BerryPicker]

Previously elsewhere on these forums:
viewtopic.php?p=941630#p941630

... I modified the lights with my own drive circuit to be strobe lights that run at the frame rate of the camera, so even fast-moving objects appear as crisp still frames instead of motion-blurred. It is not perfect because I don't have a way to synchronize the strobe to the start-of-frame, so some images show moving objects split in half by one frame exposure time. However I get far more detail on fast motion than with a normal 360 degree shutter that you typically use at night. ... Doing a short 2 msec pulse at 23 Hz on the IR LEDs allows me to run 4x more current than I could at DC.

viewtopic.php?p=902677#p902677

It would be perfect if I could synchronize my light to the RPi camera frame start, but I haven't figured out that part yet. It works OK as-is, there is just one region of most frames that has a double-exposure appearance where one rolling-shutter frame starts and another one is ending at the moment the IR strobe goes on.

There does now seem to be a way to do this as the raspivid code checks for the MMAL_BUFFER_HEADER_FLAG_FRAME_END flag. So by tapping into the code at the point where this flag is known to be set, a timing source for frame synchronisation can be generated.
As yet, I haven't found any example code for the particular synchronisation requirement of driving an IR strobe light. However example code which uses the flag check for other purposes does exist, and shows how the flag can be used. My intention is to use the pi's hardware pwm as a flywheel with its frequency adjusted to the arrival rate of the set frame end flags.
C++
For those wishing to derive sync by customising the userland code, there is a detailed guide by Martin O'Hanlon in his blog at
http://www.stuffaboutcode.com/2013/09/r ... pivid.html
Python
2. For those not wanting to disturb the installed picamera code there is a detailed guide in readthedocs for picamera. This example shows how to access to the buffer header flags and allow you to run your own code in response to every frame.
http://picamera.readthedocs.org/en/rele ... index.html

[6by9]

There does now seem to be a way to do this as the raspivid code checks for the MMAL_BUFFER_HEADER_FLAG_FRAME_END flag. So by tapping into the code at the point where this flag is known to be set, a timing source for frame synchronisation can be generated.

Sorry, that really isn't signalling what you think it is. H264 frames can be split over multiple buffers. That flag indicates that the this buffer is the last one in an H264 frame. That is so far downstream from the sensor (->CSI-2 receiver->ISP->video codec->raspivid) with some jitter and buffering at each level that it is of no use for indicating synchronisation of the sensor
From a quick read of Martin O'Hanlon's blog, it looks like he's only reading frame number (or is it timestamp?) from the Raspivid app. Nothing too exciting there over the exact frame timing.

Thinking slightly around the issue, each frame timestamp as seen in Raspivid is derived from the GPU STC (System Time Clock) for the start of frame interrupt. There's also a MMAL parameter call you can make to get the current STC. You could now derive a software PLL within Linux slaved to that, but it's non-trivial and will be prone to some jitter due to scheduling. If you know your exposure time, then you could sync illumination off that.
We've been round the loop of hardware sync lines many times before - it's not possible on OV5647.

[BerryPicker]

@6by9

Thank you for answering my post. I was hoping that the average of processing time accumulated before the end of frame flag would be a constant. But I now consider my expectations have been managed

I welcome your thoughts on an alternative where the time offset between strobe pulse and camera shutter is more likely to average to a constant. So again using the hardware PWM as a flywheel, I now intend to use the following process.
1. For each PWM cycle obtain:
- the value of the MMAL_PARAMETER_SYSTEM_TIME
- the time value of the PTS of the current frame
2. Calculate the difference in time value and average over (say) 10 PWM cycles
3. Check if the averaged time difference is < or > than the pre-determined time offset
4. If so, check if the variation is by more than a set threshold.
5. If so, change the PWM frequency to reduce the variation.

Hopefully the above process will keep variations in sync time within about 1ms

[jbeale]

Just out of curiosity- is there a fixed timing relationship between the start of frame and the crystal oscillator on the camera board? In other words, does the interval between "start of frame" moments occupy a fixed integer number of ticks of that clock (assuming you don't change camera modes)? Or is there some signal from the GPU that affects frame to frame timing, even after the frame rate and exposure time has been set?

I have noticed that different 3rd party camera boards have very slightly different frame rates from the official board, when used with the same configuration otherwise, which I assume is due to some frequency tolerance of the onboard clock used.

[BerryPicker]

I'm on my way with coding. As a start I've adapted the PiCamera recipe and can now retrieve the presentation time stamps (PTS) from each video frame. The times are in uS, so the least significant digit will be several clock cycles. In the default 30fps the stamps increase by about 33314 with very little variation. With thankful acknowledgement to those publishing the recipes in readthedocs I post my modified code here. Copy and save it as piCameraPTS.py then run it by typing python piCameraPTS.py

Code: Select all

import picamera
import picamera.mmal as mmal
from array import array

# Override PiVideoEncoder to keep track of the number of each type of frame
class MyEncoder(picamera.PiCookedVideoEncoder):
    def start(self, output, motion_output=None):
        self.parent.i_frames = 0
        self.parent.p_frames = 0
        # allow appending to file taking the timestamps
        super(MyEncoder, self).start(output, motion_output)

    def _callback_write(self, buf):
        # Only count when buffer indicates it's a pts timestamp
        if buf[0].pts: # Buffer has MMAL_BUFFER_HEADER_T._fields = pts
            self.parent.myArray.append (buf[0].pts)
            self.parent.i_frames += 1 # Legacy - kept for check
        else: # Buffer does not have pts timestamp
            self.parent.p_frames += 1 # Legacy - kept for check
        # Remember to return the result of the parent method!
        return super(MyEncoder, self)._callback_write(buf)


# Override PiCamera to use our custom encoder for video recording
class MyCamera(picamera.PiCamera):
    def __init__(self):
        super(MyCamera, self).__init__()
        self.i_frames = 0
        self.p_frames = 0
        # open a python array to take the timestamps
        self.myArray = array ("L") # unsigned long

    def _get_video_encoder(
            self, camera_port, output_port, format, resize, **options):
        return MyEncoder(
                self, camera_port, output_port, format, resize, **options)


with MyCamera() as camera:
    camera.start_recording('foo.h264')
    camera.wait_recording(10)
    camera.stop_recording()
    # legacy check
    print('Recording contains %d frames with pts timestamps and %d others' % (
            camera.i_frames, camera.p_frames))
    print ("Presentation Time Stamps (uS)")
    print camera.myArray

[6by9]

Just out of curiosity- is there a fixed timing relationship between the start of frame and the crystal oscillator on the camera board? In other words, does the interval between "start of frame" moments occupy a fixed integer number of ticks of that clock (assuming you don't change camera modes)? Or is there some signal from the GPU that affects frame to frame timing, even after the frame rate and exposure time has been set?

I have noticed that different 3rd party camera boards have very slightly different frame rates from the official board, when used with the same configuration otherwise, which I assume is due to some frequency tolerance of the onboard clock used.

Tolerance of the oscillator is almost certainly the issue.
Framing is all totally contained within the sensor - the SoC just sends it I2C commands to configure it. CSI-2 is totally unidirectional.

I think I've said before, but the original intent with phones/tablets was that the sensor clock was derived from one of the SoC PLLs (and then worked on via the sensor PLLs). On Pi, the flex to the camera became too much of an EMC issue, hence the onboard oscillator.
If the accuracy of that oscillator is in question, then all the calculations to set exposure time and frame rate will be off. All calcs are done based a 25MHz oscillator - I corrected all the numbers a while back as the originals were for 24.8MHz, as normally they run with 19.2, 24.8, or 27MHz source clocks.

[BerryPicker]

The possibility of achieving sync by comparing presentation and system time stamps is still looking promising. With GPIO 17 as input wired to GPIO 18 as hardware PWM pulsing at 30 Hz, and using a rising edge callback to retrieve system time stamps from the camera, the variation from an average over 10 seconds is typically as shown in the chart.

[jbeale]

The possibility of achieving sync by comparing presentation and system time stamps is still looking promising. With GPIO 17 as input wired to GPIO 18 as hardware PWM pulsing at 30 Hz, and using a rising edge callback to retrieve system time stamps from the camera, the variation from an average over 10 seconds is typically as shown in the chart.

This is a very interesting result! Any chance you could share your code?

[BerryPicker]

Any chance you could share your code?

Again with thankful acknowledgement to those whose shoulders I stand on.

To set the hardware PWM pulsing at 30Hz I use this code.

Code: Select all

import wiringpi as wpi

# for BCM GPIO numbers
wpi.wiringPiSetupGpio()

# PWM setup
wpi.pinMode(18,2) # pwm only works on GPIO 18
# pwmFrequency(Hz)= 19.2e6 / ( pwmClock * pwmRange)
# e.g. 50Hz = 19.2e6 / (1920 * 200)
wpi.pwmSetMode(0) # set mark space mode (not balanced - default)
# set the frequency at which the PWM counter is incremented
# 30Hz = 19.2e6 / (640 * 1000)
wpi.pwmSetClock(640) # values from 2 to 4095
# set the range of the PWM counter
wpi.pwmSetRange(1000) # values from 2 up to 4096

wpi.pwmWrite(18,32) # (pin, duty cycle) Duty between 0 and 1024

# cleanup
#wpi.pinMode(18,0) # reset GPIO 18 to input

Copy and save it as pwmHardware.py then run it by typing sudo python pwmHardware.py
Alas, sudo is needed for wiringpi, even with Jessie!

To setup a rising edge callback and use it to retrieve system time stamps from the camera I use this code

Code: Select all

import RPi.GPIO as GPIO
import time

import picamera
import ctypes as ct
import picamera.mmal as mmal
from picamera.exc import (mmal_check)

import numpy as np
import matplotlib.pyplot as plt

def my_get_timestamp(self):
    stc = ct.c_uint64()
    mmal_check(
        mmal.mmal_port_parameter_get_uint64(self._camera[0].control,
            mmal.MMAL_PARAMETER_SYSTEM_TIME, stc),
        prefix="Failed to retrieve camera time")
    return stc.value

def my_callback(channel):
    global count
    global myArray
    count += 1
    myArray [count] = my_get_timestamp(camera)


channel = 17 # wired from GPIO 18 hardware PWM
count = 0
# open a python array to take the timestamps
myArray = np.zeros(330, dtype=np.int64)

with picamera.PiCamera() as camera:
    GPIO.setmode(GPIO.BCM)
    GPIO.setup(channel, GPIO.IN)
    GPIO.add_event_detect(channel, GPIO.RISING, callback=my_callback)
    time.sleep(10)  # collect timestamps for 10 s
    GPIO.cleanup(channel)

# output the collected data    
print('Edges counted = %d ' % (count))
print ("System Time Stamps (us)")
print (myArray)

# check the average
a = 0 # holds sum of timestamp times
d = np.zeros(330,float) # holds difference in time values 
for n in range(1,count):
    d[n] = (myArray[n+1] - myArray[n] - 33333)
    # at 30 fps the expected time between frames is 33333us
    a = a + d[n]
    print d[n] # individual difference between frames
print 'The average difference is ', a / count, 'us'

# view the spread about the average
print 'Busy drawing plot'
# Plot a normalized histogram with 60 bins
plt.hist(d, bins=60, normed=1) # matplotlib
plt.xlabel('Time (us) Early - Late')
print 'Now showing plot. Close plot to finish.'
plt.show()
print 'Done'

Copy and save it as pwmTiming.py then run it from a terminal in a graphical user interface by typing python pwmTiming.py
At first, the imports take a while, but then the camera led will glow whilst the timestamps are being retreived from the camera. After the led extinguishes, there is again a short delay whilst the collected timestamps are processed for display.
Occasionally the displayed data may show there were glitches an order of magnitude bigger than as shown in the chart previously posted. I don't know why this happens, but am not phased by it as the average of all jitter remains close to zero.

[jbeale]

Great, thanks for the code examples! I look forward to playing with this. I am not surprised by the occasional long delays, other people working with interrupt-triggered callbacks have reported that also:

With Linux being Linux, we can expect the latency to change based on other priority tasks. I saw a variation of between 300 uSec up to more than 10mSec. It can vary a lot.

(excerpt from viewtopic.php?f=29&t=133740 )

I clearly haven't been paying attention, I had forgotten Pi had a hardware PWM (normally I use an Arduino when I want PWM). Anyway if there is a way to control the phase as well as frequency of the PWM output, you could have a software phase-locked-loop based on reading the camera timestamp which could use the PWM to generate an external video sync signal. Would not be perfect, but probably close enough for my purposes (within about 1 msec).

[BerryPicker]

More coding progress on an implementation of the strategy proposed by 6by9 The two charts show the error signal which a software PLL within Linux would receive. We can see the scheduling does cause some interesting jitter in the open loop error signal, but hopefully it will be eliminated when the hardware PWM is slaved by closing the loop.

[ivannaz]

This "mmal_port_parameter_get_uint64(port, MMAL_PARAMETER_SYSTEM_TIME, &t_test);" seems to be the function I am looking for.

I will try to close the software PLL loop tomorrow. I I succeed I will post the code here.


Oh, and ... BerryPicker, if you set the GPIO18 to detect rising event before setting it to PWM, you do not need the connection to GPIO17. At least here it worked

[BerryPicker]

Thanks ivannaz, it's good to have help. I look forward to seeing how you close the loop. Remember, its error signal is delayed by about 5 frames, and PiCamera has to be asked to provide 'raw' PTS. I pulled picamera from Git-Hub as this version gives user access to and control of time stamps.

I hope you find a way to avoid ' sudo', and I feel an additional PWM, triggered from the flywheel PWM, will be required to offset and time the strobe light pulse to just the right moment for the camera.

[ivannaz]

here is it:

http://pastebin.com/QZDf0RNV

I set the camera for 40FPS, and drove a LED from PWM. The LED pulse is very narrow; a few microsecs only.

With the program running and the led light reaching the sensor, there is a black stripe crossing horizontally the screen.

Without controlling the sensor fps, this stripe would slowly change position, eventually disappearing at top / bottom, reappearing at bottom / top, and reaching the initial position again. Like an old CRT TV without vsync.

Controlling the FPS, the program keeps the stripe oscillating around a target place that can be changed. Just input a number between 0 and 25000.

The oscillating amplitude is low enough to keep the stripe invisible, if you enter 2000 as target. This target changes for different fps.

So the idea 6by9 gave actually works: viewtopic.php?f=43&t=48238&start=50

Thank you for the support!

[jbeale]

Very good work! I'm snowed under here with another project right now, but I look forward to trying this example out on the weekend.
Meanwhile can you give a hint on what directories I need to set up before compiling the code? Thank you!

[ivannaz]

I am afraid I have not understood exactly what you are asking.

Anyway, I have cross tools and a sysroot built by buildroot.

So the command line for compiling this code is:

/home/ivan/buildroot-git/buildroot/buildroot/output/host/usr/bin/arm-buildroot-linux-uclibcgnueabihf-g++ -c -pipe -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -Os --sysroot=/home/ivan/buildroot-git/buildroot/buildroot/output/host/usr/arm-buildroot-linux-uclibcgnueabihf/sysroot -g -std=gnu++0x -Wall -W -fPIC -DQT_QML_DEBUG -I../syncpi -I. -I../buildroot-git/buildroot/buildroot/output/staging/usr/include -I../buildroot-git/wiringpi/wiringPi/wiringPi -I../buildroot-git/buildroot/buildroot/output/staging/usr/include/interface -I../buildroot-git/buildroot/buildroot/output/staging/usr/include/interface/vcos -I../buildroot-git/buildroot/buildroot/output/staging/usr/include/interface/vcos/pthreads -I../buildroot-git/buildroot/buildroot/output/staging/usr/include/interface/vmcs_host/linux -I../buildroot-git/buildroot/buildroot/output/host/usr/mkspecs/devices/linux-buildroot-g++ -o main.o ../syncpi/main.cpp

(notice a few -I options, passing include files directories. These are needed, otherwise the compiler complains when scanning the mmal include files)

and link with:

/home/ivan/buildroot-git/buildroot/buildroot/output/host/usr/bin/arm-buildroot-linux-uclibcgnueabihf-g++ --sysroot=/home/ivan/buildroot-git/buildroot/buildroot/output/host/usr/arm-buildroot-linux-uclibcgnueabihf/sysroot -o syncpi main.o -lrt -ldl -L/home/ivan/buildroot-git/buildroot/buildroot/output/target/usr/lib/ -lmmal -lmmal_core -lmmal_util -lvcos -lbcm_host -lpthread -lm -lwiringPi

The oscillation in stripe position annoyed me, so I changed the PLL control to "proportional". Now when you input a new offset, the strip goes fast there and stay locked without visible oscillation.

The code now is like:
http://pastebin.com/xTH5jnes

[RaTTuS]

can you post a video ?

[jbeale]

Thanks for the exact command line example. Yes, that is what I was looking for. So far I have always compiled packages like test versions of raspivid and raspistill on the RPi itself, I have not worked with cross-compilation or buildroot before. Looks like I can found out how with http://www.xappsoftware.com/wordpress/2 ... pberry-pi/

[ivannaz]

well, I believe this will compile natively too.... I do not see any reason why not.

But for now it seems this solution will only be useful to me, since the only thing I need is the temporary image buffer, which I will process with a mix of opencv and custom routines.

To be useful to BerryPicker and others.... I am not into it, but i feel they rely on V4L2 or raspivid to extract the stream or images.

If they somehow can, in their program:
1 - extract the pts of a recent frame,
2 - extract the current mmal time (this BerryPicker show it can be done), and together the current system time.
3 - induce a small fps change without stopping the capture,

then they are all set. The only thing needed is reimplement the code in lines 73 to 148 (most are comments). I do not see why it would not work outside the callback, asynchronously. The rest of the program is just some glue to somehow start the capture and the pwm. And is bad at it, since it is trimmed to my specific needs.

Regarding the video, I will try to do a small one, as soon as I reassemble everything here.

[BerryPicker]

Thank you ivannaz for closing the loop and posting your code. It's helpful and encouraging.
I've now also closed the loop on my python implementation and post a link to the video which the code made.
https://www.dropbox.com/s/04zc9wfezdica ... 1.mp4?dl=0
Like you I found it necessary to throw away timings excessively delayed by the pi's most demanding processes. I have also adopted a multi (3) level pull in rate. The main difference is I include an 'average of 8' filter in the feedback loop, and adjust the hardware pwm (not the camera) every 8 frames to lock in sync.
I think what you see is as good as it gets, or am I missing a trick?

[ivannaz]

My first try also left some oscillations, and did no lock that well.

In the second try I changed the control to "proportional", and then it locked perfectly. This means that, instead of one or 2 or (your case) 3 levels of pull, we calculate the pull level on the fly. The bigger the error, the stronger the pull. Of course this goes together with a "saturation level" to avoid pulling too much.

This is in lines 125 and 126 of my second pastebin. I think something similar would stabilize better the strip. In your case you will be calling pwmSetRange (cente_value + pull), where pull could be positive or negative. Should have similar behavior to change
fps on the fly. The nice thing is that the pwm controller behaves well with this call, not resetting the main counter. I know because I have tried

Oh, my intention was to leave the strip out screen. I suppose this is what you want too.

[BerryPicker]

@ivannaz
Thank you again for your helpful comments in closing the loop. The proportional feedback is spot on. It makes the bar come to a halt really smooth. I still get some wander of the bar, but the movement is within 1ms, so overall I'm happy. You seem to have a good feel for what's needed.
I did try the 'on the fly' every frame method. It does also work for me, but my test setup shows the hardware PWM doesn't like it, maybe because it's timing is often being changed at the wrong time, causing it to forget to pulse. The dark band does stay in position, but it flickers. Maybe changing the camera clock's trim is the better way to close the loop. But I have not tried that yet.
I'm assuming the dark band is the point where the shutter is closed. Useful to have visible as a test, but to prevent the frozen motion being split between frames, I'm thinking that in use the strobe timing will need to be @180deg. Again more trials ahead.

[ivannaz]

I believe the dark band is a side effect from the method used to emulate a given exposure time.

See, this sensors do not have some sort of "global reset gate" like CCDs. They reset the collected charge from the pixel by reading it. This is the only way (AFAIK) they could reset the pixel charge.

Now, suppose you specify, say, 20ms as exposure time under 40ms frame period (25fps). You want an exposure time of only half of the frame reading period. The way (I believe) it is done is by generating a "dummy reading", that throws away the collected information, with the only purpose of reseting the pixel charge 20ms before the official reading.

So you have something darkening your pixels fixed time period before the reading, thus creating a dark band. If you set a small exposure time, you will see a big increase in band thickness in those pmw driven lightning, like our particular setup. So try to set an exposure time just a little lower than the frame period. In the above example, the band would have size similar to the image.

Regarding the flickering, I do nor see why it would do it except by having the pwm period exactly half of the needed frequency. Perhaps you could try to call pmwSetRange inside the "ISR" (the simulated interrups call generated by the wiringpi), effectively changing the pwm limit early.