coderaemon
Posts: 11
Joined: Thu Apr 09, 2015 6:45 am

Picamera Timestamps ?

Thu Apr 09, 2015 7:10 am

I have been recording uncompressed(yuv format) videos using Picamera. I am using custom output to capture Presentation timestamps (pts) for each frame. Code Below.

Code: Select all

import io
    import picamera
    import time
    
    class PtsOutput(object):
    	def __init__(self, camera, video_filename, pts_filename):
    		self.camera = camera
    		self.video_output = io.open(video_filename, 'wb')
    		self.pts_output = io.open(pts_filename, 'w')
    		self.start_time = None
    
    	def write(self, buf):
    		self.video_output.write(buf)
    		if self.camera.frame.complete and self.camera.frame.timestamp:
    			if self.start_time is None:
    				self.start_time = self.camera.frame.timestamp
          	    self.pts_output.write('# timecode format v2\n')
       	        self.pts_output.write('%f\n' % ((self.camera.frame.timestamp - self.start_time) / 1000.0))

    	
    	def flush(self):	
    		self.video_output.flush()
    		self.pts_output.flush()
    		
    	def close(self):
    		self.video_output.close()
    		self.pts_output.close()	
    		
    with picamera.PiCamera() as camera:
    	camera.resolution = (320,240)
    	camera.framerate = 30
    	camera.start_recording( PtsOutput(camera, 'test_30fps.yuv', 'pts.txt'), format='yuv' )
    	camera.start_preview()
    	camera.wait_recording(60)
    	camera.stop_recording()
I have multiple questions:

1. What is the exact definition of a Presentation time stamp.? how accurate are these time stamps ?

2. I calculate the inter-frame duration (time duration between any two consecutive frames using presentation time stamps.) The precision is too good. 99.9% frames have same inter-frame duration. Is it expected?

3. 0.1% frames have a different inter-frame duration than the most common value. Does this mean that camera's frame rate decreases/increases ? Why this happens? And in this case also do the presentation time stamps remain consistent, or they lose their accuracy ?

Thanks in Advance.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7742
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Picamera Timestamps ?

Thu Apr 09, 2015 11:10 am

PTS is the STC value when the start of frame interrupt is received by the CSI-2 receiver (GPU side). You should only get interrupt latency jitter of the order of 10's of nsecs on that time (unlike if it were being handled under Linux). If running the sensor at a constant framerate, then yes you should expect them to be almost identical.

You don't state what the abnormal values are, but I suspect your issue comes if you as the client don't return buffers to the GPU fast enough. At that point the GPU has no option but to drop a frame that the sensor has produced. Your timestamp delta will then be twice (or if you're you're really delayed, a larger multiple) of the normal time.
SD card writing is slow, and there is minimal extra buffering within your code to smooth it out, so if the SD card has to stop to erase cells or similar you will get a drop. Confirm by writing the video to /dev/null. You may get better results if writing to a USB HDD, or an NFS share.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

User avatar
jbeale
Posts: 3572
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: Picamera Timestamps ?

Thu Apr 09, 2015 4:30 pm

If the PTS is measured by one clock (derived from oscillator on the R-Pi board) but the start-of-frame signal comes from a different clock source (oscillator on the camera board controlling frame start & read) then you will expect occasional measured frame-to-frame intervals that are one clock period longer or shorter, as the different clock edges drift past each other (the two oscillators on separate boards are not phase-locked). This is in addition to the n*(frame time) offset if you completely drop (n) frames, as 6by9 mentioned.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7742
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Picamera Timestamps ?

Thu Apr 09, 2015 5:16 pm

jbeale wrote:If the PTS is measured by one clock (derived from oscillator on the R-Pi board) but the start-of-frame signal comes from a different clock source (oscillator on the camera board controlling frame start & read) then you will expect occasional measured frame-to-frame intervals that are one clock period longer or shorter, as the different clock edges drift past each other (the two oscillators on separate boards are not phase-locked). This is in addition to the n*(frame time) offset if you completely drop (n) frames, as 6by9 mentioned.
The lowest level timings are in nsecs, whilst PTS is usec. The clock drift will be masked by the interrupt latency at the nsec level, and then it's a simple /1000 on the conversion to usec so you may see a +/-1us error, every so often anyway.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

coderaemon
Posts: 11
Joined: Thu Apr 09, 2015 6:45 am

Re: Picamera Timestamps ?

Mon Apr 13, 2015 6:39 am

Thanks for your detailed answers.
The 0.1% frames have inter-frame duration like: 65.65 usec, 131.31 usec while the 99.9% are of the order of 32.8 +-0.05 usec.
Another thing I found out is that if I record at lesser resolution then the inter-frame-duration is constant at 32.8usec. So is the discrepancy (0.1% frames) in inter-frame duration is occurring because of the writing more data at higher resolution?

Another question, is there any relationship between camera clock and pi clock? I have both the timestamps from camera clock and pi-clock but can't find any relationship. The distributions are random.

P.s: Can let me know the name of person who designed the camera board I have few more queries I can ask him directly.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7742
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Picamera Timestamps ?

Mon Apr 13, 2015 7:26 am

coderaemon wrote:Thanks for your detailed answers.
The 0.1% frames have inter-frame duration like: 65.65 usec, 131.31 usec while the 99.9% are of the order of 32.8 +-0.05 usec.
Another thing I found out is that if I record at lesser resolution then the inter-frame-duration is constant at 32.8usec. So is the discrepancy (0.1% frames) in inter-frame duration is occurring because of the writing more data at higher resolution?
32.8 * 2 = 65.6
32.8 * 3 = 98.4
32.8 * 4 = 131.2
So you're seeing a combination of 1 or 3 frames drops (I suspect there are some 98.4ms gaps in there too somewhere where 2 frames get dropped).
Just check to see if it is the SD card by writing to /dev/null instead (as I suggested in my earlier response). You appear to be writing uncompressed video, so even at 320x240 you'll be generating 3.4Mbytes/s of data = 27.6Mbit/s I don't know of any SD cards that claim to sustain that rate.
coderaemon wrote:Another question, is there any relationship between camera clock and pi clock? I have both the timestamps from camera clock and pi-clock but can't find any relationship. The distributions are random.
See viewtopic.php?f=43&t=106181&p=732407#p732407
GPU timestamp is the raw value off the STC. Linux will do some munging, but is going to be based on the same clock. Use MMAL_PARAM_TIMESTAMP_MODE_RAW_STC to get the STC and also get the Linux clock, and you've then got the offset. Unless the Linux clock gets updated somehow, then they should stay in sync.
coderaemon wrote:P.s: Can let me know the name of person who designed the camera board I have few more queries I can ask him directly.
Pass. The camera board design isn't public (although it isn't that complex anyway). Ask here on the forums and there are a few of us lurking that may be able to answer the question.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

coderaemon
Posts: 11
Joined: Thu Apr 09, 2015 6:45 am

Re: Picamera Timestamps ?

Mon Apr 13, 2015 2:06 pm

PTS is the STC value when the start of frame interrupt is received by the CSI-2 receiver (GPU side).
Sorry but I'm bit new to this jargon. Can you explain above quote in easier terms. What I understand is that PTS is the time value according to pi-camera clock when signal (frame interrupt) of a beginning of a frame is received by GPU ? STC value I tried googling but couldn't find the exact meaning.
Also my pi-clock values inter-frame duration(in usecs) are something like
39.99996185
29.99997139
29.99997139
29.99997139
The average comes out to be nearly 32.5 usec which is the inter-frame duration using PTS. I can see that on an average there is not much drift between the two clocks (1 sec drift in an hour recording) . But any day I would trust PTS (if it is from camera clock) more than time.time() that I am using to get pi-clock time as pi will be doing lot of different things?

P.s : On a higher level my problem is to somehow sync an external clock i.e a neural data acquisition clock with either pi clock or pi-camera clock. I can send a pulse from data acquisition to pi to start things off but then I will have to know what lag is there and when are the frames exactly been acquired i.e what PTS is, I am hoping.

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7742
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Picamera Timestamps ?

Mon Apr 13, 2015 2:51 pm

coderaemon wrote:
PTS is the STC value when the start of frame interrupt is received by the CSI-2 receiver (GPU side).
Sorry but I'm bit new to this jargon. Can you explain above quote in easier terms. What I understand is that PTS is the time value according to pi-camera clock when signal (frame interrupt) of a beginning of a frame is received by GPU ? STC value I tried googling but couldn't find the exact meaning.
Also my pi-clock values inter-frame duration(in usecs) are something like
39.99996185
29.99997139
29.99997139
29.99997139
The average comes out to be nearly 32.5 usec which is the inter-frame duration using PTS. I can see that on an average there is not much drift between the two clocks (1 sec drift in an hour recording) . But any day I would trust PTS (if it is from camera clock) more than time.time() that I am using to get pi-clock time as pi will be doing lot of different things?

P.s : On a higher level my problem is to somehow sync an external clock i.e a neural data acquisition clock with either pi clock or pi-camera clock. I can send a pulse from data acquisition to pi to start things off but then I will have to know what lag is there and when are the frames exactly been acquired i.e what PTS is, I am hoping.
STC = System Time Clock.
Pi (Linux) clock values may vary as there is a bundle of extra processing that will introduce variation in when the frame is delivered from the GPU to the userspace client.

With the timestamps you quote,you say that the PTS deltas are all the same but Pi (Linux) clock times vary. Did you open a file when you received the first buffer, or do some other slow operation that could have stalled Linux long enough that your timestamps are then off, but slowly catch up?

Your client can get the STC from userspace by calling

Code: Select all

uint64_t stc;
MMAL_STATUS_T status;
status = mmal_port_parameter_get_uint64(camera->control, MMAL_PARAMETER_SYSTEM_TIME, &stc);
It can be any component port (not just camera->control) as it is actually asking the core rather than the component. Ask waveform80 very nicely and it's the sort of thing he might wrap into a nice easy Python call for you (raise an issue on https://github.com/waveform80/picamera/issues). GPIOs can be accessed already from Python, so almost all of the work is already done to get everything working off the STC.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

coderaemon
Posts: 11
Joined: Thu Apr 09, 2015 6:45 am

Re: Picamera Timestamps ?

Wed Apr 15, 2015 11:20 am

If presentation time stamp is system time clock (STC) why does it always start with zero ? According to definition given in picamera module
timestamp : Returns the presentation timestamp (PTS) of the current frame as
reported by the encoder. This is represented by the number of microseconds
(millionths of a second) since video recording started. As the frame
attribute is only updated when the encoder outputs the end of a frame, this
value may lag behind the actual time since start_recording()
was called.
So its the time since the recording started ?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7742
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Picamera Timestamps ?

Wed Apr 15, 2015 12:02 pm

coderaemon wrote:If presentation time stamp is system time clock (STC) why does it always start with zero ? According to definition given in picamera module
timestamp : Returns the presentation timestamp (PTS) of the current frame as
reported by the encoder. This is represented by the number of microseconds
(millionths of a second) since video recording started. As the frame
attribute is only updated when the encoder outputs the end of a frame, this
value may lag behind the actual time since start_recording()
was called.
So its the time since the recording started ?
OK, I know the MMAL and firmware layers, and those would normally be returning the raw STC in Raspivid.

Picamera is using a slightly different mode by the looks of https://github.com/waveform80/picamera/ ... ra.py#L573 Switch it to MMAL_PARAM_TIMESTAMP_MODE_RAW_STC and you'll get the raw value off the STC as I was referring to.
(There was a bug that meant RESET_STC mode wasn't working as expected, but was fixed about 6 months back. I'm not quite sure what results were being achieved before that via Picamera)
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

coderaemon
Posts: 11
Joined: Thu Apr 09, 2015 6:45 am

Re: Picamera Timestamps ?

Fri Apr 17, 2015 7:09 am

Picamera is using a slightly different mode by the looks of https://github.com/waveform80/picamera/ ... ra.py#L573 Switch it to MMAL_PARAM_TIMESTAMP_MODE_RAW_STC and you'll get the raw value off the STC as I was referring to.
(There was a bug that meant RESET_STC mode wasn't working as expected, but was fixed about 6 months back. I'm not quite sure what results were being achieved before that via Picamera)
I tried changing the code as quoted above. But still timestamps start from zero. I asked waveform80 at a different forum http://raspberrypi.stackexchange.com/qu ... mera-clock about time stamps. He says its the time at which camera starts acquiring video hence it starts with zero. It has nothing to do with pi clock which is the system clock you are referring to, if I am not wrong.
Also in viewtopic.php?f=43&t=98541 you mention that camera oscillator is faster than pi oscillator by a factor of 25/24.8. I checked by recording a video at 20 frames-per-second(fps) for 60 secs. If everything is fine, It should give me 1200 frames. But it gives me 1216 frames, so something is wrong. I checked the duration of the video recording also (by pointing the camera at a digital clock) and it gives me exactly 60 secs. So the time for which camera will acquire video is accurate but the frame-rate is increased by the camera ? What is happening here ?

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 7742
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: Picamera Timestamps ?

Fri Apr 17, 2015 8:10 am

coderaemon wrote:I tried changing the code as quoted above. But still timestamps start from zero. I asked waveform80 at a different forum http://raspberrypi.stackexchange.com/qu ... mera-clock about time stamps. He says its the time at which camera starts acquiring video hence it starts with zero. It has nothing to do with pi clock which is the system clock you are referring to, if I am not wrong.
Why did you not link to your question on stackexchange when you first posted here?
He is correct that the pts values as produced by the stock picamera python library will be 0-based at the start of recording. Even so it is related directly to the STC, just with a start offset subtracted.
coderaemon wrote:Also in viewtopic.php?f=43&t=98541 you mention that camera oscillator is faster than pi oscillator by a factor of 25/24.8. I checked by recording a video at 20 frames-per-second(fps) for 60 secs. If everything is fine, It should give me 1200 frames. But it gives me 1216 frames, so something is wrong. I checked the duration of the video recording also (by pointing the camera at a digital clock) and it gives me exactly 60 secs. So the time for which camera will acquire video is accurate but the frame-rate is increased by the camera ? What is happening here ?
You're not up to date. viewtopic.php?f=43&t=99351&p=701280

Being blunt, I'm the one who wrote this part of the firmware having worked on the imaging stack at Broadcom for 7 years, so I can be pretty authoritative as to what the GPU side is doing. If set to RAW_STC mode, then it is spitting out the Pi STC at the point that the interrupt handler for the CSI-2 receiver got the FRAME_START event.
This is the mode that raspivid works in - investigated under viewtopic.php?f=43&t=98541&p=689513#p684870, though ethanol100's -pts option seems not to have been merged to the main userland repo as yet)

waveform80 is the authority on what the Picamera python library is doing as he wrote that bit.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

coderaemon
Posts: 11
Joined: Thu Apr 09, 2015 6:45 am

Re: Picamera Timestamps ?

Mon Apr 20, 2015 7:44 am

So you are saying RAW_STC is pi-clock time and that is being written as PTS. But I am also outputting time.time() in a separate file which I was calling the pi-clock time. Why is the distribution (inter-frame duration) of those timestamps(time.time()) different from PTS if both are from the same clock ? Also the total time of the video I calculated using both timestamps ( pi-clock & PTS) there is a small variation in that too. Is time.time() having a bit of lag while returning the time , and that is causing the variation in both timestamps.
Another question is that if all the timestamps being taken from STC then what is the use of pi-camera clock ? I have asked
waveform80 this , just wanted to know your answer too.
Thanks a lot , you have helped much already.

Return to “Camera board”