Robi_G1
Posts: 14
Joined: Wed May 15, 2019 7:23 am

Raspiraw stuttering

Wed May 15, 2019 7:56 am

I've been playing around with Raspiraw and an 'unsupported' sensor, and have so far successfully got 640x480 raw images out of it. The problem is that during records, every x amount of time (can be 10s, can be 1s, appears to be random) I get a stutter of maybe 1/2 a second (again, it varies). This problem starts appearing around 30fps, but occasionally it happens at 15fps too.

I thought I had this pinned down to write speed (12.2MB/s is pretty hefty after all). Trying a slow USB stick resulted in far more stuttering than the SD card, and writing to RAM /dev/shm resulted in no stuttering. So I went out and bought a fast SD card, but the problem persists. Not writing headers to files (decreasing transfer rate to 11.3MB/s) gives a negligible improvement.

When recording at 60fps to SD card, I get more images than at 30fps, so the problem cannot be absolute transfer rate (else you'd expect to get the same amount of photos, as indeed I do when I record at 90fps compared to 60fps). I therefore assume that some OS process is blocking writes every so often.

The application I'm using the camera for requires long, unbroken record periods. So yesterday I set up swap space to be 52GB, and recorded to /dev/shm. This solved the problem of jitters (I assume because the RAM acts as a cache to smooth out transfers to the SD card), but only the first 55478 images were saved. To save myself the pain of the limit on 'rm', I put the images in a folder in /dev/shm for a repeat run overnight. This morning I now see that only 55477 images have been saved (1 fewer).

So my current theory is that there's something limiting the number of files /dev/shm can hold, regardless of the size of the files. But I don't know if this is something in a config file, or some inherent limit to raspbian. I'm running a test at 60fps now with some extra files and folders in /dev/shm to see if the number of images I get still scales linearly (will update when that finishes running).

So my question is (finally :P ), is there a way of removing this apparent file limit, or is there a way of configuring a ramdisk to act as a cache for the SD card instead of bodging it with swap space as is currently the case. Or failing that, has anyone else come across stuttering and since solved it, or have any related information to anything above.

Many thanks,
Rob

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 22742
Joined: Sat Jul 30, 2011 7:41 pm

Re: Raspiraw stuttering

Wed May 15, 2019 8:33 am

It does sound like SD card write speed issue, although there are other possibilities. You could write your own caching in to raspiraw, allocate a truck load of memory (all of it) and write each frame in to that, then have a separate thread writing those images to SD card. Of course you will eventually run out of memory, as you are writing in to the cache faster than writing out, but it should last longer. I have no idea if that would be better than using the inbuilt OS caching system.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

Robi_G1
Posts: 14
Joined: Wed May 15, 2019 7:23 am

Re: Raspiraw stuttering

Wed May 15, 2019 8:39 am

File limit appears to be confirmed. The addition of 31 files / folders to /dev/shm resulted in 55447 images saved, summing to 55478. So If there's a way of removing this artificial (at least it looks that way) limit, then that immediately solves everything.

On my list of TODOs for today is:
* Post of rpi forums and pray (done :lol: )
* Try out ZRAM
* Get to the bottom of apparent file limit in /dev/shm
* Try partitioning most of the Pi SD card as XFS and see if writing to that (forgetting about RAM) improves things.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 22742
Joined: Sat Jul 30, 2011 7:41 pm

Re: Raspiraw stuttering

Wed May 15, 2019 9:07 am

Robi_G1 wrote:
Wed May 15, 2019 8:39 am
File limit appears to be confirmed. The addition of 31 files / folders to /dev/shm resulted in 55447 images saved, summing to 55478. So If there's a way of removing this artificial (at least it looks that way) limit, then that immediately solves everything.

On my list of TODOs for today is:
* Post of rpi forums and pray (done :lol: )
* Try out ZRAM
* Get to the bottom of apparent file limit in /dev/shm
* Try partitioning most of the Pi SD card as XFS and see if writing to that (forgetting about RAM) improves things.
Is /dev/shm simply running out of space (or perhaps inodes)? Might be possible to remount it with larger numbers of 'things' to give more space.

Or have a seperate thread copying files out of /dev/shm to storage.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

User avatar
rpdom
Posts: 14483
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Raspiraw stuttering

Wed May 15, 2019 9:12 am

File limit is usually related to the number of inodes allocated (depends on the file system).

What does

Code: Select all

df -i /dev/shm
show?

I get a total of 125022 on a recent Raspbian Stretch install. One of those will be used for /dev/shm itself, the rest should be available for files and directories under /dev/shm.

Robi_G1
Posts: 14
Joined: Wed May 15, 2019 7:23 am

Re: Raspiraw stuttering

Wed May 15, 2019 9:24 am

What does

Code: Select all

df -i /dev/shm
show?

I get a total of 125022 on a recent Raspbian Stretch install. <snip>
Well this is interesting...
55479 inodes, of which 55478 are free.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 22742
Joined: Sat Jul 30, 2011 7:41 pm

Re: Raspiraw stuttering

Wed May 15, 2019 11:06 am

Robi_G1 wrote:
Wed May 15, 2019 9:24 am
What does

Code: Select all

df -i /dev/shm
show?

I get a total of 125022 on a recent Raspbian Stretch install. <snip>
Well this is interesting...
55479 inodes, of which 55478 are free.
I think you have your answer!
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

Robi_G1
Posts: 14
Joined: Wed May 15, 2019 7:23 am

Re: Raspiraw stuttering

Wed May 15, 2019 11:15 am

jamesh wrote: I think you have your answer!
It looks that way. Thanks to the both of you for the fast replies and assistance.

Is there a way to change inode rate when installing Raspbian like there is in Debian with it's “standard”, “news”, “largefile”, "largefile4” options?

Or could I install raspbian normally, copy everything off the SD card, reformat the ext4 partition w/ more inodes, then copy everything back?

If not then I'm a bit stuck, because installing from Noobs doesn't give any options, and just writing the Raspbian image will do its own thing with the file systems.

(Does this now warrant a new thread?)

User avatar
rpdom
Posts: 14483
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Raspiraw stuttering

Wed May 15, 2019 11:55 am

I'm sure there is a way of defining parameters for /dev/shm when it gets created during the boot process, but I don't know where it is.

You can change it "on the fly" like this

Code: Select all

sudo mount -o remount,nr_inodes=1000000 /dev/shm
which will give you a lot of inodes to play with.

Hmm, I think you can put a line in /etc/fstab to define the parameters

Code: Select all

none    /dev/shm    tmpfs    defaults,nr_inodes=1000000
Caution: untested.

Just out of interest, what model of Pi are you using, and which version of Raspbian?

Robi_G1
Posts: 14
Joined: Wed May 15, 2019 7:23 am

Re: Raspiraw stuttering

Wed May 15, 2019 12:44 pm

rpdom wrote:
Wed May 15, 2019 11:55 am
I'm sure there is a way of defining parameters for /dev/shm when it gets created during the boot process, but I don't know where it is.

You can change it "on the fly" like this

Code: Select all

sudo mount -o remount,nr_inodes=1000000 /dev/shm
which will give you a lot of inodes to play with.

Hmm, I think you can put a line in /etc/fstab to define the parameters

Code: Select all

none    /dev/shm    tmpfs    defaults,nr_inodes=1000000
Caution: untested.

Just out of interest, what model of Pi are you using, and which version of Raspbian?
That's great, all the other places I've looked have said I'd need to reformat before changing. You have saved me several hours.

Will be giving everything a try now, and making sure that the files it's writing actually contain something (because that would be the next fun thing...). Will report back if everything works (it looks pretty likely that it will).

I'm running Pi Zero W v1.1, Raspbian 4.14.98+ #1200

User avatar
rpdom
Posts: 14483
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Raspiraw stuttering

Wed May 15, 2019 1:13 pm

Robi_G1 wrote:
Wed May 15, 2019 12:44 pm
I'm running Pi Zero W v1.1
Aha, that makes sense. The default sizes for the file system are based on the amount of memory available. I was running the df command on a Pi 3B, which has twice the memory of the Zero, so that would be why I had a larger default number of inodes/files than you.

Robi_G1
Posts: 14
Joined: Wed May 15, 2019 7:23 am

Re: Raspiraw stuttering

Wed May 15, 2019 3:21 pm

Good news and bad news.
Good news: I can now save all the files I like.

Bad news: Once I fill RAM (~188MB on the Pi Zero), all the files saved are empty. So I have ~450 files of 407kB (good), then a few 1000 files of 0B (not good).

So it looks like swap space isn't being used. I tried with 600 images and the same thing happened. Am I being stupid and not flicking a switch somewhere, or is /dev/shm not allowed swap space.

Robi_G1
Posts: 14
Joined: Wed May 15, 2019 7:23 am

Re: Raspiraw stuttering

Wed May 15, 2019 3:45 pm

Update:
Wrote a super-simple python script to move all files from /dev/shm to SD card every seconds. 5 minutes in and things appear to be going well.

It's been a full day of learning. Thanks for the help.

Return to “Camera board”