Massively upload small files without straining CPU?

[Cubytus]

Hello there,

this is more a general question on Linux usage.

I have a project to use Motion to record small video clips and encode them in h264, as I was told the Pi is optimized for.

However, while scriptable, my previous experience with Motion hasn't been stellar. Recording was fine, but uploading numerous files to a FTP server invariably crashed the router when open connection count reached 3000. I used a very simple command based on wput, but with so many small files to be uploaded at the same time, it clearly didn't have the time to close connections after a successful upload.

I was told to simply use a ready-made cloud storage software, but somehow I doubt many of them are compiled for ARM architecture.

So what would be a good way to upload numerous small files to a FTP server in the order they are finalized, without crashing the router, and taking into account the Pi, already encoding, can't spare much power for uploading?

[Jessie]

Is it possible for your server to unpack a tar file? If not I would say if 3000 connections crashes your router then add something to your script to limit the number of concurrent connections. May be as simple as telling it to wait a few ms after starting each upload or as complex as establishing a queue and limiting simultaneous uploads. When you say 3000 connections is that because you have 3000 files to upload or are you using something like BT that allows multiple connections per file?

[Cubytus]

Strictly speaking un-tar-ing the file isn't necessary, but I believe it could be run periodically since the FTP server I intend to use will be a WD MyBook Live, i.e. a limited, outdated Debian distro. As much as possible I'd like to avoid modifying the disk too much, as it's pretty easy to brick when one makes a mistake. Unlike the Pi, it's impossible to recover without dismantling the thing. In fact, I'm pretty apprehensive thinking about creating a FTP user with write-only capability.

On my shared hosting FTP, though, I don't have access to low-level tools, unless the "untar" function can be run in PHP. I wouldn't have the necessary knowledge to write it myself, though. My last course in PHP was more than 10 years ago, and I never really practiced it.

The main point of uploading discrete files is working around inherently limited upload bandwidth. What I could not understand is that files were created sequentially (though at a very high rate), not simultaneously, and I assumed the wput function implicitly queued them in the same order. Nothing fancy as BT was ever used, and on Motion's mailing list one suggested, basically, to give up on the issue and just let a cloud-sharing folder such as Dropbox manage upload.

How many ms does wput needs to properly close? I can't add too much of a delay since saving files remotely as fast as possible is absolutely critical in a home monitoring application, before either the video camera or uploading computer or router is destroyed or stolen (power can be managed with homemade DC UPS using a small battery).

Of course I am still open to more streamlined solutions. ZoneMinder appears fine on paper, but is an absolute nightmare to get working.

[Jessie]

Is there any way to increase the recording length so that you are generating less small files? I understand what your trying to do but if your router crashes while you are out won't it defeat the whole purpose? You may also have to consider a higher end router. Adding a small delay of 10 or so ms may be all you need if it is easy to add to your script you may give it a shot.

My point with the tar balls was to package a handful of these video files to gather so that not so many connections were open in the first place. There have to be already written scripts for taking the contents out of a tar file and deleting the tar file upon upload. If this doesn't work for you then disregaurd consider it a free bump (after stuff hits page 2 it is rarely answered)

[Cubytus]

Is there any way to increase the recording length so that you are generating less small files? I understand what your trying to do but if your router crashes while you are out won't it defeat the whole purpose?

Indeed it does. Originally it also captured still snapshots, because previous installations didn't support h264 encoding. I'd be curious to know what h264 encoder could be called with simple parameters on a non-Pi installation. Increasing the length of the command line would only lead to hard-to-debug issues.

You may also have to consider a higher end router.

This is not considered at the moment since a more recent router would set me back around $300, and that 3000 connections are just too high to begin with. I don't feel like throwing more money on hardware since the problem appears to be software.

Adding a small delay of 10 or so ms may be all you need if it is easy to add to your script you may give it a shot.

Will try that as soon as the NoIR module arrives.

My point with the tar balls was to package a handful of these video files to gather so that not so many connections were open in the first place. There have to be already written scripts for taking the contents out of a tar file and deleting the tar file upon upload. If this doesn't work for you then disregaurd consider it a free bump (after stuff hits page 2 it is rarely answered)

Well since network connection wasn't so reliable and in case the recording devices are still in place, I also wanted to keep a local copy. As I remember, Motion offered a way to cut files after a set period of time, but the system I had back then did take a non-negligible amount of time to start a new file (slow disk access time?), enough that it would miss fast movement in front of the camera. It was also rather difficult to record for a significant length of time before file was too big to be uploaded properly. A non-uploaded file (from router crash or too large a size) isn't worth a dime.

[plugwash]

Does it have to be a ftp server?

If you control what runs on the server then rsync would be my tool of choice for moving large numbers of small files arround.

[Cubytus]

It doesn't have to. I chose FTP out of compatibility concern since it's the only file transfer protocol available on most shared hosting and that having a backup based upon a MBL is novel to me. I would still be interested to verify if adding a delay after each iteration of wput allows for properly closing a connection. When I did upload many files by hand with this command, I never experienced such a problem, of course I can't type nearly fast enough to match Motion.

If rsync burns less CPU power than wput+script, then it could be a very interesting alternative on the Pi, since the MBL does have it preinstalled (thus no risk of bricking it). Do you have some data on that?