Main Raspbian repository overloaded

[aboudou]

Hello,

As a Raspbian mirror maintainer, I often encounter the following error while trying to sync with the main repository:

Code: Select all

@ERROR: max connections (10) reached -- try again later

Starting 2 am today, I'm unable to mirror the repository (I sync every 2 hours). As the list of official mirrors is still growing, is it possible to either allow more simultaneous connexions, or open another “primary” mirror with a transparent redirection system ?

Thanks.

[elatllat]

I hope you are all using something like zsync to optimize mirroring.

[plugwash]

I've just added a couple of zeros to the limit.

Thanks for notifying me of the problem.

[aboudou]

I've just added a couple of zeros to the limit.

Thanks for notifying me of the problem.

Nice, thanks

[plugwash]

I hope you are all using something like zsync to optimize mirroring.

Please DON'T try to mirror over http even using fancy tools like zsync. Please use continue to use rsync for all mirroring.

Also enabling compression in rsync is probablly not worth it as the vast majority of the data is compressed already.

[elatllat]

I hope you are all using something like zsync to optimize mirroring.

Please DON'T try to mirror over http even using fancy tools like zsync. Please use continue to use rsync for all mirroring.

Also enabling compression in rsync is probablly not worth it as the vast majority of the data is compressed already.

rsync is nice, just out of curiosity;
1. what is the recommended client command (rsync -qru -e rsh --compress-level=0 rsync:://server/src /dest/)?
2. and server command (rsync --daemon --bwlimit=100000)?
3. On the server side are rolling checksums calculations limiting (would it be worth it if the rsync daemon to cached them)?

[plugwash]

Not sure what the best set of rsync options is, i'll have to think about that a bit.

Other than raising the limit on simultanious connections I don't think we did anything special on the server side.

Not sure about the checksum caching, if's the first time I've heard of it. I suspect though that during regular updates the bottleneck is the disk. Unfortunately I don't really know of any good way to measure what proportion of our disks IOPs is in use over time.

[mpthompson]

Based on some questions, a few days ago I updated the Raspbian mirror page with a suggested rsync command. I'm not certain of those are the best options though and would like feedback.

http://www.raspbian.org/RaspbianMirrors

Also, as plugwash mentioned, I'm pretty sure we do not want to enable compression on rsync. Virtually all of the files in the archive are already highly compressed and enabling an extra level of rsync compression will likely just result in needless CPU load. Rsync compression is really more for very slow links where the increase in CPU load weighed against the gains achieved by compressing previously non-compressed data.