User avatar
aboudou
Posts: 98
Joined: Wed Aug 08, 2012 12:38 pm
Location: Cintegabelle, France
Contact: Website

Main Raspbian repository overloaded

Wed Sep 19, 2012 8:25 am

Hello,

As a Raspbian mirror maintainer, I often encounter the following error while trying to sync with the main repository:

Code: Select all

@ERROR: max connections (10) reached -- try again later
Starting 2 am today, I'm unable to mirror the repository (I sync every 2 hours). As the list of official mirrors is still growing, is it possible to either allow more simultaneous connexions, or open another “primary” mirror with a transparent redirection system ?

Thanks.
https://goddess-gate.com/ - My main website
http://raspberrypi.goddess-gate.com/ - A website about (and hosted by) my Raspberry Pi
https://twitter.com/boudouarnaud - Follow me on Twitter
http://www.fablab-sud31.fr/ - Fab Lab Sud31-Val d'Ariège

elatllat
Posts: 1337
Joined: Sat Dec 17, 2011 5:05 pm

Re: Main Raspbian repository overloaded

Wed Sep 19, 2012 8:47 am

I hope you are all using something like zsync to optimize mirroring.
SBC with 32GB RAM: https://hardkernel.com

FAQ : https://raspberrypi.stackexchange.com

Unanswered: https://www.raspberrypi.org/forums/search.php?search_id=unanswered

plugwash
Forum Moderator
Forum Moderator
Posts: 3415
Joined: Wed Dec 28, 2011 11:45 pm

Re: Main Raspbian repository overloaded

Wed Sep 19, 2012 9:05 am

I've just added a couple of zeros to the limit.

Thanks for notifying me of the problem.

User avatar
aboudou
Posts: 98
Joined: Wed Aug 08, 2012 12:38 pm
Location: Cintegabelle, France
Contact: Website

Re: Main Raspbian repository overloaded

Wed Sep 19, 2012 9:09 am

plugwash wrote:I've just added a couple of zeros to the limit.

Thanks for notifying me of the problem.
Nice, thanks :-)
https://goddess-gate.com/ - My main website
http://raspberrypi.goddess-gate.com/ - A website about (and hosted by) my Raspberry Pi
https://twitter.com/boudouarnaud - Follow me on Twitter
http://www.fablab-sud31.fr/ - Fab Lab Sud31-Val d'Ariège

plugwash
Forum Moderator
Forum Moderator
Posts: 3415
Joined: Wed Dec 28, 2011 11:45 pm

Re: Main Raspbian repository overloaded

Wed Sep 19, 2012 11:27 am

elatllat wrote:I hope you are all using something like zsync to optimize mirroring.
Please DON'T try to mirror over http even using fancy tools like zsync. Please use continue to use rsync for all mirroring.

Also enabling compression in rsync is probablly not worth it as the vast majority of the data is compressed already.

elatllat
Posts: 1337
Joined: Sat Dec 17, 2011 5:05 pm

Re: Main Raspbian repository overloaded

Wed Sep 19, 2012 7:20 pm

plugwash wrote:
elatllat wrote:I hope you are all using something like zsync to optimize mirroring.
Please DON'T try to mirror over http even using fancy tools like zsync. Please use continue to use rsync for all mirroring.

Also enabling compression in rsync is probablly not worth it as the vast majority of the data is compressed already.
rsync is nice, just out of curiosity;
1. what is the recommended client command (rsync -qru -e rsh --compress-level=0 rsync:://server/src /dest/)?
2. and server command (rsync --daemon --bwlimit=100000)?
3. On the server side are rolling checksums calculations limiting (would it be worth it if the rsync daemon to cached them)?
SBC with 32GB RAM: https://hardkernel.com

FAQ : https://raspberrypi.stackexchange.com

Unanswered: https://www.raspberrypi.org/forums/search.php?search_id=unanswered

plugwash
Forum Moderator
Forum Moderator
Posts: 3415
Joined: Wed Dec 28, 2011 11:45 pm

Re: Main Raspbian repository overloaded

Thu Sep 20, 2012 1:39 am

Not sure what the best set of rsync options is, i'll have to think about that a bit.

Other than raising the limit on simultanious connections I don't think we did anything special on the server side.

Not sure about the checksum caching, if's the first time I've heard of it. I suspect though that during regular updates the bottleneck is the disk. Unfortunately I don't really know of any good way to measure what proportion of our disks IOPs is in use over time.

User avatar
mpthompson
Posts: 620
Joined: Fri Feb 03, 2012 7:18 pm
Location: San Carlos, CA
Contact: Website

Re: Main Raspbian repository overloaded

Thu Sep 20, 2012 3:57 am

Based on some questions, a few days ago I updated the Raspbian mirror page with a suggested rsync command. I'm not certain of those are the best options though and would like feedback.

http://www.raspbian.org/RaspbianMirrors

Also, as plugwash mentioned, I'm pretty sure we do not want to enable compression on rsync. Virtually all of the files in the archive are already highly compressed and enabling an extra level of rsync compression will likely just result in needless CPU load. Rsync compression is really more for very slow links where the increase in CPU load weighed against the gains achieved by compressing previously non-compressed data.

Return to “Raspbian”