plugwash
Forum Moderator
Forum Moderator
Posts: 3418
Joined: Wed Dec 28, 2011 11:45 pm

introducing raspbmirror, the new way to mirror the raspbian archive.

Tue Mar 05, 2019 9:18 pm

I have just put together a new tool for mirroring the raspbian archive. This tool has several advantages over using rsync.

1. It will always update to a clean and consistent state, no worries about mirroring in the middle of an update.
2. It can use the mirror redirection system as a source for packages, this means that when creating a new mirror or updating a badly outdated mirror much of the data will be pulled from other local mirrors rather than the main server in the UK.
3. If you have a debian mirror on the same server it can re-use files that are common between Debian and Raspbian, saving you bandwidth and disk space.

https://github.com/plugwash/raspbian-to ... bmirror.py

basic usage:

Code: Select all

mkdir raspbian
mkdir tmp
cd raspbian
/path/to/raspbmirror.py --tmpdir ../tmp
The temporary directory must be on the same filesystem as the mirror. If the tmpdir option is not specified then temporary files will be stored within the mirror itself, this is fine for a local mirror but is not reccomended if you may have downstream mirrors.

If you only want to mirror the main raspbian repository and not the other stuff on archive.raspbian.org you can use

Code: Select all

/path/to/raspbmirror.py --tmpdir ../tmp --tlwhitelist raspbian
If you have a Debian mirror locally you can use it to save bandwidth and (if the mirrors are on the same filesystem) disk space by doing.

Code: Select all

/path/to/raspbmirror.py --tmpdir ../tmp --tlwhitelist raspbian --sourcepool /path/to/debian/pool
Any problems or questions please ask.

rezuan
Posts: 3
Joined: Mon Apr 29, 2019 10:57 pm

Re: introducing raspbmirror, the new way to mirror the raspbian archive.

Mon Apr 29, 2019 11:00 pm

Code: Select all

/path/to/raspbmirror.py --tmpdir ../tmp --tlwhitelist raspbian
I couldn't run the raspbmirror with the argument "--tlwhitelist raspbian".

Was looking in the code and couldn't find this argument too.

Is this a bug?

plugwash
Forum Moderator
Forum Moderator
Posts: 3418
Joined: Wed Dec 28, 2011 11:45 pm

Re: introducing raspbmirror, the new way to mirror the raspbian archive.

Mon Apr 29, 2019 11:50 pm

yeah it's a bug.

There is a related script "downloadsnapshot" aimed at a different usecase from raspbmirror (raspbmirror is aimed at maintaining a mirror, downloadsnapshot is aimed at maintaining a series of hardlink based snapshots). It seems I have the tlwhitelist feature in downloadsnapshot but not in raspbmirror.

Will try to get the feature ported across.

rezuan
Posts: 3
Joined: Mon Apr 29, 2019 10:57 pm

Re: introducing raspbmirror, the new way to mirror the raspbian archive.

Mon Apr 29, 2019 11:54 pm

Assuming that tlwhitelist will be ported to raspbmirror.py, will it reduced the download size significantly?

plugwash
Forum Moderator
Forum Moderator
Posts: 3418
Joined: Wed Dec 28, 2011 11:45 pm

Re: introducing raspbmirror, the new way to mirror the raspbian archive.

Tue Apr 30, 2019 12:25 am

Currently it will knock off about 20 gigs from the download, which isn't all that much in the grand scheme of things, this figure may change as non-repo materials are removed and/or added from archive.raspbian.org

I want to add support for mirroring only a subset of suites, but that requires some more thought/work.

mcbeardface
Posts: 2
Joined: Mon May 20, 2019 12:58 am

Re: introducing raspbmirror, the new way to mirror the raspbian archive.

Mon May 20, 2019 1:14 am

I've been wanting to create a local mirror as i have a few raspberry pi's now and don't want to download the updates for all 6 of them all of the time. I have unlimited download so I only want this to reduce the burden of my traffic to the update servers.

I downloaded this a while ago and it I kept getting the same error no matter what I did. I just downloaded the updated version of raspbmirror.py from github and ran it but received the same error (below).

I'm not sure if i am doing something wrong (likely), or just have no idea what I'm doing (also likely).

Running on a Raspberry pi 3B, with updated full raspbian stretch install. I have an external HDD mounted on /mnt/mirror

Code: Select all

:/mnt/mirror/raspbian $ sudo /home/pi/raspbmirror.py --tmpdir ../tmp --tlwhitelist raspbian
stage 1, scan existing
stage 2, main download
creating symlink raspbian/dists/oldstable -> jessie
Traceback (most recent call last):
  File "/home/pi/raspbmirror.py", line 499, in <module>
    os.symlink(symlinktarget,filepath)
PermissionError: [Errno 1] Operation not permitted: b'jessie' -> b'raspbian/dists/oldstable'

plugwash
Forum Moderator
Forum Moderator
Posts: 3418
Joined: Wed Dec 28, 2011 11:45 pm

Re: introducing raspbmirror, the new way to mirror the raspbian archive.

Mon May 20, 2019 11:41 am

what filesystem is your external hdd using? I suspect your problem is you are trying to mirror onto a filesystem that does not support symlinks.

hippy
Posts: 5370
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: introducing raspbmirror, the new way to mirror the raspbian archive.

Mon May 20, 2019 2:29 pm

plugwash wrote:
Tue Apr 30, 2019 12:25 am
Currently it will knock off about 20 gigs from the download, which isn't all that much in the grand scheme of things ...
Because you piqued my curiosity, caused an eyebrow to be raised; how large is a full Raspbian mirror download ?

mcbeardface
Posts: 2
Joined: Mon May 20, 2019 12:58 am

Re: introducing raspbmirror, the new way to mirror the raspbian archive.

Tue May 21, 2019 12:54 am

plugwash wrote:
Mon May 20, 2019 11:41 am
what filesystem is your external hdd using? I suspect your problem is you are trying to mirror onto a filesystem that does not support symlinks.
You're right, its a vfat filesystem. Cheers

plugwash
Forum Moderator
Forum Moderator
Posts: 3418
Joined: Wed Dec 28, 2011 11:45 pm

Re: introducing raspbmirror, the new way to mirror the raspbian archive.

Tue May 21, 2019 8:28 pm

hippy wrote:
Mon May 20, 2019 2:29 pm
Because you piqued my curiosity, caused an eyebrow to be raised; how large is a full Raspbian mirror download ?
330G currently.

rezuan
Posts: 3
Joined: Mon Apr 29, 2019 10:57 pm

Re: introducing raspbmirror, the new way to mirror the raspbian archive.

Thu May 23, 2019 2:28 am

I'm still running raspbmirror since three weeks ago and it's still downloading.

Here's df output.

Code: Select all

 Size  Used Avail Use%
 631G  485G  146G  77%
If the whole size of the repo is around 300GB, why is it still downloading stuff when it should be completed by now?

Return to “Raspbian”