Faster Debian package downloads - HOWTO

[nick.mccloud]

As we'll all be so hyper by the time our Pi arrives I was wondering how to speed up the acquisition of new packages to install as I don't have an ARM iso, not that such things exist .

At first I considered creating a mirror but I don't think I need all 16,400+ packages for ARM.

Then I fiddled with ftp'ing out the debs in the apt cache and back in to new installs - which works fine but is a bit fiddly and needs updating with new debs when I get them.

But then I found apt-cacher-ng.

Pick a Debian based box on your network with a bit of spare hd and apt-get install apt-cacher-ng

Then on a target install system once you have either a base install or you've booted from an SD card (having taken extreme care to turn off sshd first ):

echo 'acquire::http { proxy "http://YourCacheServerIP:3142"; };' > /etc/apt/apt.conf.d/02proxy

which creates a setting that points at your cache server.

And, er, that's it. Now that system will use whatever is cached on your local box for apt requests. Obviously the first time you retrieve something it will be at normal network speed. But subsequent requests, when you reimage and reinstall something (for fun or fixing) will be much faster.

I'm trying to find such a thing for Fedora but no luck just yet.

[jojopi]

nmcc said:

I'm trying to find such a thing for Fedora but no luck just yet.
yum honors http_proxy like any normal network client:http_proxy=HOST:PORT yum -y install PACKAGE
Or if your network connection is so bad maybe you should have a transparent proxy.

[nick.mccloud]

My network connection is pretty appalling, but the view out the window is amazing (countryside).

Google research shows that neither apt nor yum get on very well with caching proxy servers. However apt-cacher-ng is specific for this task. Still hunting for a yum solution.

[jojopi]

nmcc said:

Google research shows that neither apt nor yum get on very well with caching proxy servers.
You may need to comment out mirrorlist and uncomment baseurl in yum.repos.d/*.  And you might want to cache only the RPMs and not the metadata.  I assumed that anyone using a local proxy cache would be familiar with its logs and with configuring exceptions.

There is also the option of mirroring the repositories locally, which is quite common although not really worthwhile unless you are using them heavily.

[nick.mccloud]

jojopi said:


nmcc said:


Google research shows that neither apt nor yum get on very well with caching proxy servers.


You may need to comment out mirrorlist and uncomment baseurl in yum.repos.d/*.  And you might want to cache only the RPMs and not the metadata.  I assumed that anyone using a local proxy cache would be familiar with its logs and with configuring exceptions.

There is also the option of mirroring the repositories locally, which is quite common although not really worthwhile unless you are using them heavily.


This isn't about a failure on my part to be able to configure either yum or a caching proxy:

1. Yum apparently doesn't get on with normal caching proxies.

2. I don't want to run a mirror, just cache what I download so that if I use it again I don't have the same download delay

This mechanism works brilliantly for apt with apt-cacher-ng - there does not appear to be an equivalent for yum.

[Chromatix]

I normally keep a webcache at the border of my network - even on a fast ADSL2+ connection, it has the pleasant side effect of making the TCP layer effects more manageable.  When I get it back up (I'm currently rebuilding the machine it runs on) I'll be in a position to see whether yum copes with it at all well.  I am able to run on a more restricted connection to determine whether caching is actually happening.

I haven't had any particular problem with apt, at least under Ubuntu and Mint.  I haven't tried Debian itself, but I would hope that they keep up with downstream bugfixes.

Gentoo has also had zero trouble with it - same with MacOS X.

I *have* had some trouble with very specific websites.  I can only assume that they are doing something highly nonstandard which is violating the HTTP spec.  Given that they are always websites that depend heavily on advertising revenue, this is a very plausible explanation.

[augusto.beiro]

debmirror every night

[jojopi]

nmcc said:

This isn't about a failure on my part to be able to configure either yum or a caching proxy:
I never intended to imply anything to the contrary.  Indeed I did not assume that you had even attempted to use yum with a proxy.  My suggestions were addressed to the reader and it must be admitted that this thread is of interest only to those who are able to configure a caching proxy.

1. Yum apparently doesn't get on with normal caching proxies.
If by that you mean that you mean to say that my suggestions fail to work, then I would appreciate additional information on the failure modes etc.

2. I don't want to run a mirror, just cache what I download so that if I use it again I don't have the same download delay
Again, one who creates a thread must tolerate suggestions that might be acceptable to any interested party.  I never intended to suggest that you personally should mirror any repository.  Actually if anything I think I recommended against repository mirrors in general.