Page 1 of 5

OS download is rather useless

Posted: Thu Nov 26, 2020 9:43 pm
by picandies
Just downloaded the latest RPI OS, which wasn't too bad (6 or 7 gig file). However the pi installer was taking forever, literally to write to the SD card (like it would be hour++...why why)...so I aborted & used Balena etcher --took maybe 7 minutes to write & 5 to verify.

Now the rpi is running , but starts off downloading the updates...WHICH IS TAKING LONGER THAN THE OS I just downloaded!!!. If the OS is so out of date that the updates are more than the OS download, it seem like the download section is being allowed to be kept woefully out of date.
This is worse than downloading the OS twice... why bother if it is all going to be replaced with update?

Re: OS download is rather useless

Posted: Thu Nov 26, 2020 9:46 pm
by fruitoftheloom
picandies wrote:
Thu Nov 26, 2020 9:43 pm
Just downloaded the latest RPI OS, which wasn't too bad (6 or 7 gig file). However the pi installer was taking forever, literally to write to the SD card (like it would be hour++...why why)...so I aborted & used Balena etcher --took maybe 7 minutes to write & 5 to verify.

Now the rpi is running , but starts off downloading the updates...WHICH IS TAKING LONGER THAN THE OS I just downloaded!!!. If the OS is so out of date that the updates are more than the OS download, it seem like the download section is being allowed to be kept woefully out of date.
This is worse than downloading the OS twice... why bother if it is all going to be replaced with update?

Poppycock, are you sure it is not your internet connection speed ??

Re: OS download is rather useless

Posted: Thu Nov 26, 2020 9:49 pm
by kerry_s
picandies wrote:
Thu Nov 26, 2020 9:43 pm
Just downloaded the latest RPI OS, which wasn't too bad (6 or 7 gig file). However the pi installer was taking forever, literally to write to the SD card (like it would be hour++...why why)...so I aborted & used Balena etcher --took maybe 7 minutes to write & 5 to verify.

Now the rpi is running , but starts off downloading the updates...WHICH IS TAKING LONGER THAN THE OS I just downloaded!!!. If the OS is so out of date that the updates are more than the OS download, it seem like the download section is being allowed to be kept woefully out of date.
This is worse than downloading the OS twice... why bother if it is all going to be replaced with update?
your doing it wrong.

Re: OS download is rather useless

Posted: Thu Nov 26, 2020 9:54 pm
by LTolledo
fruitoftheloom wrote:
Thu Nov 26, 2020 9:46 pm
....are you sure it is not your internet connection speed ??

....maybe dial-up internet connection?

Re: OS download is rather useless

Posted: Thu Nov 26, 2020 10:00 pm
by Imperf3kt
My experience has been similar, RPi imager takes several hours to download and install Raspberry Pi OS

Downloading the image manually is much faster, but you do have to update it afterwards, though that usually takes ten minutes or so.

Re: OS download is rather useless

Posted: Thu Nov 26, 2020 10:06 pm
by drgeoff
picandies wrote:
Thu Nov 26, 2020 9:43 pm
Just downloaded the latest RPI OS, which wasn't too bad (6 or 7 gig file).
The full image is a 2523 Mbyte zip file. Why did you write "6 or 7 gig"?

Re: OS download is rather useless

Posted: Thu Nov 26, 2020 10:13 pm
by picandies
The extracted image file is 7,180,288 bytes, which was quickly used by Balena on the PC, after giving up on the rpi installer. The internet is running 200MB/sec

The rpi update started at least 1/2 hour ago & the rpi download is almost done but has been downloading to the pi for at least 1/2 hour...it will soon start to install. Why is the update download so humongous that it takes longer than the OS itself?

Re: OS download is rather useless

Posted: Thu Nov 26, 2020 10:16 pm
by trejan
picandies wrote:
Thu Nov 26, 2020 9:43 pm
Now the rpi is running , but starts off downloading the updates...WHICH IS TAKING LONGER THAN THE OS I just downloaded!!!.
There is something wrong with the repo mirror you're using if it is that slow. What server is apt trying to pull updates from? What region/country are you in?
picandies wrote:
Thu Nov 26, 2020 9:43 pm
If the OS is so out of date that the updates are more than the OS download, it seem like the download section is being allowed to be kept woefully out of date.
This is worse than downloading the OS twice... why bother if it is all going to be replaced with update?
The initial update with a high speed Internet connection and a functioning mirror is normally completed in minutes not hours.

Nobody distributes OS images that are always up to date with no post-install updating required so what Raspberry Pi OS is doing isn't unusual.

Re: OS download is rather useless

Posted: Thu Nov 26, 2020 10:47 pm
by incognitum
picandies wrote:
Thu Nov 26, 2020 10:13 pm
The extracted image file is 7,180,288 bytes, which was quickly used by Balena on the PC, after giving up on the rpi installer.
Will generally go faster if you feed Imager a compressed image.

Re: OS download is rather useless

Posted: Thu Nov 26, 2020 11:23 pm
by bls
If you're installing new Pi systems often, a better appoach is to use apt-cacher-ng so that your "site" only pulls the downloads over the net once, and all the other Pi systems updated after the first one go lightning fast.

One way to avoid waiting for a pile of updates to happen when you first boot the system is to do the apt update in the IMG file before you burn it to an SSD/SD Card. sdm (see sig) is one example of a command-line tool that makes this super-easy. I can go from "gee, I wish I had an SD Card with a fresh, good-to go install on it" to a first boot in just a few minutes.

Re: OS download is rather useless

Posted: Thu Nov 26, 2020 11:51 pm
by HawaiianPi
picandies wrote:
Thu Nov 26, 2020 9:43 pm
Just downloaded the latest RPI OS, which wasn't too bad (6 or 7 gig file). However the pi installer was taking forever, literally to write to the SD card (like it would be hour++...why why)...so I aborted & used Balena etcher --took maybe 7 minutes to write & 5 to verify.
RPi Imager caches the last OS it downloaded, so it's faster after the first time (as long as you're writing the same version). But I usually pre-download the images to my server using torrents and then I can quickly write them with Etcher.

Also, the download is only about 2.6GB for the full image (it decompresses to around 7GB, but you download a compressed .zip archive).

Don't use the version with recommended software, so you don't waste a lot of time updating software you may never use. Just install the software you need to the desktop version, and you'll have fewer updates.

FYI:
The "with desktop" image writes in roughly half the time of the "with desktop and recommended software" version (which makes sense, since the image decompresses to a little over half the size).

picandies wrote:
Thu Nov 26, 2020 9:43 pm
Now the rpi is running , but starts off downloading the updates...WHICH IS TAKING LONGER THAN THE OS I just downloaded!!!.
Updates for an OS image are not unusual, but the full image (with desktop and recommended software) will have a lot more updates than the other versions. This is why I don't use the full image. I use the "Lite" version on most of my systems that run headless, and the "with desktop" version for the one I have connected to a monitor.

picandies wrote:
Thu Nov 26, 2020 9:43 pm
If the OS is so out of date that the updates are more than the OS download, it seem like the download section is being allowed to be kept woefully out of date.
This is pretty much par for the course with any OS. Doesn't matter if it's Linux, Windows or Mac OS (they all do that).

A faster micro SD card might help, or run your system from an SSD for even better performance. I use SanDisk Ultra A1 SD cards on systems that don't need a lot of storage and they perform well. Systems that do need more storage run from a SATA-III SSD with a USB 3.0 adapter cable, and they are much faster than any SD card!

Re: OS download is rather useless

Posted: Thu Nov 26, 2020 11:53 pm
by picandies
My home PC just measured 220 meg internet....the rpi connection (google speed test on rpi chromium) measured 33 meg to outr sevice

The rpi was set up for New York & updates don't have any selection (other than asking do you want to install updates).....still tooko AT LEAST 1/2 hour just to download). I was expecting maybe 2 or 3 minutes tops.

I'm just saying when the updates start taking 25% or more of the time installing the original...then the original should be trashed & replaced with the update as OS download.

Someone at raspberry pi can certainly tell if it is time to put up a more recent download.
I will note that the old is from August...maybe it could/should be refreshed every 1st of the month.

Re: OS download is rather useless

Posted: Thu Nov 26, 2020 11:58 pm
by bls
My experience installing Windows 10 on a new system this year was similar: Download the ISO, install it, then wait for what seemed like years for all the updates to be installed to bring it up to current. I don't/won't use Macs but I suspsect that at least the update-to-current experience would be similar.

Re: OS download is rather useless

Posted: Fri Nov 27, 2020 12:37 am
by ejolson
picandies wrote:
Thu Nov 26, 2020 11:53 pm
My home PC just measured 220 meg internet....the rpi connection (google speed test on rpi chromium) measured 33 meg to outr sevice

The rpi was set up for New York & updates don't have any selection (other than asking do you want to install updates).....still tooko AT LEAST 1/2 hour just to download). I was expecting maybe 2 or 3 minutes tops.

I'm just saying when the updates start taking 25% or more of the time installing the original...then the original should be trashed & replaced with the update as OS download.

Someone at raspberry pi can certainly tell if it is time to put up a more recent download.
I will note that the old is from August...maybe it could/should be refreshed every 1st of the month.
Maybe it's faster today. Was there some sort of Internet bot festival yesterday?

Another reason for very slow updates could be a slow or failing SD card. What kind of card are you using?

Re: OS download is rather useless

Posted: Fri Nov 27, 2020 12:49 am
by HawaiianPi
picandies wrote:
Thu Nov 26, 2020 11:53 pm
My home PC just measured 220 meg internet....the rpi connection (google speed test on rpi chromium) measured 33 meg to outr sevice
Just tested my Pi4B2 on 5GHz WiFi and got 84 down and 81 up on a 100mbps connection.

My Windows laptop on the same desk got 93 down and 98 up (router is at end of hallway, two rooms away).

Not sure why your Pi4's Internet performance is so poor, but it could be the browser test you used. I have found that browser based speed tests choke on Pi computers and produce slow results. There is a command line speed test called speedtest-cli, but the version in the repo has an upload test bug (download speeds seem accurate, but upload is super low). The version on my system is newer, but I don't recall how I updated it (might be the pip3 version).

picandies wrote:
Thu Nov 26, 2020 11:53 pm
I will note that the old is from August...maybe it could/should be refreshed every 1st of the month.
Are you volunteering to do all the image maintenance work?

No OS updates their images that often. Microsoft only updates the Windows ISO tool twice a year (April and October, but this year both updates were late).

Re: OS download is rather useless

Posted: Fri Nov 27, 2020 1:45 am
by picandies
Are you volunteering to do all the image maintenance work?

I was thinking once a pi was all updated (such as mine finally is), simply create an image or zip from that updated pi & post it on raspberrypi.org so others can download the exact same thing. Does it need to be more complicated ? Maybe I'm oversimplifying. Let one person do what 100,000 will have to do otherwise. That's a HUGE efficiency boost.

Re: OS download is rather useless

Posted: Fri Nov 27, 2020 2:08 am
by trejan
There have been some complaints before that the GUI updater that runs as part of the first install wizard is quite slow. Running apt directly from the command line is much faster.
picandies wrote:
Fri Nov 27, 2020 1:45 am
I was thinking once a pi was all updated (such as mine finally is), simply create an image or zip from that updated pi & post it on raspberrypi.org so others can download the exact same thing. Does it need to be more complicated ?
It isn't that simple and several of the third party modified images based on Raspberry Pi OS are doing it wrong. Booting an image and then updating/modifying it without taking extra precautions will cause problems with the automatic root filesystem expansion, create a security vulnerability by shipping a pregenerated SSH host key and also leave traces of logs + other data in the image.

The proper way to generate an image is to use https://github.com/RPi-Distro/pi-gen as that guarantees it is a clean image since it is built from scratch each time. It'll compress better as well since you don't have old deleted data lingering in the image.

Re: OS download is rather useless

Posted: Fri Nov 27, 2020 2:10 am
by dustnbone
These seem like symptoms of a really crappy SD card to me.

Re: OS download is rather useless

Posted: Fri Nov 27, 2020 3:39 pm
by hippy
HawaiianPi wrote:
Fri Nov 27, 2020 12:49 am
picandies wrote:
Thu Nov 26, 2020 11:53 pm
I will note that the old is from August...maybe it could/should be refreshed every 1st of the month.
Are you volunteering to do all the image maintenance work?
I am never sure where such 'are you going to do it?' come from, why it's so often a response when someone feels something could be improved. It is surely something the RPT could afford to do given their £16 million gross profit in 2019.

It is rare that any download image is as up to date as it could be but the older it is the more updating is needed. More frequent image releases would reduce the amount of update needed by everyone, would collectively save time and global resources. And that has to be a Good Thing (TM).

Re: OS download is rather useless

Posted: Fri Nov 27, 2020 4:12 pm
by jamesh
hippy wrote:
Fri Nov 27, 2020 3:39 pm
HawaiianPi wrote:
Fri Nov 27, 2020 12:49 am
picandies wrote:
Thu Nov 26, 2020 11:53 pm
I will note that the old is from August...maybe it could/should be refreshed every 1st of the month.
Are you volunteering to do all the image maintenance work?
I am never sure where such 'are you going to do it?' come from, why it's so often a response when someone feels something could be improved. It is surely something the RPT could afford to do given their £16 million gross profit in 2019.

It is rare that any download image is as up to date as it could be but the older it is the more updating is needed. More frequent image releases would reduce the amount of update needed by everyone, would collectively save time and global resources. And that has to be a Good Thing (TM).
No way could it be done for the 1st of each month - takes longer than a month to test all the changes. Which is one reason why it's about every 6 months.

People always underestimate the amount of time and effort that goes in to each release. The latest one was frozen a while back and we are still testing it internally. It's not just a matter of collating all the changes from upstream, there is a new kernel to check, new Mesa libraries, desktop updates, and some other stuff that can wait for the release for announcement.

Re: OS download is rather useless

Posted: Fri Nov 27, 2020 4:37 pm
by Heater
I'm wondering where and/or how this is a problem.

Typically I download and install that latest Pi OS image and then do an "apt-get update", "apt-get upgrade".

This has worked with out muss or fuss and plenty fast enough. And I've done it in Finland, Sweden, England, California (Where they have the slowest most unreliable internet connectivity I have seen since analog land lines and modems) sometimes over crappy mobile connections in the middle of nowhere.

If ones internet connectivity is really so bad I'm not sure what the Pi Foundation can do about it. Apart from posting out PI OS images on DVD. Which I guess would not be quicker and certainly more expensive.

Re: OS download is rather useless

Posted: Fri Nov 27, 2020 4:55 pm
by bls
Heater wrote:
Fri Nov 27, 2020 4:37 pm
I'm wondering where and/or how this is a problem.

Typically I download and install that latest Pi OS image and then do an "apt-get update", "apt-get upgrade".

This has worked with out muss or fuss and plenty fast enough. And I've done it in Finland, Sweden, England, California (Where they have the slowest most unreliable internet connectivity I have seen since analog land lines and modems) sometimes over crappy mobile connections in the middle of nowhere.

If ones internet connectivity is really so bad I'm not sure what the Pi Foundation can do about it. Apart from posting out PI OS images on DVD. Which I guess would not be quicker and certainly more expensive.
Agree...there's absolutely no need for more frequent updated RasPiOS releases.

IMHO the way to minimize and effectively eliminate this issue is twofold:
  • Pre-build IMGs that have all your additional apps installed, and all RasPiOS updates installed as well. A ready-to-go SSD/SD Card is just a 'dd' away.
  • Use apt-cacher-ng so that rebuilding images and/or updating already-running Pi systems only hits the internet for the first download of each package.
Tools such as sdm (and perhaps pi-gen mentioned by trejan) make both of these solutions very easy-to-use, and provide a lot of other value as well.

Re: OS download is rather useless

Posted: Fri Nov 27, 2020 4:57 pm
by ejolson
Heater wrote:
Fri Nov 27, 2020 4:37 pm
I'm wondering where and/or how this is a problem.

Typically I download and install that latest Pi OS image and then do an "apt-get update", "apt-get upgrade".

This has worked with out muss or fuss and plenty fast enough. And I've done it in Finland, Sweden, England, California (Where they have the slowest most unreliable internet connectivity I have seen since analog land lines and modems) sometimes over crappy mobile connections in the middle of nowhere.

If ones internet connectivity is really so bad I'm not sure what the Pi Foundation can do about it. Apart from posting out PI OS images on DVD. Which I guess would not be quicker and certainly more expensive.
How many thousands of new Pi computers do this double download of the full image every week? While each person only wastes a small percentage of their own time and bandwidth, it is well known that if you multiply something small by a large enough quantity the tip soon fills up.

While one solution is to update the official image once a month so the size of the initial updates are less, another solution is to download the lite image and add to it new versions of the heavy packages only as needed. This second option has the pedagogical advantage of teaching new users a bit about the command line and Linux administration before they get stuck in a Python IDE writing code.

Re: OS download is rather useless

Posted: Fri Nov 27, 2020 5:04 pm
by ShiftPlusOne

Re: OS download is rather useless

Posted: Fri Nov 27, 2020 5:35 pm
by hippy
Heater wrote:
Fri Nov 27, 2020 4:37 pm
I'm wondering where and/or how this is a problem.

Typically I download and install that latest Pi OS image and then do an "apt-get update", "apt-get upgrade".
So do I. And then I have to wait for however long it takes to download and install however much it has to. And that can take a significant time. The older the image the more needs to be downloaded and installed and the longer it takes. Multiply that by everyone who has to do that and it becomes quite significant. Given 8 million Pi's are sold every year I imagine plenty are doing the same.
Heater wrote:
Fri Nov 27, 2020 4:37 pm
This has worked with out muss or fuss and plenty fast enough. And I've done it in Finland, Sweden, England, California (Where they have the slowest most unreliable internet connectivity I have seen since analog land lines and modems) sometimes over crappy mobile connections in the middle of nowhere.
It does seem you live a charmed life and rarely experience the problems and issues others do.

I have a cable internet connection which is fast enough, but the whole process does feel like it takes forever at times. Probably it's not so much the download as the installation time, especially when it involves a kernel upgrade with its large number of diversions and removals. Even without it can still be a lengthy process.