picandies
Posts: 266
Joined: Wed Nov 26, 2014 5:13 pm

OS download is rather useless

Thu Nov 26, 2020 9:43 pm

Just downloaded the latest RPI OS, which wasn't too bad (6 or 7 gig file). However the pi installer was taking forever, literally to write to the SD card (like it would be hour++...why why)...so I aborted & used Balena etcher --took maybe 7 minutes to write & 5 to verify.

Now the rpi is running , but starts off downloading the updates...WHICH IS TAKING LONGER THAN THE OS I just downloaded!!!. If the OS is so out of date that the updates are more than the OS download, it seem like the download section is being allowed to be kept woefully out of date.
This is worse than downloading the OS twice... why bother if it is all going to be replaced with update?

fruitoftheloom
Posts: 25761
Joined: Tue Mar 25, 2014 12:40 pm
Location: Delightful Dorset

Re: OS download is rather useless

Thu Nov 26, 2020 9:46 pm

picandies wrote:
Thu Nov 26, 2020 9:43 pm
Just downloaded the latest RPI OS, which wasn't too bad (6 or 7 gig file). However the pi installer was taking forever, literally to write to the SD card (like it would be hour++...why why)...so I aborted & used Balena etcher --took maybe 7 minutes to write & 5 to verify.

Now the rpi is running , but starts off downloading the updates...WHICH IS TAKING LONGER THAN THE OS I just downloaded!!!. If the OS is so out of date that the updates are more than the OS download, it seem like the download section is being allowed to be kept woefully out of date.
This is worse than downloading the OS twice... why bother if it is all going to be replaced with update?

Poppycock, are you sure it is not your internet connection speed ??
The information is out there....you just have to let it in.

My other Linux machines are a ChromeBox & Intel CoreDuo Desktop

User avatar
kerry_s
Posts: 1768
Joined: Thu Jan 30, 2020 7:14 pm

Re: OS download is rather useless

Thu Nov 26, 2020 9:49 pm

picandies wrote:
Thu Nov 26, 2020 9:43 pm
Just downloaded the latest RPI OS, which wasn't too bad (6 or 7 gig file). However the pi installer was taking forever, literally to write to the SD card (like it would be hour++...why why)...so I aborted & used Balena etcher --took maybe 7 minutes to write & 5 to verify.

Now the rpi is running , but starts off downloading the updates...WHICH IS TAKING LONGER THAN THE OS I just downloaded!!!. If the OS is so out of date that the updates are more than the OS download, it seem like the download section is being allowed to be kept woefully out of date.
This is worse than downloading the OS twice... why bother if it is all going to be replaced with update?
your doing it wrong.

LTolledo
Posts: 4814
Joined: Sat Mar 17, 2018 7:29 am
Location: Anime Heartland

Re: OS download is rather useless

Thu Nov 26, 2020 9:54 pm

fruitoftheloom wrote:
Thu Nov 26, 2020 9:46 pm
....are you sure it is not your internet connection speed ??

....maybe dial-up internet connection?
"Don't come to me with 'issues' for I don't know how to deal with those
Come to me with 'problems' and I'll help you find solutions"

Some people be like:
"Help me! Am drowning! But dont you dare touch me nor come near me!"

User avatar
Imperf3kt
Posts: 4230
Joined: Tue Jun 20, 2017 12:16 am
Location: Australia
Contact: Twitter

Re: OS download is rather useless

Thu Nov 26, 2020 10:00 pm

My experience has been similar, RPi imager takes several hours to download and install Raspberry Pi OS

Downloading the image manually is much faster, but you do have to update it afterwards, though that usually takes ten minutes or so.
55:55:44:44:4C
52:4C:52:42:41

Rose tinted glasses are difficult to see through.

drgeoff
Posts: 11530
Joined: Wed Jan 25, 2012 6:39 pm

Re: OS download is rather useless

Thu Nov 26, 2020 10:06 pm

picandies wrote:
Thu Nov 26, 2020 9:43 pm
Just downloaded the latest RPI OS, which wasn't too bad (6 or 7 gig file).
The full image is a 2523 Mbyte zip file. Why did you write "6 or 7 gig"?
Quis custodiet ipsos custodes?

picandies
Posts: 266
Joined: Wed Nov 26, 2014 5:13 pm

Re: OS download is rather useless

Thu Nov 26, 2020 10:13 pm

The extracted image file is 7,180,288 bytes, which was quickly used by Balena on the PC, after giving up on the rpi installer. The internet is running 200MB/sec

The rpi update started at least 1/2 hour ago & the rpi download is almost done but has been downloading to the pi for at least 1/2 hour...it will soon start to install. Why is the update download so humongous that it takes longer than the OS itself?

trejan
Posts: 3064
Joined: Tue Jul 02, 2019 2:28 pm

Re: OS download is rather useless

Thu Nov 26, 2020 10:16 pm

picandies wrote:
Thu Nov 26, 2020 9:43 pm
Now the rpi is running , but starts off downloading the updates...WHICH IS TAKING LONGER THAN THE OS I just downloaded!!!.
There is something wrong with the repo mirror you're using if it is that slow. What server is apt trying to pull updates from? What region/country are you in?
picandies wrote:
Thu Nov 26, 2020 9:43 pm
If the OS is so out of date that the updates are more than the OS download, it seem like the download section is being allowed to be kept woefully out of date.
This is worse than downloading the OS twice... why bother if it is all going to be replaced with update?
The initial update with a high speed Internet connection and a functioning mirror is normally completed in minutes not hours.

Nobody distributes OS images that are always up to date with no post-install updating required so what Raspberry Pi OS is doing isn't unusual.

incognitum
Posts: 654
Joined: Tue Oct 30, 2018 3:34 pm

Re: OS download is rather useless

Thu Nov 26, 2020 10:47 pm

picandies wrote:
Thu Nov 26, 2020 10:13 pm
The extracted image file is 7,180,288 bytes, which was quickly used by Balena on the PC, after giving up on the rpi installer.
Will generally go faster if you feed Imager a compressed image.

bls
Posts: 1162
Joined: Mon Oct 22, 2018 11:25 pm
Location: Seattle, WA
Contact: Twitter

Re: OS download is rather useless

Thu Nov 26, 2020 11:23 pm

If you're installing new Pi systems often, a better appoach is to use apt-cacher-ng so that your "site" only pulls the downloads over the net once, and all the other Pi systems updated after the first one go lightning fast.

One way to avoid waiting for a pile of updates to happen when you first boot the system is to do the apt update in the IMG file before you burn it to an SSD/SD Card. sdm (see sig) is one example of a command-line tool that makes this super-easy. I can go from "gee, I wish I had an SD Card with a fresh, good-to go install on it" to a first boot in just a few minutes.
Pi tools:
Quickly and easily build customized-just-for-you SD Cards: https://github.com/gitbls/sdm
Easily run your network's DHCP/DNS on a Pi: https://github.com/gitbls/ndm
Easy strongSwan VPN installer/manager: https://github.com/gitbls/pistrong
Lightweight Virtual VNC Config: https://github.com/gitbls/RPiVNCHowTo

User avatar
HawaiianPi
Posts: 6214
Joined: Mon Apr 08, 2013 4:53 am
Location: Aloha, Oregon USA

Re: OS download is rather useless

Thu Nov 26, 2020 11:51 pm

picandies wrote:
Thu Nov 26, 2020 9:43 pm
Just downloaded the latest RPI OS, which wasn't too bad (6 or 7 gig file). However the pi installer was taking forever, literally to write to the SD card (like it would be hour++...why why)...so I aborted & used Balena etcher --took maybe 7 minutes to write & 5 to verify.
RPi Imager caches the last OS it downloaded, so it's faster after the first time (as long as you're writing the same version). But I usually pre-download the images to my server using torrents and then I can quickly write them with Etcher.

Also, the download is only about 2.6GB for the full image (it decompresses to around 7GB, but you download a compressed .zip archive).

Don't use the version with recommended software, so you don't waste a lot of time updating software you may never use. Just install the software you need to the desktop version, and you'll have fewer updates.

FYI:
The "with desktop" image writes in roughly half the time of the "with desktop and recommended software" version (which makes sense, since the image decompresses to a little over half the size).

picandies wrote:
Thu Nov 26, 2020 9:43 pm
Now the rpi is running , but starts off downloading the updates...WHICH IS TAKING LONGER THAN THE OS I just downloaded!!!.
Updates for an OS image are not unusual, but the full image (with desktop and recommended software) will have a lot more updates than the other versions. This is why I don't use the full image. I use the "Lite" version on most of my systems that run headless, and the "with desktop" version for the one I have connected to a monitor.

picandies wrote:
Thu Nov 26, 2020 9:43 pm
If the OS is so out of date that the updates are more than the OS download, it seem like the download section is being allowed to be kept woefully out of date.
This is pretty much par for the course with any OS. Doesn't matter if it's Linux, Windows or Mac OS (they all do that).

A faster micro SD card might help, or run your system from an SSD for even better performance. I use SanDisk Ultra A1 SD cards on systems that don't need a lot of storage and they perform well. Systems that do need more storage run from a SATA-III SSD with a USB 3.0 adapter cable, and they are much faster than any SD card!
My mind is like a browser. 27 tabs are open, 9 aren't responding,
lots of pop-ups...and where is that annoying music coming from?

picandies
Posts: 266
Joined: Wed Nov 26, 2014 5:13 pm

Re: OS download is rather useless

Thu Nov 26, 2020 11:53 pm

My home PC just measured 220 meg internet....the rpi connection (google speed test on rpi chromium) measured 33 meg to outr sevice

The rpi was set up for New York & updates don't have any selection (other than asking do you want to install updates).....still tooko AT LEAST 1/2 hour just to download). I was expecting maybe 2 or 3 minutes tops.

I'm just saying when the updates start taking 25% or more of the time installing the original...then the original should be trashed & replaced with the update as OS download.

Someone at raspberry pi can certainly tell if it is time to put up a more recent download.
I will note that the old is from August...maybe it could/should be refreshed every 1st of the month.
Last edited by picandies on Fri Nov 27, 2020 12:09 am, edited 2 times in total.

bls
Posts: 1162
Joined: Mon Oct 22, 2018 11:25 pm
Location: Seattle, WA
Contact: Twitter

Re: OS download is rather useless

Thu Nov 26, 2020 11:58 pm

My experience installing Windows 10 on a new system this year was similar: Download the ISO, install it, then wait for what seemed like years for all the updates to be installed to bring it up to current. I don't/won't use Macs but I suspsect that at least the update-to-current experience would be similar.
Pi tools:
Quickly and easily build customized-just-for-you SD Cards: https://github.com/gitbls/sdm
Easily run your network's DHCP/DNS on a Pi: https://github.com/gitbls/ndm
Easy strongSwan VPN installer/manager: https://github.com/gitbls/pistrong
Lightweight Virtual VNC Config: https://github.com/gitbls/RPiVNCHowTo

ejolson
Posts: 6616
Joined: Tue Mar 18, 2014 11:47 am

Re: OS download is rather useless

Fri Nov 27, 2020 12:37 am

picandies wrote:
Thu Nov 26, 2020 11:53 pm
My home PC just measured 220 meg internet....the rpi connection (google speed test on rpi chromium) measured 33 meg to outr sevice

The rpi was set up for New York & updates don't have any selection (other than asking do you want to install updates).....still tooko AT LEAST 1/2 hour just to download). I was expecting maybe 2 or 3 minutes tops.

I'm just saying when the updates start taking 25% or more of the time installing the original...then the original should be trashed & replaced with the update as OS download.

Someone at raspberry pi can certainly tell if it is time to put up a more recent download.
I will note that the old is from August...maybe it could/should be refreshed every 1st of the month.
Maybe it's faster today. Was there some sort of Internet bot festival yesterday?

Another reason for very slow updates could be a slow or failing SD card. What kind of card are you using?

User avatar
HawaiianPi
Posts: 6214
Joined: Mon Apr 08, 2013 4:53 am
Location: Aloha, Oregon USA

Re: OS download is rather useless

Fri Nov 27, 2020 12:49 am

picandies wrote:
Thu Nov 26, 2020 11:53 pm
My home PC just measured 220 meg internet....the rpi connection (google speed test on rpi chromium) measured 33 meg to outr sevice
Just tested my Pi4B2 on 5GHz WiFi and got 84 down and 81 up on a 100mbps connection.

My Windows laptop on the same desk got 93 down and 98 up (router is at end of hallway, two rooms away).

Not sure why your Pi4's Internet performance is so poor, but it could be the browser test you used. I have found that browser based speed tests choke on Pi computers and produce slow results. There is a command line speed test called speedtest-cli, but the version in the repo has an upload test bug (download speeds seem accurate, but upload is super low). The version on my system is newer, but I don't recall how I updated it (might be the pip3 version).

picandies wrote:
Thu Nov 26, 2020 11:53 pm
I will note that the old is from August...maybe it could/should be refreshed every 1st of the month.
Are you volunteering to do all the image maintenance work?

No OS updates their images that often. Microsoft only updates the Windows ISO tool twice a year (April and October, but this year both updates were late).
My mind is like a browser. 27 tabs are open, 9 aren't responding,
lots of pop-ups...and where is that annoying music coming from?

picandies
Posts: 266
Joined: Wed Nov 26, 2014 5:13 pm

Re: OS download is rather useless

Fri Nov 27, 2020 1:45 am

Are you volunteering to do all the image maintenance work?

I was thinking once a pi was all updated (such as mine finally is), simply create an image or zip from that updated pi & post it on raspberrypi.org so others can download the exact same thing. Does it need to be more complicated ? Maybe I'm oversimplifying. Let one person do what 100,000 will have to do otherwise. That's a HUGE efficiency boost.

trejan
Posts: 3064
Joined: Tue Jul 02, 2019 2:28 pm

Re: OS download is rather useless

Fri Nov 27, 2020 2:08 am

There have been some complaints before that the GUI updater that runs as part of the first install wizard is quite slow. Running apt directly from the command line is much faster.
picandies wrote:
Fri Nov 27, 2020 1:45 am
I was thinking once a pi was all updated (such as mine finally is), simply create an image or zip from that updated pi & post it on raspberrypi.org so others can download the exact same thing. Does it need to be more complicated ?
It isn't that simple and several of the third party modified images based on Raspberry Pi OS are doing it wrong. Booting an image and then updating/modifying it without taking extra precautions will cause problems with the automatic root filesystem expansion, create a security vulnerability by shipping a pregenerated SSH host key and also leave traces of logs + other data in the image.

The proper way to generate an image is to use https://github.com/RPi-Distro/pi-gen as that guarantees it is a clean image since it is built from scratch each time. It'll compress better as well since you don't have old deleted data lingering in the image.
Last edited by trejan on Fri Nov 27, 2020 2:11 am, edited 1 time in total.

dustnbone
Posts: 465
Joined: Tue Nov 05, 2019 2:49 am

Re: OS download is rather useless

Fri Nov 27, 2020 2:10 am

These seem like symptoms of a really crappy SD card to me.

hippy
Posts: 9173
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: OS download is rather useless

Fri Nov 27, 2020 3:39 pm

HawaiianPi wrote:
Fri Nov 27, 2020 12:49 am
picandies wrote:
Thu Nov 26, 2020 11:53 pm
I will note that the old is from August...maybe it could/should be refreshed every 1st of the month.
Are you volunteering to do all the image maintenance work?
I am never sure where such 'are you going to do it?' come from, why it's so often a response when someone feels something could be improved. It is surely something the RPT could afford to do given their £16 million gross profit in 2019.

It is rare that any download image is as up to date as it could be but the older it is the more updating is needed. More frequent image releases would reduce the amount of update needed by everyone, would collectively save time and global resources. And that has to be a Good Thing (TM).

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 28358
Joined: Sat Jul 30, 2011 7:41 pm

Re: OS download is rather useless

Fri Nov 27, 2020 4:12 pm

hippy wrote:
Fri Nov 27, 2020 3:39 pm
HawaiianPi wrote:
Fri Nov 27, 2020 12:49 am
picandies wrote:
Thu Nov 26, 2020 11:53 pm
I will note that the old is from August...maybe it could/should be refreshed every 1st of the month.
Are you volunteering to do all the image maintenance work?
I am never sure where such 'are you going to do it?' come from, why it's so often a response when someone feels something could be improved. It is surely something the RPT could afford to do given their £16 million gross profit in 2019.

It is rare that any download image is as up to date as it could be but the older it is the more updating is needed. More frequent image releases would reduce the amount of update needed by everyone, would collectively save time and global resources. And that has to be a Good Thing (TM).
No way could it be done for the 1st of each month - takes longer than a month to test all the changes. Which is one reason why it's about every 6 months.

People always underestimate the amount of time and effort that goes in to each release. The latest one was frozen a while back and we are still testing it internally. It's not just a matter of collating all the changes from upstream, there is a new kernel to check, new Mesa libraries, desktop updates, and some other stuff that can wait for the release for announcement.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

Heater
Posts: 17418
Joined: Tue Jul 17, 2012 3:02 pm

Re: OS download is rather useless

Fri Nov 27, 2020 4:37 pm

I'm wondering where and/or how this is a problem.

Typically I download and install that latest Pi OS image and then do an "apt-get update", "apt-get upgrade".

This has worked with out muss or fuss and plenty fast enough. And I've done it in Finland, Sweden, England, California (Where they have the slowest most unreliable internet connectivity I have seen since analog land lines and modems) sometimes over crappy mobile connections in the middle of nowhere.

If ones internet connectivity is really so bad I'm not sure what the Pi Foundation can do about it. Apart from posting out PI OS images on DVD. Which I guess would not be quicker and certainly more expensive.
Memory in C++ is a leaky abstraction .

bls
Posts: 1162
Joined: Mon Oct 22, 2018 11:25 pm
Location: Seattle, WA
Contact: Twitter

Re: OS download is rather useless

Fri Nov 27, 2020 4:55 pm

Heater wrote:
Fri Nov 27, 2020 4:37 pm
I'm wondering where and/or how this is a problem.

Typically I download and install that latest Pi OS image and then do an "apt-get update", "apt-get upgrade".

This has worked with out muss or fuss and plenty fast enough. And I've done it in Finland, Sweden, England, California (Where they have the slowest most unreliable internet connectivity I have seen since analog land lines and modems) sometimes over crappy mobile connections in the middle of nowhere.

If ones internet connectivity is really so bad I'm not sure what the Pi Foundation can do about it. Apart from posting out PI OS images on DVD. Which I guess would not be quicker and certainly more expensive.
Agree...there's absolutely no need for more frequent updated RasPiOS releases.

IMHO the way to minimize and effectively eliminate this issue is twofold:
  • Pre-build IMGs that have all your additional apps installed, and all RasPiOS updates installed as well. A ready-to-go SSD/SD Card is just a 'dd' away.
  • Use apt-cacher-ng so that rebuilding images and/or updating already-running Pi systems only hits the internet for the first download of each package.
Tools such as sdm (and perhaps pi-gen mentioned by trejan) make both of these solutions very easy-to-use, and provide a lot of other value as well.
Pi tools:
Quickly and easily build customized-just-for-you SD Cards: https://github.com/gitbls/sdm
Easily run your network's DHCP/DNS on a Pi: https://github.com/gitbls/ndm
Easy strongSwan VPN installer/manager: https://github.com/gitbls/pistrong
Lightweight Virtual VNC Config: https://github.com/gitbls/RPiVNCHowTo

ejolson
Posts: 6616
Joined: Tue Mar 18, 2014 11:47 am

Re: OS download is rather useless

Fri Nov 27, 2020 4:57 pm

Heater wrote:
Fri Nov 27, 2020 4:37 pm
I'm wondering where and/or how this is a problem.

Typically I download and install that latest Pi OS image and then do an "apt-get update", "apt-get upgrade".

This has worked with out muss or fuss and plenty fast enough. And I've done it in Finland, Sweden, England, California (Where they have the slowest most unreliable internet connectivity I have seen since analog land lines and modems) sometimes over crappy mobile connections in the middle of nowhere.

If ones internet connectivity is really so bad I'm not sure what the Pi Foundation can do about it. Apart from posting out PI OS images on DVD. Which I guess would not be quicker and certainly more expensive.
How many thousands of new Pi computers do this double download of the full image every week? While each person only wastes a small percentage of their own time and bandwidth, it is well known that if you multiply something small by a large enough quantity the tip soon fills up.

While one solution is to update the official image once a month so the size of the initial updates are less, another solution is to download the lite image and add to it new versions of the heavy packages only as needed. This second option has the pedagogical advantage of teaching new users a bit about the command line and Linux administration before they get stuck in a Python IDE writing code.

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6291
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: OS download is rather useless

Fri Nov 27, 2020 5:04 pm


hippy
Posts: 9173
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: OS download is rather useless

Fri Nov 27, 2020 5:35 pm

Heater wrote:
Fri Nov 27, 2020 4:37 pm
I'm wondering where and/or how this is a problem.

Typically I download and install that latest Pi OS image and then do an "apt-get update", "apt-get upgrade".
So do I. And then I have to wait for however long it takes to download and install however much it has to. And that can take a significant time. The older the image the more needs to be downloaded and installed and the longer it takes. Multiply that by everyone who has to do that and it becomes quite significant. Given 8 million Pi's are sold every year I imagine plenty are doing the same.
Heater wrote:
Fri Nov 27, 2020 4:37 pm
This has worked with out muss or fuss and plenty fast enough. And I've done it in Finland, Sweden, England, California (Where they have the slowest most unreliable internet connectivity I have seen since analog land lines and modems) sometimes over crappy mobile connections in the middle of nowhere.
It does seem you live a charmed life and rarely experience the problems and issues others do.

I have a cable internet connection which is fast enough, but the whole process does feel like it takes forever at times. Probably it's not so much the download as the installation time, especially when it involves a kernel upgrade with its large number of diversions and removals. Even without it can still be a lengthy process.

Return to “General discussion”