gkreidl
Posts: 6095
Joined: Thu Jan 26, 2012 1:07 pm
Location: Germany

[solved] Latest Kernel & Firmware broke my RPi 3

Mon Mar 06, 2017 9:29 am

Edit: This problem was caused by an old setting in my config.txt.
fixup_file=fixup_x.elf
If you have been using that, you must replace it by
fixup_file=fixup_x.dat
before upgrading to the latest firmware and kernel.

This morning I ran apt-get dist-upgrade to get the latest updates including a new kernel and firmware release.

Note: I'm running my OS from USB HDD, which means that I have my boot partition on SD card and the Linux file system on the HDD. I boot to the command line, but start my own small desktop from /etc/rc.local.

The upgrade went fine (including the new wolfram version) and I rebooted to run the new kernel and firmware. That's when the trouble began. The system took ages to boot (after it had started to access the file system on HDD). The HDD was working like hell, which usually means that it has started swapping.
I managed to log in using SSH from another Pi, but this also took 2 or 3 minutes. I could finally run "free" and noticed, that most of the RAM was missing (probably using 256 MB, but I have set GPU memory to 192 in my config.txt). I finally managed to modify /boot/config.txt (removing the GPU memory setting) and my rc.local file (removing my xinit script and other stuff) and rebooted (to the command line). This time it worked without swapping, but running "free" showed that I still had less than 200 MB RAM.

I reinstalled kernel and bootloader from var/cache/apt/archives using dpkg and rebooted. No change.

I reinstalled former versions of the system files (see below), rebooted and Voila!! - my system is working again as it should (and I finally restored my config.txt and rc.local settings as well).

This is what I did to get my system running again:

Code: Select all

cd /var/cache/apt/archives

sudo dpkg -i raspberrypi-bootloader_1.20161215-1_armhf.deb raspberrypi-kernel_1.20161215-1_armhf.deb libraspberrypi0_1.20161215-1_armhf.deb libraspberrypi-doc_1.20161215-1_armhf.deb libraspberrypi-dev_1.20161215-1_armhf.deb libraspberrypi-bin_1.20161215-1_armhf.deb raspberrypi-sys-mods_20170127_all.deb
Which proves one thing: It's the new kernel + bootloader (firmware) which has been causing the problems. (I'm not sure if downgrading raspberrypi-sys-mods is really required).

Edit: it looks like this will happen to any system running the main file system from an USB device and the boot partition from SD card.
Last edited by gkreidl on Tue Mar 07, 2017 8:51 am, edited 2 times in total.
Minimal Kiosk Browser (kweb)
Slim, fast webkit browser with support for audio+video+playlists+youtube+pdf+download
Optional fullscreen kiosk mode and command interface for embedded applications
Includes omxplayerGUI, an X front end for omxplayer

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23679
Joined: Sat Jul 30, 2011 7:41 pm

Re: Latest Kernel & Firmware broke my RPi 3

Mon Mar 06, 2017 11:47 am

Any chance you can try this with a new SD card/setup? We've not seen the issue here.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

User avatar
bazza14
Posts: 65
Joined: Sun Jan 01, 2012 10:05 am

Re: Latest Kernel & Firmware broke my RPi 3

Mon Mar 06, 2017 3:13 pm

Had problems today as well on raspbian jessie, looks like it's the same files.
Preparing to unpack .../libraspberrypi-bin_1.20170303-1_armhf.deb ...
Unpacking libraspberrypi-bin (1.20170303-1) over (1.20161215-1) ...
dpkg-deb (subprocess): decompressing archive member: lzma error: compressed data is corrupt
dpkg-deb: error: subprocess <decompress> returned error exit status 2
dpkg: error processing archive /var/cache/apt/archives/libraspberrypi-bin_1.20170303-1_armhf.deb (--unpack):
cannot copy extracted data for './opt/vc/bin/vchiq_test' to '/opt/vc/bin/vchiq_test.dpkg-new': unexpected end of file or stream
Preparing to unpack .../libraspberrypi0_1.20170303-1_armhf.deb ...
Unpacking libraspberrypi0 (1.20170303-1) over (1.20161215-1) ...
dpkg-deb (subprocess): decompressing archive member: lzma error: compressed data is corrupt
dpkg-deb: error: subprocess <decompress> returned error exit status 2
dpkg: error processing archive /var/cache/apt/archives/libraspberrypi0_1.20170303-1_armhf.deb (--unpack):
cannot copy extracted data for './opt/vc/lib/libbrcmGLESv2.so' to '/opt/vc/lib/libbrcmGLESv2.so.dpkg-new': unexpected end of file or stream
Unpacking raspberrypi-bootloader (1.20170303-1) over (1.20161215-1) ...
dpkg-deb (subprocess): decompressing archive member: lzma error: compressed data is corrupt
dpkg-deb: error: subprocess <decompress> returned error exit status 2
dpkg: error processing archive /var/cache/apt/archives/raspberrypi-bootloader_1.20170303-1_armhf.deb (--unpack):
cannot copy extracted data for './boot/start.elf' to '/usr/share/rpikernelhack/start.elf.dpkg-new': unexpected end of file or stream
LINUX convert since 2003

gkreidl
Posts: 6095
Joined: Thu Jan 26, 2012 1:07 pm
Location: Germany

Re: Latest Kernel & Firmware broke my RPi 3

Mon Mar 06, 2017 3:27 pm

jamesh wrote:Any chance you can try this with a new SD card/setup? We've not seen the issue here.
Unfortunately not (I don't have any spare SD cards just now).
Minimal Kiosk Browser (kweb)
Slim, fast webkit browser with support for audio+video+playlists+youtube+pdf+download
Optional fullscreen kiosk mode and command interface for embedded applications
Includes omxplayerGUI, an X front end for omxplayer

gkreidl
Posts: 6095
Joined: Thu Jan 26, 2012 1:07 pm
Location: Germany

Re: Latest Kernel & Firmware broke my RPi 3

Mon Mar 06, 2017 3:28 pm

bazza14 wrote:Had problems today as well on raspbian jessie, looks like it's the same files.
Preparing to unpack .../libraspberrypi-bin_1.20170303-1_armhf.deb ...
Unpacking libraspberrypi-bin (1.20170303-1) over (1.20161215-1) ...
dpkg-deb (subprocess): decompressing archive member: lzma error: compressed data is corrupt
dpkg-deb: error: subprocess <decompress> returned error exit status 2
dpkg: error processing archive /var/cache/apt/archives/libraspberrypi-bin_1.20170303-1_armhf.deb (--unpack):
cannot copy extracted data for './opt/vc/bin/vchiq_test' to '/opt/vc/bin/vchiq_test.dpkg-new': unexpected end of file or stream
Preparing to unpack .../libraspberrypi0_1.20170303-1_armhf.deb ...
Unpacking libraspberrypi0 (1.20170303-1) over (1.20161215-1) ...
dpkg-deb (subprocess): decompressing archive member: lzma error: compressed data is corrupt
dpkg-deb: error: subprocess <decompress> returned error exit status 2
dpkg: error processing archive /var/cache/apt/archives/libraspberrypi0_1.20170303-1_armhf.deb (--unpack):
cannot copy extracted data for './opt/vc/lib/libbrcmGLESv2.so' to '/opt/vc/lib/libbrcmGLESv2.so.dpkg-new': unexpected end of file or stream
Unpacking raspberrypi-bootloader (1.20170303-1) over (1.20161215-1) ...
dpkg-deb (subprocess): decompressing archive member: lzma error: compressed data is corrupt
dpkg-deb: error: subprocess <decompress> returned error exit status 2
dpkg: error processing archive /var/cache/apt/archives/raspberrypi-bootloader_1.20170303-1_armhf.deb (--unpack):
cannot copy extracted data for './boot/start.elf' to '/usr/share/rpikernelhack/start.elf.dpkg-new': unexpected end of file or stream
Not the same problem. Download and installation (and re-installation using dpkg) went well - no errors.
Minimal Kiosk Browser (kweb)
Slim, fast webkit browser with support for audio+video+playlists+youtube+pdf+download
Optional fullscreen kiosk mode and command interface for embedded applications
Includes omxplayerGUI, an X front end for omxplayer

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5964
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: Latest Kernel & Firmware broke my RPi 3

Mon Mar 06, 2017 3:33 pm

What's required to reproduce the problem?

gkreidl
Posts: 6095
Joined: Thu Jan 26, 2012 1:07 pm
Location: Germany

Re: Latest Kernel & Firmware broke my RPi 3

Mon Mar 06, 2017 3:44 pm

ShiftPlusOne wrote:What's required to reproduce the problem?
I really don't know. My system is a bit different from most other system, as it takes the file system from USB HDD (all my RPis do that and this has been working for years). My cmdline.txt looks like this:
dwc_otg.lpm_enable=0 console=serial0,115200 console=tty1 rootfstype=ext4 root=PARTUUID=1971F32D-A89E-498A-961A-E27A53F2B485 elevator=deadline rootwait rootdelay=5 usbhid.mousepoll=4

I first thought it might be an installation problem and checked the file dates of both firmware and the modules directory, But both had the time of installation.

And it does boot but seems to be aware of only 256 MB RAM.
My RPi 3 is from the first series, bought within the first few weeks after it became available.
Minimal Kiosk Browser (kweb)
Slim, fast webkit browser with support for audio+video+playlists+youtube+pdf+download
Optional fullscreen kiosk mode and command interface for embedded applications
Includes omxplayerGUI, an X front end for omxplayer

gkreidl
Posts: 6095
Joined: Thu Jan 26, 2012 1:07 pm
Location: Germany

Re: Latest Kernel & Firmware broke my RPi 3

Tue Mar 07, 2017 7:31 am

Update: This morning I updated one of my RPi2s. I was prepared for the problems and had removed the GPU memory settings and also set it to boot to the command line without executing anything else.

The problem was exactly the same. After rebooting "free" showed only 120MB of available RAM. After restoring the older kernel and firmware (as shown in the OP) and rebooting the system, everything is running fine again.

So it looks like the problem does not depend on the RPi type, but will occur if you run your main file system from a hard disk and not from the SD card (the boot partition is on the SD card).

So my recommendation to reproduce the bug is the following:
Flash an older image (not the latest one!) to an SD card. Connect a HDD and move the file system to the HDD following this tutorial: viewtopic.php?f=29&t=44177
If the system is running from the HDD, update everything and reboot.
Minimal Kiosk Browser (kweb)
Slim, fast webkit browser with support for audio+video+playlists+youtube+pdf+download
Optional fullscreen kiosk mode and command interface for embedded applications
Includes omxplayerGUI, an X front end for omxplayer

User avatar
DougieLawson
Posts: 36130
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: Latest Kernel & Firmware broke my RPi 3

Tue Mar 07, 2017 7:56 am

Note: Having anything humorous in your signature is completely banned on this forum. Wear a tin-foil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

paulv
Posts: 558
Joined: Tue Jan 15, 2013 12:10 pm
Location: Netherlands

Re: Latest Kernel & Firmware broke my RPi 3

Tue Mar 07, 2017 8:03 am

I have added a warning to the procedure intro about moving the OS to a USB drive.
If possible, I hope a fix is communicated in this thread, so as soon as there is a fix, I can update the procedure again.

Too bad, there is no way to warn users before they do the upgrade...

Tks,

Paul

User avatar
DougieLawson
Posts: 36130
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: Latest Kernel & Firmware broke my RPi 3

Tue Mar 07, 2017 8:28 am

paulv wrote:I have added a warning to the procedure intro about moving the OS to a USB drive.
If possible, I hope a fix is communicated in this thread, so as soon as there is a fix, I can update the procedure again.
The code to do USB booting is now in the main stream kernel sudo apt-get update && sudo apt-get install --reinstall raspberrypi-bootloader raspberrypi-kernel should be good enough, no need to run sudo rpi-update.
Note: Having anything humorous in your signature is completely banned on this forum. Wear a tin-foil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

gkreidl
Posts: 6095
Joined: Thu Jan 26, 2012 1:07 pm
Location: Germany

Re: Latest Kernel & Firmware broke my RPi 3

Tue Mar 07, 2017 8:30 am

After going to the link Dougie posted above I think I have found the cause of the problem. Give me a few more minutes to check and I'll report it here. It was an old setting in my config.txt which was silently corrected by former firmware releases, but not any more by the latest firmware.
Minimal Kiosk Browser (kweb)
Slim, fast webkit browser with support for audio+video+playlists+youtube+pdf+download
Optional fullscreen kiosk mode and command interface for embedded applications
Includes omxplayerGUI, an X front end for omxplayer

User avatar
DougieLawson
Posts: 36130
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: Latest Kernel & Firmware broke my RPi 3

Tue Mar 07, 2017 8:39 am

That's got to be a new issue. Retired and deprecated parms in /boot/config.txt should be logged to the kernel messages and ignored.
Note: Having anything humorous in your signature is completely banned on this forum. Wear a tin-foil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

gkreidl
Posts: 6095
Joined: Thu Jan 26, 2012 1:07 pm
Location: Germany

Re: Latest Kernel & Firmware broke my RPi 3

Tue Mar 07, 2017 8:45 am

Problem solved:

I have always been using the extended firmware to get the additional codecs running. I have been using a setting from elinux.org and ported it to each new system.
start_file=start_x.elf
fixup_file=fixup_x.elf

This always worked, although I was always wondering why it shouldn't be "fixup_file=fixup_x.dat". Obviously this was silently corrected by the firmware until now. After changing it to
fixup_file=fixup_x.dat
the new firmware recognized the correct memory again.
Minimal Kiosk Browser (kweb)
Slim, fast webkit browser with support for audio+video+playlists+youtube+pdf+download
Optional fullscreen kiosk mode and command interface for embedded applications
Includes omxplayerGUI, an X front end for omxplayer

paulv
Posts: 558
Joined: Tue Jan 15, 2013 12:10 pm
Location: Netherlands

Re: Latest Kernel & Firmware broke my RPi 3

Tue Mar 07, 2017 8:49 am

gkreidl,
So can we take the warning away again?
Maybe you can edit/delete you post with the big red warning? ;)

gkreidl
Posts: 6095
Joined: Thu Jan 26, 2012 1:07 pm
Location: Germany

Re: Latest Kernel & Firmware broke my RPi 3

Tue Mar 07, 2017 8:51 am

paulv wrote:gkreidl,
So can we take the warning away again?
Maybe you can edit/delete you post with the big red warning? ;)
I'll do that in a moment!
Minimal Kiosk Browser (kweb)
Slim, fast webkit browser with support for audio+video+playlists+youtube+pdf+download
Optional fullscreen kiosk mode and command interface for embedded applications
Includes omxplayerGUI, an X front end for omxplayer

paulv
Posts: 558
Joined: Tue Jan 15, 2013 12:10 pm
Location: Netherlands

Re: [solved] Latest Kernel & Firmware broke my RPi 3

Tue Mar 07, 2017 8:56 am

gkreidel, thank you, I just took the warning away from the original post.

Dougie, I also highlighted the RPI Model 3 procedure as being different, to mke it stand out more, but at this moment, the following link is still the only reference I have :
https://github.com/raspberrypi/document ... des/msd.md Are there updates to this?

User avatar
DougieLawson
Posts: 36130
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: [solved] Latest Kernel & Firmware broke my RPi 3

Tue Mar 07, 2017 9:37 am

paulv wrote:gkreidel, thank you, I just took the warning away from the original post.

Dougie, I also highlighted the RPI Model 3 procedure as being different, to mke it stand out more, but at this moment, the following link is still the only reference I have :
https://github.com/raspberrypi/document ... des/msd.md Are there updates to this?
The main update is DON'T run rpi-update. Run apt-get to install the latest kernel and bootcode.

At the moment rpi-update will break RealVNC.
Note: Having anything humorous in your signature is completely banned on this forum. Wear a tin-foil hat and you'll get a ban.

Any DMs sent on Twitter will be answered next month.

This is a doctor free zone.

Return to “Raspbian”