alias_neo
Posts: 108
Joined: Fri Jul 29, 2011 8:55 am
Location: London, England
Contact: Website

Pi won't boot after update.

Thu Sep 20, 2012 7:46 pm

I ran rpi-update on my previously working pi and it wouldn't boot anymore. All I get is a quick flicker of the OK LED and then nothing. No output, flashes, no display, nothing.

Next, I replaced my old bootloader files, it booted up, then I ran apt-get update && apt-get dist-upgrade

After this finished and updated my bootloader files the Pi no longer booted, yet again. If I switch to the 240 split I get the rainbow screen and it stays there, any other split I get nothing.

I'm running Raspbian wheezy, 2012-08-16 on a new 32GB Sandisk Ultra Class 10.

It's causing me extreme frustration trying various firmware files, plugging and unplugged the SD card to and from pi and pc and finding none of them boot except my old bootloader files.

The problem with the old files is that because my OS is up to date it can't load the kernel modules for a bunch of stuff.

Anyone got a suggestion why it's just not booting?
http://2byt.es - Just my two bytes - RPi - Android - Arduino - Blog

alias_neo
Posts: 108
Joined: Fri Jul 29, 2011 8:55 am
Location: London, England
Contact: Website

Re: Pi won't boot after update.

Thu Sep 20, 2012 8:27 pm

I went ahead and downloaded every revision on Github since middle of August (approx 100MB each) and tested them all (keeping only my own config.txt and cmdline.txt) and not a single version since commit 0c0ef8afc1d56d172ef0b8476d26616b1096498e on the 3rd of September will boot on my Pi.

My Pi is a Rev. 2.0 (the one with D14 missing) and has the Samsung memory chipset.

Hopefully someone can pin down what has broken it in the later builds of the firmware.
http://2byt.es - Just my two bytes - RPi - Android - Arduino - Blog

rixcam
Posts: 3
Joined: Thu Jun 21, 2012 9:02 am

Re: Pi won't boot after update.

Thu Sep 20, 2012 9:17 pm

You're not alone, I've just taken a pretty barebones 2012-07-15-wheezy-raspbian SD card and ran update & upgrade this evening - just as you were making the first post in the thread. Totally dead, no flicker on the monitor. Luckily I have the .zip, it's back to that now.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5341
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Pi won't boot after update.

Thu Sep 20, 2012 10:02 pm

alias_neo wrote:I went ahead and downloaded every revision on Github since middle of August (approx 100MB each) and tested them all (keeping only my own config.txt and cmdline.txt) and not a single version since commit 0c0ef8afc1d56d172ef0b8476d26616b1096498e on the 3rd of September will boot on my Pi.

My Pi is a Rev. 2.0 (the one with D14 missing) and has the Samsung memory chipset.

Hopefully someone can pin down what has broken it in the later builds of the firmware.
Can you confirm if 0c0ef8afc1d56d172ef0b8476d26616b1096498e was the last working version or the first failing version?
Do you believe it is a change to kernel or start.elf that breaks it?

alias_neo
Posts: 108
Joined: Fri Jul 29, 2011 8:55 am
Location: London, England
Contact: Website

Re: Pi won't boot after update.

Thu Sep 20, 2012 10:05 pm

dom wrote:
alias_neo wrote:I went ahead and downloaded every revision on Github since middle of August (approx 100MB each) and tested them all (keeping only my own config.txt and cmdline.txt) and not a single version since commit 0c0ef8afc1d56d172ef0b8476d26616b1096498e on the 3rd of September will boot on my Pi.

My Pi is a Rev. 2.0 (the one with D14 missing) and has the Samsung memory chipset.

Hopefully someone can pin down what has broken it in the later builds of the firmware.
Can you confirm if 0c0ef8afc1d56d172ef0b8476d26616b1096498e was the last working version or the first failing version?
Do you believe it is a change to kernel or start.elf that breaks it?
0c0ef8afc1d56d172ef0b8476d26616b1096498e is the last working version. Everything AFTER it fails.

I'm unsure if it's the start or the kernel but give me a few minutes and I'll test it and report back.
http://2byt.es - Just my two bytes - RPi - Android - Arduino - Blog

alias_neo
Posts: 108
Joined: Fri Jul 29, 2011 8:55 am
Location: London, England
Contact: Website

Re: Pi won't boot after update.

Thu Sep 20, 2012 10:11 pm

OK, here's the results;

Latest with 0c0 start.elf = no boot
Latest with 0c0 kernel = no boot
Latest with 0c0 kernel + start.elf = BOOT.

EDIT:

Ok, with 240M start.elf from 0c0 I get a partial boot. It doesn't load fully, but hangs after the following lines;

[ 1.956299] bcm2835-cpufreq: min=0 max=700000 cur=700000
[ 1.961748] bcm2835-cpufreq: switching to governor powersave
[ 1.974766] bcm2835-cpufreq: switching to governor powersave_

That's a replica of what I see, including the cursor at the end of the third line.
http://2byt.es - Just my two bytes - RPi - Android - Arduino - Blog

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5341
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Pi won't boot after update.

Thu Sep 20, 2012 10:54 pm

alias_neo wrote:OK, here's the results;

Latest with 0c0 start.elf = no boot
Latest with 0c0 kernel = no boot
Latest with 0c0 kernel + start.elf = BOOT.

EDIT:

Ok, with 240M start.elf from 0c0 I get a partial boot. It doesn't load fully, but hangs after the following lines;

[ 1.956299] bcm2835-cpufreq: min=0 max=700000 cur=700000
[ 1.961748] bcm2835-cpufreq: switching to governor powersave
[ 1.974766] bcm2835-cpufreq: switching to governor powersave_

That's a replica of what I see, including the cursor at the end of the third line.
Okay, so I think you are saying that from working 0c0 firmware, just updating start.elf to 790681ab24260d652755901cbe8c5fe59b734b71 version breaks it.

Do you have anything in config.txt. Any non-default cmdline.txt options?

alias_neo
Posts: 108
Joined: Fri Jul 29, 2011 8:55 am
Location: London, England
Contact: Website

Re: Pi won't boot after update.

Thu Sep 20, 2012 11:13 pm

dom wrote:
alias_neo wrote:OK, here's the results;

Latest with 0c0 start.elf = no boot
Latest with 0c0 kernel = no boot
Latest with 0c0 kernel + start.elf = BOOT.

EDIT:

Ok, with 240M start.elf from 0c0 I get a partial boot. It doesn't load fully, but hangs after the following lines;

[ 1.956299] bcm2835-cpufreq: min=0 max=700000 cur=700000
[ 1.961748] bcm2835-cpufreq: switching to governor powersave
[ 1.974766] bcm2835-cpufreq: switching to governor powersave_

That's a replica of what I see, including the cursor at the end of the third line.
Okay, so I think you are saying that from working 0c0 firmware, just updating start.elf to 790681ab24260d652755901cbe8c5fe59b734b71 version breaks it.

Do you have anything in config.txt. Any non-default cmdline.txt options?
cmdline.txt is stock, and config.txt only has overscan disabled, no other changes.

Actually, what I mean is that the boot files are at 9f7cff0534c02cf7b2cab209886af3ff7be95a5c (the very latest) and if I swap the kernel and start.elf for the ones from 0c0ef8afc1d56d172ef0b8476d26616b1096498e it boots.

However, if I use kernel from 0c0ef8afc1d56d172ef0b8476d26616b1096498e and start.elf (240M) from 9f7cff0534c02cf7b2cab209886af3ff7be95a5 I get the result above. I may also get the partial boot with the other start.elf files from 9f7cff0534c02cf7b2cab209886af3ff7be95a5 when using the kernel from 0c0ef8afc1d56d172ef0b8476d26616b1096498e, but I havn't been able to get display to see.
http://2byt.es - Just my two bytes - RPi - Android - Arduino - Blog

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5341
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Pi won't boot after update.

Fri Sep 21, 2012 9:02 am

I want to know the single file change than breaks it.
I believe you have said with everything at 0c0 it works.
I think I can infer that updating start.elf to the next newer version (790681ab24260d652755901cbe8c5fe59b734b71) breaks it.
Can ou confirm that.

Also could you try the very latest files. The latest start.elf did actually fix a bug that could affect boot up.

alias_neo
Posts: 108
Joined: Fri Jul 29, 2011 8:55 am
Location: London, England
Contact: Website

Re: Pi won't boot after update.

Fri Sep 21, 2012 9:27 am

dom wrote:I want to know the single file change than breaks it.
I believe you have said with everything at 0c0 it works.
I think I can infer that updating start.elf to the next newer version (790681ab24260d652755901cbe8c5fe59b734b71) breaks it.
Can ou confirm that.

Also could you try the very latest files. The latest start.elf did actually fix a bug that could affect boot up.
I have tried with all the very latest files, from this commit: HERE, this doesn't work, nor does using ALL the files from any other commit since the 0c0 which does work.

If I get some time when I get home tonight I'll roll back fully to 0c0 and then replace individual files from the next immediate version HERE and let you know the result.
http://2byt.es - Just my two bytes - RPi - Android - Arduino - Blog

Justin72
Posts: 11
Joined: Tue Aug 14, 2012 6:49 am

Re: Pi won't boot after update.

Fri Sep 21, 2012 4:51 pm

Had the exact same thing. Luckily I had another SD card with wheezy on it so I just switched to that. Interested to see what happens, I had a few files on the old card I'd like to get back though...

JP

alias_neo
Posts: 108
Joined: Fri Jul 29, 2011 8:55 am
Location: London, England
Contact: Website

Re: Pi won't boot after update.

Fri Sep 21, 2012 5:13 pm

Ok, so I investigated as suggested by Dom.

I reverted the entire boot partition to the files from the last working (booting) revision (0c0ef8afc1d56d172ef0b8476d26616b1096498e), I then replaced the start.elf with each of the different sized start.elf files from the next immediate revision (790681ab24260d652755901cbe8c5fe59b734b71) which doesn't boot.

It wouldn't boot with any of them.

I then reverted back to a working set of files (0c0ef8afc1d56d172ef0b8476d26616b1096498e) and overwrote only the kernel.img with the one from 790681ab24260d652755901cbe8c5fe59b734b71.

It booted fine.

I'm sure on Doms part, he already has an idea what the problem is (?). Having searched around I saw references to boot_delay, and having tested this I can safely say it doesn't not help my pi to boot on any revision after the working one mentioned. Hopefully the real problem can be fixed :D

In short, start.elf from revision 790681ab24260d652755901cbe8c5fe59b734b71 is the single file that breaks the build after the 3rd of September.
http://2byt.es - Just my two bytes - RPi - Android - Arduino - Blog

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5341
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Pi won't boot after update.

Fri Sep 21, 2012 5:55 pm

alias_neo wrote:In short, start.elf from revision 790681ab24260d652755901cbe8c5fe59b734b71 is the single file that breaks the build after the 3rd of September.
Thanks. I need to narrow down which of the changes broke it for you. I have my suspicions but need to be sure:
https://dl.dropbox.com/u/3669512/temp/start_333987.elf
https://dl.dropbox.com/u/3669512/temp/start_334332.elf
https://dl.dropbox.com/u/3669512/temp/start_334519.elf
https://dl.dropbox.com/u/3669512/temp/start_334692.elf
https://dl.dropbox.com/u/3669512/temp/start_335176.elf
https://dl.dropbox.com/u/3669512/temp/start_335177.elf

So, 333987 should work and 335177 should fail. My guess is that only 335177 will fail, but that needs testing.

alias_neo
Posts: 108
Joined: Fri Jul 29, 2011 8:55 am
Location: London, England
Contact: Website

Re: Pi won't boot after update.

Sat Sep 22, 2012 10:25 am

Ok,

I'v tested these;

333987: OK
334332: OK
334519: OK
334692: OK
335176: OK
335177: OK

I'm afraid every one of them booted fine.
http://2byt.es - Just my two bytes - RPi - Android - Arduino - Blog

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5341
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Pi won't boot after update.

Sat Sep 22, 2012 1:48 pm

alias_neo wrote: 333987: OK
334332: OK
334519: OK
334692: OK
335176: OK
335177: OK

I'm afraid every one of them booted fine.
Hmm. Unexpected. Can you try this top of tree build:
https://dl.dropbox.com/u/3669512/temp/start_338522.elf

which should behave the same as github version (and so, I assume fail).

alias_neo
Posts: 108
Joined: Fri Jul 29, 2011 8:55 am
Location: London, England
Contact: Website

Re: Pi won't boot after update.

Sat Sep 22, 2012 3:40 pm

dom wrote:
alias_neo wrote: 333987: OK
334332: OK
334519: OK
334692: OK
335176: OK
335177: OK

I'm afraid every one of them booted fine.
Hmm. Unexpected. Can you try this top of tree build:
https://dl.dropbox.com/u/3669512/temp/start_338522.elf

which should behave the same as github version (and so, I assume fail).
This one boots. They do differ though, since their MD5s are different;

[ C:\Users\User\Desktop\start_338522.elf ]
MD5: C7CE29C9E37E255DE71EDC8E4B3DA64B

[ C:\Users\User\Desktop\start.elf ] (from latest github)
MD5: 858DBA945AD3C6AC18C18E1DEDCD4EC9

EDIT: There seems to be an issue with the archive of the latest files in Github, if I extract under Windows it complains about several modules existing twice (I assume it's a case insensitivity issue), and so it insists on overwriting one with the other. THe modules in question are; ipt_ECN.ko, xt_DSCP.ko, xt_HL.ko, xt_RATEEST.ko, xt_TCPMSS.ko.
Last edited by alias_neo on Sat Sep 22, 2012 3:57 pm, edited 1 time in total.
http://2byt.es - Just my two bytes - RPi - Android - Arduino - Blog

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5341
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Pi won't boot after update.

Sat Sep 22, 2012 3:47 pm

alias_neo wrote:
dom wrote:
alias_neo wrote: 333987: OK
334332: OK
334519: OK
334692: OK
335176: OK
335177: OK

I'm afraid every one of them booted fine.
Hmm. Unexpected. Can you try this top of tree build:
https://dl.dropbox.com/u/3669512/temp/start_338522.elf

which should behave the same as github version (and so, I assume fail).

This one boots. They do differ though, since their MD5s are different;

[ C:\Users\User\Desktop\start_338522.elf ]
MD5: C7CE29C9E37E255DE71EDC8E4B3DA64B

[ C:\Users\User\Desktop\start.elf ] (from latest github)
MD5: 858DBA945AD3C6AC18C18E1DEDCD4EC9
Yes, there is a version string embedded, so a recompile is not binary identical. So, latest github start.elf fails for you?
Should be the same source tree, but they are built on different machines.

alias_neo
Posts: 108
Joined: Fri Jul 29, 2011 8:55 am
Location: London, England
Contact: Website

Re: Pi won't boot after update.

Sat Sep 22, 2012 4:03 pm

Yes, there is a version string embedded, so a recompile is not binary identical. So, latest github start.elf fails for you?
Should be the same source tree, but they are built on different machines.
Latest github from 21 hours ago boots, using all files from latest github. I take it you managed to solve the issue/ build on a different machine?

2245c759858b3ab14593d6cd54c39e1827d99f35: Boots fine.

EDIT: Version details;

[email protected] ~ $ uname -a
Linux Raspbox 3.2.27+ #165 PREEMPT Thu Sep 20 22:28:17 BST 2012 armv6l GNU/Linux
[email protected] ~ $ /opt/vc/bin/vcgencmd version
Sep 20 2012 22:38:16
Copyright (c) 2012 Broadcom
version 338528 (release)
[email protected] ~ $
http://2byt.es - Just my two bytes - RPi - Android - Arduino - Blog

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5341
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: Pi won't boot after update.

Sat Sep 22, 2012 9:28 pm

alias_neo wrote:Latest github from 21 hours ago boots, using all files from latest github. I take it you managed to solve the issue/ build on a different machine?
There was a fix in last start.elf. Not specifically for your problem, I just spotted the wrong caching was being used when loading the kernel.
It made no difference to me, and was technically more correct so I changed it.

It is just possible that due to some difference in your setup (e.g. sdram or sdcard timing) the wrong data was getting read from the cache, and this change has actually fixed the problem.

I guess you should keep an eye on future updates and shout out if it breaks again.

Thanks for testing.

alias_neo
Posts: 108
Joined: Fri Jul 29, 2011 8:55 am
Location: London, England
Contact: Website

Re: Pi won't boot after update.

Sat Sep 22, 2012 10:02 pm

dom wrote:
alias_neo wrote:Latest github from 21 hours ago boots, using all files from latest github. I take it you managed to solve the issue/ build on a different machine?
There was a fix in last start.elf. Not specifically for your problem, I just spotted the wrong caching was being used when loading the kernel.
It made no difference to me, and was technically more correct so I changed it.

It is just possible that due to some difference in your setup (e.g. sdram or sdcard timing) the wrong data was getting read from the cache, and this change has actually fixed the problem.

I guess you should keep an eye on future updates and shout out if it breaks again.

Thanks for testing.
No problem, good to see its fixed now. I'll keep an eye on it of course and let you know if any more problems crop up.
http://2byt.es - Just my two bytes - RPi - Android - Arduino - Blog

alias_neo
Posts: 108
Joined: Fri Jul 29, 2011 8:55 am
Location: London, England
Contact: Website

Re: Pi won't boot after update.

Tue Sep 25, 2012 7:56 pm

Back to square one it would appear :cry:

I killed my memory card with an overclock it would seem, shut Pi down last night, wouldn't boot today, partition is showing as unknown in my PC, so I got the latest image and re-imaged the card, doesn't boot, latest github doesn't boot, went back to the infamous 0c0 and that boots fine.

I'm testing a few builds back but I don't have the patience to run through them all at the moment, there are things I wanted to get done on my Pi and keep getting held up by an unbootable Pi.

Pretty frustrating :(

EDIT: Further, even on bootable firmwares, my pi NEVER comes back up after a reboot, and unlike the older firmwares when I first got my Pi, it won't boot any more with both keyboard and HDMI connected, but will boot intermittently (after several attempts) with just HDMI connected.
http://2byt.es - Just my two bytes - RPi - Android - Arduino - Blog

alias_neo
Posts: 108
Joined: Fri Jul 29, 2011 8:55 am
Location: London, England
Contact: Website

Re: Pi won't boot after update.

Tue Sep 25, 2012 8:40 pm

Ok, so on the latest firmware it boots up fairly consistently with boot_delay=5, I never needed this before, however, reboots still fail every time, I assume because they probably don't take boot_delay into account (judging by the speed it fails at).

This is only true for the normal start.elf though as the 192M fails to boot every time even with the delay.

I'v come to recognise the OK LED flicker (2 short pulses) that means it will fail to boot, where as a success is usually two short followed by several faster, shorter, dimmer pulses before the LAN LEDs light up.

I hope this can get fixed, inconsistent booting is a real problem and 0% success on reboots is a bigger problem because I can't remote manage and re-boot with out power cycling it.

:(
http://2byt.es - Just my two bytes - RPi - Android - Arduino - Blog

alias_neo
Posts: 108
Joined: Fri Jul 29, 2011 8:55 am
Location: London, England
Contact: Website

Re: Pi won't boot after update.

Wed Sep 26, 2012 10:51 am

Anyone able to comment on whether this is something that might get fixed? The older firmwares all boot fine, but later ones have become increasingly problematic.

If I can't count on being able to boot reliably on any of the new firmwares I intend to return my Pi to element14/Farnell, I'm just waiting on a potential solution before I do so.
http://2byt.es - Just my two bytes - RPi - Android - Arduino - Blog

User avatar
RaTTuS
Posts: 10491
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK

Re: Pi won't boot after update.

Wed Sep 26, 2012 11:09 am

have you eliminated other issues i.e. power ?
what is the power as measured from TP1 / TP2
what power supply are you using ?
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

alias_neo
Posts: 108
Joined: Fri Jul 29, 2011 8:55 am
Location: London, England
Contact: Website

Re: Pi won't boot after update.

Wed Sep 26, 2012 12:13 pm

RaTTuS wrote:have you eliminated other issues i.e. power ?
what is the power as measured from TP1 / TP2
what power supply are you using ?
I have several power supplies, mains ones, all capable of sourcing at least an amp of current, and they average 4.9V between TP1/TP2, with the lowest one dropping to 4.8V.

They have all run fine in the past with 2xUSB devices, Ethernet and HDMI connected when under heavy load. Even with no peripherals connected the Pi won't boot on later firmwares.

I'm using a Rev 2 board (without poly fuses I believe) which eliminates some voltage drop there.

The SD card is a new, 32GB Sandisk Ultra Class 10.
http://2byt.es - Just my two bytes - RPi - Android - Arduino - Blog

Return to “Troubleshooting”