B+ odd issues with SD card (solved)

[Shoka]

Converting from an existing model B (rev 2 IIRC.)

Updated the original Pi which has a 32G Sandisk full size SD card.

dd the original 32G SD card on to a no name 32G microSD using my normal Linux laptop.
Not exact match on the by bytes in and out on the two cards. Not a good sign.

B+ would not boot with it, nor would the original pi with an adapter.

Paid extortionate money for a 32G Sandisk microSD card.
(more than the B+)

Re did the dd. Exact match in bytes transferred.
Looks better, but B+ will not boot on it, nor will the model B with that card in an adapter.

All very odd, put the SD card back in my laptop, which mounts the second partition with no problems, but's not happy with the boot partition. Investigated with fdisk and fsck. Partition 2 checks out ok, partition 1 seems broken. My Linux laptop does not like the boot partition, and complains that the partition is corrupt.

Upshot, eventually had to fdisk that microSD card with a new definition of the boot partition, format the boot partition as vfat using my laptop, and manually copy the files on the boot partition of the original SD card to the new dos boot partition.


At which point the B+ happily booted.

NOT A Problem with the Pi. Doubt very much its a problem with my laptop.
Could be a problem with the card adapters I'm using on the laptop, but they have worked ok before.

I have a nasty suspicion that the microSD card is doing something "helpful" with the partition blocks on the SDcard.

Any body else seen anything like this?


Harry

[Lob0426]

I updated and upgraded a 4GB SD then backed it up with Win32DiskImager. Then transferred it to a 16GB Micro SD, using a SD card adapter. It worked just fine when I put it into the RasPi B+.

You may have a bad adapter. Check the boot files in the FAT32 partition. They may have been corrupted.

[Shoka]

Sorry was possibly not clear that I've resolved the issue. I'm wondering at the cause...

The partition blocks at the beginning of the SD card were corrupted, but corrupted so that fdisk still knew that the first partition was a vfat one. Like something "knew" what the partition block should look like, but got it wrong.

I quite happily ran fdisk, format and copied the boot partition contents via that adapter, and its happily and correctly copied the whole of the 32G root partition, so the fault with the adapter is odd to say the least that it messed with the partition block only, and only while doing a disk to disk copy.

I'm polling for anyone who has encountered similar issues to see if more data will show a pattern and thus help elucidate the cause. This is not a cry for help, the issue is fixed.


Cheers Harry.

[DougieLawson]

http://www.raspberrypi.org/forums/viewt ... 29#p547029

dosfstools is fundamentally borked until you update it.

[AndrewS]

Dunno why it would mess up the first partition rather than the second partition, but when using 'dd' to copy images between two SD cards that apparently have the same size, you'll often find that e.g. one SD card is 31.9 GB and the other SD card is 31.7 GB, and so 'dd' won't be able to properly copy from the bigger card to the smaller card.

But you've already found the solution - manually create and format the partitions instead, and copy the data across at the (mounted) filesystem-layer instead of at the low-level (unmounted) block-layer.

[rpdom]

Sorry was possibly not clear that I've resolved the issue. I'm wondering at the cause...

Sorry, I've no idea what the cause may have been for this.

The partition blocks at the beginning of the SD card were corrupted, but corrupted so that fdisk still knew that the first partition was a vfat one. Like something "knew" what the partition block should look like, but got it wrong.

In the first 512 bytes of the card there is the Master Boot Record (MBR). The "Boot" bit of that isn't used on the Pi, but it also contains the Partition Table. This has a simple list of all the partitions: Where they start, how big they are, whether they are "bootable" (again, not applicable to Pi), and what type they are supposed to be.

fdisk (and friends) only change this table. They don't do anything to the partition itself. So your partition table was correctly showing partition 1 to be vfat, but the partition itself had been damaged somehow.

Good that you got it fixed

[Jim JKla]

We should take on board that SD cards are ok but have some background flakieness and there ar loads of fakes out there that are even flakier.

[Shoka]

Initially the copied SD would not mount partition 1 on my linux laptop. IIRC partition 2 mounted fine.

Facing another mega dd (32G takes hours) I had nothing to loose.

Initially tried to create a new fat file system in partition 1.
mkfs came back unable to find partition 1.

Ran fdisk which reported the boot partition as corrupted, built me a new one that matched the old in all important respects as far as I could see, like the partition sizes and types and matched the one on my reference full size 32G SD .

Wrote that back to the SD card, checked that partition 2 still mounted, tried to fsck partition 1.

The fat file system in partition 1 was comprehensively trashed, fsck still made no sense of it and was unable to establish that a vfat filesystem was there at all.

I then rebuilt the vfat file system on partition 1. File level copy of the boot files to partition 1. Did not need to do anything with partition 2.

That's why I raised the issue since I'd expect a size mismatch issue to show up at the end of the big partition, not at the start of the disk, and selectively mangling the relatively tiny fat filesystem and leaving the huge ext partition untouched seems suspiciously clever for dumb luck.

As delivered there are a nest of nasty useless utilities on the original32 gig fat partition, and possibly some protected code for the SD cards internal processor. I'm wondering if that processor messed with the initial blocks copied in by DD thinking it was preserving its "extended fat" partition.

My fdisk and mkfs would splatter anything but private hidden code for the processor I think. Fdisk is really good at shattering code that tries to hide in the rest of the partition block.

But if some activity by the SD cards processor triggered the issue, others will see the same problem, ie dd'ing into a brand spanking new big SD-card and getting the initial blocks corrupted.

It could of course be just "good luck" that only the tiny fat system was trashed.
Time and more users will tell, hence my shout.

Harry

[AndrewS]

It could of course be just "good luck" that only the tiny fat system was trashed.

Possibly!

The other thing you could try (if you wanted, and don't mind trashing /dev/sdX1 again) would be to:
# get the size (in sectors) of /dev/sdX1
sudo parted -s /dev/sdX u s p
# create a random-contents file of the same size
dd if=/dev/urandom of=random.input count=size-in-sectors
# write the random-data to the SD card
sudo dd if=random.input of=/dev/sdX1 bs=4M
sync
# read the random-data from the SD card
sudo dd if=/dev/sdX1 of=random.output bs=4M
sync
# check if the data written matches the data read - no output if identical
diff -q random.input random.output

Obviously you need to replace sdX with the device-name of your SD card, and you need to be very careful when typing these commands as you could easily trash one of the partitions on your hard drive.

[Shoka]

I'm guessing it would not be repeatable, since my hammer handed "you will take this fdisk" seems to have worked.

Think I'll wait for the statistical answer, as 32G cards get used more, sandisk ones specifically, does the problem show up elsewhere.

If it does we have this thread to help anyone else who hits it, and if we don't this thread will fade away.



Harry

[rpdom]

I have seen something like this with some of the "fake" cards.

One example (not an SD card, but a flash device) was a cheap MP3 player that was sold as 8GB, but in reality only had 2GB of storage.

Once you went past the first 2GB it just started again from the beginning and overwrote everything that was already written often including the MBD/Partition table

Once I realised this I used fdisk to create a 2GB partition and it works fine. fdisk still reports a total "disk" size of 8GB, with 6GB unallocated though.

[DougieLawson]

It could of course be just "good luck" that only the tiny fat system was trashed.
Time and more users will tell, hence my shout.

Stick it in your Windows laptop and checkdisk the FAT partition. Windows is good at repairing them.

The dosfstools package in Raspbian is borked.
http://www.raspberrypi.org/forums/viewt ... 28&t=64843

To avoid copying 32GB of, mostly empty, space shrink the card before you start, expand it after.
http://www.raspberrypi.org/forums/viewt ... 91&t=58069
http://www.raspberrypi.org/forums/viewt ... 91&t=60918

[pleriche]

I can't be sure but I seem to be seeing similar problems with using dd to copy images.

I'm trying to create a customised Raspbian config for replication for school use, with additional packages pre-intalled. This requires expanding partition 2 but not to the full 4GB of the SDCard, since the destination SDCard may be slightly smaller. I do this by deleting and recreating partition 2 bigger then using resize2fs to resize the fs within it. I then use dd to take an image of only enough of the card to include up to the end of partition 2. I haven't yet succeeded in getting a clone of the image on another card to boot successfully. That may be because I'm running out of known good cards, but I'm not entirely convinced as the resultant card seems to check out when mounted in a USB card reader.

Regards - Philip

[AndrewS]

I then use dd to take an image of only enough of the card to include up to the end of partition 2.

Are you sure you're calculating this value correctly?

[pleriche]

Another potential gotcha seems to be if the SDCard you're about to write to with Win32DiskImager already contains a vfat boot partition, Windows will attempt to mount it, and Win32DiskImager is then unable to write it. If you get in quick you can beat Windows, but I'm wondering whether it still messes around under Win32DiskImager's feet. Clobbering the first MB of the SDCard with dd if=/dev/zero of=/dev/sda count=2048 seems to avoid the problem.

Regards - Philip

[pleriche]

I checked the size calculation very carefully as it is indeed easy to get it wrong.

[Shoka]

Will the faulting cards boot the Pi at all?

I'm not clear from your description if it is partition 1 or 2 that is the problem. Or like my fault even the partition table borked.

Your dd starts at the lowest address in the SD card I believe, and works its way up.

The partition table is also at a very low address.

If the boot partition is faulty the indicator LED's on the PI stay entirely static.

If the boot partition is intact but the filesystem in partition 2 is broken, the Pi panics and stops, but there is some activity on the LED's before that happens (I think)

You seem to be messing with the size of the root partition, so miscalculations there, except for the start address, which will always be the same, since the boot partition is constant size, will kill the Linux kernel during boot, but should not stop the Pi running through its initialisation...

So I'm not sure what exactly your symptoms are, but miscalculations of the end of the root partition as suggested above should give different symptoms to a trashed partition table.

Of course if you are getting the start wrong, and trashing the top of the boot partition, all bets are off.

Harry

[AndrewS]

The partition table is also at a very low address.

Yup, the partition table is part of the MBR, and the MBR is always the first 512 bytes on the SD card.

[Shoka]

Problem is that as far as the outside world is concerned, its in the lowest addresses on the SD card. Block zero as Andrew points out, but where it really is located depends on where the processor on the SD card put it.

And its quite capable of special casing that address range.

SD cards look like discs from "outside" but you are actually accessing the storage via a "hidden actor" the processor on the SD card.

Harry

[Jim JKla]

I keep pointing people at these two guys Bunnie & Xobs their video is not quick and in some places its quite deep but it definately gives a heads up on some SD card issues.

http://media.ccc.de/browse/congress/201 ... _xobs.html

[Shoka]

Yes seen the video.
Terrifying.

I started this thread as I had some issues that looked to me like a possible attempt to be helpful by the internal processor in a big SD card, that had had unfortunate side effects, and wanted to flag it up in case others hit the same problem.

FAT at 32G size is not very efficient, and I can easily see SD card manufacturers thinking they know what is going on and putting in code to help.....

Since I took it straight out of it's packaging and started a DD down to it, I can sort of forgive it for thinking that it should try and protect its internal FAT file structure from me, if that was what it was doing. Sort of .....

So far nobody has shouted up with symptoms that match mine exactly, but I'm not really surprised.

My issue was with the first time use of a good quality (well expensive anyway) 32G SD card, that cost significantly more than the pi it's running in, so I'm guessing at the moment playing with these is a minority sport.

However time and peoples expectation of Moore's law marches on, so if there is a real issue here I expect eventually more people will hit it.

Cheers Harry

[mahjongg]

However time and peoples expectation of Moore's law marches on, so if there is a real issue here I expect eventually more people will hit it.

I'm sure of it.

Perhaps instructions saying any card must always be full formatted using the official formatter would help.

marked solved for now.

[Shoka]

Clobbering the first MB of the SDCard with dd if=/dev/zero of=/dev/sda count=2048 seems to avoid the problem.

From Phillip up thread or fdisking it with a non fat partition or a blank partition table seem more practical.

Running a formatter on it , to then trash the format with a disc image seems odd.

Or is there there some preferred "formatter" utility to prepare an SD card for copying in an image, that I've missed somewhere?

Harry

[bcw142]

One of the problems I see with what was done is that 32G isn't really a real size standard. 32G drives of any type even from the same manufacturer have different sizes often times. I ran into this on Transcend S.S.D. drives (all say they are 32G), the new ones are a few bytes smaller than the old ones (2013 vs 2014)! When you try to DD from old to new it doesn't work and I suspect that few bytes is being wrapped to the start causing the problem. It basically did the same thing you mentioned (was Fedora in this case, but still boot area first). It's only 2 or 4 bytes but that will kill the boot, keeping the basic FAT structure and partition table intact. I suspect if you did the same thing with the same two SD cards the same result would occur. I tried Clonezilla and it wouldn't do the copy no matter how I tried, DD isn't that smart and just writes blindly. It will try to write right off the end the the drive and it may end up wrapped to the start. I ended up copying using DD, compression, and separate images of both partitions rather than one continuous 'copy' like DD would otherwise do. Then made the partitions with gparted and copied the images in (in reverse order; partition 2 first) so it if walked off the end and back to start it would still be rewritten when I did partition 1 (boot) afterwards. I don't think the B+ and older B are that different when it comes to the SD card (even though the SD is now MicroSD).

[pluggy]

Use gparted (and make the main partition smaller) BEFORE you create the image from it. dd will give an error when you put the image on the slightly smaller card but since the extra space is just dead area it doesn't matter. You can stretch it again to fit the new card afterwards if you want. If you putting it on a B+ afterwards, update/upgrade the software on the B before you start.