dd takes so long...


23 posts
by sheriff076 » Sat Nov 03, 2012 9:27 pm
Hello,

I use dd to make a back up of my 16gb sd card. It does the job ok but it takes a such long time to complete.

Am i using the correct dd command in OS X to get the job done swiftly or is 6.6 hours typical for a 16 gb sd card to be dd'd to a .img?
My command and the final out put are below for reference.

Code: Select all
$ sudo dd bs=1m if=/dev/rdisk1 of=~/Desktop/ssbackup04.img

Code: Select all
15193+1 records in
15193+1 records out
15931539456 bytes transferred in 23842.935850 secs (668187 bytes/sec)


Thanks for any advice.
Posts: 34
Joined: Sat May 19, 2012 5:33 pm
Location: South Yorkshire
by PeterO » Sat Nov 03, 2012 9:33 pm
Have you tried increasing the block size ? I use bs=32M.
PeterO
User avatar
Posts: 1077
Joined: Sun Jul 22, 2012 4:14 pm
by milhouse » Sat Nov 03, 2012 11:16 pm
The dd command is fine, but obviously your read speed is awful, suggesting a hardware issue on the Mac. Can you try a different SD-to-USB adapter/card reader, maybe there's a problem with your current one.
Posts: 569
Joined: Mon Jan 16, 2012 12:59 pm
by bud-pnq » Sun Nov 04, 2012 4:19 am
SD card manufacturer and model name would be helpful to help you :) help us help you :P
Posts: 27
Joined: Sat Aug 04, 2012 3:09 am
by sheriff076 » Sun Nov 04, 2012 1:44 pm
PeterO wrote:Have you tried increasing the block size ? I use bs=32M.
PeterO


I haven't, not too sure what the bs actually did in real terms. I have seen different dd how to guides using bs=2 and bs=1.

What am i telling bs to do by changing block size to say 32?
Last edited by sheriff076 on Sun Nov 04, 2012 1:52 pm, edited 1 time in total.
Posts: 34
Joined: Sat May 19, 2012 5:33 pm
Location: South Yorkshire
by sheriff076 » Sun Nov 04, 2012 1:46 pm
bud-pnq wrote:SD card manufacturer and model name would be helpful to help you :) help us help you :P


Is a Sandisk class 6 ultra 16gb card, it works well sofar with day to day Pi tinkering.
Posts: 34
Joined: Sat May 19, 2012 5:33 pm
Location: South Yorkshire
by sheriff076 » Sun Nov 04, 2012 1:49 pm
milhouse wrote:The dd command is fine, but obviously your read speed is awful, suggesting a hardware issue on the Mac. Can you try a different SD-to-USB adapter/card reader, maybe there's a problem with your current one.



I dont have another SD card reader to try that isnt SDHC compatible, it seems to be ok in other uses. The macbook generally runs very well despite being 6 years old, I cant pin it to be a hardware issue but that said I'm no expert :o
Posts: 34
Joined: Sat May 19, 2012 5:33 pm
Location: South Yorkshire
by bud-pnq » Mon Nov 05, 2012 4:34 am
My sandisk ultra class 4 16gb SD card took only about 30 minutes or so on my pretty old macbook. Writing 2 gb image to the SD card took 10min and backing 16gb image took 30-40 min. Your Macbook should be fine and SD card should be fine. Probably your card reader. Dynex reader butchered my microsd useless, address mapping got bitched up and I ain't using dynex no mo :x its bad to assume the reader was wrong but I'm 90 percent sure, it was flaky reader before that problem.
Posts: 27
Joined: Sat Aug 04, 2012 3:09 am
by PeterO » Mon Nov 05, 2012 9:34 am
sheriff076 wrote:I haven't, not too sure what the bs actually did in real terms. I have seen different dd how to guides using bs=2 and bs=1.
What am i telling bs to do by changing block size to say 32?

With bs=1 dd will read and write one byte at time thus incurring huge OS call overheads.
With bs=32M dd will read and write 32M at a time, thus incurring 1/32M of OS call overheads.
On sata hard disks this certainly makes a difference, but on slow devices like SDcards the improvement is less as the speed is dominated by the card access, but ISTR it did make a useful improvement the last time I timed a dd.
PeterO
User avatar
Posts: 1077
Joined: Sun Jul 22, 2012 4:14 pm
by sheriff076 » Mon Nov 05, 2012 3:20 pm
PeterO wrote:With bs=1 dd will read and write one byte at time thus incurring huge OS call overheads.
With bs=32M dd will read and write 32M at a time, thus incurring 1/32M of OS call overheads.
On sata hard disks this certainly makes a difference, but on slow devices like SDcards the improvement is less as the speed is dominated by the card access, but ISTR it did make a useful improvement the last time I timed a dd.
PeterO



Hello Peter

At the risk of sounding a bit thick, this suggestion still writes the whole image file presumably?

Can I get data loss/damage or degradation from higher block sizes.

Thanks
Posts: 34
Joined: Sat May 19, 2012 5:33 pm
Location: South Yorkshire
by milhouse » Mon Nov 05, 2012 3:32 pm
Yes it still writes the whole image, no you won't get data loss/damage or degradation from higher block sizes.
Posts: 569
Joined: Mon Jan 16, 2012 12:59 pm
by bredman » Mon Nov 05, 2012 3:35 pm
I checked the manpage for dd and the command that you used is not valid
$ sudo dd bs=1m if=/dev/rdisk1 of=~/Desktop/ssbackup04.img

If you want to transfer 1Megabyte at a time you should use bs=1M instead of bs=1m.

There is a good chance that dd is ignoring the lowercase m (which is not valid). This means that it is reading and writing one byte at a time, instead of one Megabyte at a time.
Posts: 1413
Joined: Tue Jan 17, 2012 2:38 pm
by PeterO » Mon Nov 05, 2012 3:47 pm
bredman wrote:I checked the manpage for dd and the command that you used is not valid
$ sudo dd bs=1m if=/dev/rdisk1 of=~/Desktop/ssbackup04.img

If you want to transfer 1Megabyte at a time you should use bs=1M instead of bs=1m.

There is a good chance that dd is ignoring the lowercase m (which is not valid). This means that it is reading and writing one byte at a time, instead of one Megabyte at a time.


Except that if you look at the record counts that the OP posted it is clear that it is NOT transferring single bytes.
PeterO
User avatar
Posts: 1077
Joined: Sun Jul 22, 2012 4:14 pm
by bredman » Mon Nov 05, 2012 3:56 pm
PeterO wrote:Except that if you look at the record counts that the OP posted it is clear that it is NOT transferring single bytes.
PeterO

Oh dear, it was a good theory for a short while.
Posts: 1413
Joined: Tue Jan 17, 2012 2:38 pm
by lucas85cunha » Wed Dec 19, 2012 1:33 pm
:?: Hi! I tried to install the .img of raspbian on SD card (32Gb class 10). But dd did not mounted the swap partition. Why this happened?
Posts: 3
Joined: Mon Dec 10, 2012 4:23 pm
by sdjf » Wed Dec 19, 2012 1:56 pm
lucas85cunha wrote::?: Hi! I tried to install the .img of raspbian on SD card (32Gb class 10). But dd did not mounted the swap partition. Why this happened?


dd does not mount file systems, it just copies contents of the img, as far as I know. What exactly did you do? Post the code and output if possible.

Also post details about your card, manufacturer and model, in case that is an issue. Is it on the Verified Peripherals page?

http://elinux.org/RPi_VerifiedPeripherals
FORUM TIP: To view just one person's posting history, sign in, click on their user name, then click on "Search User's Posts." || This Pi owner is running Arch on 512MB Model B.
Posts: 1305
Joined: Fri Mar 16, 2012 5:20 am
Location: California
by jheaton5 » Mon Dec 24, 2012 4:43 pm
rsync (525 kb) from the debian wheezy main repository can transfer a .img file with lightning fast speed. See the script at https://gist.github.com/1004950
User avatar
Posts: 23
Joined: Mon Dec 24, 2012 3:33 pm
by lucas85cunha » Sun Jan 20, 2013 12:38 pm
sdjf wrote:
lucas85cunha wrote::?: Hi! I tried to install the .img of raspbian on SD card (32Gb class 10). But dd did not mounted the swap partition. Why this happened?


dd does not mount file systems, it just copies contents of the img, as far as I know. What exactly did you do? Post the code and output if possible.

Also post details about your card, manufacturer and model, in case that is an issue. Is it on the Verified Peripherals page?

http://elinux.org/RPi_VerifiedPeripherals


I'm just used this command "dd if=/dev/zero of=/dev/sda bs=1M". Are there script for mount perfect system partition?
Posts: 3
Joined: Mon Dec 10, 2012 4:23 pm
by lucas85cunha » Sun Jan 20, 2013 12:38 pm
sdjf wrote:
lucas85cunha wrote::?: Hi! I tried to install the .img of raspbian on SD card (32Gb class 10). But dd did not mounted the swap partition. Why this happened?


dd does not mount file systems, it just copies contents of the img, as far as I know. What exactly did you do? Post the code and output if possible.

Also post details about your card, manufacturer and model, in case that is an issue. Is it on the Verified Peripherals page?

http://elinux.org/RPi_VerifiedPeripherals


I'm just used this command "dd if=/dev/zero of=/dev/sda bs=1M". Are there script for mount perfect system partition?
Posts: 3
Joined: Mon Dec 10, 2012 4:23 pm
by ghans » Sun Jan 20, 2013 9:14 pm
Err ...
that should simply delete your whole SD card.
If you want to write a image to your SD card try this :
Code: Select all
dd bs=32M if=~/Desktop/IMAGENAME.img of=/dev/DEVICE


Don't forget to change IMAGENAME and DEVICE according to your setup.

ghans
• Don't like the board ? Missing features ? Change to the prosilver theme ! You can find it in your settings.
• Don't like to search the forum BEFORE posting 'cos it's useless ? Try googling : yoursearchtermshere site:raspberrypi.org
Posts: 4891
Joined: Mon Dec 12, 2011 8:30 pm
Location: Germany
by redhawk » Sun Jan 20, 2013 9:26 pm
Sounds like the card reader is working in USB 1.1 compatible mode.

Can you format your SD Card to fat/fat32 and copy a large file, does it also take a while to process??

Richard S.
User avatar
Posts: 3519
Joined: Sun Mar 04, 2012 2:13 pm
Location: ::1
by Alport » Mon Mar 11, 2013 6:17 pm
CTRL+t reports the progress of the dd image copy operation.
Posts: 3
Joined: Sat Aug 25, 2012 6:48 am
by RichardBronosky » Wed Mar 13, 2013 6:02 pm
Old topic, but I'll add this anyway...

Sadly it seems that no one has picked up on the fact that the OP is dd to copy data OFF of a 16GB device. Therefore it is moving all 16GB even the unused space. This is very different from using dd to put an raspbian image ONTO a 16GB image.

Consider doing this on a device that has identical read and write characteristics (which none do, but...):
  1. use dd to place 2013-02-09-wheezy-raspbian.img onto a 16GB device
  2. use raspi-config to resizefs to fill the disk
  3. use dd to make a backup of the device

In this simple senario you copy
Step 1 moves 1.8GB of data and takes about 20 minutes
Step 3 moves 16GB of data and takes about 180 minutes

This happens even though there is only 1.3GB of data on the device (as reported by du -sch).

The tool dd is not ideal for this. I suggest using bzipped tar. Full SD recovery becomes a little more complicated because you will want to reimage the device from the original image and the use tar to replace the contents on the data partition. However, it is very simple to extract a subset of files from a tar. So, it's a tradeoff.
-- Richard Bronosky
2x Raspberry Pi 512MB
User avatar
Posts: 58
Joined: Thu Feb 14, 2013 5:38 pm
Location: Atlanta, GA, USA