Copy image to multiple SD Cards.

Share and discuss resources for use in the classroom

18 posts
by morphy_richards » Fri Dec 21, 2012 10:43 am
I know there was a thread on this a while back but I cant seem to find it now ... Lots of info out there on copying a single image to a single card but ...

I've got a nice up to date working image of Raspbian that's set up with a few extras installed such as Geany copied to a hard disk.
I've copied this image successfully to another SD card using Win32 Disk Imager.

The problem is I've got fifteen SD Cards to copy.

Equipment I've got to hand is

a number of 10 port USB hubs
15 rev 2 Raspberry Pi's
Some Windows Computers (but only have admin access and therefore ability to run W32 Disk Imager on one computer.)

What's the best way of copying the image to 10 or 15 cards at once - (it's not a problem if it needs to run overnight)
User avatar
Posts: 852
Joined: Mon Mar 05, 2012 3:26 pm
Location: London
by jamiesk » Fri Dec 21, 2012 11:05 am
Pi1 (Nov 2012 loft)= 1KW immersion controller for Solar panel
Pi2 (Jan 2013 living room)=Play thing
Pi3 (Feb 2013 mobile)= Play thing with Tandy Ladder board,breakout board,Nokia display
http://www.raspberrypi.org/phpBB3/viewtopic.php?f=26&t=28193
Posts: 87
Joined: Mon Nov 26, 2012 8:48 pm
Location: Ipswich, Suffolk, England, UK.
by morphy_richards » Fri Dec 21, 2012 11:28 am
http://raspberrypi.stackexchange.com/qu ... o-many-sds
Given that no-one seems to have produced a multi-card SD card reader (one that can hold multiple SD cards at once) and the fact that that you can buy a USB hub and a bunch of single SD card USB readers very cheaply, one way to speed up the process would be to make your own SD card duplicator.


Okay ..

Theoretically ...
from http://raspberrypi.stackexchange.com/qu ... spberry-pi
Creating a disk image will preserve not only files but also the filesystem structure and when you decide to flash your new SD card, you will be able to just plug it in and it will work.

On Linux, you can use the standard dd tool:

dd if=/dev/sdx of=/path/to/image bs=1M
Where /dev/sdx is your SD card.


So , could I plug a load of SD Card readers & cards into a multi-port hub on a Raspberry Pi
Start one dd copy process going
Detach it - (using something like nohup? or Screen?)
Start the next one going...
Repeat process until all cards are being written
Go home for the weekend
Come back on Monday to 15 written SD cards?
User avatar
Posts: 852
Joined: Mon Mar 05, 2012 3:26 pm
Location: London
by morphy_richards » Fri Dec 21, 2012 12:26 pm
Thinking about this a bit more ...

I think it would be generally useful to have a Bash script that could automate the entire process. - to be shared with schools who need to setup lots of Pis in a batch job.
Assume You have a Raspberry Pi with a powered USB hub plugged in

The output might go something like this...

pi@raspberry ~ $ clonedisks
Insert the master disk into a USB port and press Enter >
Checking disk ... ok
Insert a blank disk for copying into a new USB port and press Enter >
...ok. Do you want to copy another y/n? > y
Insert a blank disk for copying into a new USB port and press Enter >
...ok. Do you want to copy another y/n? > y
Insert a blank disk for copying into a new USB port and press Enter >
...ok. Do you want to copy another y/n? > y
Insert a blank disk for copying into a new USB port and press Enter >
...ok. Do you want to copy another y/n? > n
Ready to clone disks. Press enter when ready>
Copying ...
...
...
...
Done.
pi@raspberry ~ $
User avatar
Posts: 852
Joined: Mon Mar 05, 2012 3:26 pm
Location: London
by jamiesk » Fri Dec 21, 2012 12:49 pm
A nice little script will do that for you :) If you mount all the devices then try ...

#!/bin/bash
for i in 1 2 3 4 5
do
echo "Starting to program $i"
dd if=/path/to/image of=/dev/sd$i bs=1M
echo "We have completed $i"
done

I have not tested this substitution of the loop variable $i in this ... maybe make a string of it and use "of=string"

My rPi is in the loft switching on and off the 1Kw hot water heater in this sun shine 8-)
Pi1 (Nov 2012 loft)= 1KW immersion controller for Solar panel
Pi2 (Jan 2013 living room)=Play thing
Pi3 (Feb 2013 mobile)= Play thing with Tandy Ladder board,breakout board,Nokia display
http://www.raspberrypi.org/phpBB3/viewtopic.php?f=26&t=28193
Posts: 87
Joined: Mon Nov 26, 2012 8:48 pm
Location: Ipswich, Suffolk, England, UK.
by morphy_richards » Fri Dec 21, 2012 12:53 pm
jamiesk wrote:A nice little script will do that for you :) If you mount all the devices then try ...


Excellent, thank you! I haven't made one of these since nineteen ninety something so tinkering with this will make a nice little Christmas holiday project...

jamiesk wrote:My rPi is in the loft switching on and off the 1Kw hot water heater in this sun shine 8-)


Lucky you! Most of the country here seems to now be underwater :D
User avatar
Posts: 852
Joined: Mon Mar 05, 2012 3:26 pm
Location: London
by morphy_richards » Fri Dec 21, 2012 2:38 pm
jamiesk wrote:#!/bin/bash
for i in 1 2 3 4 5
do
echo "Starting to program $i"
dd if=/path/to/image of=/dev/sd$i bs=1M
echo "We have completed $i"
done



This would do one after the other in a queue which is fine, however I'm wondering if it might be possible to make all the cards copy the from master in parallel /at the same time (not even sure if it would be possible like that with dd) by detaching the processes, and would that even make the job finish any faster.
User avatar
Posts: 852
Joined: Mon Mar 05, 2012 3:26 pm
Location: London
by jamiesk » Fri Dec 21, 2012 9:40 pm
Yes you can. I would think it would go faster due to writing to a card is slower than reading.

Google found me http://superuser.com/questions/145516/c ... ltaneously which basically says
Code: Select all
cat drive.image | tee >(dd of=/dev/sda) >(dd of=/dev/sdb) >(dd of=/dev/sdc) | dd of=/dev/sdd

But how will you catch errors ? Or do you care?

I would put together files for each copy like (NOT TESTED)
#!/bin/bash
# for SD1
dd if=/path/to/image of=/dev/sd1 bs=1M > /tmp/sd1
mail -s "Sd1 done" you@email.com <~/email.done


and then call them all from a main one
#!/bin/bash
# to run all files from SD1 to SD5
./SD1 &&
./SD2 &&
./SD3 &&
./SD4 &&
./SD5 &&


And then each will send you an email of how it did. Assuming you have email setup!!

But you said it could be an overnighter, I would have hoped that 15 times 4G cards would be written a bit quicker than an overnighter!!!
Pi1 (Nov 2012 loft)= 1KW immersion controller for Solar panel
Pi2 (Jan 2013 living room)=Play thing
Pi3 (Feb 2013 mobile)= Play thing with Tandy Ladder board,breakout board,Nokia display
http://www.raspberrypi.org/phpBB3/viewtopic.php?f=26&t=28193
Posts: 87
Joined: Mon Nov 26, 2012 8:48 pm
Location: Ipswich, Suffolk, England, UK.
by morphy_richards » Sat Dec 22, 2012 7:02 am
you said it could be an overnighter, I would have hoped that 15 times 4G cards would be written a bit quicker than an overnighter!!!

Originally I thought it could be but it would be good if we can make it go faster. If it could do a batch in a lunch break (doubtful but worth a go) that would be good.
User avatar
Posts: 852
Joined: Mon Mar 05, 2012 3:26 pm
Location: London
by jamiesk » Sat Dec 22, 2012 10:28 am
Give it a go. Have one of the science kids with a stopwatch too :)

Also make sure the master has had the file expanded to the full size of the SD card too. You will be copying it, might as well make it right in the first place.
Pi1 (Nov 2012 loft)= 1KW immersion controller for Solar panel
Pi2 (Jan 2013 living room)=Play thing
Pi3 (Feb 2013 mobile)= Play thing with Tandy Ladder board,breakout board,Nokia display
http://www.raspberrypi.org/phpBB3/viewtopic.php?f=26&t=28193
Posts: 87
Joined: Mon Nov 26, 2012 8:48 pm
Location: Ipswich, Suffolk, England, UK.
by rpdom » Sun Dec 30, 2012 9:37 am
Another way to copy to multiple cards simultaneously is to install and use the dcfldd package. It is like an enhanced version of dd with lots of useful extra options.

As an example (disclaimer - I haven't actually tired this as I don't have any spare cards right now), I'd use:

Code: Select all
dcfldd if=/path/to/image bs=1M sizeprobe=if of=/dev/sda of=/dev/sdb of=/dev/sdc (etc...)


As you can see, it supports multiple "of=" statements, and the "sizeprobe=if" means "Show a continuous progress report calculated using the size of the input file"
Posts: 2562
Joined: Sun May 06, 2012 5:17 am
Location: Essex, UK
by hagalin » Fri Sep 13, 2013 11:36 am
Premise ;

A machine with running linux and root access
6 card readers/writers
6 copies of the image to be written

The operating system is running on /dev/sda and it is the only thing that is usally mounted

#!/bin/bash
# Unmount all media / disk to start clean
umount /media/*
UNIQ_DATE_ID=`date +%d%m%Y%H%M%S_$$`
# Find what disks are there
fdisk -l /dev/sd[qwertyuiopasdfghjklzxcvbnm]|grep dev|grep Disk|grep -v sda | awk '{print $2}' | awk -F"\:" '{print $1}'>>/tmp/disk_id$UNIQ_DATE_ID
# Generate disks varibles
DISK1=`cat /tmp/disk_id$UNIQ_DATE_ID | awk 'NR==1 {print}'`
DISK2=`cat /tmp/disk_id$UNIQ_DATE_ID | awk 'NR==2 {print}'`
DISK3=`cat /tmp/disk_id$UNIQ_DATE_ID | awk 'NR==3 {print}'`
DISK4=`cat /tmp/disk_id$UNIQ_DATE_ID | awk 'NR==4 {print}'`
DISK5=`cat /tmp/disk_id$UNIQ_DATE_ID | awk 'NR==5 {print}'`
DISK6=`cat /tmp/disk_id$UNIQ_DATE_ID | awk 'NR==6 {print}'`

# spit out to standard out disk devices
echo $DISK1
echo $DISK2
echo $DISK3
echo $DISK4
echo $DISK5
echo $DISK6

# start to copy take the image /img/some_img_a.img and send it to disk which is correct
dd if=/img/some_img.A.img of=$DISK1 bs=1048576 conv=noerror >> /tmp/dd_status-$UNIQ_DATE_ID &
dd if=/img/some_img.B.img of=$DISK2 bs=1048576 conv=noerror >> /tmp/dd_status-$UNIQ_DATE_ID &
dd if=/img/some_img.C.img of=$DISK3 bs=1048576 conv=noerror >> /tmp/dd_status-$UNIQ_DATE_ID &
dd if=/img/some_img.D.img of=$DISK4 bs=1048576 conv=noerror >> /tmp/dd_status-$UNIQ_DATE_ID &
dd if=/img/some_img.E.img of=$DISK5 bs=1048576 conv=noerror >> /tmp/dd_status-$UNIQ_DATE_ID &
dd if=/img/some_img.F.img of=$DISK6 bs=1048576 conv=noerror >> /tmp/dd_status-$UNIQ_DATE_ID &

echo `date` >> /tmp/dd_status-$UNIQ_DATE_ID
Posts: 2
Joined: Fri Sep 13, 2013 11:23 am
by hagalin » Sat Sep 14, 2013 2:15 pm
One caveat.

If no linux machine is there, you can always live boot a machine from for example ubuntu livecd to gain a temporary linux machine. After use, boot the machine normaly and it will be unchanged.
http://www.ubuntu.com/download/desktop

any linux would work as most if not all will have the tools used in the script.
Posts: 2
Joined: Fri Sep 13, 2013 11:23 am
by rockandscissor » Tue Oct 29, 2013 4:46 pm
We had a similar requirement for a project where we needed to be able to write images to 10+ Raspberry Pi SD cards on a daily basis. We looked at the various Card Duplicator's available to buy but found most of them won't successfully make a bootable SD card, as most only copy files directly and not the bootable information, they are also ridiculously expensive!

So as a solution we wrote our own simple web-based software to run on a single Raspberry Pi connected to 2 x 7-port Belkin powered USB hubs (The Pi has a limitation of 2 hubs, and a maximum of 14 USB ports).

We've now released this as Open Source software to help others who have the same requirement. Currently just the source code is available but we will also be uploading an image file based on Arch Linux ARM running Nginx + php

http://www.rockandscissor.com/projects/osid
Posts: 1
Joined: Tue Oct 29, 2013 4:45 pm
by morphy_richards » Mon Nov 18, 2013 4:25 pm
rpdom wrote:Another way to copy to multiple cards simultaneously is to install and use the dcfldd package. It is like an enhanced version of dd with lots of useful extra options.

As an example (disclaimer - I haven't actually tired this as I don't have any spare cards right now), I'd use:

Code: Select all
dcfldd if=/path/to/image bs=1M sizeprobe=if of=/dev/sda of=/dev/sdb of=/dev/sdc (etc...)


As you can see, it supports multiple "of=" statements, and the "sizeprobe=if" means "Show a continuous progress report calculated using the size of the input file"


Giving this a go right now ...
fdisk -l to locate the mounted names of the sd cards and then the command as you gave it ... seems to be working, I have 5 cards being written (apparently, flashy lights going whizzy anyway) via a USB hub and 5 card reader/adapter thingies. Sadly it didn't seem able to detect any more than 5 even though the hub has 10 ports.
Image
User avatar
Posts: 852
Joined: Mon Mar 05, 2012 3:26 pm
Location: London
by morphy_richards » Tue Nov 19, 2013 9:36 am
Success.

A batch of 5 cards plugged into a multi USB hub seem to take about 15 minutes to write.
User avatar
Posts: 852
Joined: Mon Mar 05, 2012 3:26 pm
Location: London
by morphy_richards » Tue Dec 17, 2013 8:25 am
to copy the image from the working sd
dmesg | tail
use this to find the sd card
gives
Code: Select all
[80619.310254] sd 6:0:0:0: [sdb] Mode Sense: 03 00 00 00
[80619.310879] sd 6:0:0:0: [sdb] No Caching mode page found
[80619.310893] sd 6:0:0:0: [sdb] Assuming drive cache: write through
[80619.316077] sd 6:0:0:0: [sdb] No Caching mode page found
[80619.316084] sd 6:0:0:0: [sdb] Assuming drive cache: write through
[80619.317600]  sdb: sdb1 sdb2
[80619.321028] sd 6:0:0:0: [sdb] No Caching mode page found
[80619.321037] sd 6:0:0:0: [sdb] Assuming drive cache: write through
[80619.321044] sd 6:0:0:0: [sdb] Attached SCSI removable disk

we want "[sdb] Attached SCSI removable disk"
So use dd to copy the file:
sudo -i
dd if=/dev/sdb of=~/filename.bin
This copies the file into the root home directory.
When it's finished, copy the file to somewhere more accessible and then issue the commands above to copy multiple images.
User avatar
Posts: 852
Joined: Mon Mar 05, 2012 3:26 pm
Location: London
by morphy_richards » Tue Jan 14, 2014 10:12 pm
Word of warning. Pay attention when you list your sd cards using fdisk -l , I managed to accidentally over write my main linux partition on my ubuntu machine today with a copy of a rasbian image because I just assumed sdb would be a card. Doh!
User avatar
Posts: 852
Joined: Mon Mar 05, 2012 3:26 pm
Location: London