osukl1
Posts: 50
Joined: Thu Jan 02, 2020 5:00 pm

CM4 Mass Programming

Thu Mar 25, 2021 2:47 pm

Has anyone had any ideas or experience in mass programming of the CM4 modules?
We will want to program between 20 and 50 at a time - and using the USB MSD and dd for each individually would take a long time.

My thoughts so far;

Push an initrd with a minimal system with usbboot to grab the image from tftp or similar and flash the emmc with that.
Pi farm for flashing - ~20 CM4 to program ~20 CM4
Like above except with a single processor + USB hubs to push a stage 1 initrd to the farm.


It would be interesting to hear how you guys at Pi are doing this in production too if possible.

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1697
Joined: Sat Sep 10, 2011 11:43 am

Re: CM4 Mass Programming

Thu Mar 25, 2021 3:45 pm

We do have a solution we're trialling at the moment, but it's in a state of flux right now... Basically programs everything through USB device, but we're switching it to Ethernet to improve the performance...

Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

osukl1
Posts: 50
Joined: Thu Jan 02, 2020 5:00 pm

Re: CM4 Mass Programming

Thu Apr 01, 2021 9:28 am

Sounds interesting. It would be great if you are able to share any details with us, as we will need to start producing our product in volume in the next 12 weeks or so. Thank you

jmckenna
Posts: 4
Joined: Wed Feb 24, 2021 1:29 pm

Re: CM4 Mass Programming

Wed Apr 28, 2021 2:41 pm

Hi,

Is there any update on this feature?

Erosis
Posts: 2
Joined: Tue May 11, 2021 6:05 pm

Re: CM4 Mass Programming

Wed May 12, 2021 2:14 am

gsh wrote:
Thu Mar 25, 2021 3:45 pm
We do have a solution we're trialling at the moment, but it's in a state of flux right now... Basically programs everything through USB device, but we're switching it to Ethernet to improve the performance...

Gordon
I'd also like to know if there's any updates on flashing over ethernet or emulating MSD at a higher frequency for USB 2. Being limited to 5.5 MB/s is quite slow for flashing many full custom images.

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1697
Joined: Sat Sep 10, 2011 11:43 am

Re: CM4 Mass Programming

Wed May 12, 2021 12:37 pm

Yes, the provisioning system is being trialled right now with a few select customers...

Email me and I'll add you to the list... simply email

echo info@raspberrypi.com | sed -e "s/info/gordon/"
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

jmckenna
Posts: 4
Joined: Wed Feb 24, 2021 1:29 pm

Re: CM4 Mass Programming

Wed May 12, 2021 7:52 pm

gsh wrote:
Wed May 12, 2021 12:37 pm
Yes, the provisioning system is being trialled right now with a few select customers...

Email me and I'll add you to the list... simply email

echo info@raspberrypi.com | sed -e "s/info/gordon/"

Thanks for the reply!

How will the ethernet flash approach would work if using an ethernet switch chip on a custom IO board? The switch chip I'm using gets configured through SPI from the CM4 on bootup. Can this pose a problem?

Thanks,
JM

cleverca22
Posts: 3743
Joined: Sat Aug 18, 2012 2:33 pm

Re: CM4 Mass Programming

Wed May 12, 2021 8:29 pm

i think that would depend on what the switch chip defaults to

does it default to a dumb un-managed switch, or does it block all traffic until configured?

incognitum
Posts: 752
Joined: Tue Oct 30, 2018 3:34 pm

Re: CM4 Mass Programming

Wed May 12, 2021 11:17 pm

If it is one of those older 10/100 mbit switch chips, throughput may also be a concern.
May be worth testing if using USB (CDC-ECM, not MSD) doesn't perform better than Ethernet in your particular corner case.

cleverca22
Posts: 3743
Joined: Sat Aug 18, 2012 2:33 pm

Re: CM4 Mass Programming

Wed May 12, 2021 11:31 pm

incognitum wrote:
Wed May 12, 2021 11:17 pm
If it is one of those older 10/100 mbit switch chips, throughput may also be a concern.
May be worth testing if using USB (CDC-ECM, not MSD) doesn't perform better than Ethernet in your particular corner case.
usb2.0 has a raw symbol-rate of 480mhz
due to bit-stuffing, the actual bit-rate of the usb packets will vary between 384-480mbit
which comes to between 45MB/sec and 57MB/sec, minus the usb protocol overheads

from my old comments in https://github.com/raspberrypi/usbboot/issues/68
the msd/start4.elf file, can write data at 11 MB/sec
a linux based MSD gadget (i have an initrd to run it on boot), can write data at 32 MB/sec
100mbit ethernet, comes out to 11.92MB/sec, minus protocol overheads
gigabit ethernet comes out to 119MB/sec, minis protocol overheads

so 100mbit is basically tied with the official msd/start4.elf, until you subtract out the ethernet overheads

a linux based MSD gadget comes in 2nd place
and something gigabit based comes in 1st place

incognitum
Posts: 752
Joined: Tue Oct 30, 2018 3:34 pm

Re: CM4 Mass Programming

Thu May 13, 2021 10:32 am

cleverca22 wrote:
Wed May 12, 2021 11:31 pm
so 100mbit is basically tied with the official msd/start4.elf, until you subtract out the ethernet overheads
a linux based MSD gadget comes in 2nd place
and something gigabit based comes in 1st place
Comparison is not that simple.

You are not taking into account that when you use an initrd that talks to a server over Ethernet (or USB CDC-ECM) you have the option to transfer compressed images.

OTOH we do not know if that switch chip can forward packets at line speed (may even be slower than 100 mbit).

And when you are provisioning multiple units simultaneously buses may also get congested server side, and different transport methods deal better/worse with that.

cleverca22
Posts: 3743
Joined: Sat Aug 18, 2012 2:33 pm

Re: CM4 Mass Programming

Thu May 13, 2021 3:01 pm

incognitum wrote:
Thu May 13, 2021 10:32 am
And when you are provisioning multiple units simultaneously buses may also get congested server side, and different transport methods deal better/worse with that.
multi-cast might also be an option there, but that then needs far more custom protocols
incognitum wrote:
Thu May 13, 2021 10:32 am
You are not taking into account that when you use an initrd that talks to a server over Ethernet (or USB CDC-ECM) you have the option to transfer compressed images.
ah yeah, i didnt think about adding in compression, which makes a lot of the initrd based ones (even MSD, with some trickery) capable of going even faster


edit:
oh, somebody already made that 1st part: https://en.wikipedia.org/wiki/UFTP
The UDP-based File Transfer Protocol (UFTP) is a communication protocol designed to transfer files to multiple recipients.[1] To accomplish this, UFTP multicasts the files to recipients via the User Datagram Protocol (UDP).

incognitum
Posts: 752
Joined: Tue Oct 30, 2018 3:34 pm

Re: CM4 Mass Programming

Thu May 13, 2021 4:32 pm

cleverca22 wrote:
Thu May 13, 2021 3:01 pm
edit:
oh, somebody already made that 1st part: https://en.wikipedia.org/wiki/UFTP
The UDP-based File Transfer Protocol (UFTP) is a communication protocol designed to transfer files to multiple recipients.[1] To accomplish this, UFTP multicasts the files to recipients via the User Datagram Protocol (UDP).
Intel's PXE standard also defines a multi-cast tftp flavor, supported by some tftpd daemons like atftpd.

Can see such things being useful if you have a need to provision a lot of systems at the same time.
Like when you want to re-install a Windows image on all computers in a school on a periodic basis.


But don't see the direct advantage in this case.
Keep in mind that when provisioning devices there are also typically human limitations involved.

Say you have a single person handling the provisioning and:

1) are using in-system programming, where the CM4 module is inserted in the final device.
2) that device is connected to Ethernet and power, network boots, and the provisioning system does its job.
3) afterwards the person has to disconnect the device, puts some printed label with the serial on the device, puts device in a carton box, seals it.

Say it takes 3 minutes for the system to write the image, and the human handling before and after take 30 seconds in total.
Then you could provision 6 units at a time, to have one finished every 30 seconds.

Probably the number of units downloading simultaneously is still low enough then for the speed of the server's Ethernet port to not become the limiting factor in that case, provided gigabit is used and compressed images.
Last edited by incognitum on Thu May 13, 2021 4:39 pm, edited 1 time in total.

cleverca22
Posts: 3743
Joined: Sat Aug 18, 2012 2:33 pm

Re: CM4 Mass Programming

Thu May 13, 2021 4:38 pm

i did also have an idea on how to make a multicast protocol specifically for imaging, that would better deal with the async nature

basically, just blast the entire .img out over multicast, in an endless loop, with headers to define the seek position at regular intervals

the instant a blank CM4 gets on the network and hears a header, it just starts writing, at whatever position the header is at
so it may start writing at 55% into the disk image
when it gets the last byte, the transmission will wrap-around back to the 0% mark
then just keep writing, until you get back to the 55% point you started at, and now the image is complete, sync, validate, shutdown, and signal completion via some LED

that idea, allows a cm4 to be turned on at any time, and join in on a pre-existing transfer, so you can just keep shoving cm4's in as fast as you can, and pop them back out as they complete
if you have the right hw, it could also be automated with a bed of nails and a automated belt

incognitum
Posts: 752
Joined: Tue Oct 30, 2018 3:34 pm

Re: CM4 Mass Programming

Thu May 13, 2021 4:51 pm

cleverca22 wrote:
Thu May 13, 2021 4:38 pm
basically, just blast the entire .img out over multicast, in an endless loop, with headers to define the seek position at regular intervals
But it still only makes sense if you are short on bandwidth.

And it will be slower than unicast otherwise, because you will have to broadcast a bit on the slow side to take into account that storage not always has the same write speed (even if it are supposed to be same eMMC chips).
While it can go for whatever maximum speed the eMMC of that individual board is willing to accept the data if downloading individually.

cleverca22
Posts: 3743
Joined: Sat Aug 18, 2012 2:33 pm

Re: CM4 Mass Programming

Thu May 13, 2021 5:12 pm

yeah, you will want to tune the transmit speed to match the write speed, and how much you can buffer in ram on the CM4 itself

it may also need a unicast way to ask for missed packets to be resent, so you dont have to wait for a second lap

all depends heavily on the size of the .img, the ram size, and how many you want to do in parallel

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1697
Joined: Sat Sep 10, 2011 11:43 am

Re: CM4 Mass Programming

Tue May 18, 2021 5:50 am

It sounds like UFTP does everything you suggest (i.e. allows individual nodes to request retransmissions), don't know about whether it'll allow a device to enter the stream at a random point, but that doesn't sound particularly difficult to implement...

Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

gsh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 1697
Joined: Sat Sep 10, 2011 11:43 am

Re: CM4 Mass Programming

Tue May 18, 2021 5:54 am

Actually reading about the protocol, it seems the start of transfer would have to be synchronised with UFTP, so this wouldn't actually help (and will in fact make it worse since all the units would finish at the same time wasting the operators time...

So yes we'd probably have to implement our own version of that which didn't bother with encryption and allowed the random entry into the data stream...

Gordon
--
Gordon Hollingworth PhD
Raspberry Pi - Director of Software Engineering

incognitum
Posts: 752
Joined: Tue Oct 30, 2018 3:34 pm

Re: CM4 Mass Programming

Tue May 18, 2021 7:49 am

Another option can also be to just use Intel/AMD server gear with higher bandwidth ports.

Network switches with a couple 10 gbe uplink ports start around $ 300 these days, can find 40 gbe QSFP+ fiber at $ 500 (e.g. https://mikrotik.com/product/crs354_48g_4splus2qplusrm )
Add similar amounts for a matching network card server side.
Last edited by incognitum on Tue May 18, 2021 12:21 pm, edited 1 time in total.

dp11
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 520
Joined: Thu Dec 29, 2011 5:46 pm

Re: CM4 Mass Programming

Tue May 18, 2021 8:14 am

or 10 pi's less cost than a server and a switch

incognitum
Posts: 752
Joined: Tue Oct 30, 2018 3:34 pm

Re: CM4 Mass Programming

Tue May 18, 2021 8:18 am

dp11 wrote:
Tue May 18, 2021 8:14 am
or 10 pi's less cost than a server and a switch
Well, you typically will still need some sort of switch with those too, so price difference may be less impressive.

dp11
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 520
Joined: Thu Dec 29, 2011 5:46 pm

Re: CM4 Mass Programming

Tue May 18, 2021 8:19 am

10 memory sticks and 10 ethernet cables :)

Return to “Compute Module”