pmbdk
Posts: 8
Joined: Thu Jan 02, 2020 2:09 pm

Raspberry Pi 4 boot-over-USB with NVMe drive

Tue Jan 14, 2020 10:53 am

TLDR: I used https://jamesachambers.com/raspberry-pi ... sh-drives/ as a guide for setting up boot-over-USB on the RPi4. I used an externally powered USB3 hub for attaching a Samsung EVO 970 Plus 1 TB NVMe through a USB-to-NVMe bridge as the original RPi4 power supply could not handle the load.

This is just a quick post of my experience with trying to boot from USB on the Raspberry Pi 4.

I know that there are a huge amount of posts regarding this, but most of them do not apply to the raspberry pi 4 or they are simply huge (the sticky one in this forum has ~550 replies and the first post does not seem to apply to the RPi4).

Currently the RPi4 does not support boot-over-USB, so the SD-card need to contain /boot, while the USB-drive can contain the rest. I had a Samsung EVO 970 Plus 1 TB NVMe laying around, so I used an external enclosure from StarTech.com (https://www.startech.com/HDD/Enclosures ... M2E1BMU31C) with a ASMedia ASM2362 USB 3.1g2-to-NVMe bridge chip to connect to the RPi4 (and yes, I know, these two are crazy overkill for the RPi4!).

After having burned the raspbian images to both the SD-card and the USB-drive from Windows, I initially booted up on the SD-card to see that everything was working. I attached the USB-drive to the RPi4, the blue led on the enclsoure powered up, but no drive was available in Linux.

After digging around in the output of dmesg it seemed the drive was recognized and added to /dev/sda, but there were some strange messages in it as well (some unrecognized commands and stuff).

Looking into the the datasheet of the EVO 970, I found that the drive could use up to 8-9 W (!). My USB-C power supply (original RPi4 power supply) seems to be able to to deliver 3A@5.1 V with the RPi (plus keyboard/mouse) taking up around 5-8 W, so I figured it was probably marginal.

I attached an externally powered USB3 hub, and voilá! everything was working!

I then followed https://jamesachambers.com/raspberry-pi ... sh-drives/ (you can jump to the section "USB Boot Instructions" but there is some good stuff about known working SB enclosures) to use the USB drive as system drive.

I have been running it for a few days now and have observed the following:
  • The RPi4 is SO much more responsive when running from the USB-drive. It's a massive improvement. Especially when doing "heavy" disk I/O you will really feel the difference: With the stock SD-card, when doing a long apt update/upgrade/install there were times when my browser simply stopped responding for several seconds every 1-5 seconds. Long compile-jobs can now just run and I don't even notice any slowdown. :-)
  • Boot-up time is actually not that much different (but hey, it was already pretty fast).
  • Application startup is improved quite a bit.
  • I would NOT recommend using an NVMe drive for this; at least not the one I have been using. The drive enclosure gets incredibly hot (uncomfortably so). The enclosure from StarTech.com I have been using does not have ANY way of transmitting heat from the SSD to the enclosure, so the heat is purely based on air-convection; I am too much of a chicken to try to touch the actual SSD but it is surely in the ~100 degC range. No matter how you look at it, this can not be good for the longevity of the drive. Although the NVMe drive may be slightly faster than others in theory, this should not make a difference in reality due to the USB3.0 I/F on the RPi4. I will still be using it, but I will ditch the alu enclosure from StarTech and simply use the bridge PCB only and add a heat-sink with some air flow to the SSD.
  • If you use an old USB3 enabled SSD disk, you may not run into power-problems. I don't know if my problems were due to the 15W original USB-C power supply or something else, but if you have problems, try to go through an externally powered USB3 hub.

User avatar
bonzadog
Posts: 259
Joined: Wed Apr 25, 2012 9:40 am
Location: Rietberg/Germany

Re: Raspberry Pi 4 boot-over-USB with NVMe drive

Tue Jan 28, 2020 11:48 am

An interesting report. I want to do the same bus using a 320GB USB Disk.

I wonder when RPI.org will get the full USB Boot up and running?

Brad Q
Posts: 101
Joined: Mon Aug 12, 2019 12:10 am

Re: Raspberry Pi 4 boot-over-USB with NVMe drive

Tue Jan 28, 2020 10:11 pm

There is and has been for a long time, a utility here on the site to automate doing this. Do a search for usb-boot.zip here on the site. RonR does a good job of supporting issues.
Last edited by Brad Q on Wed Jan 29, 2020 6:21 am, edited 1 time in total.

mact
Posts: 3
Joined: Tue Jan 28, 2020 4:17 am

Re: Raspberry Pi 4 boot-over-USB with NVMe drive

Wed Jan 29, 2020 3:18 am

noobie here, looking into adding a pi to my menu. There is an outfit in China selling on eBay a couple of RPi addon boards. One will put a SATA SSD into the boot option and another changes this to an NVMe board.

A 128–512 SSD costs about the same, or a little less, than a similar capacity USB Thumb Drive or SD card. And is much faster on any level USB.

I don't think I ought to post the eBay item numbers. But I imagine there are those who have seen them, maybe tried them and …???

User avatar
Gavinmc42
Posts: 4547
Joined: Wed Aug 28, 2013 3:31 am

Re: Raspberry Pi 4 boot-over-USB with NVMe drive

Wed Jan 29, 2020 6:07 am

Nice trick, did not know the Pi4 USB boot option was working yet.
Should work with a Pi3.
Easy to do without having to buy a "special" from China.

Some USB3 to SATA chips do work, some don't on a Pi4.
But they still need to boot from an uSD at the moment.
I'm dancing on Rainbows.
Raspberries are not Apples or Oranges

User avatar
bonzadog
Posts: 259
Joined: Wed Apr 25, 2012 9:40 am
Location: Rietberg/Germany

Re: Raspberry Pi 4 boot-over-USB with NVMe drive

Wed Jan 29, 2020 8:52 am

Brad Q wrote:
Tue Jan 28, 2020 10:11 pm
There is and has been for a long time, a utility here on the site to automate doing this. Do a search for usb-boot.zip here on the site. RonR does a good job of supporting issues.
When I entered usb-boot.zip on the search I got 611 posts. Would you be so kind and point me the download?

I really want the raspberrypi.org version...so I will have to put up with the SD + USB HDD solution.
I have no idea when raspberrypi.org will release the "pure" boot solution.

BUT I will try the usb-boot though.

I found this https://www.diskgenius.com/manual/creat ... b-disk.php but I think it may not
work on the RPi

Brad Q
Posts: 101
Joined: Mon Aug 12, 2019 12:10 am

Re: Raspberry Pi 4 boot-over-USB with NVMe drive

Wed Jan 29, 2020 9:06 am

The ONLY way to currently boot the Pi4 involves using the SD card. Until the foundation makes a firmware to support USB boot, that is our only option. The usb-boot.zip utility I mentioned just automates the process of moving from a straight SD to a linked SD-USB boot. They have stated that they will not even start to work on the USB-boot part until they work out the network boot (PXE). While PXE has been in testing for a while I have not heard of much progress since October. I am not faulting anybody here as there have been a lot of other bugs that I am sure pulled people away from the PXE project and am well aware that things are often not as easy as we hope. That being said I would guess that July(pure guess) would probably early for a firmware USB boot availability.

https://www.raspberrypi.org/forums/view ... b+boot.zip

Edit: Not sure why the double post.
Last edited by Brad Q on Wed Jan 29, 2020 9:10 am, edited 1 time in total.

Brad Q
Posts: 101
Joined: Mon Aug 12, 2019 12:10 am

Re: Raspberry Pi 4 boot-over-USB with NVMe drive

Wed Jan 29, 2020 9:09 am

The ONLY way to currently boot the Pi4 involves using the SD card. Until the foundation makes a firmware to support USB boot, that is our only option. The usb-boot.zip utility I mentioned just automates the process of moving from a straight SD to a linked SD-USB boot. They have stated that they will not even start to work on the USB-boot part until they work out the network boot (PXE). While PXE has been in testing for a while I have not heard of much progress since October. I am not faulting anybody here as there have been a lot of other bugs that I am sure pulled people away from the PXE project and am well aware that things are often not as easy as we hope. That being said I would guess that July(pure guess) would probably early for a firmware USB boot availability.

https://www.raspberrypi.org/forums/view ... b+boot.zip

JessicaMulein
Posts: 2
Joined: Mon Dec 16, 2019 2:08 pm

Re: Raspberry Pi 4 boot-over-USB with NVMe drive

Sun Mar 15, 2020 12:03 am

Any update in the last month or so?

User avatar
Rukbat
Posts: 27
Joined: Fri Nov 11, 2011 3:59 pm

Re: Raspberry Pi 4 boot-over-USB with NVMe drive

Thu Mar 19, 2020 10:18 pm

Unfortunately, at this point (March 19th), you still have to boot from the SD card. You can set things up to mount /boot on the card and / on the SSD, so the card is used only for booting and updating the boot software.

LTolledo
Posts: 3756
Joined: Sat Mar 17, 2018 7:29 am
Location: Anime Heartland

Re: Raspberry Pi 4 boot-over-USB with NVMe drive

Thu Mar 19, 2020 10:26 pm

....and there is no real-world advantage on booting RPi4B with the root on a NVMe drive....

even if native USB booting does becomes avialable (dont ask when, it will be ready when its ready)....the condition will remain...
"Don't come to me with 'issues' for I don't know how to deal with those
Come to me with 'problems' and I'll help you find solutions"

Some people be like:
"Help me! Am drowning! But dont you dare touch me nor come near me!"

Brad Q
Posts: 101
Joined: Mon Aug 12, 2019 12:10 am

Re: Raspberry Pi 4 boot-over-USB with NVMe drive

Thu Mar 19, 2020 10:30 pm

While I would agree that it gets here when it is ready and linked boot is good enough until then, I would disagree that there is no real world advantage.
For starters you will no longer need to have access to the SD card. This is a pretty big deal(PITA) in case design.

LTolledo
Posts: 3756
Joined: Sat Mar 17, 2018 7:29 am
Location: Anime Heartland

Re: Raspberry Pi 4 boot-over-USB with NVMe drive

Thu Mar 19, 2020 11:41 pm

ooops....
the statement should have been:
with the rootfs residing on the drives, there are no real world advantage between NVMe drive in USB3.0 adapters vs. regular SSDs in USB3.0 adapters,
"Don't come to me with 'issues' for I don't know how to deal with those
Come to me with 'problems' and I'll help you find solutions"

Some people be like:
"Help me! Am drowning! But dont you dare touch me nor come near me!"

jfabernathy
Posts: 103
Joined: Thu Oct 11, 2018 10:52 am
Location: Central North Carolina

Re: Raspberry Pi 4 boot-over-USB with NVMe drive

Mon Apr 06, 2020 11:15 am

Did everyone apply the Quirks fix that was mentioned in the section of the linked guide under "Fix (some) USB Adapter Problems Using Quirks".
My performance was okay but I finally put in the Quirks fix and it really jumped.

User avatar
bonzadog
Posts: 259
Joined: Wed Apr 25, 2012 9:40 am
Location: Rietberg/Germany

Re: Raspberry Pi 4 boot-over-USB with NVMe drive

Mon Apr 06, 2020 3:07 pm

LTolledo wrote:
Thu Mar 19, 2020 10:26 pm
....and there is no real-world advantage on booting RPi4B with the root on a NVMe drive....

even if native USB booting does becomes avialable (dont ask when, it will be ready when its ready)....the condition will remain...
Is there no project plan date for the USB Boot -- we all want it.

W. H. Heydt
Posts: 12956
Joined: Fri Mar 09, 2012 7:36 pm
Location: Vallejo, CA (US)

Re: Raspberry Pi 4 boot-over-USB with NVMe drive

Mon Apr 06, 2020 5:29 pm

bonzadog wrote:
Mon Apr 06, 2020 3:07 pm
LTolledo wrote:
Thu Mar 19, 2020 10:26 pm
....and there is no real-world advantage on booting RPi4B with the root on a NVMe drive....

even if native USB booting does becomes avialable (dont ask when, it will be ready when its ready)....the condition will remain...
Is there no project plan date for the USB Boot -- we all want it.
*Some* of us want it. (And I include myself in that.) If we "all" wanted it, it would have been a higher priority than PxE boot. I have 3 Pi4B4s doing hybrid boot waiting for the full USB boot, plus a Pi3B+ the boots from USB and will get swapped out for a Pi4B2 when USB booting arrives for the Pi4B.

Brad Q
Posts: 101
Joined: Mon Aug 12, 2019 12:10 am

Re: Raspberry Pi 4 boot-over-USB with NVMe drive

Mon Apr 06, 2020 6:11 pm

While I would prefer a direct USB boot, I do not understand the impatient foot tapping. As far as I can tell the only speed difference between a hybrid boot and a direct boot would be in booting itself. Considering how short booting is now with the hybrid boot, I doubt you will see much over a ten second improvement in booting when direct boot becomes available. It is a bit of a PITA designing a good case that you either have to maintain access to the SD card or putting up with a tear down to access the SD card, but in reality it is just a speed bump. So what advantage of direct boot that has some people so excited am I missing?

User avatar
dickon
Posts: 1653
Joined: Sun Dec 09, 2012 3:54 pm
Location: Home, just outside Reading

Re: Raspberry Pi 4 boot-over-USB with NVMe drive

Mon Apr 06, 2020 6:59 pm

Meh -- if it's ethernet-connected just boot over the network... You don't even need to bother mounting /boot (other than for firmware upgrades) if you don't want to.

User avatar
bonzadog
Posts: 259
Joined: Wed Apr 25, 2012 9:40 am
Location: Rietberg/Germany

Re: Raspberry Pi 4 boot-over-USB with NVMe drive

Tue Apr 07, 2020 6:50 am

Brad Q wrote:
Mon Apr 06, 2020 6:11 pm
While I would prefer a direct USB boot, I do not understand the impatient foot tapping. As far as I can tell the only speed difference between a hybrid boot and a direct boot would be in booting itself. Considering how short booting is now with the hybrid boot, I doubt you will see much over a ten second improvement in booting when direct boot becomes available. It is a bit of a PITA designing a good case that you either have to maintain access to the SD card or putting up with a tear down to access the SD card, but in reality it is just a speed bump. So what advantage of direct boot that has some people so excited am I missing?
I think the main advantage is that the SD can only perform a number or Write operations and putting all the file in a HDD should make is faster.
Plus a lot of space. Some 32GB SD seem to be a bit slow as well, but I am trying to find some trustworthy article that has made a comparison on this.
But there again I also use the external drive ( not booted at the moment) for SQL ITE which saves SD space.

User avatar
bonzadog
Posts: 259
Joined: Wed Apr 25, 2012 9:40 am
Location: Rietberg/Germany

Re: Raspberry Pi 4 boot-over-USB with NVMe drive

Tue Apr 07, 2020 6:52 am

Brad Q wrote:
Mon Apr 06, 2020 6:11 pm
While I would prefer a direct USB boot, I do not understand the impatient foot tapping. As far as I can tell the only speed difference between a hybrid boot and a direct boot would be in booting itself. Considering how short booting is now with the hybrid boot, I doubt you will see much over a ten second improvement in booting when direct boot becomes available. It is a bit of a PITA designing a good case that you either have to maintain access to the SD card or putting up with a tear down to access the SD card, but in reality it is just a speed bump. So what advantage of direct boot that has some people so excited am I missing?
Perhaps the question is why a hybrid
boot and not a direct boot - so we are back at square one.

User avatar
rpdom
Posts: 17418
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Raspberry Pi 4 boot-over-USB with NVMe drive

Tue Apr 07, 2020 7:24 am

bonzadog wrote:
Tue Apr 07, 2020 6:50 am
Brad Q wrote:
Mon Apr 06, 2020 6:11 pm
While I would prefer a direct USB boot, I do not understand the impatient foot tapping. As far as I can tell the only speed difference between a hybrid boot and a direct boot would be in booting itself. Considering how short booting is now with the hybrid boot, I doubt you will see much over a ten second improvement in booting when direct boot becomes available. It is a bit of a PITA designing a good case that you either have to maintain access to the SD card or putting up with a tear down to access the SD card, but in reality it is just a speed bump. So what advantage of direct boot that has some people so excited am I missing?
I think the main advantage is that the SD can only perform a number or Write operations and putting all the file in a HDD should make is faster.
Plus a lot of space. Some 32GB SD seem to be a bit slow as well, but I am trying to find some trustworthy article that has made a comparison on this.
But there again I also use the external drive ( not booted at the moment) for SQL ITE which saves SD space.
These are not valid arguments for not using the "hybrid" boot method. Only the /boot file system stays on the card. That hardly ever gets written to and is only read for a few seconds when starting to boot, so speed of the card isn't really important.. The main / filesystem is on the external device with plenty of space and speed.

I'm keeping a copy of my Pi4B's /boot filesystem on my SSD in preperation for when full USB boot arrives. But I'm in no hurry.

Interestingly, I watched an interview with Eben yesterday. He was asked if it is likely that a future Pi would have a SATA interface. Eben said no and said that the USB adaptors work well for SATA so there is no point in building it in to a Pi. However he went on to add that if they did put any mass storage interface on a hypothetical future Pi, it would likely be NVME.
Unreadable squiggle

User avatar
bonzadog
Posts: 259
Joined: Wed Apr 25, 2012 9:40 am
Location: Rietberg/Germany

Re: Raspberry Pi 4 boot-over-USB with NVMe drive

Tue Apr 07, 2020 7:31 am

rpdom wrote:
Tue Apr 07, 2020 7:24 am
bonzadog wrote:
Tue Apr 07, 2020 6:50 am
Brad Q wrote:
Mon Apr 06, 2020 6:11 pm
While I would prefer a direct USB boot, I do not understand the impatient foot tapping. As far as I can tell the only speed difference between a hybrid boot and a direct boot would be in booting itself. Considering how short booting is now with the hybrid boot, I doubt you will see much over a ten second improvement in booting when direct boot becomes available. It is a bit of a PITA designing a good case that you either have to maintain access to the SD card or putting up with a tear down to access the SD card, but in reality it is just a speed bump. So what advantage of direct boot that has some people so excited am I missing?
I think the main advantage is that the SD can only perform a number or Write operations and putting all the file in a HDD should make is faster.
Plus a lot of space. Some 32GB SD seem to be a bit slow as well, but I am trying to find some trustworthy article that has made a comparison on this.
But there again I also use the external drive ( not booted at the moment) for SQL ITE which saves SD space.
These are not valid arguments for not using the "hybrid" boot method. Only the /boot file system stays on the card. That hardly ever gets written to and is only read for a few seconds when starting to boot, so speed of the card isn't really important.. The main / filesystem is on the external device with plenty of space and speed.

I'm keeping a copy of my Pi4B's /boot filesystem on my SSD in preperation for when full USB boot arrives. But I'm in no hurry.
I want a full USB Boot and not hybrid solution. This is why I would like to see a USB Boot as soon as possible. It was mentioned when the RPi4 came out.

LTolledo
Posts: 3756
Joined: Sat Mar 17, 2018 7:29 am
Location: Anime Heartland

Re: Raspberry Pi 4 boot-over-USB with NVMe drive

Tue Apr 07, 2020 11:50 am

be careful, you might already be under the radar of the mods.....

some who were quite adamant about this were given "quarantine".....a "lockdown" .....or worst "cremated" out....
you wouldn't want any of that.. do you?

as been already said on numerous topics/post on the matter...
it will be ready when its ready.... no time frame.... be patient like most of us are.
"Don't come to me with 'issues' for I don't know how to deal with those
Come to me with 'problems' and I'll help you find solutions"

Some people be like:
"Help me! Am drowning! But dont you dare touch me nor come near me!"

User avatar
bonzadog
Posts: 259
Joined: Wed Apr 25, 2012 9:40 am
Location: Rietberg/Germany

Re: Raspberry Pi 4 boot-over-USB with NVMe drive

Tue Apr 07, 2020 12:04 pm

pmbdk wrote:
Tue Jan 14, 2020 10:53 am
TLDR: I used https://jamesachambers.com/raspberry-pi ... sh-drives/ as a guide for setting up boot-over-USB on the RPi4. I used an externally powered USB3 hub for attaching a Samsung EVO 970 Plus 1 TB NVMe through a USB-to-NVMe bridge as the original RPi4 power supply could not handle the load.

This is just a quick post of my experience with trying to boot from USB on the Raspberry Pi 4.

I know that there are a huge amount of posts regarding this, but most of them do not apply to the raspberry pi 4 or they are simply huge (the sticky one in this forum has ~550 replies and the first post does not seem to apply to the RPi4).

Currently the RPi4 does not support boot-over-USB, so the SD-card need to contain /boot, while the USB-drive can contain the rest. I had a Samsung EVO 970 Plus 1 TB NVMe laying around, so I used an external enclosure from StarTech.com (https://www.startech.com/HDD/Enclosures ... M2E1BMU31C) with a ASMedia ASM2362 USB 3.1g2-to-NVMe bridge chip to connect to the RPi4 (and yes, I know, these two are crazy overkill for the RPi4!).

After having burned the raspbian images to both the SD-card and the USB-drive from Windows, I initially booted up on the SD-card to see that everything was working. I attached the USB-drive to the RPi4, the blue led on the enclsoure powered up, but no drive was available in Linux.

After digging around in the output of dmesg it seemed the drive was recognized and added to /dev/sda, but there were some strange messages in it as well (some unrecognized commands and stuff).

Looking into the the datasheet of the EVO 970, I found that the drive could use up to 8-9 W (!). My USB-C power supply (original RPi4 power supply) seems to be able to to deliver 3A@5.1 V with the RPi (plus keyboard/mouse) taking up around 5-8 W, so I figured it was probably marginal.

I attached an externally powered USB3 hub, and voilá! everything was working!

I then followed https://jamesachambers.com/raspberry-pi ... sh-drives/ (you can jump to the section "USB Boot Instructions" but there is some good stuff about known working SB enclosures) to use the USB drive as system drive.

I have been running it for a few days now and have observed the following:
  • The RPi4 is SO much more responsive when running from the USB-drive. It's a massive improvement. Especially when doing "heavy" disk I/O you will really feel the difference: With the stock SD-card, when doing a long apt update/upgrade/install there were times when my browser simply stopped responding for several seconds every 1-5 seconds. Long compile-jobs can now just run and I don't even notice any slowdown. :-)
  • Boot-up time is actually not that much different (but hey, it was already pretty fast).
  • Application startup is improved quite a bit.
  • I would NOT recommend using an NVMe drive for this; at least not the one I have been using. The drive enclosure gets incredibly hot (uncomfortably so). The enclosure from StarTech.com I have been using does not have ANY way of transmitting heat from the SSD to the enclosure, so the heat is purely based on air-convection; I am too much of a chicken to try to touch the actual SSD but it is surely in the ~100 degC range. No matter how you look at it, this can not be good for the longevity of the drive. Although the NVMe drive may be slightly faster than others in theory, this should not make a difference in reality due to the USB3.0 I/F on the RPi4. I will still be using it, but I will ditch the alu enclosure from StarTech and simply use the bridge PCB only and add a heat-sink with some air flow to the SSD.
  • If you use an old USB3 enabled SSD disk, you may not run into power-problems. I don't know if my problems were due to the 15W original USB-C power supply or something else, but if you have problems, try to go through an externally powered USB3 hub.
An interesting and informative report...well done!

input
Posts: 221
Joined: Sun Nov 02, 2014 9:18 am
Location: Netherlands

Re: Raspberry Pi 4 boot-over-USB with NVMe drive

Tue Apr 07, 2020 12:11 pm

You say you used a powered USB3.0 hub. Can you tell me what brand that is.

Thanks

Return to “General discussion”