subatomicglue
Posts: 26
Joined: Sun Oct 13, 2019 5:58 pm

Help me Fast Boot my pi-Zero MIDI "USB Gadget"

Thu Jan 16, 2020 4:40 pm

What I have: A Raspberry Pi Zero in OTG mode running as a "USB Gadget" for both MIDI and Ethernet (for SSH login for development).

The Problem: When I plug in my piZero to my MacOSX, it takes 28 seconds: from plugin, to MIDI device recognized. I'd like to be faster, in 5-10sec range. To support a nice consumer experience.

My Configuration:
- I have a readonly FS so power can be cut without filesystem corruption (easily switched back to readwrite when I want to make a change)
- Service which creates the USB Gadget MIDI definition (and Ethernet for SSH over USB)
....- Registered to run ~ #5 at top of the systemd list, right under the display service (sudo systemctl list-dependencies to see all)
....- in the service I modprobe dwc2 and libcomposite, then create 2 USB device definitions in the /sys/kernel/config/usb_gadget/ directory - one for MIDI and one for Ethernet (so I can also SSH in).

What I've tried:
- I tried a faster microsd card, same boot up speed, no change, I figure we're CPU bound (or stuck on some hardware or network init) here... maybe piZero is slow.
- I originally was running my service from /etc/rc.local, which gave me 58second "time till MIDI". Moving my service to systemd, high up in the order, got me down to 28seconds "time till MIDI".

Ideas for next steps:
- initrd apparently can boot the pi much faster from a RAM disk (you create a disk image on your microsd card, and i’m not sure, but sounds like it slams that into memory as a filesystem for (magic) faster boot times). Pipedream? idk... haven't found a good guide on doing this.
- maybe there is still something to reduce. systemd-analyze blame gives a list of the services and orders by slowest to fastest, so you can find the worst and perhaps optimize them. e.g. Keyboard service seems to be taking ~4seconds.
- maybe I could init the MIDI USB Gadget before systemd initializes services? Maybe using the init.d/rc.d system? IDK. information here is murky. I tried to run my service there but it killed the device and that was the last thing i tried. I could spend more time here. I haven't fully explored yet.
- I dont have a clear understanding what happens before my service runs - nor whether I can move my service earlier. Can you help me think of ways to diagnose / troubleshoot this?


measurements from 2 separate boots, returned by systemd-analyze blame (this list is ordered by slowest to fastest systemd service):

Code: Select all

          9.011s dev-mmcblk0p2.device
          3.858s keyboard-setup.service
          3.204s systemd-udev-trigger.service
          2.995s busybox-syslogd.service
          2.436s [[[[---this is my controller---]]].service     <-------
          2.368s systemd-logind.service
          2.273s dhcpcd.service
          2.219s systemd-journald.service
          2.027s avahi-daemon.service
          1.916s networking.service
          1.856s ssh.service
          1.835s rpi-eeprom-update.service
          1.798s systemd-modules-load.service
          1.734s triggerhappy.service
          1.496s systemd-remount-fs.service
          1.434s rng-tools.service
          1.164s wpa_supplicant.service
          1.156s systemd-user-sessions.service
          1.113s systemd-tmpfiles-setup.service
          1.052s ntp.service
           947ms rc-local.service
           917ms kmod-static-nodes.service
           900ms raspi-config.service
           865ms fake-hwclock.service
           851ms run-rpc_pipefs.mount
           849ms sys-kernel-debug.mount
           823ms dev-mqueue.mount
           813ms systemd-sysctl.service
           802ms systemd-tmpfiles-setup-dev.service
           718ms [email protected]
           689ms systemd-udevd.service
           678ms systemd-random-seed.service
           671ms systemd-fsck-root.service
           597ms alsa-restore.service
           491ms systemd-update-utmp.service
           415ms var-tmp.mount
           414ms tmp.mount
           368ms systemd-journal-flush.service
           357ms sys-kernel-config.mount
           330ms var-lib-dhcpcd5.mount
           298ms [email protected]\x2dpartuuid-6c586e13\x2d01.service
           280ms console-setup.service
           276ms nfs-config.service

Code: Select all

          13.987s dhcpcd.service
          9.364s dev-mmcblk0p2.device
          4.289s networking.service
          3.678s keyboard-setup.service
          3.597s systemd-logind.service
          3.294s systemd-udev-trigger.service
          2.827s [[[[---this is my controller---]]].service     <-------
          2.607s wpa_supplicant.service
          2.385s systemd-journald.service
          2.089s rng-tools.service
          1.958s systemd-modules-load.service
          1.827s avahi-daemon.service
          1.644s busybox-syslogd.service
          1.471s systemd-remount-fs.service
          1.391s rpi-eeprom-update.service
          1.064s systemd-tmpfiles-setup.service
          1.056s fake-hwclock.service
           950ms kmod-static-nodes.service
           932ms triggerhappy.service
           930ms run-rpc_pipefs.mount
           842ms dev-mqueue.mount
           838ms alsa-restore.service
           820ms raspi-config.service
           775ms ssh.service
           722ms systemd-fsck-root.service
Any ideas to get this to boot faster - specifically the "time till USB Gadget" of my MIDI device init?
I was also thinking to try OTG mode on Raspberry PI 4 - it should boot a lot faster... maybe.

Otherwise, i might move over to Teensy 4.0, arduinos boot very fast. I'd like to stay with raspberry pi if possible...

Can you think of ways to troubleshoot or diagnose this?

DarkElvenAngel
Posts: 234
Joined: Tue Mar 20, 2018 9:53 pm

Re: Help me Fast Boot my pi-Zero MIDI "USB Gadget"

Fri Jan 17, 2020 2:42 am

An initramfs will do what you think, however it's not a trivial task to get going.

I suggest you try looking into Buildroot and build your own Linux, you can get some great speeds out of it.

You have to way out what you need when doing either method. You could just use an initramfs to load the OTG drivers and then the rest of the system would come up more slowly.

If you have a short list of what software you need for your project please post it I'll be able to suggest something better from that and point you to some information.

Goodluck with your project

subatomicglue
Posts: 26
Joined: Sun Oct 13, 2019 5:58 pm

Re: Help me Fast Boot my pi-Zero MIDI "USB Gadget"

Fri Jan 17, 2020 7:23 am

Hi,

Once my USB MIDI gadget is registered I would load a c++ custom app I wrote based on RtMidi. I would also have a LCD hat on, the 16x2 Adafruit one with i2C And buttons. Pretty simple. Low dependencies. Headless. No internet except for development purposes

Loading the rest of system later sounds safe. I just need to figure out how to accomplish that
Would "loading OTG drivers" early, simply be to run modprobe dwc2 and libcomposite? Or is there more... not sure if those 2 depend on other systems to be up

Thank you for the ideas, and any config help/pointers you can

DarkElvenAngel
Posts: 234
Joined: Tue Mar 20, 2018 9:53 pm

Re: Help me Fast Boot my pi-Zero MIDI "USB Gadget"

Fri Jan 17, 2020 10:12 pm

subatomicglue wrote: Loading the rest of system later sounds safe. I just need to figure out how to accomplish that
Would "loading OTG drivers" early, simply be to run modprobe dwc2 and libcomposite? Or is there more... not sure if those 2 depend on other systems to be up
So using an initramfs to accomplish this should work. First you need to know what modules you need the easiest way to do this is to boot up without OTG enable and look at what is loaded then to boot with everything you need enabled and list the modules and note each difference.

Then you add them to the end of /etc/initramfs-tools/modules

Generate initramfs

Code: Select all

update-initramfs -c -k `uname -r`
Then all you need to do is add the initramfs line to your /boot/config.txt

Code: Select all

 initramfs  initrd.img-4.19.58+
you will need to make sure that you use the correct filename this is just an example I'm running from Stretch on a Pi Zero W

Try that out and see if you get the speed your after. This method has advantages and drawbacks. Doing any kernel updates tends to break initramfs especially when dealing with modules. The kernel numbers change and the update will try to make a new initramfs without deleting the old versions and doesn't update the config.txt



For clarity I gather these steps from here
https://www.raspberrypi.org/forums/view ... p?t=104108 and modified the steps for your needs, basically just loading the modules. This also gives you an idea of what to look for.

subatomicglue
Posts: 26
Joined: Sun Oct 13, 2019 5:58 pm

Re: Help me Fast Boot my pi-Zero MIDI "USB Gadget"

Sat Jan 18, 2020 1:32 am

Thank you, I will try this this soon.
boot up without OTG enable and look at what is loaded then to boot with everything you need enabled and list the modules and note each difference.
Am I listing just modules here? I know to list services.., I’ll look around for how to list modules

DarkElvenAngel
Posts: 234
Joined: Tue Mar 20, 2018 9:53 pm

Re: Help me Fast Boot my pi-Zero MIDI "USB Gadget"

Sat Jan 18, 2020 3:39 am

subatomicglue wrote:
Sat Jan 18, 2020 1:32 am
Thank you, I will try this this soon.
boot up without OTG enable and look at what is loaded then to boot with everything you need enabled and list the modules and note each difference.
Am I listing just modules here? I know to list services.., I’ll look around for how to list modules
Yes the goal is to find out which modules are loaded for your needs and have them loaded in the initramfs. The command

Code: Select all

lsmod
Should provide insight here.

subatomicglue
Posts: 26
Joined: Sun Oct 13, 2019 5:58 pm

Re: Help me Fast Boot my pi-Zero MIDI "USB Gadget"

Sat Jan 18, 2020 11:31 pm

Raspberry PI 4 gives me 13 seconds "time till midi" using a 64gb, Class10, v30 A2 UHS-1/U3 microsd card...

(working on initrd next)

subatomicglue
Posts: 26
Joined: Sun Oct 13, 2019 5:58 pm

Re: Help me Fast Boot my pi-Zero MIDI "USB Gadget"

Sun Jan 19, 2020 1:00 am

lsmod from a fresh install of Buster on Raspberry Pi Zero (ssh enabled so i could get in via ethernet USB dongle, I have no HDMI mini display connector)

Code: Select all

cat ./modules-clean-zero.txt

Module                  Size  Used by
sha256_generic         20480  0
cfg80211              610304  0
rfkill                 28672  2 cfg80211
8021q                  32768  0
garp                   16384  1 8021q
stp                    16384  1 garp
llc                    16384  2 garp,stp
ax88179_178a           24576  0
snd_bcm2835            24576  1
raspberrypi_hwmon      16384  0
hwmon                  16384  1 raspberrypi_hwmon
snd_pcm                98304  1 snd_bcm2835
bcm2835_codec          36864  0
bcm2835_v4l2           45056  0
v4l2_mem2mem           24576  1 bcm2835_codec
snd_timer              32768  1 snd_pcm
bcm2835_mmal_vchiq     32768  2 bcm2835_codec,bcm2835_v4l2
v4l2_common            16384  1 bcm2835_v4l2
videobuf2_vmalloc      16384  1 bcm2835_v4l2
videobuf2_dma_contig    20480  1 bcm2835_codec
snd                    73728  5 snd_timer,snd_bcm2835,snd_pcm
videobuf2_memops       16384  2 videobuf2_dma_contig,videobuf2_vmalloc
videobuf2_v4l2         24576  3 bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem
videobuf2_common       45056  4 bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem,videobuf2_v4l2
videodev              192512  6 bcm2835_codec,v4l2_common,videobuf2_common,bcm2835_v4l2,v4l2_mem2mem,videobuf2_v4l2
vc_sm_cma              36864  1 bcm2835_mmal_vchiq
media                  36864  3 bcm2835_codec,videodev,v4l2_mem2mem
fixed                  16384  0
uio_pdrv_genirq        16384  0
uio                    20480  1 uio_pdrv_genirq
ip_tables              24576  0
x_tables               32768  1 ip_tables
ipv6                  446464  20
lsmod from a OTG USB MIDI+Ether Gadget on Buster, on Raspberry Pi Zero

Code: Select all

$ cat modules-midi-zero.txt
Module                  Size  Used by
sha256_generic         20480  0
usb_f_midi             20480  2
snd_rawmidi            32768  1 usb_f_midi
snd_seq_device         16384  1 snd_rawmidi
cfg80211              610304  0
rfkill                 28672  2 cfg80211
usb_f_ecm              16384  2
8021q                  32768  0
u_ether                20480  1 usb_f_ecm
garp                   16384  1 8021q
stp                    16384  1 garp
llc                    16384  2 garp,stp
libcomposite           61440  10 usb_f_ecm,usb_f_midi
ax88179_178a           24576  0
snd_bcm2835            24576  1
snd_pcm                98304  1 snd_bcm2835
raspberrypi_hwmon      16384  0
snd_timer              32768  1 snd_pcm
hwmon                  16384  1 raspberrypi_hwmon
snd                    73728  8 snd_timer,snd_rawmidi,snd_seq_device,snd_bcm2835,usb_f_midi,snd_pcm
bcm2835_codec          36864  0
bcm2835_v4l2           45056  0
v4l2_mem2mem           24576  1 bcm2835_codec
bcm2835_mmal_vchiq     32768  2 bcm2835_codec,bcm2835_v4l2
v4l2_common            16384  1 bcm2835_v4l2
videobuf2_vmalloc      16384  1 bcm2835_v4l2
videobuf2_dma_contig    20480  1 bcm2835_codec
videobuf2_memops       16384  2 videobuf2_dma_contig,videobuf2_vmalloc
videobuf2_v4l2         24576  3 bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem
videobuf2_common       45056  4 bcm2835_codec,bcm2835_v4l2,v4l2_mem2mem,videobuf2_v4l2
videodev              192512  6 bcm2835_codec,v4l2_common,videobuf2_common,bcm2835_v4l2,v4l2_mem2mem,videobuf2_v4l2
media                  36864  3 bcm2835_codec,videodev,v4l2_mem2mem
vc_sm_cma              36864  1 bcm2835_mmal_vchiq
uio_pdrv_genirq        16384  0
uio                    20480  1 uio_pdrv_genirq
fixed                  16384  0
dwc2                  167936  0
udc_core               49152  5 usb_f_ecm,dwc2,u_ether,libcomposite,usb_f_midi
ip_tables              24576  0
x_tables               32768  1 ip_tables
ipv6                  446464  26
The differences here are:

Code: Select all

usb_f_midi, snd_rawmidi, snd_seq_device, usb_f_ecm, u_ether, libcomposite, dwc2, udc_core

subatomicglue
Posts: 26
Joined: Sun Oct 13, 2019 5:58 pm

Re: Help me Fast Boot my pi-Zero MIDI "USB Gadget"

Sun Jan 19, 2020 2:05 am

Alright, "time to midi" on the rPI Zero is now 34seconds (got slower, was 28).
This is what I did:

1. lsmod running under a clean install (see previous post)
2. Install my OTG MIDI+Ether config
3. lsmod on a full install (e.g. with OTG MIDI+Ether and readonly FS) (see previous post)

4. Add the difference to /etc/initramfs-tools/modules:

Code: Select all

$ sudo vi /etc/initramfs-tools/modules

# add these:
dwc2
usb_f_midi
usb_f_ecm
libcomposite
snd_rawmidi
snd_seq_device
u_ether
udc_core
5. Enable initrd:

Code: Select all

$ sudo apt-get install initramfs-tools
$ sudo update-initramfs -c -k `uname -r`
     update-initramfs: Generating /boot/initrd.img-4.19.75+

# Add to end of /boot/config.txt:
$ initramfs initrd.img-4.19.75+
6. Reboot, and use stopwatch to get the new "time to midi"

Ideas why it got slower? How to diagnose/troubleshoot this... ? See anything I did wrong?

Return to “Advanced users”