hippyau
Posts: 6
Joined: Sun Aug 01, 2021 11:11 am

CM4 very slow from power-on to kernel start time

Sun Aug 01, 2021 11:31 am

Hi all,

It seems to take me 7 seconds to get from power applied to loading from eMMC and staring the kernel, where in the CM3 this can be much, much faster < 2 seconds. This is really slow for a user appliance, looking to build a product around the CM4.

How can this be improved? I have understanding now the SDRAM controller takes ~4 seconds to initialize and I suppose 3 seconds to read my config.txt, do some overlay stuff and jump to the kernel. Is there some way around this by firmware enhancement, or is this a physical limitation of the hardware?

There is a github issue here... https://github.com/raspberrypi/firmware ... -603565098

Surely it can be faster than this?

Cheers,
Hippyau

cleverca22
Posts: 4366
Joined: Sat Aug 18, 2012 2:33 pm

Re: CM4 very slow from power-on to kernel start time

Sun Aug 01, 2021 4:19 pm

a few things of note to help with the debug

1:
if you set BOOT_UART=1 in the bootconf.txt of the SPI flash, the bootloader will print logs to the serial port (gpio 14/15) at 115200 baud
the logs dont include any timestamps, so youll need a terminal client that timestamps things for you

2:
if you set uart_2ndstage=1 in /boot/config.txt, then start4.elf will also print its own debug logs to the same serial port

Code: Select all

MESS:00:00:01.409159:0: brfs: File read: /mfs/sd/config.txt
MESS:00:00:01.413989:0: brfs: File read: 1810 bytes
the logs will look like this, and the timestamps are relative to when the cpu came out of reset (shortly before the SPI bootloader)

3:
https://github.com/librerpi/rpi-tools/b ... boottime.c
if you compile and run this on the pi, it will print 3 numbers
the 1st is the soc uptime (what start4.elf uses for its timestamps)
the 2nd is the linux uptime (starts counting when linux gained control of the arm core)
the 3rd is the difference between the 2, which is how long it took from soc exiting reset, to linux gaining control of the arm core

by running the boottime program after making changes, you can use that 3rd number to quantify exactly what effect your changes had


some thoughts on what adds the boot times
1: the bootrom must load a bootcode.bin from SPI flash (the bootloader), and execute it
2: the bootloader must load several other files from SPI flash, and initialize the ddr4 controller
3: the bootloader must load start4.elf from the configured boot device
- if there are extra entries in BOOT_ORDER, it may take time to fail and loook in the right place

Code: Select all

[clever@amd-nixos:~/apps/rpi/firmware/boot]$ ls -lh start*elf
-rw-r--r-- 1 clever users 778K Jun  6 14:12 start4cd.elf
-rw-r--r-- 1 clever users 3.6M Jun  6 14:12 start4db.elf
-rw-r--r-- 1 clever users 2.2M Jun  6 14:12 start4.elf
-rw-r--r-- 1 clever users 2.9M Jun  6 14:12 start4x.elf
- the "cut down" variant(start4cd) is less then half the size of the default (start4), having a smaller file to read from /boot may make loading faster
- the "cut down" variant may also start less background services, and boot faster
i think gpu_mem=16 makes the bootloader use the cut-down version automatically

PhilE
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 4062
Joined: Mon Sep 29, 2014 1:07 pm
Location: Cambridge

Re: CM4 very slow from power-on to kernel start time

Sun Aug 01, 2021 4:51 pm

I managed to shave almost a second off the boot time recently, (so grab the latest firmware from our firmware repo) but I can't see it getting much better than that - the memory PHY setup is lengthy.

cleverca22
Posts: 4366
Joined: Sat Aug 18, 2012 2:33 pm

Re: CM4 very slow from power-on to kernel start time

Sun Aug 01, 2021 11:06 pm

PhilE wrote:
Sun Aug 01, 2021 4:51 pm
the memory PHY setup is lengthy.
the recovery.bin recently gained a ram test function, and a new blink error code for bad ram
the code for that ram setup seems far simpler, are you able to reveal what its missing and why its so much simpler?

timg236
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 897
Joined: Thu Jun 21, 2018 4:30 pm

Re: CM4 very slow from power-on to kernel start time

Mon Aug 02, 2021 9:08 am

PhilE wrote:
Sun Aug 01, 2021 4:51 pm
I managed to shave almost a second off the boot time recently, (so grab the latest firmware from our firmware repo) but I can't see it getting much better than that - the memory PHY setup is lengthy.
Indeed. The RAM setup won't get any faster.

cleverca22
Posts: 4366
Joined: Sat Aug 18, 2012 2:33 pm

Re: CM4 very slow from power-on to kernel start time

Mon Aug 02, 2021 9:39 am

timg236 wrote:
Mon Aug 02, 2021 9:08 am
PhilE wrote:
Sun Aug 01, 2021 4:51 pm
I managed to shave almost a second off the boot time recently, (so grab the latest firmware from our firmware repo) but I can't see it getting much better than that - the memory PHY setup is lengthy.
Indeed. The RAM setup won't get any faster.
ive heard that x86 systems push it faster, by saving the timing offsets to flash, and then using those to skip some steps on startup

no chance of the bcm2711 doing the same thing?

my rough understanding, is that the phy needs each data/addr line fine-tuned to compensate for minor timing differences in trace length, and variations within the production process


oh, what if you just underclock the ram, then you can be more sloppy with the timing, and entirely skip the fine-tuning? some users may not need the ram running at peak performance!

timg236
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 897
Joined: Thu Jun 21, 2018 4:30 pm

Re: CM4 very slow from power-on to kernel start time

Mon Aug 02, 2021 10:17 am

cleverca22 wrote:
Mon Aug 02, 2021 9:39 am
timg236 wrote:
Mon Aug 02, 2021 9:08 am
PhilE wrote:
Sun Aug 01, 2021 4:51 pm
I managed to shave almost a second off the boot time recently, (so grab the latest firmware from our firmware repo) but I can't see it getting much better than that - the memory PHY setup is lengthy.
Indeed. The RAM setup won't get any faster.
ive heard that x86 systems push it faster, by saving the timing offsets to flash, and then using those to skip some steps on startup

no chance of the bcm2711 doing the same thing?
No chance.

timg236
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 897
Joined: Thu Jun 21, 2018 4:30 pm

Re: CM4 very slow from power-on to kernel start time

Mon Aug 02, 2021 10:24 am

timg236 wrote:
Mon Aug 02, 2021 9:08 am
PhilE wrote:
Sun Aug 01, 2021 4:51 pm
I managed to shave almost a second off the boot time recently, (so grab the latest firmware from our firmware repo) but I can't see it getting much better than that - the memory PHY setup is lengthy.
Indeed. The RAM setup won't get any faster.
Edit: With the latest bootloader the time from power on the first SD access from start4.elf (look at vcdbg log msg) is about 3.5 seconds. I think it's unlikely that that will change much. The rest of the time for kernel loading is more dependent on the IO speed of the boot device, kernel size etc

hippyau
Posts: 6
Joined: Sun Aug 01, 2021 11:11 am

Re: CM4 very slow from power-on to kernel start time

Mon Aug 02, 2021 1:34 pm

Thanks so much for the quick responses! And to be in touch with the Engineers also!

As this appliance rather industrial and needs to boot as quickly as possible, but also give the appearance to be alive (via HDMI) within ~3s (user acceptable before they start hitting it), I would love to hear your thoughts.

I'm really genuinely surprised by the delay, having experienced getting Raspberry Pi's up in like 2~3 second. In case anyone else is considering, we started development in Raspbian, designed boards, to then just moved to a super minimal buildroot solution to discover the huge delays where previous Raspberry Pi CM3 smashed it.

If there is no provision to store the SDRAM timings etc, then can we maybe get a user splash screen displayed in the firmware? So we are not waiting forever to see our product is actually appearing alive?

One other question I have is about appending the DTB to the kernel, is that support on CM4? Will that actually help, if the kernel can deal with loading it vs. firmware loading it? How would that go with overlays?

Thanks again,
Hippy

trejan
Posts: 3593
Joined: Tue Jul 02, 2019 2:28 pm

Re: CM4 very slow from power-on to kernel start time

Mon Aug 02, 2021 1:46 pm

hippyau wrote:
Mon Aug 02, 2021 1:34 pm
If there is no provision to store the SDRAM timings etc, then can we maybe get a user splash screen displayed in the firmware? So we are not waiting forever to see our product is actually appearing alive?
If caching the SDRAM timing isn't going to happen then a splash screen is completely out. The firmware would need a major rewrite to initialise all the needed peripherals before even starting the RAM training process.

hippyau
Posts: 6
Joined: Sun Aug 01, 2021 11:11 am

Re: CM4 very slow from power-on to kernel start time

Mon Aug 02, 2021 2:01 pm

trejan wrote:
Mon Aug 02, 2021 1:46 pm
>> a splash screen is completely out.
Yeah but "It is designed for deeply embedded and industrial applications.". My industrial embedded customers, like many, would not wait 14 seconds.

hippyau
Posts: 6
Joined: Sun Aug 01, 2021 11:11 am

Re: CM4 very slow from power-on to kernel start time

Mon Aug 02, 2021 2:13 pm

Hi All,

Is it possible that the SDRAM controller is timing out while waiting for a non existent EEPROM or something to give it the timing values, and then either using values from firmware or doing some testing to arrive at a working set of timing?

"Raspberry Pi Compute Module 4 succeeds the Compute Module 3 and 3+ from 2017 and 2019, respectively."

Cheers,
Hippyg
Last edited by hippyau on Mon Aug 02, 2021 2:29 pm, edited 1 time in total.

timg236
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 897
Joined: Thu Jun 21, 2018 4:30 pm

Re: CM4 very slow from power-on to kernel start time

Mon Aug 02, 2021 2:14 pm

hippyau wrote:
Mon Aug 02, 2021 2:01 pm
trejan wrote:
Mon Aug 02, 2021 1:46 pm
>> a splash screen is completely out.
Yeah but "It is designed for deeply embedded and industrial applications.". My industrial embedded customers, like many, would not wait 14 seconds.
The rainbow splash screen is normally started at about 5 seconds after PON with the Linux kernel running after about 8 seconds.

timg236
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 897
Joined: Thu Jun 21, 2018 4:30 pm

Re: CM4 very slow from power-on to kernel start time

Mon Aug 02, 2021 2:14 pm

hippyau wrote:
Mon Aug 02, 2021 2:13 pm
Hi All,

Is it possible that the SDRAM controller is timing out while waiting for a non existent EEPROM or something to give it the timing values, and then either using values from firmware or doing some testing to arrive at a working set of timing?

Cheers,
Hippyg
Nope

hippyau
Posts: 6
Joined: Sun Aug 01, 2021 11:11 am

Re: CM4 very slow from power-on to kernel start time

Mon Aug 02, 2021 2:19 pm

timg236 wrote:
Mon Aug 02, 2021 2:14 pm
The rainbow splash screen is normally started at about 5 seconds after PON with the Linux kernel running after about 8 seconds.
Thanks very much for the reply! So there is the possibility we can have a custom splash in 5 seconds?

timg236
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 897
Joined: Thu Jun 21, 2018 4:30 pm

Re: CM4 very slow from power-on to kernel start time

Mon Aug 02, 2021 2:31 pm

hippyau wrote:
Mon Aug 02, 2021 2:19 pm
timg236 wrote:
Mon Aug 02, 2021 2:14 pm
The rainbow splash screen is normally started at about 5 seconds after PON with the Linux kernel running after about 8 seconds.
Thanks very much for the reply! So there is the possibility we can have a custom splash in 5 seconds?
IIRC loading a small custom bitmap is relatively easy, however, the problem has always been that within a couple of seconds the Linux kernel will have started and the firmware display driver reset. That combined with resolution switches (simplefb / KMS) means that on many monitors the display isn't visible at all. Maybe it's something that can be looked into once the dust has settled for KMS.

hippyau
Posts: 6
Joined: Sun Aug 01, 2021 11:11 am

Re: CM4 very slow from power-on to kernel start time

Mon Aug 02, 2021 2:44 pm

timg236 wrote:
Mon Aug 02, 2021 2:31 pm
hippyau wrote:
Mon Aug 02, 2021 2:19 pm
timg236 wrote:
Mon Aug 02, 2021 2:14 pm
The rainbow splash screen is normally started at about 5 seconds after PON with the Linux kernel running after about 8 seconds.
Thanks very much for the reply! So there is the possibility we can have a custom splash in 5 seconds?
IIRC loading a small custom bitmap is relatively easy, however, the problem has always been that within a couple of seconds the Linux kernel will have started and the firmware display driver reset. That combined with resolution switches (simplefb / KMS) means that on many monitors the display isn't visible at all. Maybe it's something that can be looked into once the dust has settled for KMS.
Indeed! The resolution switches would be a problem... at least something would be on the screen albeit briefly, in addition I wonder if u-boot (added boot time) would get something up..... https://developer.toradex.com/knowledge ... ash-screen (unrelated to my project)

timg236
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 897
Joined: Thu Jun 21, 2018 4:30 pm

Re: CM4 very slow from power-on to kernel start time

Mon Aug 02, 2021 4:10 pm

hippyau wrote:
Mon Aug 02, 2021 2:44 pm
timg236 wrote:
Mon Aug 02, 2021 2:31 pm
hippyau wrote:
Mon Aug 02, 2021 2:19 pm


Thanks very much for the reply! So there is the possibility we can have a custom splash in 5 seconds?
IIRC loading a small custom bitmap is relatively easy, however, the problem has always been that within a couple of seconds the Linux kernel will have started and the firmware display driver reset. That combined with resolution switches (simplefb / KMS) means that on many monitors the display isn't visible at all. Maybe it's something that can be looked into once the dust has settled for KMS.
Indeed! The resolution switches would be a problem... at least something would be on the screen albeit briefly, in addition I wonder if u-boot (added boot time) would get something up..... https://developer.toradex.com/knowledge ... ash-screen (unrelated to my project)
In theory, there's nothing to stop a minimal frame-buffer / splash screen HVS + HDMI driver from being added to u-boot now that the display drivers are all open source in KMS.
In the long term, I think the bootloader will probably create a single buffered frame buffer at some HD resolution and leave the hardware running (no FKMS / dispmanx) but that's a few steps away. The ARM stage can inherit that or poke a mailbox to reset everything.

cleverca22
Posts: 4366
Joined: Sat Aug 18, 2012 2:33 pm

Re: CM4 very slow from power-on to kernel start time

Mon Aug 02, 2021 10:58 pm

one thing ive thought about

uboot could configure the mailbox framebuffer, for its own logs/splash
but then pass the address of that framebuffer to linux as a simple framebuffer that is tied to the (f)kms nodes in DT

linux will use the simple-framebuffer initially, until it finds the kms modules on the rootfs, then switch over

having a pre-configured simple-framebuffer, would allow linux to do non-accelerated gfx much sooner in its boot process, and not require baking the kms modules intothe kernel, which would have slowed down kernel loading

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 11634
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: CM4 very slow from power-on to kernel start time

Tue Aug 03, 2021 6:18 am

cleverca22 wrote:
Mon Aug 02, 2021 10:58 pm
one thing ive thought about

uboot could configure the mailbox framebuffer, for its own logs/splash
but then pass the address of that framebuffer to linux as a simple framebuffer that is tied to the (f)kms nodes in DT

linux will use the simple-framebuffer initially, until it finds the kms modules on the rootfs, then switch over

having a pre-configured simple-framebuffer, would allow linux to do non-accelerated gfx much sooner in its boot process, and not require baking the kms modules intothe kernel, which would have slowed down kernel loading
The firmware already configures a simple-framebuffer node if FKMS or KMS are enabled.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

cleverca22
Posts: 4366
Joined: Sat Aug 18, 2012 2:33 pm

Re: CM4 very slow from power-on to kernel start time

Tue Aug 03, 2021 7:11 am

6by9 wrote:
Tue Aug 03, 2021 6:18 am
The firmware already configures a simple-framebuffer node if FKMS or KMS are enabled.
ah, perfect!, i need to take a closer look at how it all behaves on the latest raspi-os

Return to “Compute Module”