hvz
Posts: 7
Joined: Thu Jun 27, 2019 3:06 pm

Newer Raspbian versions a lot slower than older ones? [Solved]

Thu Jun 27, 2019 3:21 pm

I just installed Raspbian Buster to run some performance tests of my own software on my new Pi 4. To my surprise, the performance was nearly identical to that of my older Pi 3 (not even a 3+). So, I put the Buster MicroSD card into my old Pi 3, and it turned out that CPU load of my software doubled compared to what I had been before - the boot screen says "Raspbian 2.1 2017", uname -a "4.9.59-v7+".

I made sure that I am using the same binaries of my software, and the same settings file. The software is computationally heavy, and uses a lot of float calculations, some of them using NEON code.

Oddly, I also had a question from a customer about a week ago who had similar performance issues with Raspbian Stretch - I've sent him a copy of my old image and his performance problems were gone.

vcgencmd get_throttled returns 0x0 in both images.
vcgencmd get_config arm_freq returns 1200 for the Pi 3, 1500 for the Pi 4.

What can this be?


As a side note: Since I can run the Buster image on both Pi's, I have been able to get some performance values using Buster, and that shows a 2.2 time reduction in CPU load on the Pi 4, which looks really good - if I can fix the OS image. I have not yet tested other images such as Ubuntu Mate etc.
Last edited by hvz on Fri Jun 28, 2019 9:39 pm, edited 1 time in total.

ejolson
Posts: 3382
Joined: Tue Mar 18, 2014 11:47 am

Re: Newer Raspbian versions a lot slower than older ones?

Fri Jun 28, 2019 8:34 am

hvz wrote:
Thu Jun 27, 2019 3:21 pm
I just installed Raspbian Buster to run some performance tests of my own software on my new Pi 4. To my surprise, the performance was nearly identical to that of my older Pi 3 (not even a 3+). So, I put the Buster MicroSD card into my old Pi 3, and it turned out that CPU load of my software doubled compared to what I had been before - the boot screen says "Raspbian 2.1 2017", uname -a "4.9.59-v7+".

I made sure that I am using the same binaries of my software, and the same settings file. The software is computationally heavy, and uses a lot of float calculations, some of them using NEON code.

Oddly, I also had a question from a customer about a week ago who had similar performance issues with Raspbian Stretch - I've sent him a copy of my old image and his performance problems were gone.

vcgencmd get_throttled returns 0x0 in both images.
vcgencmd get_config arm_freq returns 1200 for the Pi 3, 1500 for the Pi 4.

What can this be?


As a side note: Since I can run the Buster image on both Pi's, I have been able to get some performance values using Buster, and that shows a 2.2 time reduction in CPU load on the Pi 4, which looks really good - if I can fix the OS image. I have not yet tested other images such as Ubuntu Mate etc.
Does Raspian Buster on the Pi 3B+ use an LPAE kernel?

I've been trying to understand why my merge sort routine runs slower on the Pi 4B here. This may be related to your observation. Any help would be appreciated.

hvz
Posts: 7
Joined: Thu Jun 27, 2019 3:06 pm

Re: Newer Raspbian versions a lot slower than older ones?

Fri Jun 28, 2019 8:58 am

I'll have to look that up. In my code I'm not using a lot of thread synchronization, so I don't think that it's related.

In your case, you really need to start by ruling out the effect of gcc, by using the same binary. Assuming that you compile with an older GLIBC on the Pi 3B+ you can probably just copy the resulting binary to the Pi 4.

Having said that, your results (the Pi 4 is about the same speed as the Pi 3B+) seems to match mine. Which OS are you using on the Pi 3B+?

goodburner
Posts: 41
Joined: Sun Jun 16, 2019 3:20 am

Re: Newer Raspbian versions a lot slower than older ones?

Fri Jun 28, 2019 9:12 am

I can concur that I'm not seeing anywhere near the "3x performance" that was advertised. It's barely faster than 3b+.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 23309
Joined: Sat Jul 30, 2011 7:41 pm

Re: Newer Raspbian versions a lot slower than older ones?

Fri Jun 28, 2019 9:18 am

goodburner wrote:
Fri Jun 28, 2019 9:12 am
I can concur that I'm not seeing anywhere near the "3x performance" that was advertised. It's barely faster than 3b+.
Most benchmarks are showing a CPU speed up of x2, the wired networking is about x4 faster, USB3 access is about x10 faster in my tests. Average all those out in general usage and you get about x4 faster overall. The memory is what makes all the difference for web browsing, which was always memory rather than CPU or I/O bound.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

hvz
Posts: 7
Joined: Thu Jun 27, 2019 3:06 pm

Re: Newer Raspbian versions a lot slower than older ones?

Fri Jun 28, 2019 9:49 am

I just ran Pichart (openmp) on both OS's on the Pi 3B, and the performance is nearly identical (within 5%). I used the same binary for both, built on the old image (gcc 6.3). So it looks like I'm doing *something* that has become very heavy in the new OS, which isn't covered by these tests.

Edit: I just performed the same test on the Pi4, and it's really a lot faster.

Numbers: Linux2017pi3 / Linux2019pi3 / Linux2019pi4
Prime Sieve: 485 / 483 / 1411
Merge Sort: 301 / 301 / 540
Fourier: 157 / 149 / 258
Lorenz 96: 907 / 905 / 4610

So the Pi 4 is clearly a _lot_ faster in this test.

ejolson
Posts: 3382
Joined: Tue Mar 18, 2014 11:47 am

Re: Newer Raspbian versions a lot slower than older ones?

Fri Jun 28, 2019 9:37 pm

hvz wrote:
Fri Jun 28, 2019 9:49 am
So it looks like I'm doing *something* that has become very heavy in the new OS, which isn't covered by these tests.
The Cortex-A72 cores used in the Pi 4B are subject to Spectre-style side-channel information leakage while the A53 cores in the 3B+ are not. As a result, the default Rasbian Buster kernel may now include mitigation measures that make system calls much more expensive. It would be interesting to know for sure and whether that is the cause of the slowdown you are observing.

hvz
Posts: 7
Joined: Thu Jun 27, 2019 3:06 pm

Re: Newer Raspbian versions a lot slower than older ones? [Solved]

Fri Jun 28, 2019 9:39 pm

Ok, I found out what was wrong. I measured the old and new image speeds by generating, processing and then sending audio to the built-in sound card of the Pi. But I hadn't checked the audio that came out. I didn't think I needed to, since I was using ALSA with big enough buffers.

Anyway, when I plugged in some HifiBerry cards and selected those as output, suddenly the performance of both Raspbian images was the same. Unfortunately it's the same as the slower of the two - but at least it's now clear what's going on.

This means that my code runs pretty good on a Pi 4, but not so much on a Pi 3B. So I will have some more work to do...

Return to “Troubleshooting”