User avatar
graysky
Posts: 176
Joined: Fri Apr 05, 2013 11:43 am
Location: /run/user/1000
Contact: Website

Optimal distcc settings when compiling an RPi

Sun Aug 01, 2021 10:27 am

Has anyone looked at the impact of different distcc configurations on compilation time when using a RPi4 as the client and a much more powerful x86_64 machine as the volunteer? Wondering what is most efficient, exporting more threads to the x86_64 box, taking on some locally, etc.

User avatar
DougieLawson
Posts: 41687
Joined: Sun Jun 16, 2013 11:19 pm
Location: A small cave in deepest darkest Basingstoke, UK
Contact: Website Twitter

Re: Optimal distcc settings when compiling an RPi

Sun Aug 01, 2021 12:39 pm

Why not ask the Box86 developer(s) by opening an issue at https://github.com/ptitSeb/box86/issues?

They appear to be optimising for ArmV7 in https://github.com/ptitSeb/box86/blob/m ... aspberrypi
They should be using -mcpu=cortex-a72 -mfloat-abi=hard -mfpu=neon-fp-armv8 -mneon-for-64bits -mtune=cortex-a72 to optimise for an RPi4.
Languages using left-hand whitespace for syntax are ridiculous

DMs sent on Twitter/LinkedIn will be answered next month.
Fake doctors - are all on my foes list.

The use of crystal balls and mind reading is prohibited.

User avatar
graysky
Posts: 176
Joined: Fri Apr 05, 2013 11:43 am
Location: /run/user/1000
Contact: Website

Re: Optimal distcc settings when compiling an RPi

Sun Aug 01, 2021 2:54 pm

I was thinking more about how to setup the build itself. MAKEFLAGS and DISTCC_HOSTS. I am experimenting with some settings and will share the results when finished, just wanted to see if anyone else had any experiences.

ejolson
Posts: 8118
Joined: Tue Mar 18, 2014 11:47 am

Re: Optimal distcc settings when compiling an RPi

Sun Aug 01, 2021 3:18 pm

graysky wrote:
Sun Aug 01, 2021 2:54 pm
I was thinking more about how to setup the build itself. MAKEFLAGS and DISTCC_HOSTS. I am experimenting with some settings and will share the results when finished, just wanted to see if anyone else had any experiences.
I'm looking forward to the results. From what I understand you have 4 cores on the Pi and n cores each with 2-way symmetric multithreading on the x86 based system. Intuitively, it would seem that maxing out the cores and threads should lead to the fastest build times but maybe not.

I think the ideal setting would depend on how much of the build process involves non-compilation tasks like GNU autoconfig scripts. Thus, another plausible setup would be to do all compilation on the x86 system and reserve the cores on the Pi for everything else.

User avatar
graysky
Posts: 176
Joined: Fri Apr 05, 2013 11:43 am
Location: /run/user/1000
Contact: Website

Re: Optimal distcc settings when compiling an RPi

Sun Aug 01, 2021 9:30 pm

Client = RPi4 running Arch ARM aarch64
Volunteer = AMD 5950X running Arch Linux (16 physical cores + 16 hyperthreaded cores).
GCC on both was v11.2.0/distcc was v3.4.

Test was building mplayer 2021-08-01 snapshot.

Tested conditions were the full matrix of:
1) make flags (8, 12, 16, 20, 24, 28, and 32)
2) DISTCC_HOSTS="10.1.10.234:3636/x" where x was (8, 16, 24, and 32)
3) Finally, I tried including 'localhost' in the DISTCC_HOSTS using -j16 and /16

* Each condition (make flag + limit combo) was run nine times and mean vales and standard deviations were compared using a one-way ANOVA analysis at the 0.05 confidence level).
* For comparison, without distcc (just make -j8 on the pi) took about 91 seconds.
* Slowest runs were the ones containing the localhost entry. Its order (either first or last) did not matter. Average values were around 24 secs and these two results were not different from each other by the ANOVA.
* Best results obtained omitting localhost (the pi) from DISTCC_HOSTS and just defining the x86_64 box.
* Most of the results in the matrix combo were not significantly different from each other with the exception of those run with the limit of 8 which were among the slowest. Mean times ranges from 19.7-21.0 sec.
* The next slowest group were those run with -j8 and any limit (16, 24, and 32). Mean times ranged from 19.8-19.9 sec.
* The rest were statistically the same per the ANOVA. Mean times ranged from 19.3-19.6 sec.

The variation in the replicate mean values were such that an ANOVA did not show that they were different from each other at the 0.05 level.

So take home:
* Don't include localhost
* Pick something sane for -j and limits
* I'll probably just dial in -j20 and /24 and call it good

swampdog
Posts: 764
Joined: Fri Dec 04, 2015 11:22 am

Re: Optimal distcc settings when compiling an RPi

Wed Aug 04, 2021 5:19 pm

It depends what you're doing and gets very wishy-washy :-|

When I was building QT, I found "localhost/3" on the end of DISTCC_HOSTS worked best for the rpi4+ssd+swap=4G, essentially leaving one native rpi4 core for linking/responsiveness. distcc was in basic mode.

Solskogen
Posts: 59
Joined: Tue Sep 27, 2016 6:07 am

Re: Optimal distcc settings when compiling an RPi

Wed Aug 04, 2021 6:15 pm

Make sure that you've got /tmp on tmpfs (on the Pi) when using distcc so that it's not using the sd card.

User avatar
graysky
Posts: 176
Joined: Fri Apr 05, 2013 11:43 am
Location: /run/user/1000
Contact: Website

Re: Optimal distcc settings when compiling an RPi

Wed Aug 04, 2021 7:52 pm

Yes, my volunteer machine has plenty of memory which I use on a tmpfs mount exported by NFS for this reason.

User avatar
jahboater
Posts: 7329
Joined: Wed Feb 04, 2015 6:38 pm
Location: Wonderful West Dorset

Re: Optimal distcc settings when compiling an RPi

Wed Aug 04, 2021 9:59 pm

ejolson wrote:
Sun Aug 01, 2021 3:18 pm
I'm looking forward to the results. From what I understand you have 4 cores on the Pi and n cores each with 2-way symmetric multithreading on the x86 based system. Intuitively, it would seem that maxing out the cores and threads should lead to the fastest build times but maybe not.
Yes. Unless limited by memory and swapping starts.
Also, after the cores are all maxed out, having a task or two waiting to take over when one of the running tasks has to block for I/O is considered good.
But then too many tasks means more time wasted context switching!

These days I just do:

Code: Select all

make -j `nproc`

Return to “Advanced users”