dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5500
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

DVFS Firmware

Thu Nov 21, 2019 2:25 pm

Latest rpi-update firmware includes DVFS.

Normally on a Pi4 we pick a suitable voltage (based on silicon characteristics) on boot, and stick with it whether busy or idle.
But if all clocks in the system (e.g. arm/core/v3d/isp/h264/hevc) are below maximum you can reduce the voltage and get a saving in power and temperature. The new firmware implements this.

The cpufreq driver now has multiple frequency points (1500, 1000, 750, 600), so moderate loads (e.g. video playback in browser) may choose 1000MHz (bumping to 1500MHz whenever we become busier) which allows a lower voltage and so a lower temperature.

Also when we start to throttle, the voltage can be reduced at 1000MHz allowing the temperature to drop more quickly, and so hitting 600MHz when throttling is rarer.

We'd like to get this into the stable firmware, but it needs testing first (and we've scared away many of the rpi-update testers...)

So, volunteers are wanted. Simplest test is just to rpi-update and let us know if anything becomes less stable or performant.
If you want to look more closely, bcmstat is very useful. I run it with a config of

Code: Select all

MCxgpd1 Tye
(in ~/.bcmstat.conf) so I can see frequencies, temperature and voltage under different loads.

User avatar
rpdom
Posts: 16962
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: DVFS Firmware

Thu Nov 21, 2019 3:57 pm

I'm giving it a try. I don't stress my Pi 4B much, but I'll see how it holds up anyway.
Unreadable squiggle

jahboater
Posts: 5626
Joined: Wed Feb 04, 2015 6:38 pm
Location: West Dorset

Re: DVFS Firmware

Thu Nov 21, 2019 5:02 pm

Works fine for me (64-bit kernel), light usage (code compilations and tests).

Some measurements:
The idle voltage has dropped down to 0.81 volts with a 3-4 degree drop in temps.
Idle ...

Code: Select all

$ ./pistat.sh 
Time       Temp     CPU     Core         Health           Vcore
16:29:31  35.0'C   600MHz  200MHz  00000000000000000000  0.8100V
16:29:51  36.0'C   600MHz  200MHz  00000000000000000000  0.8100V
Running cpuburn stress test ...

Code: Select all

Time       Temp     CPU     Core         Health           Vcore
16:49:46  65.0'C  1500MHz  500MHz  00000000000000000000  0.8438V
16:50:06  65.0'C  1500MHz  500MHz  00000000000000000000  0.8438V
Remarkable!
This is with no fan and not in case.

jdb
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 2336
Joined: Thu Jul 11, 2013 2:37 pm

Re: DVFS Firmware

Thu Nov 21, 2019 5:10 pm

rpdom wrote:
Thu Nov 21, 2019 3:57 pm
I'm giving it a try. I don't stress my Pi 4B much, but I'll see how it holds up anyway.
Partially-busy workloads are likely to see the most benefit - the intermediate performance point at 1GHz doesn't need as much volts as 1.5GHz, so "busy" temperatures should be closer to idle temperatures (and idle temperatures get a reduction).
Rockets are loud.
https://astro-pi.org

User avatar
rpdom
Posts: 16962
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: DVFS Firmware

Thu Nov 21, 2019 5:40 pm

jdb wrote:
Thu Nov 21, 2019 5:10 pm
rpdom wrote:
Thu Nov 21, 2019 3:57 pm
I'm giving it a try. I don't stress my Pi 4B much, but I'll see how it holds up anyway.
Partially-busy workloads are likely to see the most benefit - the intermediate performance point at 1GHz doesn't need as much volts as 1.5GHz, so "busy" temperatures should be closer to idle temperatures (and idle temperatures get a reduction).
I seem to have a 4-5 degree drop in temps with that firmware applied :)

Pi seems stable so far.
Unreadable squiggle

noee
Posts: 18
Joined: Thu Nov 21, 2019 5:30 pm

Re: DVFS Firmware

Thu Nov 21, 2019 5:59 pm

My pi4 is running as an audio player using JRiver's MediaCenter and after this update I'm seeing about 3 degree drop in temps when in normal playback mode (some minor DSP volume leveling and equalizer stuff). I do have a 3.3V fan in vented case.

However, I'm seeing a boot issue when I enable overclocking with the following in the config.txt:

Code: Select all

[all]
arm_freq=2000
gpu_freq=600
v3d_freq=700
over_voltage=4
Specifically, when I power-on, nothing happens, no green activity light, but I do see the red pwr light. If I remove the SD card, change the config.txt to comment those lines out, it boots up. Before this rpi-update, I was not seeing this behavior, it always booted fine with these overclock settings.

One thing of note, I am booting from SD but have located the rootfs off to USB attached SSD with the appropriate entries to cmdline.txt and /etc/fstab to make sure /boot and / are mounted correctly.

timg236
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 504
Joined: Thu Jun 21, 2018 4:30 pm

Re: DVFS Firmware

Thu Nov 21, 2019 7:18 pm

noee wrote:
Thu Nov 21, 2019 5:59 pm
My pi4 is running as an audio player using JRiver's MediaCenter and after this update I'm seeing about 3 degree drop in temps when in normal playback mode (some minor DSP volume leveling and equalizer stuff). I do have a 3.3V fan in vented case.

However, I'm seeing a boot issue when I enable overclocking with the following in the config.txt:

Code: Select all

[all]
arm_freq=2000
gpu_freq=600
v3d_freq=700
over_voltage=4
Specifically, when I power-on, nothing happens, no green activity light, but I do see the red pwr light. If I remove the SD card, change the config.txt to comment those lines out, it boots up. Before this rpi-update, I was not seeing this behavior, it always booted fine with these overclock settings.

One thing of note, I am booting from SD but have located the rootfs off to USB attached SSD with the appropriate entries to cmdline.txt and /etc/fstab to make sure /boot and / are mounted correctly.
The lack of activity light is a bit suspicious because the bootloader flashes the green activity light. All of the clock frequency settings are applied in start.elf after the bootloader has completed.

Does it work if you remove v3d_freq=700 ?

pboh
Posts: 13
Joined: Wed Oct 31, 2018 2:26 pm

Re: DVFS Firmware

Fri Nov 22, 2019 5:55 am

timg236 wrote:
Thu Nov 21, 2019 7:18 pm
noee wrote:
Thu Nov 21, 2019 5:59 pm
My pi4 is running as an audio player using JRiver's MediaCenter and after this update I'm seeing about 3 degree drop in temps when in normal playback mode (some minor DSP volume leveling and equalizer stuff). I do have a 3.3V fan in vented case.

However, I'm seeing a boot issue when I enable overclocking with the following in the config.txt:

Code: Select all

[all]
arm_freq=2000
gpu_freq=600
v3d_freq=700
over_voltage=4
Specifically, when I power-on, nothing happens, no green activity light, but I do see the red pwr light. If I remove the SD card, change the config.txt to comment those lines out, it boots up. Before this rpi-update, I was not seeing this behavior, it always booted fine with these overclock settings.

One thing of note, I am booting from SD but have located the rootfs off to USB attached SSD with the appropriate entries to cmdline.txt and /etc/fstab to make sure /boot and / are mounted correctly.
The lack of activity light is a bit suspicious because the bootloader flashes the green activity light. All of the clock frequency settings are applied in start.elf after the bootloader has completed.

Does it work if you remove v3d_freq=700 ?
Hi there. I can confirm the same behavior as poster noee. With the overclock settings posted on an rpi-updated 4B, the boot behaviour including the lights, is the same. Removing v3d_freq did not fix the issue, so by process of elimination I ended up with the following and can now boot successfully. If there are any other pieces of data I can gather or permutations to try to give you a hand, I'd be happy to do it.

Code: Select all

[all]
over_voltage=4
arm_freq=2000
#gpu_freq=600
v3d_freq=700

User avatar
rpdom
Posts: 16962
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: DVFS Firmware

Fri Nov 22, 2019 8:12 am

Something I've just noticed with my 4B4G running this firmware is that the network LEDs are both off except when there is activity. It's like both of them are now performing the same function. This may not be related to this firmware, I just hadn't noticed it before. Neither seems to be acting as a steady "link/speed" indicator.

Network connection seem to be running fine. I use mine headless, so I'd notice if that wasn't working. Connection is 1000Mb/s FD.
Unreadable squiggle

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 26353
Joined: Sat Jul 30, 2011 7:41 pm

Re: DVFS Firmware

Fri Nov 22, 2019 9:50 am

rpdom wrote:
Fri Nov 22, 2019 8:12 am
Something I've just noticed with my 4B4G running this firmware is that the network LEDs are both off except when there is activity. It's like both of them are now performing the same function. This may not be related to this firmware, I just hadn't noticed it before. Neither seems to be acting as a steady "link/speed" indicator.

Network connection seem to be running fine. I use mine headless, so I'd notice if that wasn't working. Connection is 1000Mb/s FD.
There has been an update to the ethernet driver LED code (some changes pulled in from upstream), the initial checkin did as you describe, but there was a subsequent fix that should mean the functionality stays the same as previous models. I wonder if that fix has not been merged yet.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

timg236
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 504
Joined: Thu Jun 21, 2018 4:30 pm

Re: DVFS Firmware

Fri Nov 22, 2019 10:18 am

pboh wrote:
Fri Nov 22, 2019 5:55 am
timg236 wrote:
Thu Nov 21, 2019 7:18 pm
noee wrote:
Thu Nov 21, 2019 5:59 pm
My pi4 is running as an audio player using JRiver's MediaCenter and after this update I'm seeing about 3 degree drop in temps when in normal playback mode (some minor DSP volume leveling and equalizer stuff). I do have a 3.3V fan in vented case.

However, I'm seeing a boot issue when I enable overclocking with the following in the config.txt:

Code: Select all

[all]
arm_freq=2000
gpu_freq=600
v3d_freq=700
over_voltage=4
Specifically, when I power-on, nothing happens, no green activity light, but I do see the red pwr light. If I remove the SD card, change the config.txt to comment those lines out, it boots up. Before this rpi-update, I was not seeing this behavior, it always booted fine with these overclock settings.

One thing of note, I am booting from SD but have located the rootfs off to USB attached SSD with the appropriate entries to cmdline.txt and /etc/fstab to make sure /boot and / are mounted correctly.
The lack of activity light is a bit suspicious because the bootloader flashes the green activity light. All of the clock frequency settings are applied in start.elf after the bootloader has completed.

Does it work if you remove v3d_freq=700 ?
Hi there. I can confirm the same behavior as poster noee. With the overclock settings posted on an rpi-updated 4B, the boot behaviour including the lights, is the same. Removing v3d_freq did not fix the issue, so by process of elimination I ended up with the following and can now boot successfully. If there are any other pieces of data I can gather or permutations to try to give you a hand, I'd be happy to do it.

Code: Select all

[all]
over_voltage=4
arm_freq=2000
#gpu_freq=600
v3d_freq=700
It would be useful to know whether core_freq=600 works (instead of gpu_freq=600) and failing that whether core_freq=550 is ok.
Overclocking the core or using a fixed core-frequency requires a larger voltage with only a small performance benefit. It's possible that this has pushed the configuration into something that the DVFS firmware doesn't like.

lb
Posts: 263
Joined: Sat Jan 28, 2012 8:07 pm

Re: DVFS Firmware

Fri Nov 22, 2019 10:35 am

Cool stuff! I've been waiting for this. After all there has been various commits in the past that pointed to ongoing work. :)

I don't have a Pi 4 handy right now, does anyone have any numbers, e.g. current draw, before and after in idle and mixed load states yet?

Edit: for anyone interested, I found this great article with a lot of analysis and numbers: https://www.hackster.io/news/raspberry- ... 22c78e7089. Looks like the power consumption, heat and throttling issues of the Pi 4 are pretty much solved with the latest update!
Last edited by lb on Fri Nov 22, 2019 10:40 am, edited 1 time in total.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 26353
Joined: Sat Jul 30, 2011 7:41 pm

Re: DVFS Firmware

Fri Nov 22, 2019 10:40 am

lb wrote:
Fri Nov 22, 2019 10:35 am
Cool stuff! I've been waiting for this. After all there has been various commits in the past that pointed to ongoing work. :)

I don't have a Pi 4 handy right now, does anyone have any numbers, e.g. current draw, before and after in idle and mixed load states yet?
There's some power stuff in this article.

https://www.hackster.io/news/raspberry- ... 22c78e7089
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

lb
Posts: 263
Joined: Sat Jan 28, 2012 8:07 pm

Re: DVFS Firmware

Fri Nov 22, 2019 10:53 am

jamesh wrote:
Fri Nov 22, 2019 10:40 am
lb wrote:
Fri Nov 22, 2019 10:35 am
Cool stuff! I've been waiting for this. After all there has been various commits in the past that pointed to ongoing work. :)

I don't have a Pi 4 handy right now, does anyone have any numbers, e.g. current draw, before and after in idle and mixed load states yet?
There's some power stuff in this article.

https://www.hackster.io/news/raspberry- ... 22c78e7089
Yeah, already found the article myself. The results are impressive, to say the least. Hats off to all the engineers that worked on these improvements.

User avatar
rpdom
Posts: 16962
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: DVFS Firmware

Fri Nov 22, 2019 11:30 am

jamesh wrote:
Fri Nov 22, 2019 9:50 am
rpdom wrote:
Fri Nov 22, 2019 8:12 am
Something I've just noticed with my 4B4G running this firmware is that the network LEDs are both off except when there is activity. It's like both of them are now performing the same function. This may not be related to this firmware, I just hadn't noticed it before. Neither seems to be acting as a steady "link/speed" indicator.

Network connection seem to be running fine. I use mine headless, so I'd notice if that wasn't working. Connection is 1000Mb/s FD.
There has been an update to the ethernet driver LED code (some changes pulled in from upstream), the initial checkin did as you describe, but there was a subsequent fix that should mean the functionality stays the same as previous models. I wonder if that fix has not been merged yet.
Thanks Jamesh, I'll wait to see when that patch become part of the main or rpi_update release. :)
Unreadable squiggle

andrum99
Posts: 1172
Joined: Fri Jul 20, 2012 2:41 pm

Re: DVFS Firmware

Fri Nov 22, 2019 12:47 pm

timg236 wrote:
Fri Nov 22, 2019 10:18 am
pboh wrote:
Fri Nov 22, 2019 5:55 am
timg236 wrote:
Thu Nov 21, 2019 7:18 pm


The lack of activity light is a bit suspicious because the bootloader flashes the green activity light. All of the clock frequency settings are applied in start.elf after the bootloader has completed.

Does it work if you remove v3d_freq=700 ?
Hi there. I can confirm the same behavior as poster noee. With the overclock settings posted on an rpi-updated 4B, the boot behaviour including the lights, is the same. Removing v3d_freq did not fix the issue, so by process of elimination I ended up with the following and can now boot successfully. If there are any other pieces of data I can gather or permutations to try to give you a hand, I'd be happy to do it.

Code: Select all

[all]
over_voltage=4
arm_freq=2000
#gpu_freq=600
v3d_freq=700
It would be useful to know whether core_freq=600 works (instead of gpu_freq=600) and failing that whether core_freq=550 is ok.
Overclocking the core or using a fixed core-frequency requires a larger voltage with only a small performance benefit. It's possible that this has pushed the configuration into something that the DVFS firmware doesn't like.
Would it not be safer to just disable dynamic voltage scaling if the user has specified an overvoltage? Overclocking is, by its very nature, pushing at the limits of what is possible, so anything which might negatively affect stability while doing so is a bad thing.

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5500
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: DVFS Firmware

Fri Nov 22, 2019 1:34 pm

andrum99 wrote:
Fri Nov 22, 2019 12:47 pm
Would it not be safer to just disable dynamic voltage scaling if the user has specified an overvoltage? Overclocking is, by its very nature, pushing at the limits of what is possible, so anything which might negatively affect stability while doing so is a bad thing.
Depends on the reason for the fail. If it can be simply fixed so overclockers get the benefits from DVFS (e.g. less time spent throttling), then that is the better solution. I'll try to reproduce, but overclocking related issues will be lower priority than issues that affect all users.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 26353
Joined: Sat Jul 30, 2011 7:41 pm

Re: DVFS Firmware

Fri Nov 22, 2019 1:51 pm

jamesh wrote:
Fri Nov 22, 2019 9:50 am
rpdom wrote:
Fri Nov 22, 2019 8:12 am
Something I've just noticed with my 4B4G running this firmware is that the network LEDs are both off except when there is activity. It's like both of them are now performing the same function. This may not be related to this firmware, I just hadn't noticed it before. Neither seems to be acting as a steady "link/speed" indicator.

Network connection seem to be running fine. I use mine headless, so I'd notice if that wasn't working. Connection is 1000Mb/s FD.
There has been an update to the ethernet driver LED code (some changes pulled in from upstream), the initial checkin did as you describe, but there was a subsequent fix that should mean the functionality stays the same as previous models. I wonder if that fix has not been merged yet.
Hmm, just tested this and getting the same results. You can workaround to get previous behaviour using the following in config.txt

dtparam=eth_led0=0
dtparam=eth_led1=8

I'll see if I can find where the update went missing to make this the default.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

andrum99
Posts: 1172
Joined: Fri Jul 20, 2012 2:41 pm

Re: DVFS Firmware

Fri Nov 22, 2019 2:11 pm

dom wrote:
Fri Nov 22, 2019 1:34 pm
andrum99 wrote:
Fri Nov 22, 2019 12:47 pm
Would it not be safer to just disable dynamic voltage scaling if the user has specified an overvoltage? Overclocking is, by its very nature, pushing at the limits of what is possible, so anything which might negatively affect stability while doing so is a bad thing.
Depends on the reason for the fail. If it can be simply fixed so overclockers get the benefits from DVFS (e.g. less time spent throttling), then that is the better solution. I'll try to reproduce, but overclocking related issues will be lower priority than issues that affect all users.
That's really what I was getting at. If overclocking works with a static voltage then there's no need to change it, in my view, given that overclocking is pushing the envelope anyway. By simply disabling dynamic voltage scaling when overclocked (or indeed with an undervoltage specified) then there should be no negative impact on stability. But it does certainly depend on the reasons for the failure, I agree.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 26353
Joined: Sat Jul 30, 2011 7:41 pm

Re: DVFS Firmware

Fri Nov 22, 2019 4:27 pm

jamesh wrote:
Fri Nov 22, 2019 1:51 pm
jamesh wrote:
Fri Nov 22, 2019 9:50 am
rpdom wrote:
Fri Nov 22, 2019 8:12 am
Something I've just noticed with my 4B4G running this firmware is that the network LEDs are both off except when there is activity. It's like both of them are now performing the same function. This may not be related to this firmware, I just hadn't noticed it before. Neither seems to be acting as a steady "link/speed" indicator.

Network connection seem to be running fine. I use mine headless, so I'd notice if that wasn't working. Connection is 1000Mb/s FD.
There has been an update to the ethernet driver LED code (some changes pulled in from upstream), the initial checkin did as you describe, but there was a subsequent fix that should mean the functionality stays the same as previous models. I wonder if that fix has not been merged yet.
Hmm, just tested this and getting the same results. You can workaround to get previous behaviour using the following in config.txt

dtparam=eth_led0=0
dtparam=eth_led1=8

I'll see if I can find where the update went missing to make this the default.
OK, this is fixed and will be in an rpi-update in the near future. Sorry. I missed a DT change when adding the ability to change the actions of the LED's.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed.
I've been saying "Mucho" to my Spanish friend a lot more lately. It means a lot to him.

User avatar
rpdom
Posts: 16962
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: DVFS Firmware

Fri Nov 22, 2019 4:45 pm

jamesh wrote:
Fri Nov 22, 2019 4:27 pm
jamesh wrote:
Fri Nov 22, 2019 1:51 pm
jamesh wrote:
Fri Nov 22, 2019 9:50 am


There has been an update to the ethernet driver LED code (some changes pulled in from upstream), the initial checkin did as you describe, but there was a subsequent fix that should mean the functionality stays the same as previous models. I wonder if that fix has not been merged yet.
Hmm, just tested this and getting the same results. You can workaround to get previous behaviour using the following in config.txt

dtparam=eth_led0=0
dtparam=eth_led1=8

I'll see if I can find where the update went missing to make this the default.
OK, this is fixed and will be in an rpi-update in the near future. Sorry. I missed a DT change when adding the ability to change the actions of the LED's.
Ok, cool and thanks :)
Unreadable squiggle

dom
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5500
Joined: Wed Aug 17, 2011 7:41 pm
Location: Cambridge

Re: DVFS Firmware

Fri Nov 22, 2019 5:08 pm

rpi-update contains the ethernet led fix

User avatar
rpdom
Posts: 16962
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: DVFS Firmware

Fri Nov 22, 2019 5:52 pm

dom wrote:
Fri Nov 22, 2019 5:08 pm
rpi-update contains the ethernet led fix
Much better, Thanks Dom :)

Dom.
Unreadable squiggle

Eirikur
Posts: 57
Joined: Sun Sep 09, 2018 9:43 pm

Re: DVFS Firmware

Sat Nov 23, 2019 2:32 am

I updated one of my 4Bs to the latest firmware via rpi-update. Overclocking to 2147 with no GPU overclock reports a speed of 2294, with idle at 655 and two increments between. I don't know if this is the actual speed, or if there's reporting issue. I'm using the pi-enhanced htop, which I believe gets the values from /proc.

It's been running like this for about 15 minutes. I did a reboot and got the same funny numbers.
I don't think I'm going to try running cpuburn like this :-) Maybe on my 4B that has the Ice Tower.

pboh
Posts: 13
Joined: Wed Oct 31, 2018 2:26 pm

Re: DVFS Firmware

Sat Nov 23, 2019 4:43 am

timg236 wrote:
Fri Nov 22, 2019 10:18 am
It would be useful to know whether core_freq=600 works (instead of gpu_freq=600) and failing that whether core_freq=550 is ok.
Overclocking the core or using a fixed core-frequency requires a larger voltage with only a small performance benefit. It's possible that this has pushed the configuration into something that the DVFS firmware doesn't like.
Hi there. Adding core_freq=600 OR core_freq=550 (neither case using an uncommented gpu_freq) reverts back to the boot failures earlier described. Reverting back to the settings earlier upthread brings it back to bootable again. This is post application of the LED fixed rpi-update.

Hope this helps and thanks for all the hard work.

Return to “Advanced users”