VLI firmware v2.0 - powersaving features enabled

[jdb]

The fact that you flashed the EEPROM twice (with the updated version) and fixed the issue suggests that something went wrong during the first flash.

Unfortunately both EEPROMs are now in a "good" state so there's not really any information that can be gotten out of them...

[itsmedoofer]

Yea like I said a moot point for me and not a catastrophic issue.....

[Chimneyfactory]

Flashed my pi4 4gb, working fine with the caveat that there's a slight performance hit in USB SSD speed.

Code: Select all

pi@pi4:~/Downloads $ sudo ./vl805 
VL805 FW version: 00013701

pi@pi4:~ $ sudo hdparm -t /dev/sda1

/dev/sda1:
 Timing buffered disk reads: 1100 MB in  3.00 seconds = 366.48 MB/sec
pi@pi4:~ $ sudo hdparm -t /dev/sda1

/dev/sda1:
 Timing buffered disk reads: 1098 MB in  3.00 seconds = 365.39 MB/sec
pi@pi4:~ $ sudo hdparm -t /dev/sda1

/dev/sda1:
 Timing buffered disk reads: 1110 MB in  3.00 seconds = 369.80 MB/sec
pi@pi4:~ $ sudo hdparm -t /dev/sda1

/dev/sda1:
 Timing buffered disk reads: 1104 MB in  3.00 seconds = 367.57 MB/sec
pi@pi4:~ $ sudo hdparm -t /dev/sda1

/dev/sda1:
 Timing buffered disk reads: 1110 MB in  3.00 seconds = 369.61 MB/sec

So, on original firmware, average read speed there is 367.77MB/sec

Idle temperature after 10 minutes at ambient of 22.4C was 56-57C

Now, after flashing and rebooting :

Code: Select all

pi@pi4:~/Downloads $ sudo ./vl805 
VL805 FW version: 000137ab
pi@pi4:~/Downloads $ sudo hdparm -t /dev/sda1

/dev/sda1:
 Timing buffered disk reads: 1060 MB in  3.00 seconds = 352.94 MB/sec
pi@pi4:~/Downloads $ sudo hdparm -t /dev/sda1

/dev/sda1:
 Timing buffered disk reads: 1062 MB in  3.00 seconds = 353.60 MB/sec
pi@pi4:~/Downloads $ sudo hdparm -t /dev/sda1

/dev/sda1:
 Timing buffered disk reads: 1066 MB in  3.01 seconds = 354.67 MB/sec
pi@pi4:~/Downloads $ sudo hdparm -t /dev/sda1

/dev/sda1:
 Timing buffered disk reads: 1060 MB in  3.01 seconds = 352.73 MB/sec
pi@pi4:~/Downloads $ sudo hdparm -t /dev/sda1

/dev/sda1:
 Timing buffered disk reads: 1070 MB in  3.00 seconds = 356.30 MB/sec

With new firmware, average read speeds now 354.048MB/sec, so down 13.7MB/sec
Idle temperature now 54 - 55C , so down 1 - 2C


I too had shutdown issues that were repeatable after flashing the firmware.

With new firmware, sudo shutdown -h now would cause the pi to hang, with "Stopping Restore / Save the current clock..." being the last item displayed. This was repeatable.

sudo shutdown -r would work as expected.

I too found this problem has gone away! But while it existed, it was repeatable, happened maybe 5 - 7 times during my tests.

I am booting from SD card, Raspbian stored on external USB 3 SSD.

I will poke at it a bit more and see if I can make the problem come back again. I'll try flashing back to original firmware, then to new one a 2nd time.

[Chimneyfactory]

I've been trying to get the hanging on shutdown problem to come back, which I did, but not repeatably.

Running new firmware.
sudo shutdown -h now OK
reboot
flash firmware (original)
sudo shutdown -h now HANGS
reboot
sudo shutdown -h now OK
reboot
sudo shutdown -h now OK
reboot
flash firmware (new)
sudo shutdown -h now OK
reboot
sudo shutdown -h now OK
reboot
sudo ./vl805 (to show version number - does this cause problem?)
sudo shutdown -h now HANGS
reboot
sudo shutdown -h now OK
reboot
sudo ./vl805
flash firmware (original)
sudo shutdown -h now HANGS
reboot
sudo shutdown -h now OK
reboot
sudo shutdown -h now OK
reboot
sudo ./vl805
sudo shutdown -h now OK
reboot
flash firmware (new)
sudo shutdown -h now OK
reboot
wait 10 minutes
sudo shutdown -h now OK

OK means Pi shuts down, green LED flashes as expected
HANGS means text stays on screen "Stopping Restore / save the current clock..." being last item. Green LED does not flash.
reboot means power off pi, wait for red LED to extinguish, then power on again.
flash firmware means sudo ./vl805 -w firmware_image

SO, I can make it hang on shutdown sometime, and a hang was always preceded by the use of the vl805 program either to flash or display the firmware. But use of this program doesn't always cause a hang at shutdown.

(Previously, I was consistently getting hangs at shutdown)

As stated before, pi 4 4Gb, boot from SD then Raspbian on external USB3 SSD

Thanks

Ian

[jdb]

The use of the PCIe bounce buffers has been known to cause variable performance, especially when operating at near 100% utilisation of the PCIe link.

Can you add total_mem=3072 to /boot/config.txt prior to retesting? This will limit the available ARM memory to the bottom 3G which should remove the bounce buffers from the equation.

Also, if you have the EEPROM in a dodgy state again, can you run sudo ./vl805 -r vl805_fw_backup.bin to dump the contents to a file?

[PhilE]

For performance measurements I would also recommend switching to the performance governor:

Code: Select all

sudo sh -c "echo performance > /sys/devices/system/cpu/cpufreq/policy0/scaling_governor"

[Chimneyfactory]

Code: Select all

Mem=3072?	Perf Gov?	firmware	speed (ave)	Reps	Difference (MB/sec)
				00013701	364.516		3	-9.40
				00013701	367.770		5	-6.15
Y				00013701	367.424		5	-6.49
				00013701	366.410		5	-7.51
		Y		00013701	372.390		5	-1.53
		Y		00013701	371.880		5	-2.04
Y		Y		00013701	373.918		5	0.00
				000137ab	353.610		3	-20.31
				000137ab	354.048		5	-19.87
Y				000137ab	355.798		5	-18.12
				000137ab	354.720		5	-19.20
		Y		000137ab	357.236		5	-16.68
		Y		000137ab	358.988		5	-14.93
Y		Y		000137ab	360.082		5	-13.84

Results of lots of tests, with the two different firmwares, with TOTAL_MEM=3072 in config.txt and with the performance governor.

The speed test used was hdparm :

Code: Select all

sudo hdparm -t /dev/sda1

and the speed is reported in MB/sec.

The difference column notes the test speed compared to the best speed recorded.

The first two tests consisted of 3 repetitions, averaged. All other tests were 5 repetitions of the hdparm command, averaged. The 3 reps tests seem to be odd outliers, the other tests seem consistent based around the test settings - firmware version, governor etc.

My conclusion is that the new firmware slows the USB3 transfer rate by 13MB/sec or thereabouts.


During the repeated reboots and reflashing of the firmware, I had an additional 4 freezes after

Code: Select all

sudo shutdown -h now

, but this did not happen every time. Each freeze ocurred when shutting down after use of the ./vl805 command, but not every use of this command resulted in a freeze.

Following the freeze and a reboot, I issued the command

Code: Select all

sudo ./vl805 -r test

and did a binary comparison of this file to the firmware image previously written - it was identical. I did this command after another freeze and reboot, still identical.

Ian

[Gerbertus]

I think a few more +1's for success and we put this in the announcements forum.

I haven't done any USB speed testing, but my pi4 is working just fine with the new firmware, and the idle temperature dropped from 49-50C to 43C!

This is with a FLIRC case, which is a fairly ideal case for seeing improvements from reducing the USB chip power draw. (The processor is already rather well cooled and there isn't much ventilation for getting rid of heat from any other components on the board, so reducing the heat from those components will bring the Pi substantially closer to ambient. Were there a way to reduce the heat coming from e.g. the MxL PMIC, I bet that would show further notable improvements.) I imagine that people running without a case or with a case that doesn't cool the processor like this may see more limited improvements.

Still, though, it seems to me that this large a difference in power draw and temperature is more than worth a tiny performance hit on super fast external SSDs.

[asavah]

Is vl805 utility itself closed source (proprietary)?
Are its sources available anywhere?

I'm asking because I have full aarch64 os (my own project) happily running without any arm 32 bit support.
I was able to get vl805 to run successfully by grabbing armhf libs from raspbian mirror and patching rpath with patchelf into ./vl805.
Firmware was updated successfully and my pi4 is running fine.

So far so good but it would be nice to be able to build the utility for aarch64 if possible.

[trejan]

Is vl805 utility itself closed source (proprietary)?
Are its sources available anywhere?

Closed source. I'm not sure who wrote it. The VL805 documentation is only available under NDA and the vl805 tool does do some undocumented register twiddling so there might be a problem releasing it. You need to do a bunch of register writes to enable/use the SPI interface on the VL805.

[asavah]

Is vl805 utility itself closed source (proprietary)?
Are its sources available anywhere?

Closed source. I'm not sure who wrote it. The VL805 documentation is only available under NDA and the vl805 tool does do some undocumented register twiddling so there might be a problem releasing it. You need to do a bunch of register writes to enable/use the SPI interface on the VL805.

As I suspected.
Thanks for the info.

[PhilE]

The utility contains VIA proprietary information so is closed source, but for greater portability I've updated the archive with a statically linked build.

[asavah]

The utility contains VIA proprietary information so is closed source, but for greater portability I've updated the archive with a statically linked build.

Thanks!
Works nicely on aarch64 without any fiddling.

[Chimneyfactory]

I've found a sequence that, for me anyway, will consistently cause the shutdown freeze problem.

Setup :
Pi 4 4gb
Full Raspbian install on SSD
Boot from SD
New VLI firmware 000137ab already installed
VLI805 command and firmware is unzipped in Downloads directory
Pi boots to CLI

Login, then type these commands

Code: Select all

cd Downloads
sudo ./vl805
ls
cd
ls
cd Desktop
ls
cd
sudo shutdown -h now

Note that the cd and ls commands work as expected, showing that USB3 access to the SSD is probably still working, cache notwithstanding.

For me, the above sequence failed to properly shutdown 5 out of 5 attempts. Then additionally a further freeze when reading firmware into a file.

If I leave out the sudo ./vl805 step, but do everything else, it shuts down correctly.

The freeze varied slightly :

Last text on screen : "Stopping restore/save current clock..." with no red LED lit : 3 times
Last text on screen : "Stopping restore/save current clock..." with red LED lit : 2 times
Last text on screen : "target power off" with red and green LED lit :1 (green LED went out around 30 seconds later)

In all occasions of the freezing, the text remains on the screen and the power down does not complete.

Following the 5 repeated freezes, I restarted and used

Code: Select all

sudo ./vl805 -r test3
cmp test3 vl805_fw_000137ab.bin

Which reported nothing, showing the firmware does not seem to be corrupted.
(comparing the extracted firmware to the previous version 00013701 showed differences, as expected)
When I tried to shutdown after doing these commands, the Pi froze for a 6th consecutive time, as noted above.

Hope this is useful

Ian

[jdb]

OK, so that confirms there's no issue with the firmware update - just that the utility is leaving the chip in an odd state that can cause a shutdown hang.

I can replicate the hang, but only with USB3.0 devices attached. I tested with a mass storage device (filesystem not mounted) and a USB-HDMI adapter. Linux doesn't even have a driver for the USB-HDMI adapter so it's purely a function of a USB3.0 device being attached.

[CypherOz]

Installed, only saw a ~1 degree drop, but I am running a fan cooler.

[jerrm]

OK, so that confirms there's no issue with the firmware update - just that the utility is leaving the chip in an odd state that can cause a shutdown hang.

I would consider the performance hit an issue if there isn't an option to get it back.

I don't care about the temp at all as long as it is stable. A Flirc(like) or fan case is a necessity either way, so temps within reason are irrelevant.

[pagenotfound]

OK, so that confirms there's no issue with the firmware update - just that the utility is leaving the chip in an odd state that can cause a shutdown hang.

I would consider the performance hit an issue if there isn't an option to get it back.

I don't care about the temp at all as long as it is stable. A Flirc(like) or fan case is a necessity either way, so temps within reason are irrelevant.

There are many people, including at least one RPi engineer, who want to run them without a case and without a fan. Milliwatts count. It's not for the lower price that some people still use the old A models.

Also, 300 mW x 1.000.000 units produced so far = 300 kW. After Christmas we will be talking about a Megawatt. There are worse energy wasters on this planet, but over the years investing a few more engineer hours in this will pay off.

[jerrm]

There are many people, including at least one RPi engineer, who want to run them without a case and without a fan. Milliwatts count. It's not for the lower price that some people still use the old A models.

Folks still have old A's as an option if performance is not their need.

IMO, a Pi without a case is a hobbyist toy. For any real deployment the Pi is inside something - be that its own case or another product. Cooling will still be needed. A 5 degree reduction in base temps will only delay throttling for minute or two. Great for short bursty applications, not so good for anything with sustained usage.

Also, 300 mW x 1.000.000 units produced so far = 300 kW. After Christmas we will be talking about a Megawatt. There are worse energy wasters on this planet, but over the years investing a few more engineer hours in this will pay off.

But for uses that move a lot of data, will the net power usage be any better? If my job runs for an extra 15 minutes and keeps the Pi at full clock speed that much longer (along with all the related equipment - attached HDD, switch, local router, internet backbone, remote router, remote switch, remote PC, etc) has any power really been saved?

Reduced power as an option is great, for many performance is more important.

[jamesh]

IMO, a Pi without a case is a hobbyist toy. For any real deployment the Pi is inside something - be that its own case or another product. Cooling will still be needed. A 5 degree reduction in base temps will only delay throttling for minute or two. Great for short bursty applications, not so good for anything with sustained usage.

Well, I do professional work on a Pi4 every single day, and it's sitting on my desk naked as the day it was born. No case, no heatsink.

You statement about a 5 degree reduction only delaying throttling is inaccurate. If it drops peak temp below the threshold, then no throttling will occur. A lower base temp means less power going into the system and therefore more chance the inbuilt heat dissipation can keep equilibrium at a lower overall temperature. ie there is a total amount of heat and therefore power the board can dissipate (various according to ambient and board temperature) - reduce the amount going in and you are going to have lower overall equilibrium.

[jerrm]

Well, I do professional work on a Pi4 every single day, and it's sitting on my desk naked as the day it was born. No case, no heatsink.

You are not the average user. Maybe not a toy in your case, but the grand-ultimate-king hobbyist.

Our uses are not average either, but there is no scenario we would ship a Pi bare with no protection. I would think most average/non-hobbyist users don't want a bare board floating around either.

The official case causes throttling within a couple minutes of a job running. You no doubt have better data on how hot it can ultimately get if allowed to exceed the current throttling temps and therefore how much difference the 5 degrees really makes, but based on current behavior I find it hard to believe a Pi4 in a closed case will not need cooling (either passive of fan) if the CPU is stressed for more than few minutes.

[jdb]

Limit the discussion to the VLI firmware testing, please. Offtopic posts are going to get deleted.

[Chimneyfactory]

In my post of Wed Sep 11, 2019 4:01 pm, https://www.raspberrypi.org/forums/view ... 5#p1534499 I noted that I could sometimes get the Pi to hang when shutting down, but it didn't seem repeatable.

In my post of Thu Sep 12, 2019 1:43 pm https://www.raspberrypi.org/forums/view ... 5#p1534965 I showed a way that consistently made the Pi freeze when shutting down after using the vl805 utility.

I think I've found why the first tests seemed inconsistent.

If you're running the original 00013701 firmware, using the vl805 utility does not cause the pi to hang when shutting down.

If you're running the new firmware 000137ab, the pi will consistently hang when shutting down after using the vl805 utility.

Test :
Pi 4 4gb with latest Raspbian installed on SSD, booting from SD
Pi boots to CLI
VL805 utility and firmware unzipped to Downloads directory

With original 00013701 firmware installed, these commands were typed as in the previous test:

Code: Select all

cd Downloads
sudo ./vl805
ls
cd
ls
cd Desktop
ls
cd
sudo shutdown -h now

Works correctly. I repeated this 5 times, each time the Pi operates as expected for each command, and shuts down properly.

I then used the vl805 utility to flash the new 000137ab firmware, and shut the pi down. The shutdown completed successfully.

Reboot, and

Code: Select all

cd Downloads
sudo ./vl805
ls
cd
ls
cd Desktop
ls
cd
sudo shutdown -h now

On 3 out of 3 tests, the shutdown failed to complete, as noted before.

So, if the vl805 utility is leaving the chip in an unknown state, why is this not a problem with the original firmware, but is now causing problems with the new firmware? And might this have other unexpected issues?

Hope this is of interest, it was bugging me why I'd originally not been able to make it fail repeatedly, so I'm happy to have found out why.

Thanks

Ian

[invertedohm]

If you're running the original 00013701 firmware, using the vl805 utility does not cause the pi to hang when shutting down.

If you're running the new firmware 000137ab, the pi will consistently hang when shutting down after using the vl805 utility.

Are you using the same vl805 binary for both flashes? I noted that the new zip file had a new binary as well. I don't know what the difference is but the sizes and strings dump look quite different. Just checking as this may have some unexpected impact.

For the record, I'm running the new firmware on two bare pi4's and they're working just fine, one with USB-SSD as well (/boot still on the flash card). Generally 55 degrees idle. I tweaked the CPU governor a little bit:

Code: Select all

$ cat /etc/sysfs.d/ondemand.conf
devices/system/cpu/cpufreq/ondemand/ignore_nice_load = 1
devices/system/cpu/cpufreq/ondemand/sampling_rate = 266250
devices/system/cpu/cpufreq/ondemand/sampling_down_factor = 100

Maxes out around 80 degrees without any cooling and no noticeable throttling, they idle at 50 and 55 respectively.

[PhilE]

There were some internal implementation changes in the vl805 utility, and library linkage is now static, but the functionality should be identical.