PoE HAT - USB Ports not working - over-current (FIXED - New Hardware Release)

[The_Raven]

Is it the physical distance that makes the difference, for example with long leads/wires attached if you move it close to the RPi does it start to fail. Or is it the length/resistance of the cables that are making a difference?

Good question. I am not 100% sure. I think it is the distance.
In the next step i try to add the capacitors.
As told before: Have a look at the datasheet of the MP8007 chip. Specially the capacitor and the part with the red "X".

[aduvenhage]

What version of the boards have these power issues? My POE Hat has the numbering A1 7D.
If it is only this version that has the error and there are other versions around, then I will return my HAT for another one...

[Grozzie]

Hi,

I have been getting the same problems as mentioned on these post.

However, I have managed to fix the problem!!!

I removed the PoE hat from the Raspberry Pi.
Used jumpers to connect the 4pin PoE header on the Pi to the 4pin header on the PoE hat.
Stripped a micro usb cable, crimped Dupont connectors to the red and black wires. Connected these wires to the 5v and GND on the PoE hat. Plugged the micro usb connector end into the micro usb connector (power) on the Raspberry Pi. I then connected a flash drive, PoE Ethernet cable, monitor etc... to the Raspberry Pi. The Raspberry Pi booted without any errors. I then ran numerous large file copies from the Raspberry Pi to my PC over the network, without any issues.

I didn't connect any of the 40pin GPIO headers from the Raspberry Pi to the PoE hat so no I2C connections. I also noticed the the link speed indicator on my hub changed from yellow (10/100mb/s) to green (1000mb/s).

I did read somewhere (can't remember where) that the Raspberry Pi shouldn't be powered via the GPIO.

20180904_124433_001.jpg (197.51 KiB) Viewed 1768 times

Anyway this seems to work.

Have fun.

[The_Raven]

I added the missing parts, but same problem.
My board rev is "A1 8C" if this is the revision.
Now i run out of ideas...

[Blaster]

My guess is that problem lies in converter output voltage filtering, excessive noise disturbing overcurrent supervising feature of USB controller producing false "over-current" messages.
In this type of applications the current monitoring is commonly realised as voltage drop sensing, so it's usually sensitive to noise in applied voltage.
And measurements shows that 5V line at USB ports (while powered by PoE HAT) is very noisy, ringing spikes clearly produced by DC/DC converter switching at some 19kHz. Also the voltage ripple is quite high (at least to my sense).

[martinrowan]

@grozzie Nice that you found a way to make it work and I think it supports @blasters suggestion that it might be on the output power stage of the HAT that has the issue, I suspect the microUSB port may have some further components to clean up the input. Though a better solution if you're happy with a them being separate would be a Active PoE Splitter like this one: https://www.amazon.co.uk/DSLRKIT-Active ... e+splitter at £7.90 a fraction of the price of the PoE-HAT

Though I'm sure you're setup is just to help identify the problem with the HAT

[The_Raven]

@Grozzie: But this is not a real fix. This is was i have done yesterday
@Blaster: You pointed me in the right direction.
The 5V output of this Pi-Hat is horrible!
I measured the same as you. And have a look at the attached picture. This is measured on the same USB-Port, but with an external [email protected] PSU also powerd by the GPIO (not via microUSB)!
As you can see it is a clean DC voltage.

After this "find out" i connected a 1000uf/10V capacitor to the GPIO 5V pin to remove the noise. And surprise, i can pull 0.25mA of the USB-Port. But not more.
And then i found another interesting thing:
It is NOT the distance between the Pi and the Hat. It are the cables! I think they have the same effect like a capacitor and remove the noise.
Next i will try to enlarge the capacitors.

[Grozzie]

Thanks,

I wanted a small single cable solution so thought the PoE Hat would be ideal. My Raspberry Pi image shows a 'work-in-progress', well actually a backward step from what I had until I tried using the usb ports. I have ordered a splitter from Amazon but now this works I can tidy the build up. I will replace the flash drive with a ssd (hopefully) which was fine until I used the PoE Hat.. Still tons of money spent buying a Netgear PoE hub and a Pi Hat just to reduce cable count. I'm running Raspbian stretch lite with a LAMP stack and WordPress/Joomla. Anyway has kept me busy for the last day or so!

Forwards and upwards.

PS. Makes you think that maybe the PoE Hat was rushed and not fully tested NOT!

[daytona]

The observation I've made with a Raspberry Pi3B+ and PoE HAT today is that the USB over-current change messages only occur when the fan on the PoE HAT is not running. As soon as the fan runs, the messages stop.

I've had it running on my desk all day, the CPU had been sitting at about 50degC so the HAT fan was running continually. A while ago I popped the top off the case that it is housed in and noticed that the fan was now running, stopping, runnning . The CPU temperature had dropped to around 45degC. While it's stopped it generates dmesg entries like below:

Code: Select all

[19591.740035] usb 1-1.1-port2: over-current change
[19591.809916] usb 1-1-port3: over-current condition
[19591.810060] usb 1-1-port4: over-current change
[19591.980035] usb 1-1.1-port3: over-current change

Soon as the fan starts, the messages stop. I can even get it to generate the messages by stalling the running fan with my finger.

This Pi and Hat are new from RS in the last few days and I updated firmware/software this morning. I've not actually got anything attached to the USB ports, it's used as a serial terminal server and running headless.

[jamesh]

We are looking in to this at the moment. Thanks for all the input so far.

[Roken]

If it helps (and taking from daytona's post), I tried this as an overlay (EDIT: Ripped off and modified from https://www.raspberrypi.org/forums/view ... 5&t=221639):

Code: Select all

/*
 * Overlay for the Raspberry Pi POE HAT.
 */
/dts-v1/;
/plugin/;

/ {
	compatible = "brcm,bcm2708";

	[email protected] {
		target-path = "/";
		__overlay__ {
			fan0: [email protected] {
				compatible = "rpi-poe-fan";
				firmware = <&firmware>;
				cooling-min-state = <0>;
				cooling-max-state = <3>;
				#cooling-cells = <2>;
				cooling-levels = <0 50 150 255>;
				status = "okay";
			};
		};
	};

	[email protected] {
		target = <&cpu_thermal>;
		__overlay__ {
			trips {
				threshold: [email protected] {
					temperature = <5000>;
					hysteresis = <5000>;
					type = "active";
				};
				target: [email protected] {
					temperature = <50000>;
					hysteresis = <2000>;
					type = "active";
				};
				cpu_hot: [email protected] {
					temperature = <55000>;
					hysteresis = <2000>;
					type = "active";
				};
			};
			cooling-maps {
				map0 {
					trip = <&threshold>;
					cooling-device = <&fan0 0 1>;
				};
				map1 {
					trip = <&target>;
					cooling-device = <&fan0 1 2>;
				};
				map2 {
					trip = <&cpu_hot>;
					cooling-device = <&fan0 2 3>;
				};
			};
		};
	};
};

Which should keep the fan running. Now, dmesg gives me:

Code: Select all

[    7.447021] systemd[1]: Detected architecture arm.
[    7.482276] systemd[1]: Set hostname to <pi4>.
[    8.640677] systemd[1]: Reached target Swap.
[    8.648022] systemd[1]: Listening on /dev/initctl Compatibility Named Pipe.
[    8.654003] systemd[1]: Listening on udev Kernel Socket.
[    8.660713] systemd[1]: Listening on udev Control Socket.
[    8.667163] systemd[1]: Created slice System Slice.
[    8.673951] systemd[1]: Created slice system-getty.slice.
[    8.682292] systemd[1]: Mounting RPC Pipe File System...
[    9.629626] systemd-journald[103]: Received request to flush runtime journal from PID 1
[   10.647543] usb 1-1-port2: over-current change
[   10.657940] usb 1-1.1-port2: over-current change
[   10.881964] usb 1-1-port3: over-current change
[   10.892487] usb 1-1.1-port3: over-current change
[   11.121804] usb 1-1-port4: over-current change
[   11.141811] usb 1-1.1-port2: over-current change
[   11.361779] usb 1-1-port2: over-current change
[   11.382495] usb 1-1.1-port3: over-current change
[   11.601756] usb 1-1-port3: over-current change
[   11.632337] snd_bcm2835: module is from the staging directory, the quality is unknown, you have been warned.
[   11.636253] bcm2835_alsa bcm2835_alsa: card created with 8 channels
[   12.108444] brcmfmac: F1 signature read @0x18000000=0x15264345
[   12.116968] brcmfmac: brcmf_fw_map_chip_to_name: using brcm/brcmfmac43455-sdio.bin for chip 0x004345(17221) rev 0x000006
[   12.117562] usbcore: registered new interface driver brcmfmac
[   12.654610] brcmfmac: brcmf_c_preinit_dcmds: Firmware version = wl0: Feb 27 2018 03:15:32 version 7.45.154 (r684107 CY) FWID 01-4fbe0b04
[   12.655279] brcmfmac: brcmf_c_preinit_dcmds: CLM version = API: 12.2 Data: 9.10.105 Compiler: 1.29.4 ClmImport: 1.36.3 Creation: 2018-03-09 18:56:28 
[   16.006405] uart-pl011 3f201000.serial: no DMA platform data
[   16.373030] Adding 102396k swap on /var/swap.  Priority:-2 extents:1 across:102396k FS
[   16.800615] IPv6: ADDRCONF(NETDEV_UP): wlan0: link is not ready
[   16.800667] brcmfmac: power management disabled
[   23.110276] Bluetooth: Core ver 2.22
[   23.110351] NET: Registered protocol family 31
[   23.110357] Bluetooth: HCI device and connection manager initialized
[   23.110378] Bluetooth: HCI socket layer initialized
[   23.110391] Bluetooth: L2CAP socket layer initialized
[   23.110422] Bluetooth: SCO socket layer initialized
[   23.171530] Bluetooth: HCI UART driver ver 2.3
[   23.171544] Bluetooth: HCI UART protocol H4 registered
[   23.171550] Bluetooth: HCI UART protocol Three-wire (H5) registered
[   23.171773] Bluetooth: HCI UART protocol Broadcom registered
[   24.429303] Bluetooth: BNEP (Ethernet Emulation) ver 1.3
[   24.429315] Bluetooth: BNEP filters: protocol multicast
[   24.429338] Bluetooth: BNEP socket layer initialize

No more over-current warnings.

[martinrowan]

Just tried the same thing. Set the temp to 5 degress, so the fan is always on. Connected up a USB light, a USB flash drive, an iPhone and an Android phone and they all worked, and no errors logged. Switched showed it consuming 9.5W which seems low, even if dealing with 48V supply.

Disconnecting the fan from the header and the messages re-appear and devices start connecting/disconnecting. Reconnect fan and stability is restored. Stalling the fan with my finger, things stay stable. How very odd that the running of the fan induces some level of stability.

[piman314]

Personally I don't regard this as a fix, so will be sending my hats back as faulty for a refund when I get back from this work trip and will go back to using cheap splitters at a fraction of the price but without the reliability problems. I'm surprised something this bad got released.

[Roken]

Personally I don't regard this as a fix, so will be sending my hats back as faulty for a refund when I get back from this work trip and will go back to using cheap splitters at a fraction of the price but without the reliability problems. I'm surprised something this bad got released.

Surely, the issue is shown to be software and not hardware. If even a very low fan speed fixes, that's a fix.

Let's put it into perspective. If your desktop fans stop spinning, do you lose USB stability, or does your processor blow up?

[jdb]

Personally I don't regard this as a fix, so will be sending my hats back as faulty for a refund when I get back from this work trip and will go back to using cheap splitters at a fraction of the price but without the reliability problems. I'm surprised something this bad got released.

Surely, the issue is shown to be software and not hardware. If even a very low fan speed fixes, that's a fix.

Let's put it into perspective. If your desktop fans stop spinning, do you lose USB stability, or does your processor blow up?

Well, no - as the converse is also true. The fact that powering on the onboard fan causes a positive result in system stability is a critical data point. What if we shipped the PoE hat with no fan? I'd bet the same number of people would still be posting in this thread.

Can everyone who's experiencing stability issues with the USB port power on PoE-equipped Pi3B+ Pis please test the devicetree overlay that sets the fan permanently running and report back.

[piman314]

If it doesn't work without a fudge then it's more expensive but more of a fudge than a cheap POE splitter, I wanted it as a neater alternative but as it's now turned out to have significant disadvantages over a POE splitter and no real advantages, I can't see any reason to keep it or recommend it. I'll try the fudge when I get home and will report results but this has turned a marginal improvement on POE splitters into an expensive but poor option. Having to rely on a fan being overworked to mask a design flaw is not a step up over a cheap POE splitter.

[jdb]

If it doesn't work without a fudge then it's more expensive but more of a fudge than a cheap POE splitter, I wanted it as a neater alternative but as it's now turned out to have significant disadvantages over a POE splitter and no real advantages, I can't see any reason to keep it or recommend it. I'll try the fudge when I get home and will report results but this has turned a marginal improvement on POE splitters into an expensive but poor option. Having to rely on a fan being overworked to mask a design flaw is not a step up over a cheap POE splitter.

The PoE HAT is an official addon product and as such we expect it to work in the vast majority of use cases without issue.

This setting is not a "fudge", it is a diagnostic measure that generates additional data points that will aid our analysis of the issue.

[wolf.z]

My guess is that problem lies in converter output voltage filtering, excessive noise disturbing overcurrent supervising feature of USB controller producing false "over-current" messages....

Put everything together, I suggest any kind of experiments should be done by the manufacturer. Any switching power supply causes noise. Blaster showed, there is much noise with the Pi hat. Others showed, that introducing supply wires, acting as inductor weakens the problem if not cancels it. So it is noise over the 5 Volt causing trouble. The problem is either bad, not sufficiently tested design or false or missing electronic parts. That's clearly no user issue.
What's irritating me is the fan hack.
A switching supply definitely can cause problems if underloaded. At first glance, the fan is supposed to draw enough power to load the supply sufficiently. But why does any USB load do not put enough load on the supply?
That points to a design issue. Again, this is no user job. It is disconcerting.

[MikeCH]

Can everyone who's experiencing stability issues with the USB port power on PoE-equipped Pi3B+ Pis please test the devicetree overlay that sets the fan permanently running and report back.

I can confirm that a fan running all the time stops the overcurrent / USB issue most of the time. I was still able to trigger it by changing the CPU load from idle to 100%. Four threads "yes > /dev/null &" still stop USB from working but within a few seconds the system stabilises again. Killing these four threads at once trigger the overccurrent again.

[6by9]

Though a better solution if you're happy with a them being separate would be a Active PoE Splitter like this one: https://www.amazon.co.uk/DSLRKIT-Active ... e+splitter at £7.90 a fraction of the price of the PoE-HAT

Minor niggle there

Data Transfer Rate: 10/100Mbps

So no support for gigabit.

[Brasse83]

This is for sure a hardware problem, measuring with an oscilloscope between +5V and GND on the PoE hat gives +/-1.5V ripple, that is insane.
Nothing done in software would compensate for this.

[daytona]

...
Can everyone who's experiencing stability issues with the USB port power on PoE-equipped Pi3B+ Pis please test the devicetree overlay that sets the fan permanently running and report back.

With the new devicetree the fan is running continually, there are no overcurrent warnings.

With the CPU idling, (frequency(45)=600000000, temp=39.2'C) if I stall the running fan, the messages start; once the fan spins up again they stop.

If I get the CPU working with stress -c 4 -t 60s (frequency(45)=1400000000, temp=53.2'C) then I don't get any messages even if I stall the fan.

I've tried this 10 times over 30 minutes this morning and it has been consistent and repeatable.

[Brasse83]

...
Can everyone who's experiencing stability issues with the USB port power on PoE-equipped Pi3B+ Pis please test the devicetree overlay that sets the fan permanently running and report back.

With the new devicetree the fan is running continually, there are no overcurrent warnings.

With the CPU idling, (frequency(45)=600000000, temp=39.2'C) if I stall the running fan, the messages start; once the fan spins up again they stop.

If I get the CPU working with stress -c 4 -t 60s (frequency(45)=1400000000, temp=53.2'C) then I don't get any messages even if I stall the fan.

I've tried this 10 times over 30 minutes this morning and it has been consistent and repeatable.

This is expected, a little more power usage should decrease the ripple just enough to keep things running smoothly enough. The construction of the PoE Hat is lacking a filter.

[jamesh]

Thanks for all the diagnostics everyone, it's all very useful stuff to help with figuring out what is going on. We are looking at it here as well, and it's....odd.....

[The_Raven]

I added a second 1000uf capacitor between the 5V and the GND.
And as expected it reduces the noise and i can pull more or less 500mA from the USB.

To say it is a software-problem it's totally wrong. Of course, i would be happy if it would be a software-problem but. But it is hardware related! If you look at the oscilloscope it looks more like AC than DC.
And yes, it looks like it was not tested before.
As told in my first post, i am very disappointed about that, but we can discuss now one month or more about why it happens, who makes a fault and so on. But it brings nothing.
So let's focus on the problem and fix it!
Ah and running the fan always is not a real fix, it is quick and (very) dirty

BTW: Sorry again for my bad english.