Page 6 of 14

Re: PoE HAT - USB Ports not working - over-current

Posted: Thu Sep 13, 2018 10:30 am
by jamesh
martinrowan wrote:
Wed Sep 12, 2018 7:25 am
I'm really pleased to see the comprehensive response to the Register and the repost here. The upcoming blog will hopefully be even more enlightening. With respect to one line segment of the response:
jamesh wrote:
Tue Sep 11, 2018 2:22 pm
We missed it in product testing because (dumb luck) our heavy-load testing was done on boards with the other brand of switch, and most of our field testers were only using the board to power mice and keyboards, which works fine on all the HAT/Pi pairs we've tested.
This sort of makes sense, with the exception that I (and others) have reported the over-current warnings being triggered without any device being attached to the USB ports and my initial report of the problem was experienced with just a Logitech wireless keyboard/mouse adaptor, which I'd expect to be fairly low current (though I don't have a USB power tester to measure it).

So in the release to the field, problems were seen even at zero and low load. Just an observation.
roken wrote: Except - I see the overvoltage, with literally nothing connected. The one Pi that I have a hat connected to is entirely independent of anything else. All that it relies on is the network. I mean that, inasmuch as even the sdcard does nothing more than boot, and hand over to an NFS boot.

This is, incidentally, why the issue is not currently a problem to me, but for my other PIs will be.
Hi MartinRowan, Roken

We are interested in getting hold of devices that are showing the issue with nothing attached to the USB, as this is not something we have seen/replicated. To that end could you contact me via the [email protected] email address so we can have a discussion. Mark email as FAO James Hughes. Thanks.

Re: PoE HAT - USB Ports not working - over-current

Posted: Thu Sep 13, 2018 2:16 pm
by that1guyrob
jamesh wrote:
Thu Sep 13, 2018 10:30 am

Hi MartinRowan, Roken

We are interested in getting hold of devices that are showing the issue with nothing attached to the USB, as this is not something we have seen/replicated. To that end could you contact me via the [email protected] email address so we can have a discussion. Mark email as FAO James Hughes. Thanks.
I have one of these PoE HATs that is showing the issue on a headless Pi with no USB devices connected. I know this post wasn't directed specifically to me, but I went ahead and sent an email in to the address above anyway, in case you could use another.

Thanks,
Rob

Re: PoE HAT - USB Ports not working - over-current

Posted: Thu Sep 13, 2018 2:45 pm
by 6by9
shawaj wrote:
Wed Sep 12, 2018 7:44 pm
For anyone who needs a quick fix, here is our PoE HAT (tried and tested for 3+ years) which does not face the same issue and as a plus works with any 40 pin Pi (not just 3B+) - https://uk.pi-supply.com/products/pi-po ... spberry-pi

Will be back in stock in around a week :-)
Gigabit ethernet supported?
That's the one that many of the plug in PoE widgets miss out on and only support 10 or 100Mbit/s.

Re: PoE HAT - USB Ports not working - over-current

Posted: Thu Sep 13, 2018 7:27 pm
by martinrowan
At least those experimenting here didn't blow up their HATs like Dave EEVblog accidentally did: https://www.youtube.com/watch?v=Tpvjo6wDFUA
I wonder if the foundation will honour his warranty ;-)

@jamesh I've dropped you an email. Feel free to get in touch. Sadly I'm not in Cambridge as often as I used to be, else I would have offered to drop in.

Re: PoE HAT - USB Ports not working - over-current

Posted: Thu Sep 13, 2018 7:32 pm
by Roken
Email also sent. Happy to help out, if I can.

Re: PoE HAT - USB Ports not working - over-current

Posted: Thu Sep 13, 2018 7:56 pm
by jamesh
Thanks Guys, will email you tomorrow.

James

Re: PoE HAT - USB Ports not working - over-current

Posted: Fri Sep 14, 2018 1:53 am
by raymate
Just got mine, not read all the posts yet, is it right a revised board is being shipped?
How can I tell if I have the revised board ?

Re: PoE HAT - USB Ports not working - over-current

Posted: Fri Sep 14, 2018 2:01 am
by Imperf3kt
raymate wrote:
Fri Sep 14, 2018 1:53 am
Just got mine, not read all the posts yet, is it right a revised board is being shipped?
How can I tell if I have the revised board ?
Since the problem has only just been diagnosed, you do not have a revised edition.
The silkscreening will probably show this information.

Re: PoE HAT - USB Ports not working - over-current

Posted: Fri Sep 14, 2018 8:50 am
by hippy
raymate wrote:
Fri Sep 14, 2018 1:53 am
How can I tell if I have the revised board ?
More importantly; how will people be able to easily tell if they receive or encounter a faulty or revised board in the future ?

That's going to be important as retailers are continuing to sell faulty boards and we will inevitably end up with boards which do and don't work with particular Pi boards circulating in the wild.

Re: PoE HAT - USB Ports not working - over-current

Posted: Fri Sep 14, 2018 9:10 am
by weust
I would expect something like "Revision B" printed on the PCB.
But let's not jump to conclusion before anything official is posted.

"Assumption is the mother of all f*ckups"

Re: PoE HAT - USB Ports not working - over-current

Posted: Fri Sep 14, 2018 10:03 am
by jamesh
Thanks for everyone who has been in contact re: boards showing the undervoltage with no USB devices attached. I hopefully now have enough victimsvolunteers.


Been checking up on some of the questions.

The boad had not been respun yet, so there are no new revisions. A new revision will have a new version number on it. There are no safety issues with continuing to use the boards as they stand - if they work for you, continue to use them.

Re: PoE HAT - USB Ports not working - over-current

Posted: Fri Sep 14, 2018 11:10 am
by raymate
jamesh wrote:
Fri Sep 14, 2018 10:03 am
Thanks for everyone who has been in contact re: boards showing the undervoltage with no USB devices attached. I hopefully now have enough victimsvolunteers.


Been checking up on some of the questions.

The boad had not been respun yet, so there are no new revisions. A new revision will have a new version number on it. There are no safety issues with continuing to use the boards as they stand - if they work for you, continue to use them.
Good to know.... so far I have not had issue so nice to know they are safe to use mine is destine for a remote location so once in place it not going to be access for months at a time

Re: PoE HAT - USB Ports not working - over-current

Posted: Fri Sep 14, 2018 12:02 pm
by Blaster
So to sum it up a little.

The only issue with PoE HAT is faulty design - inability to at least copy recommended design component values - followed with lack of final design tests. And that "different chip vendors" mumbo jumbo is only smoke screen. There must not be any significant voltage ripple on RPi supply rail, it is vital part of overall reliability.
With any switching power supply one of the first measurements to prove that it's on the right way is to measure output ripple at different loads, what we can see here (posts with oscilloscope measurements) is really horrible, at least from the HW designer point of view.

Now some facts.

Reference design for MP8007 5V/2,5A power supply recommends three output capacitors 47uF + 47uF + 470uF = 564uF nominal value

RPi HAT designers used output capacitors with nominal values of 2*22uF + 4*47uF = 232uF so they supposed that 41% of the recommended value should be fine. But they probably didn't know "secret" related to multilayer ceramic capacitors (MLCC) - decrease of capacitance with bias voltage - so things are even worse.

I measured capacitance values of two types of ceramic capacitors used in output stage of PoE HAT, and how it drops when DC voltage is applied:
(measured on RLCG bridge at 1kHz frequency, 1V excitation level)

Code: Select all

22uF MLCC capacitor

Bias [V] | C [uF] | ESR [Ohm] | C [%]
    0    |  20,7  |   0,597   | 100
    1    |  19,2  |   0,628   | 92,8
    2    |  16,2  |   0,701   | 78,3
    3    |  13,0  |   0,797   | 62,8
    4    |  10,3  |   0,889   | 49,8
    5    |   8,1  |   0,963   | 39,1


47uF MLCC capacitor

Bias [V] | C [uF] | ESR [Ohm] | C [%]
    0    |  46,4  |   0,253   | 100
    1    |  43,5  |   0,266   | 93,8
    2    |  36,7  |   0,287   | 79,1
    3    |  29,6  |   0,312   | 63,8
    4    |  23,1  |   0,350   | 49,8
    5    |  19,1  |   0,368   | 41,2

Now, the overall capacity at the output is 2*8,1uF + 4*19,1uF = 92,6uF and it is only 16,4%(!) of the recommended reference design value.
The measured ESR values is another thing to consider.

Rally someone believe that this is adequate?
There are presumably some capacitors at RPi input which were maybe considered as part of PoE HAT output capacitance, but again, it didn't work.

Below is attached graph taken from document about this capacitors behaviour (I posted link before). By coincidence they evaluated the same capacitor as used in HAT and you can clearly see that values I measured (capacitance drop with applied bias voltage) are almost identical.

Two lines (red and dashed) represents two capacitors, one rated at 6,3V (smaller physical size) and another at 16V (bigger size), both evaluated at the same DC voltages. For comprehensive info see original document, link in my older post.

Re: PoE HAT - USB Ports not working - over-current

Posted: Fri Sep 14, 2018 12:23 pm
by jamesh
Blaster wrote:
Fri Sep 14, 2018 12:02 pm
So to sum it up a little.

The only issue with PoE HAT is faulty design - inability to at least copy recommended design component values - followed with lack of final design tests. And that "different chip vendors" mumbo jumbo is only smoke screen. There must not be any significant voltage ripple on RPi supply rail, it is vital part of overall reliability.
With any switching power supply one of the first measurements to prove that it's on the right way is to measure output ripple at different loads, what we can see here (posts with oscilloscope measurements) is really horrible, at least from the HW designer point of view.

Now some facts.

Reference design for MP8007 5V/2,5A power supply recommends three output capacitors 47uF + 47uF + 470uF = 564uF nominal value

RPi HAT designers used output capacitors with nominal values of 2*22uF + 4*47uF = 232uF so they supposed that 41% of the recommended value should be fine. But they probably didn't know "secret" related to multilayer ceramic capacitors (MLCC) - decrease of capacitance with bias voltage - so things are even worse.

I measured capacitance values of two types of ceramic capacitors used in output stage of PoE HAT, and how it drops when DC voltage is applied:
(measured on RLCG bridge at 1kHz frequency, 1V excitation level)

Code: Select all

22uF MLCC capacitor

Bias [V] | C [uF] | ESR [Ohm] | C [%]
    0    |  20,7  |   0,597   | 100
    1    |  19,2  |   0,628   | 92,8
    2    |  16,2  |   0,701   | 78,3
    3    |  13,0  |   0,797   | 62,8
    4    |  10,3  |   0,889   | 49,8
    5    |   8,1  |   0,963   | 39,1


47uF MLCC capacitor

Bias [V] | C [uF] | ESR [Ohm] | C [%]
    0    |  46,4  |   0,253   | 100
    1    |  43,5  |   0,266   | 93,8
    2    |  36,7  |   0,287   | 79,1
    3    |  29,6  |   0,312   | 63,8
    4    |  23,1  |   0,350   | 49,8
    5    |  19,1  |   0,368   | 41,2

Now, the overall capacity at the output is 2*8,1uF + 4*19,1uF = 92,6uF and it is only 16,4%(!) of the recommended reference design value.
The measured ESR values is another thing to consider.

Rally someone believe that this is adequate?
There are presumably some capacitors at RPi input which were maybe considered as part of PoE HAT output capacitance, but again, it didn't work.

Below is attached graph taken from document about this capacitors behaviour (I posted link before). By coincidence they evaluated the same capacitor as used in HAT and you can clearly see that values I measured (capacitance drop with applied bias voltage) are almost identical.

Two lines (red and dashed) represents two capacitors, one rated at 6,3V (smaller physical size) and another at 16V (bigger size), both evaluated at the same DC voltages. For comprehensive info see original document, link in my older post.
I thought we had already summed things up, but hey ho. Have a pound of flesh on us.

Re: PoE HAT - USB Ports not working - over-current

Posted: Fri Sep 14, 2018 2:50 pm
by enigma1406
Having the same issues with no USB devices attached. I'll need a replacement so will follow this thread.

Re: PoE HAT - USB Ports not working - over-current

Posted: Fri Sep 14, 2018 5:33 pm
by hippy
jamesh wrote:
Fri Sep 14, 2018 10:03 am
There are no safety issues with continuing to use the boards as they stand - if they work for you, continue to use them.
I agree there do not seem to be any safety issues but not so sure about "if they work for you, continue to use them". In the short term yes; no need to rush to a decision. Longer term it's more nuanced.

The PoE HAT one has might work with the Pi one also currently has but it might not work with a different Pi if one has to replace it or one decides to relocate the PoE HAT to another Pi. At that point any warranty and ability to get the PoE HAT easily replaced may be well gone.

It would also mean one could not resell it or pass it on without the risk that it doesn't work with the PI a buyer has which could create issues and extra costs, that may also reduce its resale value. There's no easy way to tell which Pi boards it will work with and "sold as seen" doesn't necessarily alleviate a seller from responsibilities as may be believed.

Don't forget that, unless the Foundation offers to replace defective boards themselves, any warranty replacement claim will have to be through the retailer or seller.

I would like to say we can all have faith in retailers to do the right thing but that's tempered by those I checked still selling the PoE HAT without any warning that it is defective, might not work with the Pi the buyer has, might require a replacement claim as soon as it's received, with the buyer expected to pay the postage costs involved.

I would be inclined to get a PoE HAT replaced with a revised board as soon as they become available to avoid those issues. But that's just me; everyone will have to decide what they will do for themselves.

Re: PoE HAT - USB Ports not working - over-current

Posted: Fri Sep 14, 2018 6:18 pm
by alias_neo
I have seen an interesting issue that no one here seems to have mentioned thus far.

I had two HATs with apparently distant batch numbers. I connected them to my Ubiquiti PoE switch with a ~30cm ethernet cable (amongst others, for testing) and yes they exhibited the USB issues.

Both with and without USB devices attached they had the USB issues, BUT, more importantly;

Both the HATs caused the network connection on my Pi to bounce up and down, on one of the HATs with the Pi under medium load and the fan spinning constantly, the network stayed up for about an hour.

After leaving them off to cool for a while, and switching between them I found both HATs would drop the network connection. Sometimes, on startup the Pi wouldn't connect at all, other times it would connect and drop the connection, which would come up again and go again seconds later.

When I say up and down, I mean the interface state found by testing using

Code: Select all

ip -br -c a
which shows "UP" or "DOWN" and an IP or no IP, and also by pinging from another machine on the LAN.

After a couple of hours testing and the Pis has "warmed up" it became almost impossible to get the LAN to come up while using the PoE HAT.

Unfortunately I've returned the HATs today as they both are totally unusable if they can't even maintain a network connection, which is slightly more worrisome than the USB issues for my use case.

Re: PoE HAT - USB Ports not working - over-current

Posted: Sat Sep 15, 2018 6:37 am
by jamesh
alias_neo wrote:
Fri Sep 14, 2018 6:18 pm
I have seen an interesting issue that no one here seems to have mentioned thus far.

I had two HATs with apparently distant batch numbers. I connected them to my Ubiquiti PoE switch with a ~30cm ethernet cable (amongst others, for testing) and yes they exhibited the USB issues.

Both with and without USB devices attached they had the USB issues, BUT, more importantly;

Both the HATs caused the network connection on my Pi to bounce up and down, on one of the HATs with the Pi under medium load and the fan spinning constantly, the network stayed up for about an hour.

After leaving them off to cool for a while, and switching between them I found both HATs would drop the network connection. Sometimes, on startup the Pi wouldn't connect at all, other times it would connect and drop the connection, which would come up again and go again seconds later.

When I say up and down, I mean the interface state found by testing using

Code: Select all

ip -br -c a
which shows "UP" or "DOWN" and an IP or no IP, and also by pinging from another machine on the LAN.

After a couple of hours testing and the Pis has "warmed up" it became almost impossible to get the LAN to come up while using the PoE HAT.

Unfortunately I've returned the HATs today as they both are totally unusable if they can't even maintain a network connection, which is slightly more worrisome than the USB issues for my use case.
The USB and Ethernet are both supplied by the same chip, so I suspect this is a manifestation of the same power issue, but I an not an expert! I'll pass this onto the engineers working on the issue.

Re: PoE HAT - USB Ports not working - over-current

Posted: Sat Sep 15, 2018 7:23 am
by Burngate
The fault appears to be with the PoE HAT, interacting with the chip on the Pi - any PoE HAT, some Pis.

So if one gets a HaT and it doesn't play ball, there's no point in getting it replaced, yet - the next HAT will be no better.
We'll have to wait for the re-spin.

Is that the state of affairs at the moment?

Re: PoE HAT - USB Ports not working - over-current

Posted: Sat Sep 15, 2018 10:21 am
by martinrowan
Is that the state of affairs at the moment?
Yep pretty much, as the problem is with the PoE HAT and their isn't a new design out yet, you are likely to still have the same problem with any replacement. So better to source a refund, or await the blog post that is apparently coming with some possible soldering suggestions to work around the problem.

If you have multiple RPi3 b+ around, it would be worth testing the HAT on different ones to see if it works with any (hasn't for me).

Re: PoE HAT - USB Ports not working - over-current

Posted: Tue Sep 18, 2018 3:16 pm
by leol
Any news on the promised blog?

Re: PoE HAT - USB Ports not working - over-current

Posted: Wed Sep 19, 2018 8:59 am
by jamesh
leol wrote:
Tue Sep 18, 2018 3:16 pm
Any news on the promised blog?
We are still investigating the issue - it seems to be a lot more complicated that first thought and we want to make sure we have a decent handle on the situation. We've put a hold on sales for the moment, and we are geting some returned here which show up a particular issue we want to get to the bottom of.

Re: PoE HAT - USB Ports not working - over-current

Posted: Wed Sep 19, 2018 12:23 pm
by hippy
jamesh wrote:
Wed Sep 19, 2018 8:59 am
We've put a hold on sales for the moment
Just to note that while most of the resellers are now "sold out", "awaiting stock" or have removed the product completely, Pimoroni appear to still be offering them for sale -

Edited: Now "out of stock" - I'm guessing someone had words 8-)

I did note the official Raspberry Pi page for the PoE HAT has reverted to "Coming Soon", but couldn't find anything in the blog or elsewhere which details the problem or lets people know they may have purchased Pi PoE HATs which are problematic.

Re: PoE HAT - USB Ports not working - over-current

Posted: Thu Sep 20, 2018 6:40 am
by enigma1406
Not all vendors are showing out of stock

https://www.alliedelec.com/raspberry-pi ... /71191981/

Re: PoE HAT - USB Ports not working - over-current

Posted: Thu Sep 20, 2018 2:01 pm
by hippy
enigma1406 wrote:
Thu Sep 20, 2018 6:40 am
Not all vendors are showing out of stock

https://www.alliedelec.com/raspberry-pi ... /71191981/
I'd spotted that one but did not mention it. If you try and buy one it returns "[!] Material number does not exist" and I couldn't find any link or reference to it from their own product pages, only via Google links, so would consider that "no longer available".