jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 20698
Joined: Sat Jul 30, 2011 7:41 pm

Re: PoE HAT - USB Ports not working - over-current

Thu Sep 13, 2018 10:30 am

martinrowan wrote:
Wed Sep 12, 2018 7:25 am
I'm really pleased to see the comprehensive response to the Register and the repost here. The upcoming blog will hopefully be even more enlightening. With respect to one line segment of the response:
jamesh wrote:
Tue Sep 11, 2018 2:22 pm
We missed it in product testing because (dumb luck) our heavy-load testing was done on boards with the other brand of switch, and most of our field testers were only using the board to power mice and keyboards, which works fine on all the HAT/Pi pairs we've tested.
This sort of makes sense, with the exception that I (and others) have reported the over-current warnings being triggered without any device being attached to the USB ports and my initial report of the problem was experienced with just a Logitech wireless keyboard/mouse adaptor, which I'd expect to be fairly low current (though I don't have a USB power tester to measure it).

So in the release to the field, problems were seen even at zero and low load. Just an observation.
roken wrote: Except - I see the overvoltage, with literally nothing connected. The one Pi that I have a hat connected to is entirely independent of anything else. All that it relies on is the network. I mean that, inasmuch as even the sdcard does nothing more than boot, and hand over to an NFS boot.

This is, incidentally, why the issue is not currently a problem to me, but for my other PIs will be.
Hi MartinRowan, Roken

We are interested in getting hold of devices that are showing the issue with nothing attached to the USB, as this is not something we have seen/replicated. To that end could you contact me via the [email protected] email address so we can have a discussion. Mark email as FAO James Hughes. Thanks.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.

that1guyrob
Posts: 4
Joined: Wed Apr 18, 2018 6:24 pm

Re: PoE HAT - USB Ports not working - over-current

Thu Sep 13, 2018 2:16 pm

jamesh wrote:
Thu Sep 13, 2018 10:30 am

Hi MartinRowan, Roken

We are interested in getting hold of devices that are showing the issue with nothing attached to the USB, as this is not something we have seen/replicated. To that end could you contact me via the [email protected] email address so we can have a discussion. Mark email as FAO James Hughes. Thanks.
I have one of these PoE HATs that is showing the issue on a headless Pi with no USB devices connected. I know this post wasn't directed specifically to me, but I went ahead and sent an email in to the address above anyway, in case you could use another.

Thanks,
Rob

6by9
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 5799
Joined: Wed Dec 04, 2013 11:27 am
Location: ZZ9 Plural Z Alpha, aka just outside Cambridge.

Re: PoE HAT - USB Ports not working - over-current

Thu Sep 13, 2018 2:45 pm

shawaj wrote:
Wed Sep 12, 2018 7:44 pm
For anyone who needs a quick fix, here is our PoE HAT (tried and tested for 3+ years) which does not face the same issue and as a plus works with any 40 pin Pi (not just 3B+) - https://uk.pi-supply.com/products/pi-po ... spberry-pi

Will be back in stock in around a week :-)
Gigabit ethernet supported?
That's the one that many of the plug in PoE widgets miss out on and only support 10 or 100Mbit/s.
Software Engineer at Raspberry Pi Trading. Views expressed are still personal views.
Please don't send PMs asking for support - use the forum.
I'm not interested in doing contracts for bespoke functionality - please don't ask.

martinrowan
Posts: 40
Joined: Wed Aug 26, 2015 11:00 am
Location: Berkshire, UK
Contact: Website Twitter

Re: PoE HAT - USB Ports not working - over-current

Thu Sep 13, 2018 7:27 pm

At least those experimenting here didn't blow up their HATs like Dave EEVblog accidentally did: https://www.youtube.com/watch?v=Tpvjo6wDFUA
I wonder if the foundation will honour his warranty ;-)

@jamesh I've dropped you an email. Feel free to get in touch. Sadly I'm not in Cambridge as often as I used to be, else I would have offered to drop in.

Roken
Posts: 232
Joined: Sun Dec 31, 2017 4:35 pm

Re: PoE HAT - USB Ports not working - over-current

Thu Sep 13, 2018 7:32 pm

Email also sent. Happy to help out, if I can.
Headless PI. OMG, someone cut it's head off. Oh, hang on. it didn't have one to start with.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 20698
Joined: Sat Jul 30, 2011 7:41 pm

Re: PoE HAT - USB Ports not working - over-current

Thu Sep 13, 2018 7:56 pm

Thanks Guys, will email you tomorrow.

James
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.

raymate
Posts: 58
Joined: Sat Sep 28, 2013 3:56 pm
Location: Canada / UK
Contact: Website

Re: PoE HAT - USB Ports not working - over-current

Fri Sep 14, 2018 1:53 am

Just got mine, not read all the posts yet, is it right a revised board is being shipped?
How can I tell if I have the revised board ?
__________________________________
Pi model B / Pi 2 / Pi 3 / Pi Zero W

User avatar
Imperf3kt
Posts: 1373
Joined: Tue Jun 20, 2017 12:16 am
Location: Australia

Re: PoE HAT - USB Ports not working - over-current

Fri Sep 14, 2018 2:01 am

raymate wrote:
Fri Sep 14, 2018 1:53 am
Just got mine, not read all the posts yet, is it right a revised board is being shipped?
How can I tell if I have the revised board ?
Since the problem has only just been diagnosed, you do not have a revised edition.
The silkscreening will probably show this information.
Stop plugging your fan directly into the GPIO 5v
https://www.electronics-tutorials.ws/power/transient-suppression.html

hippy
Posts: 3882
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: PoE HAT - USB Ports not working - over-current

Fri Sep 14, 2018 8:50 am

raymate wrote:
Fri Sep 14, 2018 1:53 am
How can I tell if I have the revised board ?
More importantly; how will people be able to easily tell if they receive or encounter a faulty or revised board in the future ?

That's going to be important as retailers are continuing to sell faulty boards and we will inevitably end up with boards which do and don't work with particular Pi boards circulating in the wild.

User avatar
weust
Posts: 14
Joined: Tue Sep 04, 2018 10:44 am

Re: PoE HAT - USB Ports not working - over-current

Fri Sep 14, 2018 9:10 am

I would expect something like "Revision B" printed on the PCB.
But let's not jump to conclusion before anything official is posted.

"Assumption is the mother of all f*ckups"

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 20698
Joined: Sat Jul 30, 2011 7:41 pm

Re: PoE HAT - USB Ports not working - over-current

Fri Sep 14, 2018 10:03 am

Thanks for everyone who has been in contact re: boards showing the undervoltage with no USB devices attached. I hopefully now have enough victimsvolunteers.


Been checking up on some of the questions.

The boad had not been respun yet, so there are no new revisions. A new revision will have a new version number on it. There are no safety issues with continuing to use the boards as they stand - if they work for you, continue to use them.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.

raymate
Posts: 58
Joined: Sat Sep 28, 2013 3:56 pm
Location: Canada / UK
Contact: Website

Re: PoE HAT - USB Ports not working - over-current

Fri Sep 14, 2018 11:10 am

jamesh wrote:
Fri Sep 14, 2018 10:03 am
Thanks for everyone who has been in contact re: boards showing the undervoltage with no USB devices attached. I hopefully now have enough victimsvolunteers.


Been checking up on some of the questions.

The boad had not been respun yet, so there are no new revisions. A new revision will have a new version number on it. There are no safety issues with continuing to use the boards as they stand - if they work for you, continue to use them.
Good to know.... so far I have not had issue so nice to know they are safe to use mine is destine for a remote location so once in place it not going to be access for months at a time
__________________________________
Pi model B / Pi 2 / Pi 3 / Pi Zero W

Blaster
Posts: 4
Joined: Tue Sep 04, 2018 12:22 pm

Re: PoE HAT - USB Ports not working - over-current

Fri Sep 14, 2018 12:02 pm

So to sum it up a little.

The only issue with PoE HAT is faulty design - inability to at least copy recommended design component values - followed with lack of final design tests. And that "different chip vendors" mumbo jumbo is only smoke screen. There must not be any significant voltage ripple on RPi supply rail, it is vital part of overall reliability.
With any switching power supply one of the first measurements to prove that it's on the right way is to measure output ripple at different loads, what we can see here (posts with oscilloscope measurements) is really horrible, at least from the HW designer point of view.

Now some facts.

Reference design for MP8007 5V/2,5A power supply recommends three output capacitors 47uF + 47uF + 470uF = 564uF nominal value

RPi HAT designers used output capacitors with nominal values of 2*22uF + 4*47uF = 232uF so they supposed that 41% of the recommended value should be fine. But they probably didn't know "secret" related to multilayer ceramic capacitors (MLCC) - decrease of capacitance with bias voltage - so things are even worse.

I measured capacitance values of two types of ceramic capacitors used in output stage of PoE HAT, and how it drops when DC voltage is applied:
(measured on RLCG bridge at 1kHz frequency, 1V excitation level)

Code: Select all

22uF MLCC capacitor

Bias [V] | C [uF] | ESR [Ohm] | C [%]
    0    |  20,7  |   0,597   | 100
    1    |  19,2  |   0,628   | 92,8
    2    |  16,2  |   0,701   | 78,3
    3    |  13,0  |   0,797   | 62,8
    4    |  10,3  |   0,889   | 49,8
    5    |   8,1  |   0,963   | 39,1


47uF MLCC capacitor

Bias [V] | C [uF] | ESR [Ohm] | C [%]
    0    |  46,4  |   0,253   | 100
    1    |  43,5  |   0,266   | 93,8
    2    |  36,7  |   0,287   | 79,1
    3    |  29,6  |   0,312   | 63,8
    4    |  23,1  |   0,350   | 49,8
    5    |  19,1  |   0,368   | 41,2

Now, the overall capacity at the output is 2*8,1uF + 4*19,1uF = 92,6uF and it is only 16,4%(!) of the recommended reference design value.
The measured ESR values is another thing to consider.

Rally someone believe that this is adequate?
There are presumably some capacitors at RPi input which were maybe considered as part of PoE HAT output capacitance, but again, it didn't work.

Below is attached graph taken from document about this capacitors behaviour (I posted link before). By coincidence they evaluated the same capacitor as used in HAT and you can clearly see that values I measured (capacitance drop with applied bias voltage) are almost identical.

Two lines (red and dashed) represents two capacitors, one rated at 6,3V (smaller physical size) and another at 16V (bigger size), both evaluated at the same DC voltages. For comprehensive info see original document, link in my older post.
Attachments
C_graph.png
C_graph.png (28.38 KiB) Viewed 3134 times

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 20698
Joined: Sat Jul 30, 2011 7:41 pm

Re: PoE HAT - USB Ports not working - over-current

Fri Sep 14, 2018 12:23 pm

Blaster wrote:
Fri Sep 14, 2018 12:02 pm
So to sum it up a little.

The only issue with PoE HAT is faulty design - inability to at least copy recommended design component values - followed with lack of final design tests. And that "different chip vendors" mumbo jumbo is only smoke screen. There must not be any significant voltage ripple on RPi supply rail, it is vital part of overall reliability.
With any switching power supply one of the first measurements to prove that it's on the right way is to measure output ripple at different loads, what we can see here (posts with oscilloscope measurements) is really horrible, at least from the HW designer point of view.

Now some facts.

Reference design for MP8007 5V/2,5A power supply recommends three output capacitors 47uF + 47uF + 470uF = 564uF nominal value

RPi HAT designers used output capacitors with nominal values of 2*22uF + 4*47uF = 232uF so they supposed that 41% of the recommended value should be fine. But they probably didn't know "secret" related to multilayer ceramic capacitors (MLCC) - decrease of capacitance with bias voltage - so things are even worse.

I measured capacitance values of two types of ceramic capacitors used in output stage of PoE HAT, and how it drops when DC voltage is applied:
(measured on RLCG bridge at 1kHz frequency, 1V excitation level)

Code: Select all

22uF MLCC capacitor

Bias [V] | C [uF] | ESR [Ohm] | C [%]
    0    |  20,7  |   0,597   | 100
    1    |  19,2  |   0,628   | 92,8
    2    |  16,2  |   0,701   | 78,3
    3    |  13,0  |   0,797   | 62,8
    4    |  10,3  |   0,889   | 49,8
    5    |   8,1  |   0,963   | 39,1


47uF MLCC capacitor

Bias [V] | C [uF] | ESR [Ohm] | C [%]
    0    |  46,4  |   0,253   | 100
    1    |  43,5  |   0,266   | 93,8
    2    |  36,7  |   0,287   | 79,1
    3    |  29,6  |   0,312   | 63,8
    4    |  23,1  |   0,350   | 49,8
    5    |  19,1  |   0,368   | 41,2

Now, the overall capacity at the output is 2*8,1uF + 4*19,1uF = 92,6uF and it is only 16,4%(!) of the recommended reference design value.
The measured ESR values is another thing to consider.

Rally someone believe that this is adequate?
There are presumably some capacitors at RPi input which were maybe considered as part of PoE HAT output capacitance, but again, it didn't work.

Below is attached graph taken from document about this capacitors behaviour (I posted link before). By coincidence they evaluated the same capacitor as used in HAT and you can clearly see that values I measured (capacitance drop with applied bias voltage) are almost identical.

Two lines (red and dashed) represents two capacitors, one rated at 6,3V (smaller physical size) and another at 16V (bigger size), both evaluated at the same DC voltages. For comprehensive info see original document, link in my older post.
I thought we had already summed things up, but hey ho. Have a pound of flesh on us.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.

enigma1406
Posts: 3
Joined: Fri Sep 14, 2018 2:49 pm

Re: PoE HAT - USB Ports not working - over-current

Fri Sep 14, 2018 2:50 pm

Having the same issues with no USB devices attached. I'll need a replacement so will follow this thread.

hippy
Posts: 3882
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: PoE HAT - USB Ports not working - over-current

Fri Sep 14, 2018 5:33 pm

jamesh wrote:
Fri Sep 14, 2018 10:03 am
There are no safety issues with continuing to use the boards as they stand - if they work for you, continue to use them.
I agree there do not seem to be any safety issues but not so sure about "if they work for you, continue to use them". In the short term yes; no need to rush to a decision. Longer term it's more nuanced.

The PoE HAT one has might work with the Pi one also currently has but it might not work with a different Pi if one has to replace it or one decides to relocate the PoE HAT to another Pi. At that point any warranty and ability to get the PoE HAT easily replaced may be well gone.

It would also mean one could not resell it or pass it on without the risk that it doesn't work with the PI a buyer has which could create issues and extra costs, that may also reduce its resale value. There's no easy way to tell which Pi boards it will work with and "sold as seen" doesn't necessarily alleviate a seller from responsibilities as may be believed.

Don't forget that, unless the Foundation offers to replace defective boards themselves, any warranty replacement claim will have to be through the retailer or seller.

I would like to say we can all have faith in retailers to do the right thing but that's tempered by those I checked still selling the PoE HAT without any warning that it is defective, might not work with the Pi the buyer has, might require a replacement claim as soon as it's received, with the buyer expected to pay the postage costs involved.

I would be inclined to get a PoE HAT replaced with a revised board as soon as they become available to avoid those issues. But that's just me; everyone will have to decide what they will do for themselves.

alias_neo
Posts: 108
Joined: Fri Jul 29, 2011 8:55 am
Location: London, England
Contact: Website

Re: PoE HAT - USB Ports not working - over-current

Fri Sep 14, 2018 6:18 pm

I have seen an interesting issue that no one here seems to have mentioned thus far.

I had two HATs with apparently distant batch numbers. I connected them to my Ubiquiti PoE switch with a ~30cm ethernet cable (amongst others, for testing) and yes they exhibited the USB issues.

Both with and without USB devices attached they had the USB issues, BUT, more importantly;

Both the HATs caused the network connection on my Pi to bounce up and down, on one of the HATs with the Pi under medium load and the fan spinning constantly, the network stayed up for about an hour.

After leaving them off to cool for a while, and switching between them I found both HATs would drop the network connection. Sometimes, on startup the Pi wouldn't connect at all, other times it would connect and drop the connection, which would come up again and go again seconds later.

When I say up and down, I mean the interface state found by testing using

Code: Select all

ip -br -c a
which shows "UP" or "DOWN" and an IP or no IP, and also by pinging from another machine on the LAN.

After a couple of hours testing and the Pis has "warmed up" it became almost impossible to get the LAN to come up while using the PoE HAT.

Unfortunately I've returned the HATs today as they both are totally unusable if they can't even maintain a network connection, which is slightly more worrisome than the USB issues for my use case.
http://2byt.es - Just my two bytes - RPi - Android - Arduino - Blog

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 20698
Joined: Sat Jul 30, 2011 7:41 pm

Re: PoE HAT - USB Ports not working - over-current

Sat Sep 15, 2018 6:37 am

alias_neo wrote:
Fri Sep 14, 2018 6:18 pm
I have seen an interesting issue that no one here seems to have mentioned thus far.

I had two HATs with apparently distant batch numbers. I connected them to my Ubiquiti PoE switch with a ~30cm ethernet cable (amongst others, for testing) and yes they exhibited the USB issues.

Both with and without USB devices attached they had the USB issues, BUT, more importantly;

Both the HATs caused the network connection on my Pi to bounce up and down, on one of the HATs with the Pi under medium load and the fan spinning constantly, the network stayed up for about an hour.

After leaving them off to cool for a while, and switching between them I found both HATs would drop the network connection. Sometimes, on startup the Pi wouldn't connect at all, other times it would connect and drop the connection, which would come up again and go again seconds later.

When I say up and down, I mean the interface state found by testing using

Code: Select all

ip -br -c a
which shows "UP" or "DOWN" and an IP or no IP, and also by pinging from another machine on the LAN.

After a couple of hours testing and the Pis has "warmed up" it became almost impossible to get the LAN to come up while using the PoE HAT.

Unfortunately I've returned the HATs today as they both are totally unusable if they can't even maintain a network connection, which is slightly more worrisome than the USB issues for my use case.
The USB and Ethernet are both supplied by the same chip, so I suspect this is a manifestation of the same power issue, but I an not an expert! I'll pass this onto the engineers working on the issue.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.

User avatar
Burngate
Posts: 5467
Joined: Thu Sep 29, 2011 4:34 pm
Location: Berkshire UK
Contact: Website

Re: PoE HAT - USB Ports not working - over-current

Sat Sep 15, 2018 7:23 am

The fault appears to be with the PoE HAT, interacting with the chip on the Pi - any PoE HAT, some Pis.

So if one gets a HaT and it doesn't play ball, there's no point in getting it replaced, yet - the next HAT will be no better.
We'll have to wait for the re-spin.

Is that the state of affairs at the moment?

martinrowan
Posts: 40
Joined: Wed Aug 26, 2015 11:00 am
Location: Berkshire, UK
Contact: Website Twitter

Re: PoE HAT - USB Ports not working - over-current

Sat Sep 15, 2018 10:21 am

Is that the state of affairs at the moment?
Yep pretty much, as the problem is with the PoE HAT and their isn't a new design out yet, you are likely to still have the same problem with any replacement. So better to source a refund, or await the blog post that is apparently coming with some possible soldering suggestions to work around the problem.

If you have multiple RPi3 b+ around, it would be worth testing the HAT on different ones to see if it works with any (hasn't for me).

User avatar
leol
Posts: 147
Joined: Fri Jan 13, 2012 4:27 pm
Location: Haute-Vienne, France

Re: PoE HAT - USB Ports not working - over-current

Tue Sep 18, 2018 3:16 pm

Any news on the promised blog?

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 20698
Joined: Sat Jul 30, 2011 7:41 pm

Re: PoE HAT - USB Ports not working - over-current

Wed Sep 19, 2018 8:59 am

leol wrote:
Tue Sep 18, 2018 3:16 pm
Any news on the promised blog?
We are still investigating the issue - it seems to be a lot more complicated that first thought and we want to make sure we have a decent handle on the situation. We've put a hold on sales for the moment, and we are geting some returned here which show up a particular issue we want to get to the bottom of.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Please direct all questions to the forum, I do not do support via PM.

hippy
Posts: 3882
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: PoE HAT - USB Ports not working - over-current

Wed Sep 19, 2018 12:23 pm

jamesh wrote:
Wed Sep 19, 2018 8:59 am
We've put a hold on sales for the moment
Just to note that while most of the resellers are now "sold out", "awaiting stock" or have removed the product completely, Pimoroni appear to still be offering them for sale -

Edited: Now "out of stock" - I'm guessing someone had words 8-)

I did note the official Raspberry Pi page for the PoE HAT has reverted to "Coming Soon", but couldn't find anything in the blog or elsewhere which details the problem or lets people know they may have purchased Pi PoE HATs which are problematic.


hippy
Posts: 3882
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: PoE HAT - USB Ports not working - over-current

Thu Sep 20, 2018 2:01 pm

enigma1406 wrote:
Thu Sep 20, 2018 6:40 am
Not all vendors are showing out of stock

https://www.alliedelec.com/raspberry-pi ... /71191981/
I'd spotted that one but did not mention it. If you try and buy one it returns "[!] Material number does not exist" and I couldn't find any link or reference to it from their own product pages, only via Google links, so would consider that "no longer available".

Return to “HATs and other add-ons”