Raspberry Pi 4 possible h.265 decoder?

[W. H. Heydt]

Honestly if new hardware were to be planned now, I'd rather the Foundation wait till AV1(https://en.wikipedia.org/wiki/AOMedia_Video_1) is complete. HEVC is a mess due to patent licensing and would lower any profit that can be made with the Pi. VP9 is also at best a stop gap right now due to a flawed standard (All of the bugs currently being addressed in AV1).

I have no doubt that hardware is being planned even as we write these posts. I would be well and truly shocked if the SoC for the Pi4 has not only been already developed, but is in Silicon (even if another iteration or two is stilled needed to have everything work just right). What would be in the works at the SoC level at this point (or very soon) will be the SoC for the Pi5. Planning ahead...it's what the smart companies do.

[W. H. Heydt]

Use A35's in the next Zero/A+/B+, full 64 bit family

It is extraordinarily unlikely that SoCs will be developed specifically for any board other than the projected "top of the line". The follow on for the A+ is already known. It'll be the Pi3A (and there's your A-series 64-bit CPU). The B+ is a legacy device and is unlikely to change. One could see a successor to the Pi2B2 in the form of a de-rated Pi4 SoC 6 to 18 months after a Pi4B comes out. Again, the Pi2B "series" is maintained for compatibility for those that still need it (mostly, I think, because of the change in LED location).

As for the Pi0...No way anyone is going to pour money into developing an SoC *just* for the Pi0. The only thing I can see happening there would be a developmental chip to "get their feet wet" doing 28nm. If a single core 28nm A53 chip were made that could take a PoP RAM package *and* it could be produced cheaply enough, then *maybe* a revised Pi0 would be feasible. And if that happened, one could envision the same SoC being applied to a revised B+ and CM1.

And to get all this back on topic...all of this would come down whether or not such an SoC would include H.265 hardware decode in the upgraded "VC4".

[CarlRJ]

I find it amusing when folks suggest the Pi absolutely has to add X or Y new feature, or it will "lose out" to competitors. The Foundation's goal is education, not to win the PC race. The specs of a typical desktop PC will blow the Pi out of the water, but that doesn't matter - the Pi, as is, is tremendous for education and low-end computing. It's a (very) happy accident that the Pi is also a great fit for the hobbyist market (and that helps channel some funds back into the Foundation to forward their goals). I have a bunch of Pi's running in my home (weather station, home automation, MQTT broker/database server and several clocks), not because they have the best specs, but because there is a huge and thriving culture built up around them, of information and accessories, something which the "competition" can't match.

Predicting that the Pi 4 will arrive eventually is a pretty safe bet. There's no hurry to beat the competition. As several here have said, a Pi 4 that was little more than a Pi 3 with USB3 (thus opening the door to much faster communications and mass storage) would be a huge win, but I presume that once the Foundation has the cover pried off the SoC they'll sprinkle some magic upgrade dust around on the other bits as well. Despite the pleas for improved GPUs, GbE, USB-C, and die shrinks and such, there are a near infinite number of interesting projects that can be done with the Pi hardware as is (and the field has widened even further with the arrival of the ZeroW). If the current Pi's don't meet your needs, then no problem, they're on the low end of the computing/IO scale by design. There are an enormous number of machines available that will work for you in exchange for merely a bit more money. Don't confuse your desire to run a 4K set-top box (or whatever) on the Pi with a need for the Pi to add new features.

[W. H. Heydt]

Predicting that the Pi 4 will arrive eventually is a pretty safe bet.

One to two more years, according to what Eben said last year about this time.

As several here have said, a Pi 4 that was little more than a Pi 3 with USB3 (thus opening the door to much faster communications and mass storage) would be a huge win, but I presume that once the Foundation has the cover pried off the SoC they'll sprinkle some magic upgrade dust around on the other bits as well.

That is my take.

When I got my first look at the original Pi specs, I saw that it could be used for a particular application I had in mind. The original (256MB!) Model B did the job. Those Pis have been since upgraded to Pi2Bs. And they are just fine as they stand. So, yes, I look forward to seeing what the RPF/RPT do with the Pi4, but the Pis that exist now do everything I need them for.

[jdb]

Honestly if new hardware were to be planned now, I'd rather the Foundation wait till AV1(https://en.wikipedia.org/wiki/AOMedia_Video_1) is complete. HEVC is a mess due to patent licensing and would lower any profit that can be made with the Pi. VP9 is also at best a stop gap right now due to a flawed standard (All of the bugs currently being addressed in AV1).

One thing we absolutely have to be sensitive to is market adoption of codecs. For example, HD TV in Germany is planning a switchover to DVB-T2/HEVC sometime next year, with complete transition by 2019. A significant fraction of our export market is the EU (mostly Germany) and if TV in Germany is delivered only via HEVC streams then we're going to have to support it.

HEVC is fundamentally awkward because of the CABAC compression used on the resulting bitstream. The predictive entropy coding means that decompression requires either extremely high single-threaded software decode performance or dedicated hardware, so much so that an A72 at 2GHz will probably not be able to keep up with 1080p60.

We can't wait for arbitrary "better" codecs to become mainstream - if major broadcast networks start using technically inferior but sufficient codecs then we're forced into considering implementing hardware solutions to keep that market segment going.

RPTL's job is double-edged - sell Pis, but maintain the price point and relevance in education.

[bstrobl]

Honestly if new hardware were to be planned now, I'd rather the Foundation wait till AV1(https://en.wikipedia.org/wiki/AOMedia_Video_1) is complete. HEVC is a mess due to patent licensing and would lower any profit that can be made with the Pi. VP9 is also at best a stop gap right now due to a flawed standard (All of the bugs currently being addressed in AV1).

One thing we absolutely have to be sensitive to is market adoption of codecs. For example, HD TV in Germany is planning a switchover to DVB-T2/HEVC sometime next year, with complete transition by 2019. A significant fraction of our export market is the EU (mostly Germany) and if TV in Germany is delivered only via HEVC streams then we're going to have to support it.

HEVC is fundamentally awkward because of the CABAC compression used on the resulting bitstream. The predictive entropy coding means that decompression requires either extremely high single-threaded software decode performance or dedicated hardware, so much so that an A72 at 2GHz will probably not be able to keep up with 1080p60.

We can't wait for arbitrary "better" codecs to become mainstream - if major broadcast networks start using technically inferior but sufficient codecs then we're forced into considering implementing hardware solutions to keep that market segment going.

RPTL's job is double-edged - sell Pis, but maintain the price point and relevance in education.

Who are you going to pay off to not get sued into oblivion? Even Apple retracted HEVC support from their A8 chips and newer, although they are patent holders. Most implementations seem to work with the assumption that any payments to rights holders can simply be delayed or renegotiated in the future. That is not a very good basis to plan a chip on unless you have plenty of transistors that you not care about.

By the way, dvb-t2 has been active since a few days here in Germany, with dvb-t1 already shut off. I don't watch any antenna TV, but the amount of channels available are limited and i believe all the private ones will require a fee for continued usage in the future, hence additional costs for those which many may not be willing to pay.

If HEVC decode is being added, VP9 decode should be lumped in as well as protection against insane patent fees. Pretty much all chipsets nowadays come with both.

[fruitoftheloom]

If HEVC decode is being added, VP9 decode should be lumped in as well as protection against insane patent fees. Pretty much all chipsets nowadays come with both.

How does lack of support for these must haves at any cost effect the stated aims of the Raspberry Pi Foundation ??

The Raspberry Pi Foundation works to put the power of digital making into the hands of people all over the world, so they are capable of understanding and shaping our increasingly digital world, able to solve the problems that matter to them, and equipped for the jobs of the future.

We provide low-cost, high-performance computers that people use to learn, solve problems and have fun. We provide outreach and education to help more people access computing and digital making. We develop free resources to help people learn about computing and how to make things with computers, and train educators who can guide other people to learn.

[Gavinmc42]

And they are just fine as they stand.

Yep, I still have an original model A in the field, still working fine after nearly 5years.
h.265 is totally irrelevant to what I use them for, that said I use them within their limits.
If a Pi4 or 5 or 6 comes with h.265, AV1 or whatever I am sure I can come up with a way to use them

We can speculate and wish for rainbow colored pcbs or Sata or what ever.
We can wish and crystal ball gaze until the cows come home, that's part of the fun too.
But we get what we get and it is up to us to make the Pi's work for what WE want to do.

If someone wants h.265 now, there are plenty of other options.
Don't complain about Pi's because the other options may suck, you cannot blame RPF for Odroid or NEO or xxx.not working like you want.

I am sure RPF know what they are doing hardware wise most of the time, they did pick up a few smart guys along the way.
But they have grabbed a Tiger by the tail, up to them to hang on for the ride.

Will the next Pi do h.265? maybe. Does it need to do h.265? Maybe.
Maybe it will be fast enough to do it in software? maybe Pi5 will do it h.265/AV1 in hardware.
RPF is not telling us and they don't have to say anything, it keeps the other guys guessing too.
The ripple effect of the Pi's on the SBC industry is worth $millions maybe even $billions.

RPF don't have to do anything we want, but they do seem to delivery what we need if we are realistic.
Realistically they ARE making Zero's as fast as they can too

We can make a few guesses based on currently technology.
28nm tick, ARMv? tick, USB3 tick?
28nm means the silicon is smaller, much smaller, that means higher yields per wafer = cheaper chips.
Or stick more stuff in the die, bigger caches, higher clock speeds, more cores?

Will the chip be a Broadcom chip?
I bet most of BCM did not expected to be in the SBC market, with the possible exception of the first sales guy who sold those first 10,000 to Eben.

The hardware is secondary to the aims of RPF, they still have a very, very long way to go in the education market.

[W. H. Heydt]

The ripple effect of the Pi's on the SBC industry is worth $millions maybe even $billions.

By the end of this year, somewhere around an accumulated $0.5 billion, and--at the current rate--upwards of $125 million per year.

28nm means the silicon is smaller, much smaller, that means higher yields per wafer = cheaper chips.

Smaller, yes, Runs on lower voltage, yes, Uses less current, probably (depends on how the clock is pushed). More chips per wafer, yes. Higher yield? Sort of. Might hit the same percentage yield in time, though that is really subject to the experience of the fab more than it is of any skills by the folks at Broadcom. Since this is a shrink to about 2/3 of the prior node, the same design would cover roughly 1/2 the area, so the actual chip count on the wafer will (approx.) double. Therefore if the yield of good chips is over 50% of what it was at the larger process node, then the total yield will be higher, even if the percentage of good chips is lower.

Or stick more stuff in the die, bigger caches, higher clock speeds, more cores?

Bigger caches only helps to the extent that the cores can make effective use of it. Higher clock speed is the most likely "specification" upgrade. However, since Broadcom seems to be pretty conservative, I would expect--at least at first--that they would stick with 1.2GHz to cut down on the thermal issues, and the RPF/RPT would set up non-warranty-voiding overclocks, possibly to the 1.4 to 1.5GHz range. More cores...what would anybody *do* with them? Four cores seems to be the "sweet spot" for the Pi. Adding SoC complexity will drive down yields (and, therefore, drive up cost).

Much of this is why, at least for the first foray into 28nm, I think that the Pi4 SoC will be much the same as the Pi3 SoC. Improve the (very) few things that are crying out to be improved: I/O speed, larger physical address space, thermal issues. Anything beyond that falls in the "nice to have" category.

Will the chip be a Broadcom chip?

I think that is as close to a certainty as one could expect. For one thing, if the RPT gets an SoC from anyone else, there have to be a huge software effort to port to a new boot sequence and work with a different (and completely incompatible) GPU. Just to consider a very, very small issue...What effect would moving to an SoC from anyone other than Broadcom have on the "boot over USB" that we've just gotten for the BCM2837?

[Gavinmc42]

there have to be a huge software effort to port to a new boot sequence and work with a different (and completely incompatible) GPU.

Good point, as a hardware guy I keep forgetting the effort needed to do the software

So die shrink and maybe 1.5Ghz but still run cooler, overclock to 1.8GHz?
VC4 will run faster? USB change to USB-3.

Maybe first pass just try current design on 28nm, no USB 3.
Why do we need USB 3? Don't need it for mouse or keyboard.
Maybe USB 3 mem stick will make for faster USB boot?
Not sure how hard it would be to go USB 3, but since the VC4 needs to be USB aware this could be the stumbling block.
Double the boot rom?

2GB memory I think requires hardware change or cheaper chips?

A die shrunk 3, that could actually be soon, it's been over a year since the 3.
Any other change will take longer, but they did say a year or two.
Split difference is 18months, mid year plus.
Any problems will add months.

Faster ARM cores = h.265 in software? Maybe 1080?
4K? not unless they have been working on it for a year already
Hmm, not much heard from the hardware guys in the news, have they been too busy?

Prediction Pi 4 Oct 2017 28nm, running cooler at 1.6GHz, VC4 bit faster too, no other changes.

[W. H. Heydt]

there have to be a huge software effort to port to a new boot sequence and work with a different (and completely incompatible) GPU.

Good point, as a hardware guy I keep forgetting the effort needed to do the software

Hardware engineers do tend to forget that...and that's behind a lot of issues with SBCs. Even the Pi had teething problems early on (remember the USB issues?), but other board makers, without the resources to throw at the software have struggled and generally not done as well.

So die shrink and maybe 1.5Ghz but still run cooler, overclock to 1.8GHz?
VC4 will run faster? USB change to USB-3.

I would think 1.2GHz with overclocks to--maybe--1.5GHz. At or above 1.5GHz you'd probably be in the mandatory heat sink region. VC4 faster...probably somewhat. The VC4 is heavily tuned to conserve power (and hence stay within TDP limits). I'm *hoping* for USB 3, as I think that is the current bottleneck on the Pi. Even Eben has indicated as much in at least one interview.

Maybe first pass just try current design on 28nm, no USB 3.

Well.... I am given to understand by people actually do chip design that a die shrink is *almost* as big a job as designing a whole new chip. That being the case, making changes to alleviate known bottlenecks would--at least--be very tempting and one would hope that swapping USB 3 in for USB 2 would be, not trivial, but close enough to a standard operation as not to affect the schedule very much.

Why do we need USB 3? Don't need it for mouse or keyboard.
Maybe USB 3 mem stick will make for faster USB boot?

USB 3 wouldn't be for speeding up boot (in the general case, speeding up the SD card interface would have more effect there), but for those who have network intensive tasks (and that would also cover...streaming H.265), it would make GbE possible, even if that isn't part of the design (so long as there is a USB 3 equivalent to the LAN9514 chip). For the subset of use cases that use SSDs, HDDs, or even USB sticks for mass storage, you could go (on an HDD) from 30MB/s transfer rates to 80MB/s rates (that's from tests I've run using a Pi3B and a Roseapple Pi, which has a USB 3 port). With SSDs, you can get up around 125MB/s transfers. On a Pi1, USB 3 isn't all that useful because there isn't enough compute capacity to *need* it. It's the faster CPUs in the Pi3 where the slow external I/O starts choking off the CPUs.

Not sure how hard it would be to go USB 3, but since the VC4 needs to be USB aware this could be the stumbling block.
Double the boot rom?

I'm not sure either. I don't know how different the VC4 would have to be as it would be looking at the internal bus.

2GB memory I think requires hardware change or cheaper chips?

Yes. There would have to be a change to the VC4, which can only address 1GB and--perhaps--cheaper RAM packages. There are SBCs with 2GB RAM for prices pretty close to the Pi, so the RAM can't be all that much more expensive, but the design of the Pi is extremely sensitive the price of its components. If it were up to me (which it isn't), I would try to design the memory interface to be able to handle at least 4GB, and--if I could get away with it--more, perhaps as much as 16GB. Note that is in no way a suggestion of a 4GB (or more) Pi, but a way not to have to redesign the memory bus circuits for a good ways into the future. For the initial builds, I would expect the RPT to stick with a 1GB board and increase the RAM as pricing and availability permitted, just as the Pi1 started with 256MB (and the Model A was originally slated to have 128MB!), and later got 512MB.

A die shrunk 3, that could actually be soon, it's been over a year since the 3.
Any other change will take longer, but they did say a year or two.
Split difference is 18months, mid year plus.
Any problems will add months.

Last year, when the Pi3B was launched, Eben said not to expect a new model for "2 to 3 years", so--yes--another 1 to 2 years. I would expect that the work on the "Pi4" SoC was probably at least in the planning stages at least 2 years ago. So figure 4 to 5 years from concept start to board. Note that it took about 7 years to go from the beginning of the Pi effort until boards went up for sale, and 4 years from launch until the Pi3B. So a 5 year development cycle starts to look what passes for normal.

Looking farther out, there is probably already ground work laid for whatever eventually becomes the Pi5.

Faster ARM cores = h.265 in software? Maybe 1080?
4K? not unless they have been working on it for a year already
Hmm, not much heard from the hardware guys in the news, have they been too busy?

I would guess that there are a lot of "irons in the fire". Don't forget that within the last year we've had the A+512MB, Pi2B2, CM3, CM3L, CMIO3, and the Pi0W, with the Pi3A still to come. So...yeah...I think the hardware guys have been pretty busy. And again, I think the next SoC has been in the works for at least 2 years. Whether or not it will have VC hardware blocks for H.265...dunno. I wouldn't rule it out. I also wouldn't rule out that it might be an individual license deal like unlocking existing hardware decoders.

Prediction Pi 4 Oct 2017 28nm, running cooler at 1.6GHz, VC4 bit faster too, no other changes.

I may be too conservative, but I would predict Pi4B in February 2018, 28nm, running cooler at 1.2GHz, permitted overclock to 1.4GHz within warranty, USB 3, 1GB RAM (but able to address more, without telling us), modestly improved VC4.

[Gavinmc42]

USB 3 the must have for future I/O speed?
No 1Gbs Lan9514, but there is a USB3 LAN7801 but it has no USB hub.
So maybe two USB ports USB3 and USB2?
Anything with a Linux driver can be recompiled for ARM.
So perhaps a LAN/Hub chip change?

Not keen on RJ45 sockets for Lan, they seem so big these days.
USB-C dongle, With Wifi on board do we need Lan jack too?

[quote]but other board makers, without the resources to throw at the software have struggled and generally not done as well.
[/quote]
Yep support from the Chinese for clone/copies, good luck with that.
Allwinner got lucky because Olimex broke the ground for them.
Olimex is mostly a hardware company where RPF is software focused, the difference shows in the numbers.
Not that Allwinner should complain they are in most other CE SBC, tablet etc.

You are probably closer with the dates, I am wishful thinking.

[AntonAV]

USB 3 the must have for future I/O speed?
No 1Gbs Lan9514, but there is a USB3 LAN7801 but it has no USB hub.
So maybe two USB ports USB3 and USB2?
Anything with a Linux driver can be recompiled for ARM.
So perhaps a LAN/Hub chip change?
Not keen on RJ45 sockets for Lan, they seem so big these days.
USB-C dongle, With Wifi on board do we need Lan jack too?
.

Schools use RJ45 sockets as they beat Wifi in many fields (through put, number of different users, radiation), so please keep them. Raspberry Pis are anyway not designed to be mobile devices so they will be multi wire connected anyhow.
<= 28 nm ARM die: yes of course. Even sell the Pi standard with passive heat sink (s) on the hottest component(s), like the Odroid C2. These two measures can enable a higher standard clock speed and thus improve software video handling.
For the vital GPU / boot chip: if that too would go <= 28 nm there would be more space for good Ethernet / SD card less booting (1st in boot order from this chip!) plus some video tweaks. Extra goodies could be dedicating the existing bus to Ethernet alone and adding a USB 2 / 3 bus, apart from increasing the RAM capacity from 1 to 4 GB max.

[jamesh]

This thread has gone all wish list which is waste of bandwidth. Suffice to say, RPF has a roadmap, and is run by people who have a lot of experience in chip design and manufacture. Who have also sold 13m devices​ to become the third best selling computer brand ever. So we know what we are doing, we know what the overall market wants, and we know what to do. Some will coincide with what people have been asking for, some will come as a surprise, but it's all pretty cool.