SD Card performance in R-Pi onboard slot

[selsinork]

I should start off by saying that I'm not really interested in theoretical speeds, class 4 vs 6 vs 10 issues, or benchmarks in windows.

What I am interested in is actual card performance in the onboard sdcard slot on a R-Pi.

I noticed that all of the different cards I have when tested with a simple, highly unscientific, sequential read test like:

dd if=/dev/mmcblk0 of=/dev/null bs=1M count=100

topped out at just under 6MB/s which seemed rather strange and somewhat slower than expected.

Putting the same cards into a usb-sd adapter and plugging that into my Pi and doing the same test (with mmcblk0 changed to sda) gives a read speed of just under 20MB/s on most cards, or a roughly 3-4 times improvement.

Write speeds seem similarly limited to around 6MB/s on cards that will manage 10-12MB/s in the same usb adapter

So there's some evidence that the cards are capable of a lot more than I can get in the onboard slot.

Opening up the USB reader and using an oscilloscope to probe the sdcard CLK line shows it driving the card at ~50Mhz.

Doing the same on the R-Pi onboard slot shows it only driving the card at 10Mhz or 12.5Mhz depending on the card.

The kernel driver seems to suggest 20MHz as the maximum, and yet the peripherals data sheet suggests higher speeds could be possible. page 82 for anyone still reading

So the question is, has anyone with an R-Pi been able to get > 6MB/s sequential read from a sd card in the onboard slot ?

Searching the various threads, I can't find anyone who has..

Probably only someone from the Foundation with inside h/w knowledge can answer this one, but is there a reason for driving the onboard slot this slowly ?  Bug ?  H/W limitation ? EMC consideration ?  Nobody's wrote that bit of s/w yet ?

[AlexPT]

Good point here.

But that not the most important "speeds"… witch is random 4k/512k read/writes.

Would be nice to have a topic whit the testes on the R-Pi, whit sequential and random read/writes.

[mahjongg]

True, but such a test would not test the card slot interface hardware, but mainly the SD-card itself.

But yes, testing random read/write speeds of several cards is worth of a topic in itself (testing different cards against each other), but hardware interface speed doesn't have much to do with how well cards handle different types of uses.

This topic is about why the interface, which is supposed to be a fast (nibble mode interface) is so slow (if its really slow that is). Maybe there is something the SD-card driver developers overlooked, or could improve, for example they made the decision to throttle the communication speed to an overly conservative value.

[selsinork]

? l ? x said:


Good point here.

But that not the most important "speeds"… witch is random 4k/512k read/writes.


While I agree that random r/w is important....  my point was more that it appears that communication with the card is slower than it could be.  It's easy to notice and replicate with simple sequential reads.

If we could talk to the card at 50Mhz then maybe we get to a point where we've removed some of the latency and maybe that helps improve the random r/w numbers too.

However if there's a hard limitation on the speed, it's a moot point.  So I left it there.

Answers to the first question always lead to the more interesting questions later

[jbeale]

Great research! If I had to guess, it would be that the firmware uses conservative settings which are more likely to work with more different types of cards. But as I recall, the board does not work with many "Class 10" SD cards due to an unsupported voltage signalling level(?) In other words, there may be a hardware difference with the USB reader, it may be capable of a read mode involving different voltage levels that the SoC does not support.

[lb]

The standard SD interface that runs at 3.3V tops at 50 MHz clock speed, for about 200 Mbps raw bandwidth. So the I/O voltage is not an issue here. 1.8V I/O voltage is only required for SD 3.0 / UHS.

[JonB]

I'm keen to see this thread develop. Maybe we can ask Eben / Gert about it? I know zero about SD interfaces but I'm definitely interested in anything that will improve the overall performance of the Pi. I don't know if any of the standard distros have or expect swap partitions on the card, but if they do, we need it to run fast. Since it only has 256MB of RAM, swap is likely a distinct requirement for certain applications (especially dev toolchains like g++ which - let's face it - is what the Pi is supposed to be for).

I wonder how long it takes to compile the kernel? That would be a very good comparative test. I'm certainly not going to be using a cross compiler on a bigger machine - where is the fun in that?

[selsinork]

JonB said:


I wonder how long it takes to compile the kernel?


It takes a while, I'll time it later today. Even a very cut down config takes several hours.

Reminds me of doing linux compiles on a floppy with a slow 386 back around 92 or so, and the kernel has got a lot bigger since then

However, I wouldn't expect a lot of people to do it, apart from the sort of techies who like to tinker with this stuff

[rew]

Count me in: I'm interested in this as well. I've measured 5.5 Mb/sec max on the internal SD card, and something like 10Mbyte per second on an USB SD reader. That reader gets 16Mbyte per second on my PC.

(on the other hand, as long as people are having reliability problems with the internal SD card, keeping the settings "conservative" is a good plan... )

[selsinork]

rew said:

(on the other hand, as long as people are having reliability problems with the internal SD card, keeping the settings "conservative" is a good plan... )
Absolutely, and I have no problem with conservative settings if that's the reason, or if it's EMI or any other sensible reasons.

I had a very crap card at first, ~500kB/s read and write which made me somewhat sensitive to the card being the bottleneck, but even with a better card it still felt 'slow' somehow.

Hence starting to investigate and starting this thread.  So it's good to get confirmation from someone else who has one that they see the same as I do, removes the 'is it just me?' question.

[jamesh]

I did ask Dom about this - he was of the opinion that whilst we won't be getting the 50Mhz mentioned we shoud be bit faster than the numbers reported.

I'll try and remember to do a bit more investigation on Monday.

[selsinork]

Thanks James !

[GrahamC]

For anybody who wants to test their SD card to make sure what is written to it is read back identically this program (F3) is quite useful. It is meant for detecting fake cards (those formatted with a false size) but it also makes a good SD card tester/exerciser. It also displays estimates of the device's write and read speeds.

http://oss.digirati.com.br/f3/

It should compile and run on any Linux system. To change the size of each file created modify line 368 of  f3write.c

fine = create_and_fill_file(path, i, GIGABYTES, &fw);

[GrahamC]

It does completely fill the card's file system, so I don't know if that will cause a problem when the card is the root file system.

[Max]

Was wondering if you also measured the frequency the bootloader uses to access the card, before the Linux kernel is started?

[FreakHavoc]

Nice job researching this, selsinork I'm going to watch this thread.

[mahjongg]

rew said:


Count me in: I"m interested in this as well. I"ve measured 5.5 Mb/sec max on the internal SD card, and something like 10Mbyte per second on an USB SD reader. That reader gets 16Mbyte per second on my PC.

(on the other hand, as long as people are having reliability problems with the internal SD card, keeping the settings "conservative" is a good plan… )


I'm wondering if its true that there are many reliability problems with cards, I have designed some SD-Card interfaces myself, (using only the SPI mode mind you) and learned that especially the switching from low speed (300 KHz) initialization and resetting mode to "full speed (SPI) working mode" can be quite tricky. And I presume that using "nibble mode" may have its own pitfalls. But anyway, I learned that almost all SD-cards (but not the early MMC cards) can clock data in at at least 20MHz, in nibble mode that means at least 10MBps.

Perhaps there needs to be done some final software tweaking with many different cards to make sure these all initialize and work well. I presume what is left are the cards that at one time or another want to have 1.8Volt as their operating voltage, and the current interface is hard-wired for 3,3V card operation only. That can be fixed by a re-design, but in the mean-time I understand that its possible to ignore the cards wishes for a 1V8 supply and keep working at 3V3 (at a lower than possible speed).

Perhaps the software can aquire some "smarts" to determine the capabilities of the card, so it can handle each kind of card at its maximum potential.

[lb]

I'm intrigued by these SD performance issues, too. The SDHCI driver limits the clock to 20 Mhz by default, and clock selection is also restricted to be a divider of the eMMC base clock, which is 80 MHz by default. But still, with 4-bit transfers at 20 MHz, I'm only getting about 4.5 MB/s for sequential reads, while at this clock I should be getting almost 10 MB/s. The same SD card achieves over 17 MB/s in a USB card reader!

Anyway, I raised the eMMC base clock to 100 MHz (using the init_emmc_clock option in config.txt) and forced 50 MHz clock in the driver (by hacking the set_clock function in sdhci-bcm2708.c), and this worked fine for me. Now I'm seeing solid 10.5 MB/s for sequential reads. This is definitely not ideal, but it's quite a bit more like it.

I'd really like to know why the actual bandwidth is so much worse than the interface speed, though. dom? jamesH? anyone?

[aocallwill]

Well I'm not a chip expert, but it seems that the VideoCore IV has a USB core built in (Synopsys apparently, see also http://www.raspberrypi.org/wp-content/u ... herals.pdf page 200)

Maybe this is what is causing the performance issues with the SD card performance?

[selsinork]

Well I'm not a chip expert, but it seems that the VideoCore IV has a USB core built in (Synopsys apparently, see also http://www.raspberrypi.org/wp-content/u ... herals.pdf page 200)

Except that the onboard SD card interface uses the Arasan eMMC controller, not USB (http://www.raspberrypi.org/wp-content/u ... herals.pdf page 65)

So I'm not sure what you're getting at... Indeed, if you plug a usb SD Card reader into the Pi you see much better performance, but you can't boot from it so it's somewhat irrelevant.

[lb]

Alright, I've applied an upstream fix for proper handling of SD 3.0 cards now, and patched the Broadcom eMMC driver to take advantage of the full clock range. No hacky, nasty clock forcing necessary anymore. Here's the code and I prepared a pre-built, compressed kernel.img.

If you feel adventurous, extract the file, backup and overwrite your current kernel.img, add "init_emmc_clock=100000000" to config.txt and reboot. If everything worked out correctly and your Pi still boots, "hdparm -t /dev/mmcblk0" should now report a sequential read rate of 10 MB/s, or more.

I'm really interested in some feedback, so please test - I've been running with high-speed SD clock for two days now, and everything has been perfectly stable.

[jbeale]

Great work! I don't have my Pi yet, but I'm looking forward to hearing how this works. Doubling the speed of the SD card interface must be good for some noticeable improvement, I would think.

Note: the link to the source code at https://bitbucket.org/grigorig/raspberry-linux just gives me an error message.

[shirro]

The sd card performance is really weird.

The same model of sd card reports 15MB/s read with hdparm on one arm board and less than 5 on the Pi. I tried the patches above but they didn't work for me. Bumping up init_emmc_clock seems to work nicely though and I am 2/3 of the way there at 10MB/sec at 200000000 and wondering how high I should go? How much of the performance is due to the clock and how much is some other fault in the driver?

[shirro]

It turns out I probably can't push my card much over hdparm of 10-12MB/sec as I bumped up a bit higher and it panicked. I am guessing there is a fair bit more to the performance disparity than just the clock speed and I am probably running it well out of spec (can this kill an sd card?). If things like this can be fixed the Pi will be a hell of a lot more computer than it is at present.

[whirpool]

I'm really interested in some feedback, so please test - I've been running with high-speed SD clock for two days now, and everything has been perfectly stable.

Heya! Thx for the effort, but sadly it's making things even worse for me.
With your kernel and variable set in config.txt r/w speeds went from 4.5MBs down to 1.5MBs

The card in use is an class 10 SDHC which reports as "mmcblk0: mmc0:b368 SMI 15.4 GiB"

But much alike shirro i played around with "init_emmc_clock" and got it stable with a value of 250000000.
Inserted into my laptop i get r/w-speeds of 21/11, with pi default settings ~4.5/4.5 and now ~11/13. Seems like i maxed out the write speed while it reads only at half speed.
Anyhow.. the performance gained is huge and the system presents itself much snappier now.

Here the benchmarks before and after:

[email protected]:~$ sudo hdparm -tT /dev/mmcblk0
/dev/mmcblk0:
Timing cached reads: 148 MB in 2.02 seconds = 73.36 MB/sec
Timing buffered disk reads: 14 MB in 3.15 seconds = 4.45 MB/sec
Timing cached reads: 174 MB in 2.01 seconds = 86.53 MB/sec
Timing buffered disk reads: 40 MB in 3.09 seconds = 12.96 MB/sec

[email protected]:~$ sudo hdparm -tT --direct /dev/mmcblk0
/dev/mmcblk0:
Timing O_DIRECT cached reads: 10 MB in 2.19 seconds = 4.57 MB/sec
Timing O_DIRECT disk reads: 14 MB in 3.06 seconds = 4.57 MB/sec
Timing O_DIRECT cached reads: 28 MB in 2.03 seconds = 13.83 MB/sec
Timing O_DIRECT disk reads: 42 MB in 3.04 seconds = 13.81 MB/sec

[email protected]:~$ dd if=/dev/zero of=tempfile bs=1M count=512 conv=fdatasync,notrunc
512+0 records in
512+0 records out
536870912 bytes (537 MB) copied, 130.234 s, 4.1 MB/s
536870912 bytes (537 MB) copied, 50.1165 s, 10.7 MB/s

[email protected]:~$ echo 3 | sudo tee /proc/sys/vm/drop_caches 3

[email protected]:~$ dd if=tempfile of=/dev/null bs=1M count=512
512+0 records in
512+0 records out
536870912 bytes (537 MB) copied, 121.065 s, 4.4 MB/s
536870912 bytes (537 MB) copied, 42.5498 s, 12.6 MB/s

cheers,
whirpool