mjohnst
Posts: 1
Joined: Mon Aug 29, 2011 8:24 pm

Re: Additional RAM from USB?

Mon Aug 29, 2011 9:41 pm

Would it be possible/worth it to add additional RAM by attaching a USB Flash Drive to either Model (mostly interested about B).

User avatar
Lob0426
Posts: 2198
Joined: Fri Aug 05, 2011 4:30 pm
Location: Susanville CA.
Contact: Website

Re: Additional RAM from USB?

Mon Aug 29, 2011 10:09 pm

It is possible but there are a number of considerations that make it unlikely. Creating a swap file (virtual memory) on Flash will destroy the drive in time. There are an infinite number of READ cycles but a finite number of WRITE cycles for flash memory.

Check out this thread to see the details.
http://www.raspberrypi.org/?pa.....#038;t=117
512MB version 2.0 as WordPress Server
Motorola Lapdock with Pi2B
Modded Rev 1.0 with pin headers at USB

http://rich1.dyndns.tv/
(RS)Allied ships old stock to reward its Customers for long wait!

obarthelemy
Posts: 1407
Joined: Tue Aug 09, 2011 10:53 pm

Re: Additional RAM from USB?

Tue Aug 30, 2011 4:50 am

RAM and a flash drive are not the same thing. a flash drive is more like a hard disk.

So.. No.

Svartalf
Posts: 596
Joined: Fri Jul 29, 2011 6:50 pm

Re: Additional RAM from USB?

Tue Aug 30, 2011 1:31 pm

Heh... You could design something that did a RAM drive on a USB "stick" that would work after a fashion. Nothing out there will do it, mind. I'll make the following observations on this subject:

The performance of the stuff will be similar to what you'd get over USB on a PC.
Flash thumbs, etc. have wear leveling. This means that while it IS correct that they have a limited write endurance, it typically means that it will simply last 2-3 years in service as a swap device. SSD's, etc. can go 10+ years with the current tech with 2Gb per day constant write load on them...
If you DID come up with a battery backed (I'd go that route...) RAM thumb, it will be faster than the Flash ones and last indefinitely...
At 128-256Mb of RAM, if you DID add a comparable swap partition/file on a Flash or RAM thumb, so long as you didn't try to hit the USB hard, you'd have a higer level of interactivity with a few more apps because of the swap...

Archangel5089
Posts: 36
Joined: Mon Aug 15, 2011 8:37 am

Re: Additional RAM from USB?

Tue Aug 30, 2011 1:57 pm

I think the wear and tear point is kind of moot here... How much extra memory are you going to want, really? 2GB-4GB before it becomes overkill? You can pick up a stick with a decent IO speed for less than £10 and it will last 2-3 years which sounds like a bargain to me!

However... You already have an SD card that you are running the system off, why don't you just get a bigger card and create a Swap partition on that? Hmmmm, now I think about it that might be a more expensive option...

Archangel5089
Posts: 36
Joined: Mon Aug 15, 2011 8:37 am

Re: Additional RAM from USB?

Tue Aug 30, 2011 2:03 pm

Quote from obarthelemy on August 30, 2011, 05:50
RAM and a flash drive are not the same thing. a flash drive is more like a hard disk.

So.. No.

Search for ReadyBoost... If they were doing it in Vista do you really think it hasn't been done in Linux first? ;)

Anyway, this link sums up the argument for it being a waste of time unless you don't have much RAM to begin with... Oh wait! :o

http://ubuntuforums.org/archiv.....86870.html

willwill017
Posts: 1
Joined: Tue Aug 30, 2011 1:28 pm

Re: Additional RAM from USB?

Tue Aug 30, 2011 2:32 pm

how do hook up pis together to make a super computer

ShiftPlusOne
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 6183
Joined: Fri Jul 29, 2011 5:36 pm
Location: The unfashionable end of the western spiral arm of the Galaxy

Re: Additional RAM from USB?

Tue Aug 30, 2011 2:38 pm

duct tape

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 25002
Joined: Sat Jul 30, 2011 7:41 pm

Re: Additional RAM from USB?

Tue Aug 30, 2011 2:51 pm

Quote from Archangel5089 on August 30, 2011, 14:57
I think the wear and tear point is kind of moot here... How much extra memory are you going to want, really? 2GB-4GB before it becomes overkill? You can pick up a stick with a decent IO speed for less than £10 and it will last 2-3 years which sounds like a bargain to me!

However... You already have an SD card that you are running the system off, why don't you just get a bigger card and create a Swap partition on that? Hmmmm, now I think about it that might be a more expensive option...

That's how the current file system is set up - three partitions on the SD card, boot, rootfs and swap. Boot is FAT so you can add data to it from a Windows PC, the other two are EXT3, to access them you need a Linux box (or as I just did, SCP'ed from desktop to board)

I'm using a 4GB card., but the same on the 2GB build.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
“I own the world’s worst thesaurus. Not only is it awful, it’s awful."

User avatar
RobinJ
Posts: 68
Joined: Mon Aug 08, 2011 4:21 pm
Location: Belgium
Contact: Website

Re: Additional RAM from USB?

Tue Aug 30, 2011 3:24 pm

Linux allows you to do this ;) Just format the USB drive as a swap filesystem and swap it on :p
Rule #1: If a human tries to argue with a logical device like a computer, the device always wins.
Rule #2: A big bucket of water, window, or hammer can nullify Rule #1.

User avatar
Lob0426
Posts: 2198
Joined: Fri Aug 05, 2011 4:30 pm
Location: Susanville CA.
Contact: Website

Re: Additional RAM from USB?

Tue Aug 30, 2011 4:17 pm

There is limited number of P/E (Program/Erase cycles) cycles for each type of flash memory. How fast this limit is going to be reached depends on type and use of the flash. Most SD cards are rated at about 10,000 P/E cycles. So the question becomes how hard are you going to use your swap file? If you overload memory on the RasPi you will be replacing that SD card frequently, not 2 to 3 years. If you are circumspect at how you load the RasPi the SD Card will last a good long time, as it will use the swap file infrequently. When advising people about a swap file based on flash it is best to weigh on the conservative side of the argument. If not what will happen is "you told me it would be ok to fill memory as full as I wanted because the swap file is there". When I am asked I lean towards a USB external HDD. Remember the Model A has half the memory of Model B also when you give advice. Model A is that much more likely to use the swap file.

I never said you can not do it, because I have. ReadyBoost is not a swap(virtual memory) file. It is a cache file. Loads of difference for the discussion here. The number of reads from flash is nearly infinite.
Wikipedia:
===Write endurance===
The write endurance of SLC floating-gate NOR flash is typically equal or greater than that of NAND flash, while MLC NOR and NAND flash have similar endurance capabilities. Example Endurance cycle ratings listed in datasheets for NAND and NOR flash are provided.
*SLC NAND flash is typically rated at about 100k cycles (Samsung OneNAND KFW4G16Q2M)
*MLC NAND flash is typically rated at about 5–10k cycles (Samsung K9G8G08U0M)
*SLC floating-gate NOR flash has typical endurance rating of 100k to 1M cycles (Numonyx M58BW 100k; [[Spansion]] S29CD016J 1,000k)
*MLC floating-gate NOR flash has typical endurance rating of 100k cycles (Numonyx J3 flash)

However, by applying certain algorithms and design paradigms such as wear leveling and memory over-provisioning, the endurance of a storage system can be tuned to serve specific requirements.<ref>[http://www.wdc.com/WDProducts/SSD/white ... n_0812.pdf] Western Digital White Paper describing calculation and effects of SSD endurance</ref>
512MB version 2.0 as WordPress Server
Motorola Lapdock with Pi2B
Modded Rev 1.0 with pin headers at USB

http://rich1.dyndns.tv/
(RS)Allied ships old stock to reward its Customers for long wait!

obarthelemy
Posts: 1407
Joined: Tue Aug 09, 2011 10:53 pm

Re: Additional RAM from USB?

Tue Aug 30, 2011 5:08 pm

- ReadyBoost is a persistent disk cache, where Windows "intelligently" copies lotsa small files, to take advantage of flash's faster access times, while not being penalized by its lower throughput (over USB2, compared with sata1/2/3). It's certainly not RAM. The Ubuntu thread actually has that info. ReadyBoost, as the name implies, speeds up booting and launching apps.

- Swap space is almost-ram, but if you reach the "churning" point when you're needing so much RAM that the OS is pretty much spending all its time swapping code/data in and out of RAM to execute/work on it, and then swapping again , and then again...the performance drops so badly the computer becomes unusable. That's why swap is not equal to RAM. Swap (to simplify) works well when you're launching plenty of programs, but using one at a time. It doesn't work well if you're using only one program, with a huge dataset, that doesn't fit in RAM. It'll try to swap out part of the OS, of the program, of the data, and end up un-swapping and trying to swap some other parts instead. rinse, repeat, don't get any work done.

PS: the big question is, and has been for a while, why MS have not adapted ReadyBoost for SSDs. That would seem a very smart why to get the proverbial 80% benefit out of the 20% expense of buying a very small SSD, as opposed to a large one.

User avatar
Lob0426
Posts: 2198
Joined: Fri Aug 05, 2011 4:30 pm
Location: Susanville CA.
Contact: Website

Re: Additional RAM from USB?

Tue Aug 30, 2011 6:23 pm

ReadyDrive is a window 7 feature for hybrid drives, not sure it works on SSD's but it wouldn't really be needed on a SSD. You really do not need ReadyBoost if your primary drive is a SSD. The flash on a SSD is faster than the flash on say a USB drive.

obarthelemy: I think that explains it better than I did. You hit it on the head about being a slave to your hard disk when you rely to heavily on your swap file. I have used and worked on numerous systems that their main performance problem was lack of memory. Anytime I see the HDD light going constantly I recommend some more memory, if I can not find another reason for excessive disk use other than virtual memory. There are other strategies that work as well. I remember paying $200+ for 4MB, yes 4MB of RAM and over $900 for 16MB of SIMM (Single Inline Memory Module).

Using a swap file from the SD card or USB flash does work, just be sure you understand it's limitations.
512MB version 2.0 as WordPress Server
Motorola Lapdock with Pi2B
Modded Rev 1.0 with pin headers at USB

http://rich1.dyndns.tv/
(RS)Allied ships old stock to reward its Customers for long wait!

Archangel5089
Posts: 36
Joined: Mon Aug 15, 2011 8:37 am

Re: Additional RAM from USB?

Wed Aug 31, 2011 8:52 am

Quote from jamesh on August 30, 2011, 15:51
Quote from Archangel5089 on August 30, 2011, 14:57
I think the wear and tear point is kind of moot here... How much extra memory are you going to want, really? 2GB-4GB before it becomes overkill? You can pick up a stick with a decent IO speed for less than £10 and it will last 2-3 years which sounds like a bargain to me!

However... You already have an SD card that you are running the system off, why don't you just get a bigger card and create a Swap partition on that? Hmmmm, now I think about it that might be a more expensive option...

That's how the current file system is set up - three partitions on the SD card, boot, rootfs and swap. Boot is FAT so you can add data to it from a Windows PC, the other two are EXT3, to access them you need a Linux box (or as I just did, SCP'ed from desktop to board)

I'm using a 4GB card., but the same on the 2GB build.

Actually the Swap is optional, usually there's no good reason to leave it out but if your SD is only 2GB then I figured it might be something people consider...

I installed Mandriva without a Swap, lasted 3 hours before it annoyed me and I re-installed it with one again!

Archangel5089
Posts: 36
Joined: Mon Aug 15, 2011 8:37 am

Re: Additional RAM from USB?

Wed Aug 31, 2011 9:15 am

Quote from obarthelemy on August 30, 2011, 18:08
- ReadyBoost is a persistent disk cache, where Windows "intelligently" copies lotsa small files, to take advantage of flash's faster access times, while not being penalized by its lower throughput (over USB2, compared with sata1/2/3). It's certainly not RAM. The Ubuntu thread actually has that info. ReadyBoost, as the name implies, speeds up booting and launching apps.


My point was that you were arguing semantics, RAM and caching provide the same function and this thread is really about adding memory to speed up access (methodology irrelevant) to the R-pi. Swap isn't true RAM either technically but it'll do the job. Your original comment wasn't helpful but this one is.

I'm guessing MS have never bothered much with boosting from SSDs because most people in the market for SSDs will be putting them in good computers. Maybe they don't see the profit in targeting hobbyists?

obarthelemy
Posts: 1407
Joined: Tue Aug 09, 2011 10:53 pm

Re: Additional RAM from USB?

Wed Aug 31, 2011 9:30 am

Nope, RAM and caching don't provide the same function, read the bit about churning. that's why semantics are important: actually understanding what the question is does help answering it.
I'm still not sure if OP, who never chimed back, actually wants more RAM for large data/code, or more storage, or more performance... I have no clue, indeed, what "speeding up access to the RPi" means. Sounds like a pre-order question ^^

Archangel5089
Posts: 36
Joined: Mon Aug 15, 2011 8:37 am

Re: Additional RAM from USB?

Wed Aug 31, 2011 11:39 am

Semantics:
An L2 processor cache provides the same function as RAM. They store data, same function. If you want to be more specific then yes, they are different and the debate is pointless because we are not arguing at the same level of meaning so let's move on...

Churning:
I haven't considered the limited ram, churning point will happen a lot sooner; are you basically saying that there is no point in a swap partition or that there's no point in one on such a small RAM set? Hell, will accessing a swap partition on the SD card really be any faster than just accessing the data on the SD card in the first place?

Virtual Memory:
Churning aside is swap partition is still the best bet to add virtual memory?

obarthelemy
Posts: 1407
Joined: Tue Aug 09, 2011 10:53 pm

Re: Additional RAM from USB?

Wed Aug 31, 2011 12:52 pm

A processor cache (L1, L2 or L3, by the way this was never mentioned in this thread and is not the subject at all, unless I'm mistaken, we were talking about the ReadyBoost disk cache) does not provide the same function as RAM: it duplicates what already is in the RAM to make access to that faster. It does not expand the RAM nor is interchangeable with it. The same way a ReadyBoost cache is not extra hard disk space, it's just a faster copy of what's on the HD.

As I said, swap in useful in some cases, useless in others. The alternative, directly accessing data on-disk, requires rewriting the applications to do that. Swap is transparent, as the application does not need to be aware of whether the data (or the code for that matter, swap also swaps code) has been swapped out, the OS handles that transparently. If you know for sure you're gonna be swapping out, and can change the code, it might make sense to keep the data on-disk and access it there. But it really depends on what your data access patterns are: if you have a huge dataset, but are only working on small parts of it a time, and making lots of consecutive changes to that same data, you might be better off betting the OS will pull up your "active" data in RAM, and let you make your changes in RAM very quickly; leaving the data on-disk means a very slow disk write for every change. Also, if your "swapping" software later moves on to a less RAM-starved computer, it will automatically use that extra RAM. We might complicate the issue by adding the disk cache into the mix, but we won't !

Regarding virtual memory, a swap partition is pretty much the *only* way to add virtual memory. The question is not whether it is the best way or not, but whether it will work at all, and again that depends on whether churning happens or not.
In any case, I'm assuming SD card access will be much faster (both latency and throughput) than USB disk access, so the SD card is probably the right place to put the swap partition. Plus it solves concurrency issues, since it is my understanding that while USB ports and the Ethernet port will be competing for bandwidth, the SD card has its own dedicated channel to he processor. And the overhead for SD is lower than for USB. there are several SD card speeds available, so paying a little extra for a higher "class" might be worth it. I don't know how high the RPi can go, though.

Svartalf
Posts: 596
Joined: Fri Jul 29, 2011 6:50 pm

Re: Additional RAM from USB?

Wed Aug 31, 2011 1:04 pm

Quote from willwill017 on August 30, 2011, 15:32
how do hook up pis together to make a super computer


Ethernet. You might want to check in on the Bramble discussion over in the collab area. ;)

Timo
Posts: 8
Joined: Sat Aug 27, 2011 9:04 am

Re: Additional RAM from USB?

Wed Aug 31, 2011 1:35 pm

I think having some extra swap space (or place for /tmp) not on the SD is nice to have. But as mentioned earlier flash-sticks have a limited number of writes. So I searched around for an USB device with volatile memory (DRAM or similar), but I just found a SATA version, which get close: http://techreport.com/articles.x/16255
Further search only delivered a bunch of posts about whether there is a use case for that or not.

Such a device could be delivered as an extension module for the r-pi. How much bandwidth do the gpio (or other connectors) give? Connecting through these extra pins would avoid sharing bandwidth with ethernet and usb devices.
It could be realized as a simple version of a SoC with PoP-RAM as the r-pi itself has. Hence the extra costs of 10$ for model B splits to 128MiB RAM and the LAN chip, 128MiB on a very simple SoC shouldn't cost that much (given there is a SoC which simply forwards the RAM as a Ramdrive).

Svartalf
Posts: 596
Joined: Fri Jul 29, 2011 6:50 pm

Re: Additional RAM from USB?

Wed Aug 31, 2011 5:14 pm

Quote from Timo on August 31, 2011, 14:35
I think having some extra swap space (or place for /tmp) not on the SD is nice to have. But as mentioned earlier flash-sticks have a limited number of writes. So I searched around for an USB device with volatile memory (DRAM or similar), but I just found a SATA version, which get close: http://techreport.com/articles.x/16255
Further search only delivered a bunch of posts about whether there is a use case for that or not.

Most flash based devices have wear levelling these days. It's cheap enough to produce on the varying on-board controllers for the devices that you're going to have something REALLY old or really, really cheap (read: Do not use...ever...) these days.

Since that's the case, you're going to go 3-ish to 5-ish before it might become a bit of a problem with SD's or with any USB thumb you can purchase these days that's not a dodgy thing to use.

Having said this...if you could find someone that could make a 512-2048MB SRAM based and battery/supercap backed thumb and have it be priced reasonably...heh...it'd make for a super thumb and swap space.


Such a device could be delivered as an extension module for the r-pi. How much bandwidth do the gpio (or other connectors) give? Connecting through these extra pins would avoid sharing bandwidth with ethernet and usb devices.
It could be realized as a simple version of a SoC with PoP-RAM as the r-pi itself has. Hence the extra costs of 10$ for model B splits to 128MiB RAM and the LAN chip, 128MiB on a very simple SoC shouldn't cost that much (given there is a SoC which simply forwards the RAM as a Ramdrive).

Theoretically, there's enough bandwidth out the GPIO's to drive 1Gbit Ethernet rates, if you can push the peak signalling rate. Problem is, there's no hardware to help you do this there and the CPU'd just be pinned dealing with the data if it COULD manage the data rate. Not likely to be a pathway for things there.

obarthelemy
Posts: 1407
Joined: Tue Aug 09, 2011 10:53 pm

Re: Additional RAM from USB?

Wed Aug 31, 2011 5:43 pm

I think we're starting to be unrealistic. I'm not even sure what applications and use cases we're talking about: Is it to run a specific off-the-shelf software ? For custom-developed stuff with a huge codebase or dataset ? Just as a lark ? 256Megs (probably 200+ available to the user after the OS and screen buffer) is a lot, especially for our own code.
IMHO, a fast SD card for swapping and/or as a filesystem will be hard to beat; One caveat is it might burn out every few years (by which we'll be on RPi model "Z" anyway ^^); and it might be good to split accesses between SD and USB to get some concurrency (I don't know what the cutoff point for that is though).

Svartalf
Posts: 596
Joined: Fri Jul 29, 2011 6:50 pm

Re: Additional RAM from USB?

Wed Aug 31, 2011 6:00 pm

Quote from obarthelemy on August 31, 2011, 18:43
I think we're starting to be unrealistic. I'm not even sure what applications and use cases we're talking about: Is it to run a specific off-the-shelf software ? For custom-developed stuff with a huge codebase or dataset ? Just as a lark ? 256Megs (probably 200+ available to the user after the OS and screen buffer) is a lot, especially for our own code.
IMHO, a fast SD card for swapping and/or as a filesystem will be hard to beat; One caveat is it might burn out every few years (by which we'll be on RPi model "Z" anyway ^^); and it might be good to split accesses between SD and USB to get some concurrency (I don't know what the cutoff point for that is though).

Indeed. I've been wondering why we had this conversation, truth be known here. The SD will have wear leveling so you can go 2-5 years under normal usage with a swap partition or file. If you don't have enough room on your SD, you can get away with a slight performance penalty with most USB thumbs or HD's stuck off the end of the R-Pi.

User avatar
Lob0426
Posts: 2198
Joined: Fri Aug 05, 2011 4:30 pm
Location: Susanville CA.
Contact: Website

Re: Additional RAM from USB?

Wed Aug 31, 2011 11:57 pm

Storage for a code base or as use as a database, the flash will probably work for a long time but not as a heavily used swap file IMHO. This keeps coming up. The best idea will be to use the RasPi within it's known limitations ie. Memory and you will have a robust reliable device.
Wear leveling is not a magic cure neither is the fact that some of them can replace a certain amount of memory with memory above the formatted value. Wikipedia is a great source for information on flash memory.
512MB version 2.0 as WordPress Server
Motorola Lapdock with Pi2B
Modded Rev 1.0 with pin headers at USB

http://rich1.dyndns.tv/
(RS)Allied ships old stock to reward its Customers for long wait!

patrickhwood
Posts: 26
Joined: Wed Aug 31, 2011 2:12 am

Re: Additional RAM from USB?

Thu Sep 01, 2011 3:25 am

Hmmm, let's do a little math:

For a 1GB swap partition, if we assume 100% write usage at 10MB/sec for one year, we get just over 300K GB of data, or ~300K write cycles. This is in the range (100k to 1m erase cycles) for most current flash devices. Now note that this assumes your system is spending all its time writing to the swap partition and never *reading* from it or doing anything else at all (it also assumes 10MB/sec write speeds, which (marketing claims aside) even class 10 devices can't do continuously. 2-5 years is quite conservative, IMHO. Make the swap partition 2G, and you essentially double the life span.

Also, the erase/write speed of flash decreases as the number of cycles adds up, so the max number of write cycles per unit time decreases with it. Now you could argue this point either way -- that the slower write speeds extends the life of the part, or the value of the part decreases over time and may require replacement before it actually fails.

Return to “General discussion”