evilkitty
Posts: 380
Joined: Tue Apr 15, 2014 11:39 pm

Re: weird file corruption problem with flash drive??

Mon May 12, 2014 4:48 am

I started using a flash drive with my pi the other day, before i added the flash drive voltage tested at 5.1v now it shows 5.0
i also have a fan hooked up to my psu it did lower my voltage a little in the range of 0.005 to 0.01, my guess is flash drives eat more power than you would think, if low voltage is the cause i should be fine based on voltage levels it is possible that is causing the issues here
cheap power cables have a vdrop of ~0.25 (better cables will have less, but still have some), if you start with 5v you barely have the necessary voltage for the pi, by the time the voltage gets to the flash drive there is probably more vdrop for it that is not known and it is below the 5% threshold for acceptable usb 2 spec, so while the pi has proper voltage the flash drive does not

has anyone formated a flash drive to say ext2 and had any issues with corruption?
My Pi Server: http://imgur.com/a/6xIUI | Thermostat: http://imgur.com/a/4LVnT

gmottram
Posts: 2
Joined: Fri May 16, 2014 1:11 pm

Re: weird file corruption problem with flash drive??

Fri May 16, 2014 2:08 pm

Hi,

I came across this thread with a Google search for "linux flash drive corruption". I suspect this is an old Linux problem and is not specific to Raspberry PI. I have spent hours recently with this same problem using the DD-WRT and Tomato-USB router software. I thought a USB flash drive would be a perfect way to expand my Netgear WNR3500 router (mips based) but kept running into file system corruption with horrible results (system becomes unusable if you store executable programs on the flash drive). I am using the ext2 file system, so given your problems with FAT, it does not appear to be file system related. I can also reproduce my problem with lots of disk activity -- I don't have to wait hours, the problem will occur in minutes. My solution was to switch to an external (spinning) hard disk -- it works perfectly. That would rule out a generic USB problem. I also have a Raspberry PI but I have always used the SD drive for storage without encountering these issues.

I suspect there is some trick to using USB flash drives that Linux has not mastered but Microsoft has. I have used flash drives for supplemental storage on a laptop running Windows without any problems. Flash drives have restrictions on how data is cleared and my experience suggests problems when the same flash location is rewritten in quick succession. Perhaps FAT and NTFS, as used by Microsoft, writes to successive parts of a flash drive, ostensibly to avoid wearing out the flash drive (a technique called leveling) but which has the side effect that the same location is not written to twice in quick succession and prevents data corruption.

I think my next step will be to write a program to write to the same disk location in a few times using the raw device instead of going through the file system. Another possibility would be to use a file system like jffs that supports leveling. See the Wikipedia page for more information (including issues when working with flash memory):

http://en.wikipedia.org/wiki/JFFS

If there is any interest here, I will follow up with my results.

I find it hard to believe, however, that this could be a generic Linux problem. Have people just not used flash drives for Linux storage until recently with the increased popularity of embedded systems?.

Geoff

gmottram
Posts: 2
Joined: Fri May 16, 2014 1:11 pm

Re: weird file corruption problem with flash drive??

Tue May 20, 2014 7:36 pm

Hi,

A follow up to my previous post. I have tested various USB drive options and even recompiled the router firmware with USB storage debugging turned on. I have concluded that the problem I was having was with two different no-name brand flash thumb drives that were unreliable -- they would not fail in a consistent manner. When debugging was turned off, the device driver would actually get error messages from the sticks. When debugging was turned on, the amount of data being sent from the log was sufficient to slow down the flash writing process and there would not be error messages sent from the sticks to the device driver. Instead, I would get ext2 file system errors -- presumably because the data being read from the sticks was bad.

I tried a Windows program called "Check Flash" which found problems on one of the flash drives that was giving me issues but not the other. My conclusion is to use a hard disk if you can and if not, stay with name brand flash drives. The two flash drives I had problems with were repurposed swag that probably work fine for writing marketting material once but are not so great for use as working file system. I was successful with a SanDisk flash drive and a repurposed USB backup hard disk. I should also point out that all I was doing was loading Optware, which entails downloading packages and installing them -- no serious disk thrashing involved (like compiling).

I would be careful about using flash drives for any application that writes log files -- you will eventually wear out the flash drive. You could try a file system like jffs which writes to new locations on the stick (leveling) which avoids this issue to a point. I would also recommend unmounting the file system periodically and running fsck.

Geoff

evilkitty
Posts: 380
Joined: Tue Apr 15, 2014 11:39 pm

Re: weird file corruption problem with flash drive??

Thu May 22, 2014 12:20 am

i have a flash drive on my ddwrt router (build 23939 on a WZR-300HP), i have not had any issue accept a LED bright enough to light my room at night
it is formated to ext2
i have had a flash drive on my pi for 9 days no issues (3.12.18+ kernel)

flash drives/sd cards/ssds will wear out in high right environments eventually, SSDs are made using higher grade nand for this reason
My Pi Server: http://imgur.com/a/6xIUI | Thermostat: http://imgur.com/a/4LVnT

JonAbbott
Posts: 14
Joined: Sat Mar 29, 2014 7:00 am

Re: weird file corruption problem with flash drive??

Sat May 31, 2014 8:10 am

The USB sticks I've tried that I've seen the issue on are:

Verbatim 16GB Micro USB
Kingston 4GB DataTraveler 100 G2

I tried one other, which I don't have to hand, but will post details once I do.

becks0815
Posts: 2
Joined: Fri Jan 17, 2014 8:41 am

Re: weird file corruption problem with flash drive??

Wed Jun 11, 2014 8:43 am

Hi all,
I can also confirm that there is a problem. I am using a raspberry with a 3A USB power supply, the system is barely used (it reads out 3 sensors over i2c every 10 minutes) but I constantly run into data corruption even when using a Kingston data traveler 16GB USB stick.

Anyone knows a possible solution for this? I really don't like the idea but after being forced to reformat/reinstall my system the 5th time within 2 months I am tired of this and might replace the raspi with something else.

JonAbbott
Posts: 14
Joined: Sat Mar 29, 2014 7:00 am

Re: weird file corruption problem with flash drive??

Wed Jun 11, 2014 5:10 pm

becks0815 wrote:Anyone knows a possible solution for this?
The solution I'm using is to run everything from a network drive, its not particularly responsive due to the NIC being off the USB stack, but good enough for the odd read/write.

bust
Posts: 76
Joined: Mon Mar 17, 2014 12:31 am

Re: weird file corruption problem with flash drive??

Wed Jun 18, 2014 2:03 am

Hi
The problem will occur when we take PSU from the wire of the usb
if you use an storage that have his PSU autonomous, the problem of corruption could be solved.
Remember that we don't have battery , date time could corrupt file system when the PSU is overcharged
That is strange small HD USB2 with two wires connected on hub with his PSU not solve the problem
Currently I test an small hd USB3 that have only 1 wire (connected on an hub with his PSU)
I have move all root fs to this hd (/dev/sda1 & /dev/sda2), sdcard have only boot on /dev/mmcblk0p1
Remark: this new HD USB3 with only one wire connected use the half of consumption compared with
old disk USB2 (with two wires, one reserved PSU)
When i use the two partitions only on his /dev/mmcblkxx with using an sd card 32 Go i don't have the problem of the corruption.

Some informations:
hdparm -t /dev/sda
/dev/sda:
Timing buffered disk reads: 88 MB in 3.03 seconds = 29.07 MB/sec

Sys. de fichiers blocs de 1K Utilisé Disponible Uti% Monté sur
/dev/root 30555524 8604728 20584444 30% /
devtmpfs 215824 0 215824 0% /dev
tmpfs 44820 392 44428 1% /run
tmpfs 5120 0 5120 0% /run/lock
tmpfs 89620 0 89620 0% /run/shm
/dev/mmcblk0p1 57102 19003 38099 34% /boot
/dev/sda2 240227840 7262588 220739256 4% /sdc3

root@raspberrypi:/usr/local/httpd/bin# ./httpd -v
Server version: Apache/2.4.9 (Unix)
Server built: Apr 16 2014 15:32:29

root@raspberrypi:/usr/local/httpd/bin# ./httpd -l
Compiled in modules:
core.c
mod_so.c
http_core.c
worker.c
(using (event) seems unable to give better result)

/sdc32/sources/postgresql-9.3.4 $ time make check
=======================
All 136 tests passed.
=======================

make[1]: quittant le répertoire « /sdc32/sources/postgresql-9.3.4/src/test/regress »

real 5m45.603s
user 1m10.870s
sys 0m21.800s

The hard tests with apache & postgresql (backend C++ CGI) seem working perfectly without corruption, just it's an little *slow,
but i am very happy from the raspeberry and to work resulting of team linux Raspbian (congratulation for quality his work)..

** (backend C++ postgresql with CGI seem answer it's little slow on the browsers, i search now how to solve this problem,
i have upgrade xul24, iceweasel etc ... and kernel 3.12.20 but problem seem persisting **

Regards

JonAbbott
Posts: 14
Joined: Sat Mar 29, 2014 7:00 am

Re: weird file corruption problem with flash drive??

Sat Jun 21, 2014 11:52 am

bust wrote:if you use an storage that have his PSU autonomous, the problem of corruption could be solved.
I've confirmed the Pi is getting enough voltage, so if the problem is power related as you're indicating, there's a problem with the Pi supplying enough power to the USB ports to power a memory stick.

This can probably be confirmed by plugging the USB stick into a USB hub and seeing if the corruption continues. Sadly, I don't have a USB hub to hand to confirm this, perhaps someone else that's seeing the problem can test this.

xavalex
Posts: 1
Joined: Mon Jan 11, 2016 4:12 pm

Re: weird file corruption problem with flash drive??

Mon Jan 11, 2016 4:21 pm

I have something very similar. I have bought 3.5 USB enclosures and a 2Tb disk, formatted in MBR/EXT4, connected to the Raspberry (Model B). I have then performed rsync with a external network drive as a backup. The rsync process was expected to take a few days (1.5Gb to back up). After a few days, the disk was corrupted and showed bad data. I could not repair it even when completely redoing the partitioning and low level formatting.

I then bought a second drive thinking this was somehow related to the hardware, and formatted this one in MBR/NFS. The exact same phenomenon happened after a few days.

The possibility to have 2 drives fail in this way cannot possibly be linked to hardware. So I am at a loss how to explain this. I have tried with 2 different USB enclosures, in case the firmware of the enclosure was at fault, and the drives were both externally powered.

We are right at the limit for MBR, and I may have to try with GPT next time, but I don't want to lose another drive. Any suggestion?

rmast
Posts: 14
Joined: Fri Dec 25, 2015 11:52 am

Re: weird file corruption problem with flash drive??

Wed Feb 08, 2017 8:39 pm

Today I have a corruption/freeze issue within OSMC using a Sandisk Cruzer Extreme 32GB together with the internal SD-card.
I thought the sandisk would help me compile faster with make -j4 on my RPi3, on the extreme as 'external drive' but building gstreamer it just freezes and shows a corrupt, even unreachable log file on an unreachable disk. Only a reboot makes the disk reachable again. The kernel version is 4.4.13-5.

I wonder whether the same happens if I just copy the complete SD root to USB and only work from USB.

rmast
Posts: 14
Joined: Fri Dec 25, 2015 11:52 am

Re: weird file corruption problem with flash drive??

Wed Feb 08, 2017 10:25 pm

The gstreamer compilation just ends on some compilation-failure when I fully run from a root-fs copied to USB, as it runned quicker than before I don't know whether a compilation of the same duration would corrupt the USB-disk as well, but I suspect the mixing of SD and USB-diskaccess to be a potential corrupting factor.

rmast
Posts: 14
Joined: Fri Dec 25, 2015 11:52 am

Re: weird file corruption problem with flash drive??

Wed Feb 08, 2017 10:26 pm

By the way, an external hub didn't make that difference. This was just a run with the USB-stick right in the RPi3.

User avatar
RaTTuS
Posts: 10565
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK
Contact: Twitter YouTube

Re: weird file corruption problem with flash drive??

Thu Feb 09, 2017 7:36 am

this is a old threrad
best start again wit a new one
and post the errors you ge on compilation
also show all

Code: Select all

dmesg
errors
and everything else that is relevent
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

Return to “Raspberry Pi OS”