User avatar
redhawk
Posts: 3465
Joined: Sun Mar 04, 2012 2:13 pm
Location: ::1

weird file corruption problem with flash drive??

Thu Jan 02, 2014 6:43 pm

I've had my Pi for nearly 1 1/2 years now and for the most part it has ran pretty much without fail.
However, now and again my USB flash drive would corrupt itself so that files would not open or vanish out of thin air.
If I run chkdsk on my Windows PC it deletes the offending files and they're usually recoverable from lost clusters .chk files.
I have done a full low level scan of the flash drive, every block is readable and read speed is pretty constant throughout.
My flash drive is used for an Apache local web server it contains media files and useful links for my Android phone to access.
The drive itself is mounted with read/write flags but the contents for the web sever is read only i.e. no database or storing data via cgi scripts.

Today I edited my useful links HTML file to add a new website, I saved it, refreshed my browser and the added link was clearly visible.
I then left the Pi to idle for nearly 6 hours without a reboot, I accessed the same page again but it had somehow reverted back to it's previous saved state.
I though this was odd so I cleared my browser cache and tried again but no change Apache was still delivering my old HTML file.
I logged into the Pi and checked the file contents myself, sure enough the added link had vanished even the last edited time stamp was wrong.

What the hell happened, do I have Gremlin inside my box or something??

Anyway I fixed it with chkdsk yet again but I'm am scratching my head over this one i.e. a) file gets edited b) Apache serves changed HTML file c) time passes by Apache serves HTML file as if it was never changed.

Does Linux have some kind of weird disk write cache buffering issues??

Do I have to purge the disk cache or something because I getting a little pe-d off with this recurring problem it doesn't make any sense.

Richard S.

User avatar
redhawk
Posts: 3465
Joined: Sun Mar 04, 2012 2:13 pm
Location: ::1

Re: weird file corruption problem with flash drive??

Sun Jan 19, 2014 10:46 pm

I don't bloody believe this but Raspbian has just shafted my USB flash drive yet again with another edited file that has mysteriously reverted back to a previous saved state.

Why does this keep happen??

Is it bad idea to edit files stored on a flash drive in Linux while Apache is still running??
mount (for sda1)
/dev/sda1 on /media/flash type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
[email protected]:/media/flash/www# dosfsck -t -a -w /dev/sda1
dosfsck 3.0.13, 30 Jun 2012, FAT32, LFN
There are differences between boot sector and its backup.
Differences: (offset:original/backup)
65:01/00
Not automatically fixing this.
/WWW/videos.html
Contains a free cluster (17553). Assuming EOF.
/WWW/videos.html
File size is 13899 bytes, cluster chain length is 0 bytes.
Truncating file to 0 bytes.
/WWW/links.html
Contains a free cluster (17552). Assuming EOF.
/WWW/links.html
File size is 2176 bytes, cluster chain length is 0 bytes.
Truncating file to 0 bytes.

C:\>chkdsk g: /f
The type of the file system is FAT32.
Volume Serial Number is 18EB-A6E5
Windows is verifying files and folders...
\www\videos.html first allocation unit is not valid. The entry will be truncated.
\www\links.html first allocation unit is not valid. The entry will be truncated.
File and folder verification is complete.
Convert lost chains to files (Y/N)? y
32768 bytes in 2 recovered files.
Windows has made corrections to the file system.

3,998,826,496 bytes total disk space.
16,384 bytes in 1 hidden files.
245,760 bytes in 15 folders.
3,208,265,728 bytes in 151 files.
790,282,240 bytes available on disk.

16,384 bytes in each allocation unit.
244,069 total allocation units on disk.
48,235 allocation units available on disk.
Richard S.

User avatar
redhawk
Posts: 3465
Joined: Sun Mar 04, 2012 2:13 pm
Location: ::1

Re: weird file corruption problem with flash drive??

Thu Feb 20, 2014 9:08 am

I'm completely bloody sick of this, my Pi has screw up the flash drive yet again. :x
Always the same problem I edit a html, save it, run sync to flush pending writes to disk, reboot my Pi and data is lost due to a invalid FAT entry.
I run chkdsk on Windows and 99.9% of the time the file is recoverable as lost cluster containing the full edit of my file plus attached garbage.
Given the fact that this has happen nearly 10 times in a row I think I can say with certainty this is not a random occurrence.

Is Kernel support for fat32 broken or demented??

Richard S.

JonAbbott
Posts: 14
Joined: Sat Mar 29, 2014 7:00 am

Re: weird file corruption problem with flash drive??

Sat Mar 29, 2014 7:17 am

I think it's wider than FAT, as I'm seeing filesystem corruption on a USB stick using ext4. It's been happening since I set the thing up a year ago.

Random sections of files will change, files that are read only but accessed frequently. I've tried updating the firmware and Debian, no change. It's not overclocked and is only used to host an apache server and does nothing else. It's powered via the USB power brick supplied with it, which is powered via a UPS. The server is never rebooted either, it just sits there serving web pages.

I noticed it again last night, which will be the third time in the past year, having rebuilt the whole server only a month ago after the last corruption event.

User avatar
redhawk
Posts: 3465
Joined: Sun Mar 04, 2012 2:13 pm
Location: ::1

Re: weird file corruption problem with flash drive??

Sat Mar 29, 2014 2:18 pm

Fortunately the data corruption is limited to just the flash drive, files on the SD card seem unaffected.
It's really odd because on saving data this is clearly being written to the flash but for reasons unknown to me the fat copies are not correctly synced.
I can only see 2 possible things happening here: Linux is caching or deferred disk writes but runs into trouble when it comes to do the actual writing, or both fat copies are correctly written but the kernel randomly decides to overwrite the flash with an old fat copy.

If it happens again I might change the file system to ext3 and see if that helps. :?

Richard S.

JonAbbott
Posts: 14
Joined: Sat Mar 29, 2014 7:00 am

Re: weird file corruption problem with flash drive??

Sat Mar 29, 2014 7:09 pm

In my case, the files seem to be written okay. You can read them back and the MD5's match. Several weeks later however, repeat the process and the files are corrupt part way through with what looks like bit rot. It will start say half way through the file and then be corrupt right through to the end.

They seem to be written okay initially so are subsequently being corrupted. When I started recovering it last night, I noticed there was catalogue corruption on the USB stick as well.

The irony is that I decided to use a USB stick to avoid the known corruption that happens with SD cards. I'm almost inclined to say the issue is wider than SD and is possibly firmware related.

bust
Posts: 76
Joined: Mon Mar 17, 2014 12:31 am

Re: weird file corruption problem with flash drive??

Sat Mar 29, 2014 11:28 pm

Hi all
I can confirm that i have also observed exactly the same problem of file system corrupted that you describe.
I have doubt that this problem have relation with specifical hardware support of sdcard.
With an hard drive standard added on usb (ext4) i have observe also some other problem of corruption
I have add new partitions xfs,jfs,reiserfs that are reserved to passive storage files, for understand if
this problem have an relation precise with ext4. (currently in test)
He seem that the problem start after an long time latency when file system is not solicited.
I afraid it's suppose that this problem could have relation side udev...
I have build apr and apache 2.4.7 (workers (event)) and he works traced ,I think problem have no relation with him.
I agree with you that this problem of corruption file system is catastrophic.
one other way could be try its to store and restore files system (tar lzma) after is formatting natively partition.
I am not sure that make copy image partition (bit) that could have origin to several type models of storage will result
perfectly aligned file system on new support.
Maybe it could be also his internal PSU on card unable to manage correctly stabilized.
We must not lose hope we will surely find the origin of the problem.
Regards

JonAbbott
Posts: 14
Joined: Sat Mar 29, 2014 7:00 am

Re: weird file corruption problem with flash drive??

Tue Apr 01, 2014 7:52 am

It looks like it's corrupting the FAT. I've just rebuilt my Pi again, copied all the files access and checked them - all okay.

I then imported a mysql database, checked the files again - several were corrupt. So I then copied the files back across and then the mysql database was corrupt.

I can only conclude from that, that the files have somehow got crosslinked.

User avatar
RaTTuS
Posts: 10242
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK

Re: weird file corruption problem with flash drive??

Tue Apr 01, 2014 8:25 am

Jon what is the Voltage as measured from TP1 to TP2
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

JonAbbott
Posts: 14
Joined: Sat Mar 29, 2014 7:00 am

Re: weird file corruption problem with flash drive??

Tue Apr 01, 2014 9:21 am

4.68v

The PSU is a Farnell Ltd "Raspberry Pi PSU"
Part# RPI-PSU-UK-MK1
Rated at 5.0v 1.0A

EDIT: Sounds low to me, so I've now swapped it out for an iPad PSU and am seeing exactly 5.0v

User avatar
RaTTuS
Posts: 10242
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK

Re: weird file corruption problem with flash drive??

Tue Apr 01, 2014 2:40 pm

5V +/- 5% i.e. 4.75V - 5.25V is in range...

try another micro USB lead, you probably are seeing a lot of voltage drop on that one
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

JonAbbott
Posts: 14
Joined: Sat Mar 29, 2014 7:00 am

Re: weird file corruption problem with flash drive??

Tue Apr 01, 2014 5:18 pm

RaTTuS wrote:try another micro USB lead, you probably are seeing a lot of voltage drop on that one
The official Pi PSU's have the lead built into them, so you can't change it. They're obviously not fit for purpose if they're putting out below 4.7v !!

JonAbbott
Posts: 14
Joined: Sat Mar 29, 2014 7:00 am

Re: weird file corruption problem with flash drive??

Mon Apr 21, 2014 10:33 am

I'm still seeing the corruption after replacing the PSU with one that definitely supplies 5v, so can only conclude its an issue in either Linux or the firmware.

It seems to mainly corrupt files that are accessed frequently and are read only.

Andya111
Posts: 3
Joined: Thu Mar 13, 2014 9:51 am

Re: weird file corruption problem with flash drive??

Fri May 02, 2014 7:06 pm

Did you get this problem sorted, Ive got Raspbmc installed to a sandisk extreme 3.0 16gb usb stick, it appears to work for a while, but after a while and without warning I turn the Pi on and I have no internet connection or shortcuts under any of the main headings, the time also changes to some random world time. This is starting to annoy me, any ideas what could be causing the problems or how to stop it from happening?

Thanks
Andy

JonAbbott
Posts: 14
Joined: Sat Mar 29, 2014 7:00 am

Re: weird file corruption problem with flash drive??

Sat May 03, 2014 6:31 pm

No I've not resolved the problem, its corrupting USB sticks on a daily basis. I'm having to keep a copy of the files on a PC and FTP them all back to the Pi every day.

The problem has to be in either Linux or the Pi firmware.

User avatar
algorithm
Posts: 176
Joined: Mon Nov 25, 2013 9:09 pm
Location: Flatland

Re: weird file corruption problem with flash drive??

Tue May 06, 2014 10:53 am

This is why I abandoned the use of a RPi as a media centre with locally connected storage :( In my case the attached hdd (via a powered usb hub) was hfs+ formatted, but I got the same sort of file corruption issues. The drive was auto-mounted as read/write on sda but flipped to read-only on sdb after corruption. Needed fsck and reboot every time. It seemed to be related to heavy usb/ethernet traffic and my guess was that the usb and/or ethernet hardware/system drivers caused the corruption at some level. I certainly tried to eliminate power issues with separate 5V/1-2A adapters for the RPi and the hub (but grounded at the same power strip).

User avatar
RaTTuS
Posts: 10242
Joined: Tue Nov 29, 2011 11:12 am
Location: North West UK

Re: weird file corruption problem with flash drive??

Tue May 06, 2014 11:00 am

it's possible you have a bad RPi ...
get a digital multimeter and measure TP1 TP2 - should be 4.75V - 5.25V Ideally 5.0V
don't count on the USB lead - measure the test points

I've got a couple of RPi's that hammer the network - [85-95% of the link for 24x7]
and not had any SD card corruption this is a specific home brewed program but involves a some writes to the SD card but is mostly for pushing data
.
How To ask Questions :- http://www.catb.org/esr/faqs/smart-questions.html
WARNING - some parts of this post may be erroneous YMMV

1QC43qbL5FySu2Pi51vGqKqxy3UiJgukSX
Covfefe

User avatar
algorithm
Posts: 176
Joined: Mon Nov 25, 2013 9:09 pm
Location: Flatland

Re: weird file corruption problem with flash drive??

Tue May 06, 2014 11:06 am

RaTTuS wrote:I've got a couple of RPi's that hammer the network - [85-95% of the link for 24x7]
and not had any SD card corruption this is a specific home brewed program but involves a some writes to the SD card but is mostly for pushing data.
Right, but I do not have any SD card trouble, it's specifically a usb-connected 2.5" hdd. (Via powered hub. I also tried another hub.)

JonAbbott
Posts: 14
Joined: Sat Mar 29, 2014 7:00 am

Re: weird file corruption problem with flash drive??

Sat May 10, 2014 5:42 am

Its definitely not power related and almost certainly firmware related. The issues with the USB stack are well known, quite frankly the USB module on the Broadcom chip is not fit for purpose.

Unfortunately, the knock on effect is that the Pi is next to useless for anything that requires heavy USB traffic.

I've just noticed mine has now trashed the USB storage again, having replaced all the hardware (Pi, PSU, memory stick), which pretty much points the blame at the USB driver, firmware or module.

This problem is USB specific, I've never seen SD card corruption.

User avatar
DougieLawson
Posts: 35126
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website Twitter

Re: weird file corruption problem with flash drive??

Sat May 10, 2014 9:31 am

There's been a USB rewrite by Broadcom and the Foundation. Start by getting to the latest firmware/kernel with
sudo apt-get install rpi-update
sudo rpi-update
sudo reboot

And see if that makes things better.
Note: Having anything remotely humorous in your signature is completely banned on this forum.

Any DMs sent on Twitter will be answered next month.

This is a hippy & doctor free zone.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 22061
Joined: Sat Jul 30, 2011 7:41 pm

Re: weird file corruption problem with flash drive??

Sat May 10, 2014 10:12 am

JonAbbott wrote:Its definitely not power related and almost certainly firmware related. The issues with the USB stack are well known, quite frankly the USB module on the Broadcom chip is not fit for purpose.

Unfortunately, the knock on effect is that the Pi is next to useless for anything that requires heavy USB traffic.

I've just noticed mine has now trashed the USB storage again, having replaced all the hardware (Pi, PSU, memory stick), which pretty much points the blame at the USB driver, firmware or module.

This problem is USB specific, I've never seen SD card corruption.
What a load of rubbbish. Please do not keep repeating this nonsense. As Dougie says above, almost ALL USB problems are fixed, and whilst the USB HW isn't the greatest, the USB software is now of a sufficient standard that the huge majority of devices work fine, even at high USB loads.

Now, there are a couple of unusual use cases that are impssible to fix, but they are rare, and are NOT related to the load on the USB.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

JonAbbott
Posts: 14
Joined: Sat Mar 29, 2014 7:00 am

Re: weird file corruption problem with flash drive??

Sat May 10, 2014 3:48 pm

This issue occurs on the latest firmware with fully patched up linux.

User avatar
redhawk
Posts: 3465
Joined: Sun Mar 04, 2012 2:13 pm
Location: ::1

Re: weird file corruption problem with flash drive??

Sun May 11, 2014 10:35 am

This is certainly not a stress problem it happens even when the processor is lightly loaded with only a single flash drive connected to the USB port.
As I have already stated the files I edit or copy (with WinSCP) are always 100% successful but for some reason the fat or the fat copy does not update correctly.
This behaviour doesn't look like a USB driver issue more like a bug in the Linux kernel for reasons that are beyond my understanding (failed to commit fat update or sent wrong copy of fat to flash drive??).

Richard S.

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 22061
Joined: Sat Jul 30, 2011 7:41 pm

Re: weird file corruption problem with flash drive??

Sun May 11, 2014 10:48 am

JonAbbott wrote:This issue occurs on the latest firmware with fully patched up linux.
In which case very specific reproduction details are need so the issue can be investigated. So, the USB topology, the type of stick, power supply, software being used etc.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Contrary to popular belief, humorous signatures are allowed. Here's an example...
"My grief counseller just died, luckily, he was so good, I didn't care."

JonAbbott
Posts: 14
Joined: Sat Mar 29, 2014 7:00 am

Re: weird file corruption problem with flash drive??

Sun May 11, 2014 6:58 pm

jamesh wrote:In which case very specific reproduction details are need so the issue can be investigated. So, the USB topology, the type of stick, power supply, software being used etc.
Its all in the posts above, simply repeatedly read/write to any USB stick formatted as FAT or ext4 (not tried others) plugged directly into the USB ports, no hub, and eventually the file allocation table will become corrupt, resulting in corrupt files and directories.

PSU has already been ruled out, both are detailed above though. No software is involved going by redhawk's description, I'm running an apache web server and mySQL serving a phpBB site that's vanilla. I can post details next week on the USB devices I've tried once I'm back home.

Return to “Raspbian”