Rilhas
Posts: 36
Joined: Sat Aug 04, 2012 7:52 am

Weirdest thing ever with some specific file names

Thu Jul 27, 2017 5:10 pm

Hi,

I've been using the Pi for a long time now, went through multiple hardware versions, multiple Raspbian versions, and one thing that happens every now and then is the inability to create specific files. This is hard to see, because the occurrence is rare to watch, but today I got a series of them and was able to track them down.

One instance of the problem happened while trying to create the file:

../HTML_Server_Cache/http___www.era.pt_imoveis__pg_1966_o_1_t__a__dd__cc__ff__z__cp__nq__p__ar__ca_00000000_ct_0000_or_41_idioma_pt1.txt

The attempt was made with fopen(), and it failed (returns NULL). The current directory is /RILHAS-SRV/app/HTML_Server so the actual file is:

/RILHAS-SRV/app/HTML_Server_Cache/http___www.era.pt_imoveis__pg_1966_o_1_t__a__dd__cc__ff__z__cp__nq__p__ar__ca_00000000_ct_0000_or_41_idioma_pt1.txt

I have been assuming this to be caused by some software error on my part, but just now I went to directory /RILHAS-SRV/app/HTML_Server and from there I typed:

touch ../HTML_Server_Cache/http___www.era.pt_imoveis__pg_1966_o_1_t__a__dd__cc__ff__z__cp__nq__p__ar__ca_00000000_ct_0000_or_41_idioma_pt1.txt

... and I got the result:

touch: cannot touch `../HTML_Server_Cache/http___www.era.pt_imoveis__pg_1966_o_1_t__a__dd__cc__ff__z__cp__nq__p__ar__ca_00000000_ct_0000_or_41_idioma_pt1.txt': No space left on device

The device has a lot of space available:

df -h
Filesystem Size Used Avail Use% Mounted on
/dev/root 19G 4.1G 14G 24% /
devtmpfs 427M 0 427M 0% /dev
tmpfs 87M 276K 86M 1% /run
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 173M 4.0K 173M 1% /run/shm
/dev/mmcblk0p1 56M 21M 36M 38% /boot
/dev/mmcblk0p2 7.0G 4.1G 2.7G 61% /SDCard
/dev/sda2 899G 835G 24G 98% /RILHAS-SRV <-- **** this one ****

I also tried "sudo touch", same result.

Typing:

ls ../HTML_Server_Cache/http___www.era.pt_imoveis__pg_1966_o_1_t__a__dd__cc__ff__z__cp__nq__p__ar__ca_00000000_ct_0000_or_41_idioma_pt1.txt

... gets the result:

ls: cannot access ../HTML_Server_Cache/http___www.era.pt_imoveis__pg_1966_o_1_t__a__dd__cc__ff__z__cp__nq__p__ar__ca_00000000_ct_0000_or_41_idioma_pt1.txt: No such file or directory

The file is supposed to be new, it is the cache for an HTML page that has never been accessed before, but I also tried to "rm" the file, just in case, and also "sudo rm" the file, but it failed to remove the non-existing file, of course.

Then I tried playing with the name a little, changing the extension from .txt to .txu, and it worked, the file was created. Changing letters and characters randomly always worked, so I assume there is a specific problem with this specific file name.

The folder is full of files (thousands of cache files for various crawlers, maybe hundreds of thousand's - I don't know because it takes quite a long time to do something in that folder), so I don't see anything practical I can do with the folder or any tests I can do that won't take forever.

Nevertheless, the problem is with specific file names, not the amount of files already there, because after I got the error with this specific file and did these manual tests another 2000 files have been added to the folder without any problems.

Also, the problem is not with filename length, because adding more characters to the faulty filename solves it and a file with a longer filename is created without any problem. For example, changing the extension from ".txt" to ".txts" works. However, changing the extension from ".txt" to ".tx" doesn't work, and the file doesn't get created. But changing it from ".txt" to ".t" works, and the file gets created. Also, changing from ".txt" to ".txx" or "ttx" (and many other variations) works.

Here is a small list of other related file names that don't work (although I sometimes run into other cases of unrelated file names that don't work):
../HTML_Server_Cache/http___www.era.pt_imoveis__pg_7_o_1_t__a__dd__cc__ff__z__cp__nq__p__ar__ca_00000000_ct_0000_or_41_idioma_pt1.txt
../HTML_Server_Cache/http___www.era.pt_imoveis__pg_13_o_1_t__a__dd__cc__ff__z__cp__nq__p__ar__ca_00000000_ct_0000_or_41_idioma_pt1.txt
../HTML_Server_Cache/http___www.era.pt_imoveis__pg_15_o_1_t__a__dd__cc__ff__z__cp__nq__p__ar__ca_00000000_ct_0000_or_41_idioma_pt1.txt
../HTML_Server_Cache/http___www.era.pt_imoveis__pg_18_o_1_t__a__dd__cc__ff__z__cp__nq__p__ar__ca_00000000_ct_0000_or_41_idioma_pt1.txt
../HTML_Server_Cache/http___www.era.pt_imoveis__pg_20_o_1_t__a__dd__cc__ff__z__cp__nq__p__ar__ca_00000000_ct_0000_or_41_idioma_pt1.txt
../HTML_Server_Cache/http___www.era.pt_imoveis__pg_21_o_1_t__a__dd__cc__ff__z__cp__nq__p__ar__ca_00000000_ct_0000_or_41_idioma_pt1.txt
../HTML_Server_Cache/http___www.era.pt_imoveis__pg_23_o_1_t__a__dd__cc__ff__z__cp__nq__p__ar__ca_00000000_ct_0000_or_41_idioma_pt1.txt
../HTML_Server_Cache/http___www.era.pt_imoveis__pg_26_o_1_t__a__dd__cc__ff__z__cp__nq__p__ar__ca_00000000_ct_0000_or_41_idioma_pt1.txt
../HTML_Server_Cache/http___www.era.pt_imoveis__pg_28_o_1_t__a__dd__cc__ff__z__cp__nq__p__ar__ca_00000000_ct_0000_or_41_idioma_pt1.txt
../HTML_Server_Cache/http___www.era.pt_imoveis__pg_48_o_1_t__a__dd__cc__ff__z__cp__nq__p__ar__ca_00000000_ct_0000_or_41_idioma_pt1.txt
../HTML_Server_Cache/http___www.era.pt_imoveis__pg_49_o_1_t__a__dd__cc__ff__z__cp__nq__p__ar__ca_00000000_ct_0000_or_41_idioma_pt1.txt

Please note that there is a pattern, the file names change depending on the page number (*pg_*). Although the first file on the previous list (with *pg_7*) doesn't get created, the next cache file (with *pg_8*) works, and so do all others until we get to the second one on the list (with *pg_13*). So, in practice, this specific crawler is going through a number of pages, and no matter how many times I run it if fails on pages 7, 13, 15, 18, 20, and so on, but it works on intermediate pages 1 to 6, 8 to 12, 14, 16 to 17, 19, and so on.

Here is cat /proc/version (although this has happened every once in a while over the years and across versions):

Linux version 4.1.19-v7+ (dc4@dc4-XPS13-9333) (gcc version 4.9.3 (crosstool-NG crosstool-ng-1.22.0-88-g8460611) ) #858 SMP Tue Mar 15 15:56:00 GMT 2016

So my questions:

Is there any specific Linux file naming rules that these file names are breaking?

Any other reason for me to be unable to create these files?

ericcooper
Posts: 148
Joined: Sat Apr 08, 2017 6:23 pm

Re: Weirdest thing ever with some specific file names

Thu Jul 27, 2017 6:32 pm

Is that on an ext4 filesystem? Try doing "df -i" to see if you've run out of inodes?

User avatar
B.Goode
Posts: 11197
Joined: Mon Sep 01, 2014 4:03 pm
Location: UK

Re: Weirdest thing ever with some specific file names

Thu Jul 27, 2017 8:32 pm

ericcooper wrote:
Thu Jul 27, 2017 6:32 pm
Is that on an ext4 filesystem? Try doing "df -i" to see if you've run out of inodes?
That's what I thought as I was reading. However -
Nevertheless, the problem is with specific file names, not the amount of files already there, because after I got the error with this specific file and did these manual tests another 2000 files have been added to the folder without any problems.
(Unless there is some sort of scavenger or garbage collector in the background freeing up unneeded inodes from time to time?)

Rilhas
Posts: 36
Joined: Sat Aug 04, 2012 7:52 am

Re: Weirdest thing ever with some specific file names

Sun Jul 30, 2017 10:12 am

Hi!

Thank you for your reply.

Yes, it is an EXT4 file system:

pi@RILHAS-SRV ~ $ df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/root 1221600 100267 1121333 9% /
devtmpfs 109302 390 108912 1% /dev
tmpfs 110381 314 110067 1% /run
tmpfs 110381 2 110379 1% /run/lock
tmpfs 110381 3 110378 1% /run/shm
/dev/mmcblk0p1 0 0 0 - /boot
/dev/mmcblk0p2 474208 115307 358901 25% /SDCard
/dev/sda2 59834368 7857509 51976859 14% /RILHAS-SRV

It seems like there are plenty of inodes left. Also, the fact that the problem is with specific file names also suggests that there is effect of any garbage collection, otherwise it would happen that a specific file name that fails now would start working later, and that is not the case (if a file name fails I can create 2000 other files afterwards, and the file name will still fail after all those files have been created).

Any other ideas?

If you have the time (and the curiosity), would you mind trying this out on your end? It seems to me that all you need is to create the directory structure and then try to create one of the failing file names inside it (touch seems to manifest the problem).

User avatar
DougieLawson
Posts: 40823
Joined: Sun Jun 16, 2013 11:19 pm
Location: A small cave in deepest darkest Basingstoke, UK
Contact: Website Twitter

Re: Weirdest thing ever with some specific file names

Sun Jul 30, 2017 10:21 am

No problems here on a ext4 filesystem

Code: Select all

pi@pioneer:~$ cd /tmp
pi@pioneer:/tmp$ mkdir HTML_Server_Cache
pi@pioneer:/tmp$ touch HTML_Server_Cache/http___www.era.pt_imoveis__pg_1966_o_1_t__a__dd__cc__ff__z__cp__nq__p__ar__ca_00000000_ct_0000_or_41_idioma_pt1.txt
pi@pioneer:/tmp$ cd HTML_Server_Cache/
pi@pioneer:/tmp/HTML_Server_Cache$ ls
http___www.era.pt_imoveis__pg_1966_o_1_t__a__dd__cc__ff__z__cp__nq__p__ar__ca_00000000_ct_0000_or_41_idioma_pt1.txt
pi@pioneer:/tmp/HTML_Server_Cache$
Any language using left-hand whitespace for syntax is ridiculous

Any DMs sent on Twitter will be answered next month.
Fake doctors - are all on my foes list.

Any requirement to use a crystal ball or mind reading will result in me ignoring your question.

Rilhas
Posts: 36
Joined: Sat Aug 04, 2012 7:52 am

Re: Weirdest thing ever with some specific file names

Sun Jul 30, 2017 1:34 pm

In your experiment you tried with a different path, so I'm not surprised you didn't run into the problem. As I explained, if I change a character the file succeeds, and changing the path results in changing lots of characters.

I also tried doing this experiment from a different source folder, and if the resulting file name is one on the list then I get the same result, no matter how I got to it (combinations of . and ..). However, if the destination file is on a different folder, then the problem goes away.

Rilhas
Posts: 36
Joined: Sat Aug 04, 2012 7:52 am

Re: Weirdest thing ever with some specific file names

Sun Jul 30, 2017 1:36 pm

... trying the absolute path also fails, as long as the path is /RILHAS-SRV/app/HTML_Server_Cache/http___www.era.pt_imoveis__pg_1966_o_1_t__a__dd__cc__ff__z__cp__nq__p__ar__ca_00000000_ct_0000_or_41_idioma_pt1.txt

User avatar
DougieLawson
Posts: 40823
Joined: Sun Jun 16, 2013 11:19 pm
Location: A small cave in deepest darkest Basingstoke, UK
Contact: Website Twitter

Re: Weirdest thing ever with some specific file names

Sun Jul 30, 2017 1:43 pm

No problems with that

Code: Select all

root@pioneer /RILHAS-SRV/app/HTML_Server_Cache # ls -ls
total 0
root@pioneer /RILHAS-SRV/app/HTML_Server_Cache # su pi
pi@pioneer:/RILHAS-SRV/app/HTML_Server_Cache$ touch http___www.era.pt_imoveis__pg_1966_o_1_t__a__dd__cc__ff__z__cp__nq__p__ar__ca_00000000_ct_0000_or_41_idioma_pt1.txt
pi@pioneer:/RILHAS-SRV/app/HTML_Server_Cache$ ls -la
total 8
drwxr-xr-x 2 pi pi 4096 Jul 30 14:42 .
drwxr-xr-x 3 pi pi 4096 Jul 30 14:40 ..
-rw-r--r-- 1 pi pi    0 Jul 30 14:42 http___www.era.pt_imoveis__pg_1966_o_1_t__a__dd__cc__ff__z__cp__nq__p__ar__ca_00000000_ct_0000_or_41_idioma_pt1.txt
pi@pioneer:/RILHAS-SRV/app/HTML_Server_Cache$
Any language using left-hand whitespace for syntax is ridiculous

Any DMs sent on Twitter will be answered next month.
Fake doctors - are all on my foes list.

Any requirement to use a crystal ball or mind reading will result in me ignoring your question.

Rilhas
Posts: 36
Joined: Sat Aug 04, 2012 7:52 am

Re: Weirdest thing ever with some specific file names

Sun Jul 30, 2017 1:48 pm

Thanks for your help!

That means that it is some problem specific to me... maybe something is corrupted, although I have no idea how that results in the refusal to create specific files, complaining that there is not enough free space.

Thanks again!

Rilhas
Posts: 36
Joined: Sat Aug 04, 2012 7:52 am

Re: Weirdest thing ever with some specific file names

Sun Jul 30, 2017 1:54 pm

Well, now adding to weird (or maybe less weird after all) if I copy the file name from your message it works on my system, and the file is created.

My theory now is that my software must be using some invalid characters that are not visible, and that are lost in the copy/paste from the forum and the file name becomes legal. Not sure how that can happen, but at least now I have a clue.

Rilhas
Posts: 36
Joined: Sat Aug 04, 2012 7:52 am

Re: Weirdest thing ever with some specific file names

Sun Jul 30, 2017 1:55 pm

No, scratch that. The copy/paste was not being done properly. Pasting the file name from your message doesn't work after all.

User avatar
DougieLawson
Posts: 40823
Joined: Sun Jun 16, 2013 11:19 pm
Location: A small cave in deepest darkest Basingstoke, UK
Contact: Website Twitter

Re: Weirdest thing ever with some specific file names

Sun Jul 30, 2017 1:56 pm

Stare at your code for ten minutes. Then post it on here.

The other thing you can do is some forensic examination of the device at /dev/sda2. What type of filesystem is on there?

What do you get if you umount /dev/sda2 and use a fdisk -l /dev/sda command? What happens if you run fsck -f -y /dev/sda2 (note: that's why it needs to be unmounted)?
Any language using left-hand whitespace for syntax is ridiculous

Any DMs sent on Twitter will be answered next month.
Fake doctors - are all on my foes list.

Any requirement to use a crystal ball or mind reading will result in me ignoring your question.

Rilhas
Posts: 36
Joined: Sat Aug 04, 2012 7:52 am

Re: Weirdest thing ever with some specific file names

Sun Jul 30, 2017 2:22 pm

The code is in use for years, and used extensively on Raspberry Pi's, production servers, my Windows desktop and laptop, and for various applications. It is a kind of C++ "platform", but ultimately what it does is:

1) replace all invalid file name characters with an underscore (like '/' and ':') thus transforming the URL to a file name full of underscores, and give it a path;

2) FILE* f=fopen(file_name, "wb"), and signal an error when f==NULL.

I'm now sure it is not a problem with my code, because "touch" can't create the file either. That reduces the span of the troubleshooting tremendously. Also, it is typical to create similar cache files when I'm developing in Windows, and in Windows I don't have files fail, so I'm thinking it is something specific to the Raspberry Pi (like a corrupted disk, or some obscure file system bug). Nevertheless, file names on Windows are never the same as on Linux, but with 10K files created I may get 100 failing in Linux and I always get 0 failing in Windows.

The file system is EXT4, and I'll try the checks you indicated. I can boot from the SD card, and check it from there.

Thanks!

Rilhas
Posts: 36
Joined: Sat Aug 04, 2012 7:52 am

Re: Weirdest thing ever with some specific file names

Sun Jul 30, 2017 2:44 pm

Here are the result:

pi@RILHAS-SRV ~ $ sudo fdisk -l /dev/sda

Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders, total 1953525168 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0xcb92aadd

Device Boot Start End Blocks Id System
/dev/sda1 * 63 39064547 19532242+ 83 Linux
/dev/sda2 39065600 1953523711 957229056 83 Linux

pi@RILHAS-SRV ~ $ sudo fsck -f -y /dev/sda2
fsck from util-linux 2.20.1
e2fsck 1.42.5 (29-Jul-2012)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
RILHAS-SRV-DATA: 7857509/59834368 files (0.1% non-contiguous), 222638032/239307264 blocks

I was expecting a lot of complaints (when I need to turn it off I usually just pull the plug!), but it seems there are no complaints at all, right?

Anyway, after running the commands the manual touch command still fails, just like before. I guess this throws my "disk corrupted" theory out of the window, right?

Martin Frezman
Posts: 1009
Joined: Mon Oct 31, 2016 10:05 am

Re: Weirdest thing ever with some specific file names

Sun Jul 30, 2017 2:49 pm

(when I need to turn it off I usually just pull the plug!)
You just said the magic words.

Once you say those words, no one can or will help you.


Short summary of the above: You need to shutdown properly. If you don't, we have no choice but to assume that that is what is causing your filesystem to get corrupted.

And, no, fsck and friends don't always find all problems.
If this post appears in the wrong forums category, my apologies.

Rilhas
Posts: 36
Joined: Sat Aug 04, 2012 7:52 am

Re: Weirdest thing ever with some specific file names

Sun Jul 30, 2017 3:13 pm

Ok, thanks.

I will keep on pulling the plug though, Windows can take it perfectly, and I think the Raspberry Pi should too. I understand SD card corruptions if I pull the plug (there is not enough electronics in them to handle power outages gracefully), but I don't find it acceptable that in the 21st century EXT4 on a spinning HDD gets corrupted when power goes out. Sometimes that is even unavoidable (when the Pi hangs beyond any recovery).

So I will keep on abusing it, and when it dies on me I'll get a backup and move on to something else more robust.

Thanks for all the help and patience, I really appreciate it.

hippy
Posts: 9186
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Weirdest thing ever with some specific file names

Sun Jul 30, 2017 3:23 pm

Rilhas wrote:
Sun Jul 30, 2017 3:13 pm
I will keep on pulling the plug though, Windows can take it perfectly, and I think the Raspberry Pi should too.
From bitter and unwanted experience I can assure you Windows can't take it perfectly, and neither can the Pi be expected to.

You might get away with it but that's merely good fortune, far from a certainty.

User avatar
rpdom
Posts: 18148
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Weirdest thing ever with some specific file names

Sun Jul 30, 2017 3:35 pm

Rilhas wrote:
Sun Jul 30, 2017 3:13 pm
I will keep on pulling the plug though, Windows can take it perfectly, and I think the Raspberry Pi should too.
You're on your own now. We can't help you if you do that. Windows doesn't like it and the Pi isn't happy either. It is a great way to corrupt data and possibly damage your SD card beyond recovery.

Rilhas
Posts: 36
Joined: Sat Aug 04, 2012 7:52 am

Re: Weirdest thing ever with some specific file names

Sun Jul 30, 2017 3:41 pm

I respect your opinion (and I understand that I was the one coming here asking for help), but for the last 15 years I've pulled the plug on hundreds of Windows machines (used in professional environments, with power cut automatically through relay switches) and I ran into exactly 0 file system problems. So I can guarantee that Windows handling it properly is not luck, it is normal expected behavior, and if you did get corruptions then that I attribute to bad luck (incredible bad luck, because I have hundreds of samples in my history that never got a problem since XP).

Nevertheless - and remembering that I do respect your opinion - I don't think that the file system is corrupted on the Pi just yet. The disk is full of sensitive stuff, like compressed audio files, compressed video files, executables, subversion server, ..., all that would show catastrophic failures or noticeable artifacts if a byte got messed up in them. Subversion is used every day without problems, my kids watch videos all the time and never get a hickup, and I listen to music afternoons on end without the slightest click or pop. Executables do their thing normally, and the Pi is restarted every week, so if executables got corrupted on disk then I would see them failing when they got loaded back to RAM after the next restart.

So, I don't have any real reason to suspect that the Pi is fragile when it comes to power outages (running from an HDD, but I've seen it break quite often running from SD cards), but the fact that everyone seems to treat the file system as the most fragile thing in the world does scare me, and if power ever goes out on it then all confidence is lost forever, and as you said that leaves me all alone in the world fighting these weird problems because everyone gives up after that point. So I definitely absolutely don't want to be the user of such fragile things, and when it craps out I will take my leave to other things more robust (proven beyond doubt to be more robust, and they exist) and not look back again.

Anyway, I'm now following a lead a friend of mine gave me when he asked if the file system supports UTF-8. The question in itself is still unanswered (don't know how to check), but I noticed that the file names run into problems with pairs of underscores. For example, __41__ has a problem that _41_ doesn't.

So I will pursue this a bit longer, all the while assuming that the file system is not corrupted just yet (my kid is watching a compressed movie beside me right now!!) and try to figure out any pattern to this file name problem.

User avatar
DougieLawson
Posts: 40823
Joined: Sun Jun 16, 2013 11:19 pm
Location: A small cave in deepest darkest Basingstoke, UK
Contact: Website Twitter

Re: Weirdest thing ever with some specific file names

Sun Jul 30, 2017 3:56 pm

Rilhas wrote:
Sun Jul 30, 2017 2:44 pm
I was expecting a lot of complaints (when I need to turn it off I usually just pull the plug!), but it seems there are no complaints at all, right?

Anyway, after running the commands the manual touch command still fails, just like before. I guess this throws my "disk corrupted" theory out of the window, right?
That's not good practice.
sudo apt-get install eject
sudo eject /dev/sda


Use eject every time before you pull the drive [that's just like Windows "safely remove this hardware"] (there's an icon for it on the Raspbian desktop).

In your "C++" program what file operations are you doing? Do you check errno after every action? Is it possible to add a display of errno whenever it's non-zero?
Any language using left-hand whitespace for syntax is ridiculous

Any DMs sent on Twitter will be answered next month.
Fake doctors - are all on my foes list.

Any requirement to use a crystal ball or mind reading will result in me ignoring your question.

Rilhas
Posts: 36
Joined: Sat Aug 04, 2012 7:52 am

Re: Weirdest thing ever with some specific file names

Sun Jul 30, 2017 4:14 pm

I didn't explain myself correctly. I meant "pull the plug" as in "pull the Raspberry Pi power plug", not the USB power plug to which the disk is connected to. Nevertheless, I can't avoid power outages, and a big UPS isn't practical (and I've bought 3 over the years and they all failed when I most needed them not to fail, one of them even started to create outages on its own by tripping the circuit breaker at random times).

So, of this I'm sure: I will continue to allow the Pi to suffer power outages, and if that proves to be too much for the Pi and verify files get damaged then I'll move on to something that tolerates outages. Note that the problem is that I can't create specific files, if I do create them then I can read and write and never have a problem with them (these cache files are also accessed quite frequently, that is the whole point of their existence, so I will detect when a cache file is created but then gets corrupted).

I understand losing files that are not flushed to disk when power goes out, but I don't understand (and have trouble believing) that existing files on an EXT4 file system on a spinning hard disk get corrupted when power goes out. That was very common in the FAT days, but I don't use that anymore (for quite some time now).

The C++ operations are simple:

1) fix the file name so that it doesn't include invalid characters, or directory separators, (replace with '_') and add a path to it;

2) FILE* f=fopen(file_name, "wb"), and in these cases I get f==NULL.

I haven't checked errno, but I'm sure it will be incorrect (because "touch" says the error is that there is no disk space when there is plenty of disk space).

In any case, I've ruled out any problem with my software (because touch also manifests the problem), and I'm now working manually with touch to figure out any patterns.

User avatar
davidcoton
Posts: 5866
Joined: Mon Sep 01, 2014 2:37 pm
Location: Cambridge, UK
Contact: Website

Re: Weirdest thing ever with some specific file names

Sun Jul 30, 2017 4:28 pm

Work on that double underscore hypothesis. I don't know all the UTF-8 escape sequences off-hand, and I don't know if I would expect a problem, but update your code to allow only a single underscore? Or use hyphen instead?

As to saying you haven't had a problem when pulling power without shutdown, that's like saying you haven't been killed while running across a main road blindfold. You are right that the problem is with pending write operations, and SDCards are vulnerable because of the internal writes to do wear levelling, but sooner or later your hard disc will be damaged by an incomplete write, unless you always take steps to prevent it. Linux and Windows both have shutdown routines for good reasons. Both also have disk recovery options that might rescue you, but good practice is to shutdown before power off. Data centres use UPSs for a good reason. (Ask British Airways.) Of course, you can take the view that your system is not mission critical, and the downtime and repair cost is less than the prevention cost. But writing a routine that monitors a pushbutton and shuts a system down is not hard or expensive. You will never be able to prove that it has saved you from a rebuild, but ignore the hard won industry standard advice at your own peril (it may not apply to you, but people have lost jobs over this sort of issue).
Location: 345th cell on the right of the 210th row of L2 cache

Rilhas
Posts: 36
Joined: Sat Aug 04, 2012 7:52 am

Re: Weirdest thing ever with some specific file names

Sun Jul 30, 2017 5:16 pm

I think we are going a bit off-topic here, but I've seen a lot of people still believe on old myths and old practices of the past century, and never got to learn the new features and new protections on modern equipment.

I'm not saying that this is your case, but the fact is that no one here that commented on the power outages asked me what type of HDD I'm using. Well, I'm using a laptop HDD, that when power goes out automatically completes a pending write (only 1), still has enough energy in its capacitors to park the heads, turn off all magnetics and controllers, and is ready to fall to the floor and survive. Pretty standard stuff these days, and, actually, about 10 years ago accelerometers started to become standard in these disks to stop operations and park the heads whenever there was a more abrupt movement (not necessarily falling, but "knee jerking" and such). So, broken write operations only happen these days if the portable disk is malfunctioning.

So yes, this disk is power outage-proof, and trusting that it survives power outages is not not running blindfolded in the street and trusting that I don't get run over, and it will be very hard to convince me that manufacturing a hard disk these days that can't take power outages is common practice. One can say that trusting these hard disks is a bit like trusting that drivers stop when the light is green for pedestrians, and although I wouldn't trust my life with it and cross blindfolded, I would surely trust my movies and music to cross blindfolded. In practice, that is not trusting luck, because the standard practice is for all drivers to stop on the red light, so you would need to be very unlucky for one driver (exactly the first one in front of the red light for cars) not stopping when your hard disk with movies and music is crossing the street (blindfolded).

As I'm sure you can imagine, given my history of hundreds of power cuts and 0 file system problems I've had many such conversations with co-workers and friends, and everyone who ever told me that I shouldn't do that wasn't talking from personal experience, only things that they heard somewhere, 10 or 20 years before. Furthermore, when you get to the bottom of those stories it seems like the disks were already failing before the power outage (SMART errors increasing, and taking longer to repair, and suddenly a catastrophic failure).

As an example, I once collected 4 of those machines (that I usually powered down with relays) for maintenance. It was a longer procedure that time, so we took them from the client to the lab, to do the work in the lab. When we got to the lab all 4 machines were dead, the disks not even spinning. Obviously, it might look like the systematic power outage finally killed them, and it would be so obvious that such a story about how the 4 machines died all at the same time from the power outage would fly, but actually what happened was that the machines had been on for many years, always hot, and the 30 minute trip to the lab was enough to cool it to the point where motor got stuck and didn't move again. So, a deep thermal cycle issue, nothing more (technology from 1998). Applying a hammer to the disks released the mechanism and they started spinning happily, and functioning without further problems (and all data in them was well). Of course, we replaced them, but the point is that they had not been killed by the power cuts, although it might seem so.

I'm not saying that shutdown procedures are useless, even more so given that they are the only way to ensure that files get flushed to disk, but I've tested and verified time and time again that they are not needed for the survival of existing files. Of course, if this Pi was controlling the launch and targeting of intercontinental missiles, then I would put in a UPS (or 4, given the 3 failed on me), but as you said, this is not critical (and not even a single video file got damaged so far).

Maybe this helps understand why I say that if EXT4 can't take a power outage, then it really is a serious design fault, given that portable hard drives have been manufactured to cope with them since a long time ago. Furthermore, I've read about it (and talked to "pundit" friends) who explained all kinds of mirroring and backups and journaling in EXT4, so it is supposed to be fairly robust. So, if EXT4 + all its features + modern HDDs can't take power outages then the solution is flimsy, weak, unreliable, and I need to start using better. But I sincerely believe that I can keep doing this and that it will not kill my data nor hardware, as I said it is the least I expect from modern equipment and software.

I also just ran checks between the current data and the previous backup, and the data on my Pi is not corrupted (well, the data that exists in both the disk in use and the backup, I have nothing to compare the new data against, of course, and these caches that are giving me a hard time are not backed up either). So, my disk hasn't been runover by a truck yet, you call it "luck" but I call it "as expected ~17 years into in the 21st century". :-)

Rilhas
Posts: 36
Joined: Sat Aug 04, 2012 7:52 am

Re: Weirdest thing ever with some specific file names

Sun Jul 30, 2017 5:36 pm

... crap... replacing the underscores with hyphens doesn't solve it. Now the problem happens with some other random file (page 2):

../HTML_Server_Cache/http---www.era.pt-imoveis--pg-2-o-1-t--a--dd--cc--ff--z--cp--nq--p--ar--ca-00000000-ct-0000-or-41-idioma-pt1_ck-0x3c51fac3.txt

My next test is to rename the whole cache folder to something else, and create a clean empty new one.

Rilhas
Posts: 36
Joined: Sat Aug 04, 2012 7:52 am

Re: Weirdest thing ever with some specific file names

Sun Jul 30, 2017 5:48 pm

Renaming the old directory and creating a new directory solved it.

I would really like to know why, though. I'm sure I'll run into this problem again as the new folder becomes populated, and I'm sure it will let me still create more files except for some specific names. I've already done this a few times in the past, but I never realized the problem disappeared, but still it always came back.

I'm sure I've ran into some EXT4 limitation, when it has too many files in one directory then file name restrictions appear. Doesn't make any sense, but I don't have any other explanation right now.

Return to “Troubleshooting”