User avatar
RPi_Mike
Posts: 100
Joined: Sat Dec 09, 2017 12:57 am
Location: United States

TUTORIAL: Make Your Own Commands to Reveal a File's Hidden Creation Time!

Sun Jun 17, 2018 3:42 pm

INTRODUCTION: Many people know about the stat command. Among other things, it shows you 3 separate timestamps for a file – Modify Time (mtime), Change Time (ctime), and Access Time (atime). But what many people don't know is that there is a hidden 4th timestamp called Creation Time (crtime)! It reveals exactly when a file was originally created. That's what my tutorial explores. More importantly, I will show you how to make five brand-new commands that display a file's creation time and other critical information you wouldn't normally see! Three of the commands will show you the creation times for specific files or entire directories – sorted by your choice of name, ascending time, or descending time. The remaining two commands are turbocharged versions of the stat command – showing you ALL 4 timestamps and other hidden information! But first, here's a brief review of the 4 timestamps – what they offer and how they behave:
xstat_Command_RPi_Mike.png
xstat_Command_RPi_Mike.png (115.69 KiB) Viewed 764 times
This screenshot reveals the xstat command – one of five new commands this tutorial will allow you to create. Right-click image for full-size viewing.

mtime: By far the most useful and commonly used of all the timestamps is Modify Time (mtime). It simply shows the most recent time that a file's CONTENTS were edited or modified in any way. For example, if you open a simple text file, change one letter or sentence inside it, and then save the file, the mtime timestamp will be updated to reflect the change. This is usually the most practical time to know. If you think about the way most people use their computers, the most relevant bit of time-related information would center around basic questions like: "When was the last time I edited that file?". Or, "let me see all the most recent files I've been working on!" Not only that, mtime is also the default timestamp that both File Manager and the ls command use to sort files by time. It's no exaggeration to say that mtime is the king of timestamps!

ctime: Then there's Change Time (ctime). This mostly useless timestamp is doubly unfortunate because a vast number of people think the "c" in "ctime" stands for "creation" time. This is understandable given that both "change" and "creation" misleadingly begin with the same letter! But there's another stroke of bad luck that compounds the confusion: Very few people on Linux systems have ever seen a file's "creation" time! That's because there are no simple commands that will tell you a file's creation time! This state of affairs is due to the arbitrary politics, history and technology of Linux. The stat command, however – which people do use all the time – constantly puts this "ctime" stuff in front of their face. So many just assume "oh, that must be the creation time!". What ctime actually does is pretty lame. If you simply change the name of a file, for example, ctime will reflect that. If you change the contents of a file, ctime will also reflect that. In other words, if you change a file's metadata OR a file's contents, ctime will reflect that. The reason I say it's lame is because if you change a file's contents, mtime will tell you that anyway! And, really, who cares when a file name was changed? I'm sure there are a few "use cases" out there, but it's very limited. The other lame aspect of ctime is that it's non-specific and vague. In other words, if you see that the ctime changed, does that mean the file's contents were changed at that time – or does it mean that only the file name was changed? In isolation – without the aid of other timestamp data to compare it to – you'll never know. Ctime likes to keep things all vague and mysterious like that! In contrast, if mtime indicates a change, you can know for a fact that the file's contents, specifically, were changed!

atime: On the Raspberry and most other Linux-based systems, Access Time (atime) is also quite lame. One might think "oh, this will tell me the last time I opened or "accessed" that file! The problem is that to properly record and keep access times continuously updated, your computer would have to write data to a file every single time you (or your computer) accesses or "reads" it. If you think about it, that could easily DOUBLE the I/O – the input / output activity with your storage device. So every single time a simple "read" occurred, there would also have to be a forced "write" to the same file in order to keep the access time freshly updated. Not only does that add to the wear and tear of flash memory cells (which have a finite number of write cycles before they finally die) – but more importantly, it can dramatically slow down your system. And for what purpose? So you can check out the "access time" on a random file once in a blue moon? As a result, most Linux distributions understandably change the default atime behavior to something less than "constantly updated" – or they switch it off entirely. In general, they either choose a "relatime" setting or a "noatime" setting. I won't bother explaining the complexities and implications of all that because it's simply not relevant to this "creation time" tutorial!

crtime: Mtime is still the king, but if I had to pick a distant second place in the usefulness competition, it would have to be Creation Time (crtime). To be honest, that's not necessarily saying too much, considering how lame both ctime and atime are! Nonetheless, be sure to make note of the barely visible but GIGANTIC little detail that differentiates "crtime" from "ctime". There's an "r" in it! As in Crrrrreation Time! Not Change Time!

Creation time has a big catch – it only works if the file was created on a Linux-based system using the ext2, ext3, or ext4 file systems and has not been transferred to another device or partition. That means if you took a bunch of pictures on your camera, for example, you will NOT be able to get any "real" creation times on the files. That's because most cameras (and many other devices) still use a FAT-based or other non-Linux file system on their storage cards. [There's one obvious exception of course – in the rare event your camera or device happens to use a Linux-based file system, then you might have genuine creation times! But only if you don't transfer them from your camera's storage card. Because even if you hypothetically had a Linux-based camera, once you transfer a file to your Linux-based computer, it will automatically be assigned a new inode and will thus receive a brand-new creation time.]

But just because the pictures (or other files) won't have any "real" creation times, your Linux system will look at things very differently as soon as you TRANSFER the pictures to your system!

You see, from the standpoint of Linux, once a picture has been transferred to your Raspberry, that picture was just "born" – even if you took the picture five years ago! There's at least some rationale for this, after all – that file has just arrived and is completely new to the system in its present form.

So don't be fooled by the false "creation time" of any file – picture or otherwise – once you've transferred it to your Linux computer!

Nonetheless, crtime can occasionally be quite useful – BUT ONLY FOR FILES YOU MADE ON YOUR RASPBERRY THAT HAVEN'T BEEN MOVED TO ANOTHER DEVICE OR PARTITION. A good example of this would be if you kept a personal diary in the form of a text file that you created and maintain on your Raspberry. If you run the stat command on the file, the only majorly useful timestamp that appears will be mtime – which is still, in all likelihood, the most personally relevant time information. After all, it lets you know the last time you updated your diary and added some thoughts to it! But there is one basic question that mtime, ctime, and atime will never answer. And that would be a very basic question: "Hmmmm... when did I start writing my diary? Was it back in October or December? I just can't remember!" That's where crtime can come in handy!




THE PROBLEM: Linux has no convenient, built-in way to determine a Linux file's creation time (crtime). Theodore Ts'o – the MIT-educated lead developer and maintainer of the Linux ext4 userspace utilities – put it this way: "While it is easy to add an extra creation-date field in the inode..... it is more difficult to modify or add the necessary system calls, like stat()..... These changes would require coordination of many projects. So even if ext4 developers implement initial support for creation-date timestamps, this feature will not be available to user programs for now."

As a result, you have to use a very inconvenient 2-step method with very specific parameters to get a Linux file's creation time. To do it, you have to first run the stat command to determine the file's inode number (you can also use the ls -di command, but that doesn't make things any easier). For example, let's say we want to know the creation time for a file called My_Diary.txt. Let's also assume that the file is located in the Raspberry's standard home path – /home/pi (as opposed to being on a USB thumb drive, for example). The first step would be this:

stat My_Diary.txt

File: My_Diary.txt
Size: 48 Blocks: 8 IO Block: 4096 regular file
Device: b307h/45831d Inode: 1195887 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1000/ pi) Gid: ( 1000/ pi)
Access: 2018-06-20 09:06:57.695587026 -0400
Modify: 2018-06-20 09:10:25.358522713 -0400
Change: 2018-06-20 09:10:25.358522713 -0400
Birth: -

Do you see the inode number? In this random example, it's "1195887". You then take that inode number and plug it into the following command line. Before you do that, however, you must also make sure you have the correct "device path" at the end – such as /dev/mmcblk0p7. That's the partition on which the file resides. I'll have much more on this extremely important detail in a moment. But for now, in this example, you would simply run this line in Terminal. And BINGO – we now have something many of you have probably never seen before: crtime! Creation Time!

sudo debugfs -R 'stat <1195887>' /dev/mmcblk0p7

Inode: 1195887 Type: regular Mode: 0644 Flags: 0x80000
Generation: 953692646 Version: 0x00000000:00000001
User: 1000 Group: 1000 Project: 0 Size: 48
File ACL: 0 Directory ACL: 0
Links: 1 Blockcount: 8
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x5b2a5241:557a7d64 -- Wed Jun 20 09:10:25 2018
atime: 0x5b2a5171:a5d74348 -- Wed Jun 20 09:06:57 2018
mtime: 0x5b2a5241:557a7d64 -- Wed Jun 20 09:10:25 2018
crtime: 0x5b2a5171:a5d74348 -- Wed Jun 20 09:06:57 2018
Size of extra inode fields: 32
Inode checksum: 0x2707bc46
EXTENTS:
(0):4763709




THE SOLUTION – ALMOST: The above example is quite the cumbersome procedure just to find out a simple creation time! Fortunately, a very clever applied mathematician and web developer named Igor Moiseev has developed an automated script that gives Linux computers a brand-new command that will tell you the creation time of any Linux-based file. There's just one problem: His script doesn't work on the Raspberry! Here's his original code:

xstat() {
for target in "${@}"; do
inode=$(ls -di "${target}" | cut -d ' ' -f 1)
fs=$(df "${target}" | tail -1 | awk '{print $1}')
crtime=$(sudo debugfs -R 'stat <'"${inode}"'>' "${fs}" 2>/dev/null |
grep -oP 'crtime.*--\s*\K.*')
printf "%s\t%s\n" "${crtime}" "${target}"
done
}




MY SOLUTION: Unfortunately, when testing the author's script on my Raspberry, it refused to generate any timestamps. Instead, it only regurgitated the file names – with a blank, tabbed space just before it.

After considerable investigation, I determined this was due to the somewhat unusual way that the df command behaves on the Raspberry. Most versions of Linux behave "normally" in this regard. But every distribution of Linux is different. The Raspberry is no exception!

As you can see in the author's code, the bash script relies on properly identifying the "file system" that the file resides on. It does this by invoking the df command. In this context, the "file system" simply means the full "device path" that happens to contain the file. Notice I'm not saying the "file path" – I'm saying the "device path". Basically, it's the partition that the file is sitting on. The format it follows is the "device name" (such as mmcblk0) + the partition number appended to the end (such as p7). This path – such as /dev/mmcblk0p7 – is then assigned to the "fs" variable as a character string. The script then uses that string to run the debugfs command on the file's inode number (which it determines by running ls -di on the file). That in turn generates text output from which the file's creation time is extracted!

Unfortunately, on the Raspberry, the df command does not display the actual "device path" for the MAIN partition that people actually use – otherwise known as the "root" partition. For people that use NOOBS, that's partition 7 (p7) on the internal SD card. For people that use "pure" Raspbian (without NOOBS), that's partition 2 (p2) on the internal SD card. Other than an external storage device like a thumb drive, for example, partition 7 (or 2) is where everyone puts their files! As Murphy's Law would dictate, this of course would be the ONLY partition where the script fails! In other words, instead of listing it as "/dev/mmcblk0p7" in the case of NOOBS – or "/dev/mmcblk0p2" in the case of "pure" Raspbian – the df command lists it as "/dev/root" – even though it explicitly lists the actual path for all the other partitions – such as "/dev/mmcblk0p1", "/dev/sda1", etc. Oh well!

To resolve this on my NOOBS-based system, I inserted 3 lines of "if / then" translation code into the middle of the author's script:

if [ "$fs" = "/dev/root" ]; then
fs="/dev/mmcblk0p7"
fi

CRITICAL NOTE: If you use "pure" Raspbian without NOOBS, the partition value in the "fs=" line should be "p2" instead of "p7".

This way, if the fs variable is "inappropriately" assigned "/dev/root" by the df command, it will automatically change the value of the fs variable to "/dev/mmcblk0p7". That completely fixes the problem! Fortunately, the df command returns perfectly "appropriate" results for all other partitions. I thoroughly tested the revised script on both the internal SD card and external storage devices as well (such as a USB thumb drive). Everything works perfectly!

As you'll see, by heavily modifying the original script, I also created four additional commands. One of them, for example, produces a detailed, stat-like output. It also displays crtime -- but it has the added advantage of also displaying ctime, atime, and mtime in a very clean, easy-to-read listing. I modeled its presentation after the stat command.

UPDATE: Despite the complete effectiveness of my original solution, some commenters wanted me to come up with an even "slicker" solution – a method that would completely circumvent the shortcomings of the df command without any need for the 3-line translation of /dev/root to /dev/mmcblk0p7 (in the case of NOOBS-based systems) or /dev/mmcblk0p2 (in the case of "pure" Raspbian systems). I have now accomplished this by replacing the df command in the "fs=" line with the findmnt command! Nonetheless, for historical and educational purposes, I am deliberately leaving my original explanation and notes "as is" – because the issue of the df command displaying an "alias" for the root partition, instead of the explicit device path, is a fundamental problem that is unique to the Raspberry and a minority of other Linux distributions.




INSTALL THE SCRIPT: To make the brand-new crtime, crtime-at, crtime-dt, xstat, and xstat2 commands available in Terminal, open /home/pi with File Manager and click View | Show Hidden to make sure hidden files are visible. Then, right-click the file named ".bashrc" and then click Text Editor. At the very end of the file's pre-existing text, tap the Enter key a few times for some visual separation and paste in the following script for all five commands. Make sure you leave one blank line at the end of the file. Then save the file and close Terminal. That's it! The next time you open Terminal, all the new commands will be available!


crtime() {
for target in "${@}"; do
inode=$(ls -di "${target}" | cut -d ' ' -f 1)
fs=$(findmnt -n -o SOURCE --target "${target}")
crtime=$(sudo debugfs -R 'stat <'"${inode}"'>' "${fs}" 2>/dev/null |
grep -oP 'crtime.*--\s*\K.*')
crtime=$(date -d "${crtime}" +%Y-%m-%d\ %H:%M:%S)
printf "%s\t%s\n" "${crtime}" "${target}"
done
}


crtime-at() {
for target in *; do
inode=$(ls -di "${target}" | cut -d ' ' -f 1)
fs=$(findmnt -n -o SOURCE --target "${target}")
crtime=$(sudo debugfs -R 'stat <'"${inode}"'>' "${fs}" 2>/dev/null |
grep -oP 'crtime.*--\s*\K.*')
crtime=$(date -d "${crtime}" +%Y-%m-%d\ %H:%M:%S)
printf "%s\t%s\n" "${crtime}" "${target}" >> /tmp/crtime
done
crtime=$(sort -t$'\t' -k1 -n /tmp/crtime)
printf "${crtime}"
rm /tmp/crtime
echo
}


crtime-dt() {
for target in *; do
inode=$(ls -di "${target}" | cut -d ' ' -f 1)
fs=$(findmnt -n -o SOURCE --target "${target}")
crtime=$(sudo debugfs -R 'stat <'"${inode}"'>' "${fs}" 2>/dev/null |
grep -oP 'crtime.*--\s*\K.*')
crtime=$(date -d "${crtime}" +%Y-%m-%d\ %H:%M:%S)
printf "%s\t%s\n" "${crtime}" "${target}" >> /tmp/crtime
done
crtime=$(sort -t$'\t' -k1 -nr /tmp/crtime)
printf "${crtime}"
rm /tmp/crtime
echo
}


xstat() {
for target in "${@}"; do
inode=$(ls -di "${target}" | cut -d ' ' -f 1)
fs=$(findmnt -n -o SOURCE --target "${target}")
stat=$(stat "${target}")
debugfs=$(sudo debugfs -R 'stat <'"${inode}"'>' "${fs}" 2>/dev/null)
crtime=$(echo "$debugfs" | grep -oP "(?<=crtime: ).*?(?=:)")
crnano=$(echo "$debugfs" | grep -oP "(?<=crtime: ).*?(?<=:)\K(?<=:).*?(?= -- )")
crnano="0x$crnano"
crtime=$(date -d @$(printf %d "$crtime").$(( $(printf %d "$crnano") / 4 )) +'%F %T.%N %z')
printf "${stat::-9}"
printf "Create: ${crtime}"
echo
done
}


xstat2() {
for target in "${@}"; do
inode=$(ls -di "${target}" | cut -d ' ' -f 1)
fs=$(findmnt -n -o SOURCE --target "${target}")
stat=$(stat "${target}")
debugfs=$(sudo debugfs -R 'stat <'"${inode}"'>' "${fs}" 2>/dev/null)
crtime=$(echo "$debugfs" | grep -oP "(?<=crtime: ).*?(?=:)")
crnano=$(echo "$debugfs" | grep -oP "(?<=crtime: ).*?(?<=:)\K(?<=:).*?(?= -- )")
crnano="0x$crnano"
crtime=$(date -d @$(printf %d "$crtime").$(( $(printf %d "$crnano") / 4 )) +'%F %T.%N %z')
tput bold; echo -e "\e[97mXSTAT RESULTS:"; tput sgr0
printf "${stat::-9}"
printf "Create: ${crtime}"
echo; echo
tput bold; echo -e "\e[97mINODE STRUCTURE:"; tput sgr0
printf "${debugfs}"
echo
done
}


UPDATE: I have now made the above script "slicker" by eliminating the 3-line translation of /dev/root to /dev/mmcblk0p7 (in the case of NOOBS-based systems) or /dev/mmcblk0p2 (in the case of "pure" Raspbian systems). Instead, I have now replaced the df command in the "fs=" line with the findmnt command – thus eliminating the need for any translation of the device path.




CRTIME COMMAND: The crtime command is what you want if you're looking for the creation time of a specific file. It's also what you want for getting the creation times for an entire directory – sorted alphanumerically by file name. Be aware that crtime is the only command that requires an asterisk to see an entire directory listing. I did that on purpose because the crtime command is designed to be flexible in showing you creation time for files by name. The other two commands, however, are focused exclusively on directories – sorting their files by ascending time (crtime-at) or descending time (crtime-dt). They do not need an asterisk, nor will they even work with one (again, that's by design). But first, let's say you want to know the creation time of your diary. Open Terminal inside the file's folder and run this:


crtime My_Diary.txt

You will then get this response (in Terminal, a tabbed space, not dots, appears between the 2 fields):

2018-06-20 09:06:57..............My_Diary.txt



The crtime command also supports the full path to a file (if there are spaces in the file name or path, be sure to place the entire path inside double-quote marks):

crtime /home/pi/My_Diary.txt



This command generates an alphanumeric sort of an entire directory by file name (note the space between the command and the asterisk):

crtime *

It then generates a nice clean listing of creation times like this (in Terminal, a tabbed space, not dots, appears between the 2 fields):

2018-06-20 07:32:04..............Eighth Picture Taken.jpg
2018-06-20 07:29:18..............Fifth Picture Taken.jpg
2018-06-20 07:25:32..............First Picture Taken.jpg
2018-06-20 07:28:26..............Fourth Picture Taken.jpg
2018-06-20 07:26:25..............Second Picture Taken.jpg
2018-06-20 07:31:20..............Seventh Picture Taken.jpg
2018-06-20 07:30:07..............Sixth Picture Taken.jpg
2018-06-20 07:27:14..............Third Picture Taken.jpg



You can also use the full path to a directory (note that the directory path must end with a forward slash + asterisk; also note that if you do things this way, it will list the file names along with their full paths – so if you want a cleaner presentation with just the file names, be sure to avoid this method and simply open Terminal inside the directory first and then run "crtime *" as shown above):

crtime /home/pi/Downloads/*




CRTIME-AT COMMAND: This lists the contents of a directory by Ascending (Creation) Time. Hence the "AT":

crtime-at

2018-06-20 07:25:32..............First Picture Taken.jpg
2018-06-20 07:26:25..............Second Picture Taken.jpg
2018-06-20 07:27:14..............Third Picture Taken.jpg
2018-06-20 07:28:26..............Fourth Picture Taken.jpg
2018-06-20 07:29:18..............Fifth Picture Taken.jpg
2018-06-20 07:30:07..............Sixth Picture Taken.jpg
2018-06-20 07:31:20..............Seventh Picture Taken.jpg
2018-06-20 07:32:04..............Eighth Picture Taken.jpg




CRTIME-DT COMMAND: This lists the contents of a directory by Descending (Creation) Time. Hence the "DT":

crtime-dt

2018-06-20 07:32:04..............Eighth Picture Taken.jpg
2018-06-20 07:31:20..............Seventh Picture Taken.jpg
2018-06-20 07:30:07..............Sixth Picture Taken.jpg
2018-06-20 07:29:18..............Fifth Picture Taken.jpg
2018-06-20 07:28:26..............Fourth Picture Taken.jpg
2018-06-20 07:27:14..............Third Picture Taken.jpg
2018-06-20 07:26:25..............Second Picture Taken.jpg
2018-06-20 07:25:32..............First Picture Taken.jpg




XSTAT COMMAND: STAT + CREATION TIME: This gives you full, unmodified results from the stat command PLUS creation time – in a time & date format that perfectly matches stat's format! All 4 timestamps are displayed – Access Time (atime), Modify Time (mtime), Change Time (ctime), and Create Time (crtime). From now on, I will personally use this instead of the standard stat command. Why? Because there's no reason not to! It's a superset of stat. It gives you everything the stat command does – in exactly the same way – PLUS it gives you the file's creation time in the last line. When it comes to timestamps, your brand-new xstat command is like a turbocharged version of the stat command!

UPDATE: I have now upgraded the xstat command to display the creation time out to the nanosecond! Previously, it only displayed the time out to the second. To accomplish this, I developed a variety of regular expressions that allowed me to make use of a clever technique described by user don_crissti at unix.stackexchange.com (who, in turn, based his findings on the work of Hal Pomeranz, a computer security expert). Instead of extracting the human-friendly version of crtime that's displayed in the debugfs output, it derives the crtime directly from the timestamp's hexadecimal code (which is also listed in the debugfs output). Unlike the human-friendly format, however, the hex code also provides nanosecond data!

xstat My_Diary.txt

File: My_Diary.txt
Size: 48 Blocks: 8 IO Block: 4096 regular file
Device: b307h/45831d Inode: 1195887 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1000/ pi) Gid: ( 1000/ pi)
Access: 2018-06-20 09:06:57.695587026 -0400
Modify: 2018-06-20 09:10:25.358522713 -0400
Change: 2018-06-20 09:10:25.358522713 -0400
Create: 2018-06-20 09:06:57.695587026 -0400




XSTAT2 COMMAND: XSTAT + INODE STRUCTURE: This gives you all the benefits of the xstat command PLUS the inode structure (which is generated by the debugfs command). As you can see, the inode structure also reveals all 4 timestamps – Change Time (ctime), Access Time (atime), Modification Time (mtime), and Creation Time (crtime). It presents them in the native time & date format preferred by debugfs.

xstat2 My_Diary.txt

XSTAT RESULTS:
File: My_Diary.txt
Size: 48 Blocks: 8 IO Block: 4096 regular file
Device: b307h/45831d Inode: 1195887 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1000/ pi) Gid: ( 1000/ pi)
Access: 2018-06-20 09:06:57.695587026 -0400
Modify: 2018-06-20 09:10:25.358522713 -0400
Change: 2018-06-20 09:10:25.358522713 -0400
Create: 2018-06-20 09:06:57.695587026 -0400

INODE STRUCTURE:
Inode: 1195887 Type: regular Mode: 0644 Flags: 0x80000
Generation: 953692646 Version: 0x00000000:00000001
User: 1000 Group: 1000 Project: 0 Size: 48
File ACL: 0 Directory ACL: 0
Links: 1 Blockcount: 8
Fragment: Address: 0 Number: 0 Size: 0
ctime: 0x5b2a5241:557a7d64 -- Wed Jun 20 09:10:25 2018
atime: 0x5b2a5171:a5d74348 -- Wed Jun 20 09:06:57 2018
mtime: 0x5b2a5241:557a7d64 -- Wed Jun 20 09:10:25 2018
crtime: 0x5b2a5171:a5d74348 -- Wed Jun 20 09:06:57 2018
Size of extra inode fields: 32
Inode checksum: 0x2707bc46
EXTENTS:
(0):4763709




NOTE #1: As you may have noticed, I changed the name of Igor's original "xstat" script to "crtime". I thought this made more sense since it's not really producing a full, stat-like output on the file. Instead, the output is very clean and simple – it simply gives you the creation time! So I decided to call it the crtime command – because that's what it is! But I am calling the "xstat" commands xstat and xstat2 – because they really are generating a very detailed, stat-like output!

NOTE #2: If your brand-new commands don't return a timestamp – and instead display a blank, tabbed space before the file name – it means there is no creation time in the file's metadata – or it means the file is on a FAT-based or other non-Linux partition! Such a result, therefore, is not a bug – it means the data itself does not exist! As the man pages indicate, "The debugfs program is an interactive file system debugger. It can be used to examine and change the state of an ext2, ext3, or ext4 file system." In other words, it's definitely not intended for FAT or other non-Linux file systems!

NOTE #3: I ran "sudo fdisk -l" on my system so that I could explain to everyone where the debugfs command works (and doesn't work) – and therefore where the script will work (and won't work). The bottom line is that it works EVERYWHERE that's relevant! Keep in mind that when it comes to your internal SD card, the only partition that's relevant in terms of the script is either partition 7 (p7) in the case of NOOBS or partition 2 (p2) in the case of "pure" Raspbian without NOOBS (I explain my reasoning on all this below). Also, rest assured that the script AUTOMATICALLY identifies the correct partition – so if you're flipping back and forth between an external Linux-based thumb drive partition and the internal SD card partition, for example, everything will work perfectly on its own. Just be aware that the debugfs command doesn't work when it's simply not applicable – so you should understand the details I've listed below. My examples are drawn from my NOOBS-based system – but the underlying concepts are equally applicable to a "pure" Raspbian system without NOOBS:


THIS IS A TINY LINUX EXT 4 PARTITION. IT'S ONLY 32 MB. FILE CREATION TIMES DON'T EXIST FOR THESE YEAR 1969 / 1970 SYSTEM FILES – SO THE SCRIPT IS OBVIOUSLY NOT APPLICABLE TO THEM:

mmcblk0p5......32M....../media/pi/SETTINGS


THIS IS A TINY W95 FAT-BASED PARTITION. IT'S ONLY 69 MB. LINUX FILE CREATION TIMES DO NOT EXIST ON FAT PARTITIONS! SO THE SCRIPT IS OBVIOUSLY NOT APPLICABLE TO THEM. INTERNALLY, THE DEBUGFS COMMAND WILL THROW A "Bad magic number" ERROR IF IT'S ASKED TO ANALYZE A FILE ON A NON-LINUX PARTITION. [On a "pure" Raspbian system without NOOBS, this would be the rough equivalent of mmcblk0p1 instead.]

mmcblk0p6......69M....../boot


THIS IS YOUR MAIN (OR "ROOT") PARTITION ON YOUR INTERNAL SD CARD – THE ONE YOU ACTUALLY USE ALL THE TIME. ON A 32 GB SD CARD, IT'S OVER 28 GB IN SIZE. FILE CREATION TIMES SHOULD EXIST FOR ALL (OR ALMOST ALL) FILES. [On a "pure" Raspbian system without NOOBS, it would be mmcblk0p2 instead.]

mmcblk0p7....28.4G....../


THIS IS AN EXAMPLE OF A USB THUMB DRIVE. ASSUMING YOU FORMATTED IT TO A LINUX-BASED ext2, ext3, or ext4 FILE SYSTEM, FILE CREATION TIMES SHOULD EXIST FOR ALL (OR ALMOST ALL) FILES. HOWEVER, IF YOU'RE USING YOUR THUMB DRIVE "AS IS" – IN OTHER WORDS, THE WAY IT CAME WHEN YOU FIRST BOUGHT IT – THERE'S AN EXCELLENT CHANCE IT CAME PRE-FORMATTED AS A FAT DEVICE. IF SO, LINUX-BASED CREATION TIMES WILL NOT EXIST. MOST STORAGE CARDS ARE SOLD IN FAT-32 FORMAT BECAUSE IT'S STILL THE CLOSEST THING TO A "UNIVERSAL" FILE SYSTEM THAT MOST COMPUTERS CAN RECOGNIZE.

sda1......29.9G....../media/pi/THUMB_DRIVE




TIMESTAMP PRESERVATION BEHAVIOR: As you can see, I've extensively tested and analyzed how all 4 timestamps are affected by common scenarios. All tests were performed on a pristine, completely standard copy of Raspbian Stretch on my Raspberry 3. Both the internal partition and the external partition (a USB thumb drive, in my case) were using the standard Linux ext4 filesystem. If you transfer files to a partition running an "alien" filesystem, such as FAT, these results will not be applicable. In the case of File Manager, "copying" means "copy and paste" – and "moving" means "cut and paste":
Timestamp_Preservation_Behavior_RPi_Mike.png
Timestamp_Preservation_Behavior_RPi_Mike.png (91.32 KiB) Viewed 711 times
Last edited by RPi_Mike on Sat Jun 30, 2018 6:08 am, edited 19 times in total.

ejolson
Posts: 2050
Joined: Tue Mar 18, 2014 11:47 am

Re: TUTORIAL: Make Your Own Command to Reveal a File's Hidden Creation Time!

Sun Jun 17, 2018 9:13 pm

RPi_Mike wrote:
Sun Jun 17, 2018 3:42 pm
To resolve this, I inserted 3 lines of "if / then" code into the middle of the author's script:

if [ "$fs" = "/dev/root" ]; then
fs="/dev/mmcblk0p7"
fi
Thanks! I learned something new about Linux ext filesystems. Does this work for NOOBS installs as well as standard Raspbian installs? Do any other filesystems, such as BTRFS, ZFS or XFS support crtime? Does GNU tar preserve them? What about cp -p?

By the way, are you planning to write a book?

User avatar
RPi_Mike
Posts: 100
Joined: Sat Dec 09, 2017 12:57 am
Location: United States

Re: TUTORIAL: Make Your Own Commands to Reveal a File's Hidden Creation Time!

Mon Jun 18, 2018 2:11 pm

ejolson wrote:
Sun Jun 17, 2018 9:13 pm
Thanks! I learned something new about Linux ext filesystems.

I use NOOBS exclusively, so that's what I used to develop and test this tutorial. In fact, I wrote the "sticky" on how to update NOOBS without re-installing it. You'll see it at the very top of the General Discussion forum.

However, I realize that a lot of people also use "pure" Raspbian without NOOBS. So I took the time this morning to validate the script for that platform as well. Pure Raspbian has only 2 partitions instead of the 5 partitions you'll find on a NOOBS-based system, so the partition number in the script needs to reflect that. [p7 on NOOBS-based systems is the equivalent of p2 on "pure" Raspbian systems.]

I have also added a brand-new command to the script – so definitely check out my newly-updated tutorial!

As for your technical question, I have never used the BTRFS, ZFS or XFS file systems. But I do know that the debugfs command explicitly states in its documentation that it is solely designed to work with ext2, ext3, and ext4 file systems. Given that the only reputable method to extract creation time relies on the debugfs command, my guess would be that it does not work on other file systems.

As for your GNU tar and cp -p questions, it sounds like some fun trivia to investigate. Maybe if you look into it, you can let me know the answer.

And finally, no – I have no plans to write any books!

ejolson
Posts: 2050
Joined: Tue Mar 18, 2014 11:47 am

Re: TUTORIAL: Make Your Own Commands to Reveal a File's Hidden Creation Time!

Tue Jun 19, 2018 7:33 pm

RPi_Mike wrote:
Mon Jun 18, 2018 2:11 pm
Pure Raspbian has only 2 partitions instead of the 5 partitions you'll find on a NOOBS-based system, so the partition number in the script needs to reflect that. [p7 on NOOBS-based systems is the equivalent of p2 on "pure" Raspbian systems.]
Rather than using df to find the device for the root partition it might be more reliable to parse the output of mount.

n67
Posts: 839
Joined: Mon Oct 30, 2017 4:55 pm

Re: TUTORIAL: Make Your Own Commands to Reveal a File's Hidden Creation Time!

Tue Jun 19, 2018 9:43 pm

ejolson wrote:
Tue Jun 19, 2018 7:33 pm
RPi_Mike wrote:
Mon Jun 18, 2018 2:11 pm
Pure Raspbian has only 2 partitions instead of the 5 partitions you'll find on a NOOBS-based system, so the partition number in the script needs to reflect that. [p7 on NOOBS-based systems is the equivalent of p2 on "pure" Raspbian systems.]
Rather than using df to find the device for the root partition it might be more reliable to parse the output of mount.
ITYM, rather than hardcoding the devicename for the root filesystem (i.e., doing an inplace translation of /dev/root to /dev/mmcblk0p7 {*}), you should use mount or lsblk or something like that to do a correct translation.

The point is that you want this to work for any/all filesystems you might have mounted on your machine, so you want to first using something like "df" to do the initial translation.

In fact, I think using df is pretty awkward. And, in fact, stat (which OP relies on to get the inode number) also gets us the device number. Observe:

Code: Select all

$ lsblk | awk  -v majmin=$(stat /boot/config.txt  | awk -F/ '/^Device:/ { t = int($2/256);print t":"($2-t*256) }') '$2 == majmin { print "/dev/" substr($1,3) }'
/dev/mmcblk0p6

{*} With, of course, the need for a further correction if it happens to be: /dev/mmcblk0p2.
"L'enfer, c'est les autres"

If a post offends you, just put that poster on your foes list, and be done with it (and with them).

To do otherwise, risks being banned.

User avatar
RPi_Mike
Posts: 100
Joined: Sat Dec 09, 2017 12:57 am
Location: United States

Re: TUTORIAL: Make Your Own Commands to Reveal a File's Hidden Creation Time!

Wed Jun 20, 2018 2:01 am

ejolson wrote:
Tue Jun 19, 2018 7:33 pm
Rather than using df to find the device for the root partition it might be more reliable to parse the output of mount.

I wouldn't use the term "more reliable" – because my original script is perfectly reliable. I say this with great confidence because I put in the HARD WORK to thoroughly test my method on BOTH NOOBS-based and "pure" Raspbian systems. In fact, even though I never use "pure" Raspbian, I actually spent an hour of my time installing a "pure" copy of Raspbian from scratch – just to confirm that my instructions work on that platform as well! Amazingly enough, my script varies by only ONE character between the two platforms! If you're using a NOOBS-based system, that single character is a "7". If you're using "pure" Raspbian, it's a "2". That's about as laid-back as any requirement could possibly get for two different platforms!

Nonetheless, I know what you're getting at. Although it doesn't make my method any more "reliable", you're looking for a SLICKER solution!

If you check out my tutorial again, you'll see that I've done exactly that.



n67 wrote:
Tue Jun 19, 2018 9:43 pm
ITYM, rather than hardcoding the devicename for the root filesystem (i.e., doing an inplace translation of /dev/root to /dev/mmcblk0p7 {*}), you should use mount or lsblk or something like that to do a correct translation. {*} With, of course, the need for a further correction if it happens to be: /dev/mmcblk0p2.

There is no "further correction". It's simply a matter of two different platforms – one NOOBS, the other "pure" Raspbian. My original script requires a difference of only ONE character between them! If you use NOOBS, you use one version of the script. If you use "pure" Raspbian, you use the other version of the script. We're talking a total difference of a "7" versus a "2"! Slightly different instructions – insanely slight – for two different platforms. Sounds extremely reasonable to me!



n67 wrote:
Tue Jun 19, 2018 9:43 pm
The point is that you want this to work for any/all filesystems you might have mounted on your machine, so you want to first using something like "df" to do the initial translation.

Please see above. I know what you're getting at – but assuming people actually follow my instructions, my original tutorial works just fine on the two official Raspberry platforms. As for wanting it to work on "all" filesystems: Wow, that's pretty demanding! Even FAT-16 and FAT-32? As I explained in great detail in my tutorial, the debugfs command's documentation clearly indicates that it is only intended to work on the ext2, ext3, and ext4 Linux file systems.



n67 wrote:
Tue Jun 19, 2018 9:43 pm
In fact, I think using df is pretty awkward. And, in fact, stat (which OP relies on to get the inode number) also gets us the device number. Observe:

Code: Select all

$ lsblk | awk  -v majmin=$(stat /boot/config.txt  | awk -F/ '/^Device:/ { t = int($2/256);print t":"($2-t*256) }') '$2 == majmin { print "/dev/" substr($1,3) }'
/dev/mmcblk0p6

That's quite the long and complicated code! On my pristine system, your command line generates this output:

/dev/�─mmcblk0p6

That weird symbol, which is hard to make out in a small font, is a black square with a white question mark inside it. It's also followed, oddly enough, by what appears to be an em dash. Those weird characters would definitely throw off the script!

Anyway, trust me -- I know what both of you guys are getting at! You're looking for a solution that's SLICKER than a tiny bit of translation code. So instead of using lsblk and awk and complicated string parsing, you can simply use this elegant solution based on the findmnt command:

findmnt -n -o SOURCE --target /boot/config.txt

That produces the following very clean output:

/dev/mmcblk0p6

More importantly, it does NOT generate an "alias" for the root partition. As you'll see in my tutorial, I have now updated the script and added the following paragraph:

UPDATE: Despite the complete effectiveness of my original solution, some commenters wanted me to come up with an even "slicker" solution – a method that would completely circumvent the shortcomings of the df command without any need for the 3-line translation of /dev/root to /dev/mmcblk0p7 (in the case of NOOBS-based systems) or /dev/mmcblk0p2 (in the case of "pure" Raspbian systems). I have now accomplished this by replacing the df command in the "fs=" line with the findmnt command! Nonetheless, for historical and educational purposes, I am deliberately leaving my original explanation and notes "as is" – because the issue of the df command displaying an "alias" for the root partition, instead of the explicit device path, is a fundamental problem that is unique to the Raspberry and a minority of other Linux distributions.

ejolson
Posts: 2050
Joined: Tue Mar 18, 2014 11:47 am

Re: TUTORIAL: Make Your Own Commands to Reveal a File's Hidden Creation Time!

Wed Jun 20, 2018 2:40 am

RPi_Mike wrote:
Wed Jun 20, 2018 2:01 am
My original script requires a difference of only ONE character between them!
The findmnt command seems like a good choice. Thanks for taking the time to update your post.

User avatar
scruss
Posts: 1831
Joined: Sat Jun 09, 2012 12:25 pm
Location: Toronto, ON
Contact: Website

Re: TUTORIAL: Make Your Own Commands to Reveal a File's Hidden Creation Time!

Wed Jun 20, 2018 2:53 am

Does crtime stay the same if you move it to a different filesystem that supports it?

also:
  • s/Raspberry/& Pi/g;
  • l*me: you're better than this, don't use it here.
‘Remember the Golden Rule of Selling: “Do not resort to violence.”’ — McGlashan.

User avatar
RPi_Mike
Posts: 100
Joined: Sat Dec 09, 2017 12:57 am
Location: United States

Re: TUTORIAL: Make Your Own Commands to Reveal a File's Hidden Creation Time!

Wed Jun 20, 2018 3:28 am

scruss wrote:
Wed Jun 20, 2018 2:53 am
Does crtime stay the same if you move it to a different filesystem that supports it?

Although I've taken a fairly deep dive on the topic of crtime, it's only been in the context of common Linux systems like the Raspberry. So I'm certainly not some kind of guru that's familiar with the esoteric timestamping behavior of other filesystems. Unless you're asking a purely academic question that has nothing to do with anything you personally use, perhaps you can do a quick test and let us know. All it would take to find out is a quick transfer of a single test file to another filesystem!

User avatar
scruss
Posts: 1831
Joined: Sat Jun 09, 2012 12:25 pm
Location: Toronto, ON
Contact: Website

Re: TUTORIAL: Make Your Own Commands to Reveal a File's Hidden Creation Time!

Wed Jun 20, 2018 1:11 pm

Yeah, I really should've tried it, shouldn't I …? So:

Original file has crtime = mtime.

Copying from one ext4 filesystem to another:
  • with cp, no arguments: crtime = ctime = new mtime (in other words, all is lost)
  • with cp -p or cp --preserve=all: crtime = new ctime, mtime = original crtime
So to me, mtime looks a little more robust than crtime. I kind-of expected this, but didn't know for sure until now.

Filesystem metadata is fleeting, explicit timestamps or version control info less so.
‘Remember the Golden Rule of Selling: “Do not resort to violence.”’ — McGlashan.

n67
Posts: 839
Joined: Mon Oct 30, 2017 4:55 pm

Re: TUTORIAL: Make Your Own Commands to Reveal a File's Hidden Creation Time!

Wed Jun 20, 2018 1:54 pm

As is often the case, the real question is: How would you use this new information?

Note: OP may have answered this somewhere. I don't claim to have read every word.

Historically, whenever this question has come up on various forums, people have raised objections as to what exactly creation time would mean. If you copy a file, is it preserved? If so, then it's not accurate, since obviously the copy was created "now" and not "then". If not, then the copy isn't really a copy. Note that some OSes address this by having a second sort of "copy" command called something like "clone" - that is specified as making an exact copy of the original - including all metadata.

Similarly, if you restore the file from backup. Should it restore the original creation time? Same paradox.

So, tying these two points together, if you (i.e., software) use(s) (i.e., depend(s) on) "crtime", what semantics will it have? It is, paradoxically, one of those things where correct semantics makes it more or less useless. It's only useful if the semantics aren't quite right. And down that road lies chaos.

A couple of other notes:

1) The "stat" output contains something called "Birth" - which is supposed to be the same thing as what we are talking about here - but (IME) it is always blank. Why can't that be what we are seeking?

2) I like how OP threw various bits of shade on "atime" and "ctime". Very amusing. Well done. FWIW, I once built a system that depended on ctime. It failed badly when the system administrator restored from a backup tape. I imagine anything based on this "crtime" would fail similarly. Which goes back to my question of "How would you use this?"
"L'enfer, c'est les autres"

If a post offends you, just put that poster on your foes list, and be done with it (and with them).

To do otherwise, risks being banned.

User avatar
RPi_Mike
Posts: 100
Joined: Sat Dec 09, 2017 12:57 am
Location: United States

Re: TUTORIAL: Make Your Own Commands to Reveal a File's Hidden Creation Time!

Wed Jun 20, 2018 2:21 pm

scruss wrote:
Wed Jun 20, 2018 1:11 pm
Copying from one ext4 filesystem to another:

When you asked if crtime stays the same "if you move it to a different filesystem that supports it", I thought you meant a different filesystem – as in some kind of different, exotic filesystem technology I'm not familiar with that somehow "supports" crtime. I didn't know you simply meant transferring stuff from one ext4 partition to another!

When I have the chance, I will look into this matter further and produce a graphic chart with "false" but conceptually accurate time values that make it super simple to understand exactly what happens to each timestamp under different conditions.

User avatar
RPi_Mike
Posts: 100
Joined: Sat Dec 09, 2017 12:57 am
Location: United States

Re: TUTORIAL: Make Your Own Commands to Reveal a File's Hidden Creation Time!

Wed Jun 20, 2018 3:38 pm

MAJOR UPDATE: I have now created a total of 5 commands. Even the ones that existed before have been enhanced.

One key improvement was converting the awkward-looking, unsortable date format generated by the debugfs command – which separates the year from the date and sticks the time in the middle:

Wed Jun 20 09:06:57 2018

To this enhanced, numerically sortable version:

2018-06-20 09:06:57

By putting this in my script:

crtime=$(date -d "${crtime}" +%Y-%m-%d\ %H:%M:%S)

I also made extensive use of the sort command to create a full "File Manager like" flexibility in sorting files by creation date – both ascending and descending.

So check out my 5 new commands in both the script and examples section!

PS: Be sure to scroll up to the very top – I just added a shiny new screenshot that I'm sure everyone will find mesmerizing. Haha.

User avatar
RPi_Mike
Posts: 100
Joined: Sat Dec 09, 2017 12:57 am
Location: United States

Re: TUTORIAL: Make Your Own Commands to Reveal a File's Hidden Creation Time!

Thu Jun 21, 2018 6:46 pm

YET ANOTHER UPDATE: I have now added a detailed graphic to my tutorial that reveals how all 4 timestamps are affected by common scenarios. So scroll up and check it out!

Return to “General programming discussion”