User avatar
OneMadGypsy
Posts: 326
Joined: Wed Apr 28, 2021 1:57 am
Location: New Orleans, Louisiana
Contact: Website

Time

Sat May 22, 2021 3:33 am

When files are created with open the creation date is in the 4 billions. I put my card in a pc and the date it spit out was correct, but the number is way too high for the pico. It gives me an OverflowError. I have tried to figure out what 4billion+ is in reference to, but to no satisfactory answer. If I divide the number by 2.5568 it gets me really close, but WTH is a 2.5568? Surely that's just wrong, and unless time starts counting 2.5568 times faster that number is only going to be right for a very short amount of time. I've tried, shifting, masking, reading the bits in a different endian, attempting to determine what time before Christ this is supposed to refer to :D. I can't figure it out. What's the correct formula? There aren't 4billion+ seconds between right now and 1/1/1970. According to some rough math, that doesn't account for leap years, 4billion+ seconds is more like 1/1/1900. That can't be right. There's something I need to shift out of this number, or something. Help.
Last edited by OneMadGypsy on Sat May 22, 2021 3:45 am, edited 1 time in total.
"Focus is a matter of deciding what things you're not going to do." ~ John Carmack

User avatar
jbeale
Posts: 3884
Joined: Tue Nov 22, 2011 11:51 pm
Contact: Website

Re: Time

Sat May 22, 2021 3:43 am

Sometimes I've had endian-ness problems that sound like that (number looks right on one architecture, and very wrong on another) but I don't know if that's the case here.

User avatar
OneMadGypsy
Posts: 326
Joined: Wed Apr 28, 2021 1:57 am
Location: New Orleans, Louisiana
Contact: Website

Re: Time

Sat May 22, 2021 3:56 am

I tried the endian thing, and I don't believe that's the problem. My reason for not believing that is because dividing by 2.5568 spits out almost correct dates across a few days of dates. If it was an endian problem I would expect the dates to be all over the place., as the MSB's would be the ones changing constantly forcing the years to radically change from one date to the next. However, I know dividing by that nonsensical number can't be the answer either. I mean, what does 2.5568 refer to? Nothing, that's what. Nothing I am aware of, anyway. It just how much longer from 1970 to now has passed since whatever year it was that date started from, if I take it on face value. I've never heard of an epoch of "1900" (more like 1890) so, there is something else going on.
"Focus is a matter of deciding what things you're not going to do." ~ John Carmack

User avatar
OneMadGypsy
Posts: 326
Joined: Wed Apr 28, 2021 1:57 am
Location: New Orleans, Louisiana
Contact: Website

Re: Time

Sat May 22, 2021 10:26 am

This is an ordeal. I have a lot of info. I'm still not to a full conclusion, but I am very close. The top script is from oofatfs/ff.c and the lower is from oofatfs/ff.h. As you can see from the little chunk of console, tm has overflowed. This is exactly where the date is coming from. There is actually something else that is interesting. When you call uos.stat() it doesn't give you access, modified and creation time. It only gives you modified time in all 3 spots, and modified is the only one that is wrong. You may think, OK well just go look at GET_FATTIME alias of get_fattime and see what the code is doing. Well, that's the really weird part ~ not a single file that is included, nor any files that those include actually define get_fattime. It's like magic. The chain has ended and the only thing that even remotely resembles a definition is that little declaration in the bottom script. Maybe that's the whole problem, but the overflow number keeps changing, as if it it really is getting something like time from somewhere. I've even changed the tm in my print to get_fattime() and I get the same overflowed results.

EDIT:
Ya' know. I used to program in QuakeC (which is a very sub subset of C), and in QC you could declare something all the way in the beginning use it all over the middle and define it at the very end. I'm going the wrong direction. I'm worried about what ff.c includes. I need to be looking at what includes ff.c or maybe even what includes that. I'm going to find this function and fix this bug. Out of the like 50 get_fattime definitions you will find scattered throughout the entire micropython source, at least one of them is garbage and it's the one I need. Of course, right? What I don't understand is how accessed and created are correct, but modified is not. I mean, is one file really using 2 different ways to get time?

Image
"Focus is a matter of deciding what things you're not going to do." ~ John Carmack

hippy
Posts: 10294
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Time

Sat May 22, 2021 10:55 am

OneMadGypsy wrote:
Sat May 22, 2021 3:33 am
When files are created with open the creation date is in the 4 billions. I put my card in a pc and the date it spit out was correct, but the number is way too high for the pico. It gives me an OverflowError.
4 billion is an unsigned 32-bit value (0xEE6B2800) and a Pico is able to handle that, and larger.
OneMadGypsy wrote:
Sat May 22, 2021 3:33 am
I have tried to figure out what 4billion+ is in reference to, but to no satisfactory answer.
It looks like you are reading a FAT time stamp, where -

Time is stored in 16-bits as - 'hhhh-hmmm-mmms-ssss' - where seconds(s) has two second granularity, minutes (m) is 0-59, hour (h) is 0-23.

Date is stored in 16-bit as - 'yyyy-yyym-mmmd-dddd' - where year (y) is 0-127, which should be read as 1980+y, month (m) is 1-12, day (d) is 1-31.
Last edited by hippy on Sat May 22, 2021 11:36 am, edited 1 time in total.

User avatar
OneMadGypsy
Posts: 326
Joined: Wed Apr 28, 2021 1:57 am
Location: New Orleans, Louisiana
Contact: Website

Re: Time

Sat May 22, 2021 11:04 am

4 billion is an unsigned 32-bit value (0xEE6B2800) and a Pico is able to handle that, and larger.
localtime and gmtime do not like this number. Both throw an OverflowError
"Focus is a matter of deciding what things you're not going to do." ~ John Carmack

hippy
Posts: 10294
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Time

Sat May 22, 2021 11:34 am

OneMadGypsy wrote:
Sat May 22, 2021 11:04 am
localtime and gmtime do not like this number. Both throw an OverflowError
Ah yes; I recall seeing someone note that issue. It might be worth raising it to get it fixed if no one else has.

You may be able to subtract a value to get it into a range it accepts. Google probably has an answer for what such a value should be which avoids leap year issues and means you only have to add to the year when done.

When you read the time stamp on your PC; what date and time did it report ? That will help greatly in trying to figure out how -222942907 maps to that.

User avatar
OneMadGypsy
Posts: 326
Joined: Wed Apr 28, 2021 1:57 am
Location: New Orleans, Louisiana
Contact: Website

Re: Time

Sat May 22, 2021 11:49 am

Well, I tried what you suggested and I'm either doing something wrong or the modified date is wrong. I even tried flipping the bytes (the commented line). My pico time is correct. RShell sets it and tells me what it sets it to. Plus prinitng the values on PC shows that the accessed and created are correct but the modified is wrong.

Image
"Focus is a matter of deciding what things you're not going to do." ~ John Carmack

hippy
Posts: 10294
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Time

Sat May 22, 2021 11:54 am

OneMadGypsy wrote:
Sat May 22, 2021 10:26 am
that's the really weird part ~ not a single file that is included, nor any files that those include actually define get_fattime. It's like magic. The chain has ended and the only thing that even remotely resembles a definition is that little declaration in the bottom script.
Au contraire ...

~/pico/micropython/ports/rp2/fatfs_port.c - Line 30

Code: Select all

MP_WEAK DWORD get_fattime(void) {
    datetime_t t;
    rtc_get_datetime(&t);
    return ((2000 + t.year - 1980) << 25) | ((t.month) << 21) | ((t.day) << 16) | ((t.hour) << 11) | ((t.min) << 5) | (t.sec / 2);
}

User avatar
OneMadGypsy
Posts: 326
Joined: Wed Apr 28, 2021 1:57 am
Location: New Orleans, Louisiana
Contact: Website

Re: Time

Sat May 22, 2021 12:05 pm

It's not really an "Au contraire". The only files included in ff.c are ff.h, diskio.h and string.h. ff.h and disio.h don't include anything. The part you posted is coming from above these files.
"Focus is a matter of deciding what things you're not going to do." ~ John Carmack

hippy
Posts: 10294
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Time

Sat May 22, 2021 12:11 pm

OneMadGypsy wrote:
Sat May 22, 2021 12:05 pm
It's not really an "Au contraire". The only files included in ff.c are ff.h, diskio.h and string.h. ff.h and disio.h don't include anything. The part you posted is coming from above these files.
That's true - The association is done at link time; file handling and other code calls functions which the port must provide.
OneMadGypsy wrote:
Sat May 22, 2021 11:49 am
Well, I tried what you suggested and I'm either doing something wrong or the modified date is wrong. I even tried flipping the bytes (the commented line). My pico time is correct. RShell sets it and tells me what it sets it to. Plus prinitng the values on PC shows that the accessed and created are correct but the modified is wrong.
We need hard numbers; need to know what values are showing as which dates, which are right, which are wrong, whether the code is being run on a PC or on the Pico, what that code is.

Your posted image shows "Created" with "2103" while seemingly suggesting that's correct, but its obviously not.

User avatar
OneMadGypsy
Posts: 326
Joined: Wed Apr 28, 2021 1:57 am
Location: New Orleans, Louisiana
Contact: Website

Re: Time

Sat May 22, 2021 12:20 pm

We need hard numbers; need to know what values are showing as which dates, which are right, which are wrong, whether the code is being run on a PC or on the Pico, what that code is.

Your posted image shows "Created" with "2103" while seemingly suggesting that's correct, but its obviously not.
Ah. No problem. The "created" is static to my little table and has no actual connection to the file. The code is being run on a Pico, and the below shows the code stripped down to exactly the problem function.

Image
"Focus is a matter of deciding what things you're not going to do." ~ John Carmack

User avatar
OneMadGypsy
Posts: 326
Joined: Wed Apr 28, 2021 1:57 am
Location: New Orleans, Louisiana
Contact: Website

Re: Time

Sat May 22, 2021 12:30 pm

If it helps, here is the equivalent being done on a RPi 400. Notice m_time is negative. The file I am processing in both of my images is the exact same file, and it was created with open on the Pico, today. Also notice that my last image shows the exact same number for all 3 times, whereas below shows different times and only m_time is wrong. This is because uos.stat() on the pico puts m_time in all 3 spots. If I could actually get the created time, I really wouldn't care how wrong modified is. I don't understand how the pico can get the created time right and fail so hard at the modified time. All these times should be coming from the same place. LOL, actually, upon further inspection, every single number the Pico version of stat is spitting out is wrong except size. Not even the flag is right.

Image
"Focus is a matter of deciding what things you're not going to do." ~ John Carmack

hippy
Posts: 10294
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Time

Sat May 22, 2021 1:09 pm

It would really help if you could post code as text rather than images.

I wasn't sure what's going on but think I have cracked it. This is my FAT time stamp conversion code -

Code: Select all

def get_fattime(yy, mm, dd, h, m, s):
  yy = yy - 1980
  d = (yy <<  9) | (mm << 5) | (dd << 0)
  t = (h  << 11) | (m  << 5) | (s  >> 1)
  return (d << 16) | t

def unget_fattime(n):
  d  = (n >> 16) & 0xFFFF
  t  = (n >>  0) & 0xFFFF

  yy = (d >>  9) & 0x7F
  mm = (d >>  5) & 0x0F
  dd = (d >>  0) & 0x1F

  h  = (t >> 11) & 0x1F
  m  = (t >>  5) & 0x3F
  s  = (t <<  1) & 0x3F

  return yy + 1980, mm, dd, h, m, s

def strftime(yy, mm, dd, y, m, s):
  return "{}-{:0>2}-{:0>2} {:0>2}:{:0>2}:{:0>2}".format(yy, mm, dd, h, m, s)

yy, mm, dd, h, m, s = 2021, 05, 21, 13, 19, 22
print("{}".format(strftime(yy, mm, dd, h, m, s)))
n = get_fattime(yy, mm, dd, h, m, s)
print("Is stored as {} ( {} )".format(n, hex(n)))

print("")
print("{} ( {} )".format(n, hex(n)))
yy, mm, dd, h, m, s = unget_fattime(n)
print("Decodes to {}".format(strftime(yy, mm, dd, h, m, s)))

s = -222942907
n = ( 1 << 32) + s
print("")
print("{} ( {}, {} )".format(s, n, hex(n)))
yy, mm, dd, h, m, s = unget_fattime(n)
print("Decodes to {}".format(strftime(yy, mm, dd, h, m, s)))

n = 4146181536
print("")
print("{} ( {} )".format(n, hex(n)))
yy, mm, dd, h, m, s = unget_fattime(n)
print("Decodes to {}".format(strftime(yy, mm, dd, h, m, s)))
The first operation translates 'about now' to a 32-bit time stamp, the second back again to prove the algorithms are correct. The third for your original -222942907 timestamp, the last for the 4146181536 your os.stat reports -

Code: Select all

2021-05-21 13:19:22
Is stored as 1387620971 ( 0x52b56a6b )

1387620971 ( 0x52b56a6b )
Decodes to 2021-05-21 13:19:22

-222942907 ( 4072024389, 0xf2b62945 )
Decodes to 2101-05-22 05:10:10

4146181536 ( 0xf721b5a0 )
Decodes to 2103-09-01 22:45:00
Those numbers are odd. But let's go back ...

Code: Select all

MP_WEAK DWORD get_fattime(void) {
    datetime_t t;
    rtc_get_datetime(&t);
    return ((2000 + t.year - 1980) << 25) | ((t.month) << 21) | ((t.day) << 16) | ((t.hour) << 11) | ((t.min) << 5) | (t.sec / 2);
}
Hmm; 2000 + t.year - 1980 when datetime_t.year is a int16_t with value 0 to 4095 - https://raspberrypi.github.io/pico-sdk- ... me__t.html

That would be right if t.year is an offset from year 2000, but not if an absolute. Unfortunately the SDK doc doesn't say which it is - An issue raised with Raspberry Pi to get that clarified would be welcome there.

If I add the 2000 in my code I'm seeing the same '4 billion' you are seeing -

Code: Select all

2021-05-21 13:19:22
Is stored as 4071975531 ( 0xf2b56a6b )
And, converted back without compensating for having added 2000 -

Code: Select all

4071975531 ( 0xf2b56a6b )
Decodes to 2101-05-21 13:19:22
The same out by 80 years you are seeing. Though I can't explain why the PC is showing it correctly if that's what is written to the card.

I don't know where the issue actually is but I would lay my money on one thing having treated t.year as absolute while something else is treating it as an offset from the year 2000.

I guess an issue needs raising with MicroPython to get that sorted.

User avatar
OneMadGypsy
Posts: 326
Joined: Wed Apr 28, 2021 1:57 am
Location: New Orleans, Louisiana
Contact: Website

Re: Time

Sat May 22, 2021 1:18 pm

It would really help if you could post code as text rather than images.

I normally would, but it was one line of code and I wanted you to see the printed results with everything else that happened. In this case, the code is literally just uos.stat(filename).
I can't explain why the PC is showing it correctly if that's what is written to the card.

The PC is showing created and accessed time correctly. Pico isn't getting either of those times. Pico is only getting modified time ... ironically the only time it also wrote wrong. It's even sillier that the modified timestamp should be identical to the created one, cause it was never modified.
"Focus is a matter of deciding what things you're not going to do." ~ John Carmack

hippy
Posts: 10294
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Time

Sat May 22, 2021 1:40 pm

OneMadGypsy wrote:
Sat May 22, 2021 12:30 pm
upon further inspection, every single number the Pico version of stat is spitting out is wrong except size. Not even the flag is right.
Not sure what 'flag' but the numbers aren't all wrong. MicroPython isn't Python and MicroPython libraries aren't Python libraries. MicroPython libraries do not always return the same data as a Python library would, especially where there is no equivalent in MicroPython or the FAT system.

The data from MicroPython os.stat() is almost certainly correct for what it defines its returned data to be. [0] does mean something different, uses different encoding, [1]-[5] don't seem to be used, [7], [8], and [9] are all set to the same single timestamp. That timestamps are '4 billion' isn't os.stat's fault if that's what the file entry claims.

Code: Select all

import os
import time

def Pad(s, w):
    s = str(s)
    if len(s) < w : return s + (" " *(w-len(s)))
    else          : return s
        
for file in os.listdir("/"):
    # 0 = mode   - Type, 0x80000=File, 0x4000=Dir
    # 1 = inode  - Not used
    # 2 = dev    - Not used
    # 3 = nlink  - Not used
    # 4 = uid    - Not used
    # 5 = gid    - Not used
    # 6 = size   - File size
    # 7 = atime  - Access time       \
    # 8 = mtime  - Modification time  | All have the same value
    # 9 = ctime  - Creation time     /
    stat = os.stat(file)
    if   stat[0] == 0x8000 : filetype = "File   "
    elif stat[0] == 0x4000 : filetype = "Dir    "
    else                   : filetype = "Unknown"
    filesize  = stat[6]
    timestamp = stat[8]
    yy, mm, dd, h, m, s, x, y = time.gmtime(timestamp)
    s = "{}-{:0>2}-{:0>2} {:0>2}:{:0>2}:{:0>2}".format(yy,mm,dd,h,m,s)
    print("{} {} {} : {} ({})".format(filetype, Pad(file, 20), Pad(filesize, 10), s, timestamp))

Code: Select all

File    MyTestData.txt       29         : 2021-05-18 14:58:52 (1621349932)
File    RPi_GPIO_test.py     254        : 2021-01-01 00:00:08 (1609459208)
Dir     RPi                  536899112  : 1970-01-01 00:00:00 (0)
File    hardhang.py          210        : 2021-04-28 14:34:48 (1619620488)
File    hermannsw.mpy        262        : 2021-01-01 00:00:07 (1609459207)
Dir     lib                  536899112  : 1970-01-01 00:00:00 (0)
File    load.py              5088       : 2021-04-14 14:03:34 (1618409014)
File    tiziana976.mpy       262        : 2021-04-10 18:58:14 (1618081094)
File    uf2hack.cfg          19         : 2021-01-01 00:00:05 (1609459205)
File    uf2hack.py           44         : 2021-01-01 00:00:01 (1609459201)
Last edited by hippy on Sat May 22, 2021 1:52 pm, edited 1 time in total.

User avatar
OneMadGypsy
Posts: 326
Joined: Wed Apr 28, 2021 1:57 am
Location: New Orleans, Louisiana
Contact: Website

Re: Time

Sat May 22, 2021 1:42 pm

Look at this. It's even commented for the three timestamps but is just sticking the same thing in every spot

EDIT: we posted simultaneously. I didn't see your post til after I sent this one.


extmod/vfs_fat.c ~ starts at line 313

Code: Select all


    mp_int_t seconds = timeutils_seconds_since_epoch(
        1980 + ((fno.fdate >> 9) & 0x7f),
        (fno.fdate >> 5) & 0x0f,
        fno.fdate & 0x1f,
        (fno.ftime >> 11) & 0x1f,
        (fno.ftime >> 5) & 0x3f,
        2 * (fno.ftime & 0x1f)
        );
    t->items[0] = MP_OBJ_NEW_SMALL_INT(mode); // st_mode
    t->items[1] = MP_OBJ_NEW_SMALL_INT(0); // st_ino
    t->items[2] = MP_OBJ_NEW_SMALL_INT(0); // st_dev
    t->items[3] = MP_OBJ_NEW_SMALL_INT(0); // st_nlink
    t->items[4] = MP_OBJ_NEW_SMALL_INT(0); // st_uid
    t->items[5] = MP_OBJ_NEW_SMALL_INT(0); // st_gid
    t->items[6] = mp_obj_new_int_from_uint(fno.fsize); // st_size
    t->items[7] = mp_obj_new_int_from_uint(seconds); // st_atime
    t->items[8] = mp_obj_new_int_from_uint(seconds); // st_mtime
    t->items[9] = mp_obj_new_int_from_uint(seconds); // st_ctime

"Focus is a matter of deciding what things you're not going to do." ~ John Carmack

User avatar
OneMadGypsy
Posts: 326
Joined: Wed Apr 28, 2021 1:57 am
Location: New Orleans, Louisiana
Contact: Website

Re: Time

Sat May 22, 2021 1:50 pm

OK. Well, what do you suggest I do? The modification date is not correct. I did not see anything in your code that implied that you figured out a way around it. If I understood it, you just figured out potentially why it's wrong. Where do you suggest I go from here?
"Focus is a matter of deciding what things you're not going to do." ~ John Carmack

hippy
Posts: 10294
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Time

Sat May 22, 2021 2:08 pm

OneMadGypsy wrote:
Sat May 22, 2021 1:50 pm
Where do you suggest I go from here?
Raise an issue against https://raspberrypi.github.io/pico-sdk- ... me__t.html to have it clarified whether year is an offset from the year 2000 or an absolute year value.

Raise an issue against https://github.com/micropython/micropyt ... tfs_port.c to have it clarified whether that is correct or not, to get it modified if not, and to rectify the consequential issues of incorrect timestamping if it is correct.

For a quick fix after reading the timestamp ... 'timestamp -= (80 << 25)'

User avatar
OneMadGypsy
Posts: 326
Joined: Wed Apr 28, 2021 1:57 am
Location: New Orleans, Louisiana
Contact: Website

Re: Time

Sat May 22, 2021 2:30 pm

Ok. I can do those things. This (timestamp -= (80 << 25)) isn't going to work, but I get what you're saying ~ figure out the (hopefully) constant difference and subtract it. Which is something like 82 years 3 months and 10 days, or something. I'm probably going to just skip the timestamp in the python version til they fix it. From my C port though I can fix it. All the timestamps are there, it's just stat isn't reading them. I just wont use stat. I'll go get the timestamp manually. It's kind of strange that stat doesn't get all 3 times. I could point you directly to the very lines they are being set on, their offsets, all of it. It's all in ff.c which is what they are already using.
"Focus is a matter of deciding what things you're not going to do." ~ John Carmack

hippy
Posts: 10294
Joined: Fri Sep 09, 2011 10:34 pm
Location: UK

Re: Time

Sat May 22, 2021 5:06 pm

OneMadGypsy wrote:
Sat May 22, 2021 2:30 pm
This (timestamp -= (80 << 25)) isn't going to work, but I get what you're saying ~ figure out the (hopefully) constant difference and subtract it. Which is something like 82 years 3 months and 10 days, or something.
I think I have cracked that as well -

os.stat() returns a 'from 1970 epoch value', hence my 'Dir RPi' is showing '1970-01-01 00:00:00 (0)' earlier for my LittleFS file system on Flash.

With FAT it seems os.stat is returning the raw FAT encoded timestamp, not converting that to a 1970 epoch value.

That appears to be another bug found.

If we look at my 'uf2hack.py' file which is '2021-01-01 00:00:01 (1609459201)' that is correct for a write soon after RTC started from its epoch of 2021.

2021-01-01 00:00:01 gives a FAT encoding of 1377894400 - That's close enough to the 1970 epoch value of 1609459201 that I hadn't noticed the discrepancy.

So it seems it's a two-part issue -

1) An incorrect FAT encoded timestamp is being written
2) os.stat is returning a FAT encoded timestamp value, not a 1970 epoch value.

Return to “MicroPython”