blippy
Posts: 123
Joined: Fri Nov 03, 2017 3:07 pm

Seen one FAT32, seen them all?

Wed Nov 25, 2020 11:44 am

I've always found using SD cards with microcontrollers to be a bit of a hit-and-miss affair, and never really gotten to the bottom of it.

I was looking through someone's else code the other day, and discovered that there is more than one FAT32 partition Id. This got me to wondering if, actually, there are some quite different and incompatible layouts for the FAT32 file system. So maybe some filesystem drivers out there are doing completely the wrong thing.

Comments?

cleverca22
Posts: 2880
Joined: Sat Aug 18, 2012 2:33 pm

Re: Seen one FAT32, seen them all?

Wed Nov 25, 2020 11:49 am

that is one reason why the rpi firmware cant deal with a fat32 partition over 32gig (only a problem if your using pinn/noobs on a 64g card)

the mkfs util automatically switches to exfat when the fs is over 32gig, and the boot firmware lacks the ability to read that

off the top of my head, there is at least fat12, fat16, fat32, and exfat, and the main difference is the size of a single slot in the main FAT table

bzt
Posts: 607
Joined: Sat Oct 14, 2017 9:57 pm

Re: Seen one FAT32, seen them all?

Wed Nov 25, 2020 12:36 pm

blippy wrote:
Wed Nov 25, 2020 11:44 am
I was looking through someone's else code the other day, and discovered that there is more than one FAT32 partition Id. This got me to wondering if, actually, there are some quite different and incompatible layouts for the FAT32 file system.
No. There's only one FAT32 layout. The partition Ids were just there to hint MS-DOS (whether to use CHS/LBA addressing or is the partition hidden or not). There's not much relevance in these days, as CHS is totally obsolete just as MS-DOS. Likewise whether a partition is hidden or not does not influence the layout either.

The RPi doesn't care what's in the partition Id. You can use a FAT16 LBA identifier for example and place a FAT32 layout on the partition, the RPi firmware can use it just fine.
blippy wrote:
Wed Nov 25, 2020 11:44 am
So maybe some filesystem drivers out there are doing completely the wrong thing.
Yes there are incompatible FAT drivers out there. That's mostly because the FAT spec is very badly written and full of inconsistencies and contradictions. It worth mentioning that there's a "new" FAT standard, exFAT, which uses different layout than FAT12/16/32, so it is by definition incompatible. Unlike the old FAT, exFAT is well defined and its spec is well written, and that's what most SDcards and USB sticks use these days (mp3 players, cameras, etc.) Old FAT (also referred to as vfat under Linux) is basically limited to UEFI and to some microcontrollers (like RPi).

Cheers,
bzt
Last edited by bzt on Wed Nov 25, 2020 12:59 pm, edited 1 time in total.

LdB
Posts: 1670
Joined: Wed Dec 07, 2016 2:29 pm

Re: Seen one FAT32, seen them all?

Wed Nov 25, 2020 12:49 pm

There are up to 4 partitions under FAT32 but only one can be marked as active meaning and that is used by boot managers to indicate which to kick. In most cases you will have only one partition which will be the one marked active.

The only issue I have had with SD cards is the well known problem of linux corrupting it from time to time with its temporary files, why the issue comes up on SD cards on the Pi I have never found out. The only solution around that is to load the partitions on the SD_CARD read_only in linux so it can't put temp files on the SD card. I have around 40 kiosk type app that don't use the sd card at all 25% of them will corrupt the SD card within a year just by whatever linux is doing on the SD card if I don't mark them read only.
Last edited by LdB on Wed Nov 25, 2020 12:52 pm, edited 1 time in total.

cleverca22
Posts: 2880
Joined: Sat Aug 18, 2012 2:33 pm

Re: Seen one FAT32, seen them all?

Wed Nov 25, 2020 12:51 pm

LdB wrote:
Wed Nov 25, 2020 12:49 pm
There are up to 4 partitions under FAT32 but only one can be marked as active meaning and that is used by boot managers to indicate which to kick. In most cases you will have only one partition which will be the one marked active.
i think you mean MBR/msdos, not FAT32

GlowInTheDark
Posts: 1364
Joined: Sat Nov 09, 2019 12:14 pm

Re: Seen one FAT32, seen them all?

Wed Nov 25, 2020 12:52 pm

to some microcontrollers (like RPi).
Letters. You're going to get letters.

(For those who don't catch the cryptic reference above, this means that people are going to get upset with you for calling the RPi a "microcontroller")
GitD's list of things that are not ready for prime time:
1) IPv6
2) 64 bit OSes
3) USB 3
4) Bluetooth

Loves Linux; loves to dance.

LdB
Posts: 1670
Joined: Wed Dec 07, 2016 2:29 pm

Re: Seen one FAT32, seen them all?

Wed Nov 25, 2020 12:54 pm

cleverca22 wrote:
Wed Nov 25, 2020 12:51 pm
i think you mean MBR/msdos, not FAT32
Yes you know the story you can effectively have up to 4 FAT32 or any other partition on the card.

Unrelated to linux but on embedded system the nastiest part can be dealing with the unicode because most embedded compilers don't have the support for it. You get one user on a foreign keyboard and they can kill your entire file system because you can't find the file because of the weird characters.
Last edited by LdB on Wed Nov 25, 2020 12:58 pm, edited 1 time in total.

cleverca22
Posts: 2880
Joined: Sat Aug 18, 2012 2:33 pm

Re: Seen one FAT32, seen them all?

Wed Nov 25, 2020 12:56 pm

LdB wrote:
Wed Nov 25, 2020 12:54 pm
cleverca22 wrote:
Wed Nov 25, 2020 12:51 pm
i think you mean MBR/msdos, not FAT32
Yes you know the story you can effectively have up to 4 FAT32 or any other partition on the card.
until you get into extended partitions, which is a scheme to break one of those 4 primary up into many smaller ones

or you switch over to GPT and put all of those crazy old designs to rest
you can still use FAT32 on GPT, and the pi4 is perfectly capable of booting with GPT, older pi's may not be though

for the pi4 to boot from GPT, the fat32 must be tagged as the "efi system partition", and thats basically the same as giving it a typecode of 0xC under MBR, as far as the pi firmware is concerned

bzt
Posts: 607
Joined: Sat Oct 14, 2017 9:57 pm

Re: Seen one FAT32, seen them all?

Wed Nov 25, 2020 1:01 pm

GlowInTheDark wrote:
Wed Nov 25, 2020 12:52 pm
For those who don't catch the cryptic reference above, this means that people are going to get upset with you for calling the RPi a "microcontroller"
My apologies! Mini computer then? The point is, FAT32 is not widespread these days any more, and mostly used by some firmware only.

Cheers,
bzt

bzt
Posts: 607
Joined: Sat Oct 14, 2017 9:57 pm

Re: Seen one FAT32, seen them all?

Wed Nov 25, 2020 1:05 pm

cleverca22 wrote:
Wed Nov 25, 2020 12:56 pm
you can still use FAT32 on GPT, and the pi4 is perfectly capable of booting with GPT, older pi's may not be though
Since when? This is good news, I've written a a small tool that reads and parses the GPT looking for ESP and then maps it in the MBR. Does this mean I no longer need to use this tool?

Cheers,
bzt

LdB
Posts: 1670
Joined: Wed Dec 07, 2016 2:29 pm

Re: Seen one FAT32, seen them all?

Wed Nov 25, 2020 1:08 pm

FAT32 is patented and Microsoft started enforcing the patent is why exFAT became popular real quick ... look up Microsoft vs Tom Tom Navigation systems.

The Pi bootloader is technically in breach of MS patent but as the Pi Foundation is a not for profit, MS would likely never pursue them :-)

bzt
Posts: 607
Joined: Sat Oct 14, 2017 9:57 pm

Re: Seen one FAT32, seen them all?

Wed Nov 25, 2020 1:13 pm

LdB wrote:
Wed Nov 25, 2020 1:08 pm
The Pi bootloader is technically in breach of MS patent but as the Pi Foundation is a not for profit, MS would likely never pursue them :-)
No, they are not. The patent is limited to creating long file names with backward compatibility (the infamous ~1, ~2 etc. scheme). The Pi bootloader does not generate any FAT dir entries, so it's okay, the patent doesn't apply :-) (plus even if it were, all Pi related filenames are 8+3, so it does not need LFN in the first place).

Here's some more detail on how the Linux FAT driver workarounded this patent issue by not generating ~1, ~2 names.

Cheers,
bzt

LdB
Posts: 1670
Joined: Wed Dec 07, 2016 2:29 pm

Re: Seen one FAT32, seen them all?

Wed Nov 25, 2020 1:27 pm

Your own link explains the problem, there is no guarantee of what you said it needs to be tested in a court.

The loader writes a log file as per your link the fact you don't have long files names is not important you terminate the long file name entries so if they were read by a FAT32 system they would work. It is arguable that it breaches the MS patent because it supports FAT32 LFN. You should note they altered the linux driver to write random values in the LFN entries so it crashes if you used a linux FAT32 written disk on a FAT32 system like windows and tried to add LFN entries ... can you understand why they did that?

Not something PI Foundation has to worry about but for commercial companies getting into court with MS is never a good move so everyone avoided FAT32 like the plague.

bzt
Posts: 607
Joined: Sat Oct 14, 2017 9:57 pm

Re: Seen one FAT32, seen them all?

Wed Nov 25, 2020 2:28 pm

LdB wrote:
Wed Nov 25, 2020 1:27 pm
It is arguable that it breaches the MS patent because it supports FAT32 LFN.
Could you please elaborate? It is my understanding that the patent is restricted to "creating" and "storing" long file names only.

From my link:
Q1. Has this patch undergone legal review?
A1. Both the original patch and the new patch that we posted today
have been through legal review by several lawyers who specialize
in this area.
...

Q5. Why do you believe that this patch avoids the patent?
A5. ...
The claims of both of the VFAT patents involve the creation (or
storing) of both a long filename and a short filename for a file.
The 2nd patch only creates/stores either a short filename or a
long filename for a file, but never both.
...
It is a fact that this patch was merged 11 years ago, and MS did not sue nor win any patent-lawsuit battle with Linux since. They have realized this is futile and has no use, and they have called off all the ongoing cases as well 2 years ago.
LdB wrote:
Wed Nov 25, 2020 1:27 pm
everyone avoided FAT32 like the plague.
Just to walk into the trap. FAT32 with 8+3 is completely patent-free, and the LFN patents (U.S. patents 5579517, 5758352) must expire eventually (actually they should have expired already in 2016 according to the U.S. law). Some of exFAT patents (for example U.S. patent 8321439) on the other hand won't expire until 2029, which gives MS a legal background to sue all the world using exFAT.

BTW, exFAT has even more strict patent conditions than FAT32 LFN, and MS has only agreed to freely open source it for OIN members because they wanted WSL and there was no other way to make their built-in Linux kernel handle exFAT (since the Linux kernel mandates GPL).

This exception does not apply to the RPi firmware, since it is not open source, and it is sold for profit, not an OIN member, so using exFAT for boot partition would be indeed a bad thing and MS could legally sue RPi's firmware provider.

It's better if RPi firmware sticks with FAT32 and 8+3 and then MS can't have any hold on them.

Cheers,
bzt

Return to “Bare metal, Assembly language”