tyrionbean
Posts: 15
Joined: Sat Jun 12, 2021 8:50 am

Mirror ext4 to NTFS HDD advice?

Wed Jun 16, 2021 8:44 am

Hi there,

Got a question about mirroring a drive. I'm using my Pi 400 as an NAS with an external HDD formatted as an ext4 (works great - no complaints). What I'd like to do is mirror everything automatically to an NTFS drive (I have it mounted with a mount point in fstab - that works fine).

I know I can use rsync, but I've read various threads and advice all over the place with varying reports about how it has questionable results with NTFS. I'm not sure how to proceed. I'm curious if anyone has a line of code of rsync for me to use to try out and see if it works? Or any other advice about it?

I'd appreciate any input on the matter. Thanks kindly in advance.

LTolledo
Posts: 5542
Joined: Sat Mar 17, 2018 7:29 am
Location: Anime Heartland

Re: Mirror ext4 to NTFS HDD advice?

Wed Jun 16, 2021 10:22 am

the questionable result with NTFS formatted drive is lower transfer rate...

as I am trying now on my RPi4B-4GB, with 1 HDD formatted as EXT4 (source) and another HDD formatted as NTFS (target)

using the command

Code: Select all

rsync -auvh --progress /media/pi/MSL4TB/DBZ /media/pi/USB2TB/DBZ

am getting just about 20MB/s (below is an excerpt of that transfer)

Code: Select all

        259.02M 100%   20.57MB/s    0:00:12 (xfr#17, to-chk=287/316)

        257.81M 100%   20.40MB/s    0:00:12 (xfr#18, to-chk=286/316)
so you'll have to live with that kind of transfer rate....
"Don't come to me with 'issues' for I don't know how to deal with those
Come to me with 'problems' and I'll help you find solutions"

Some people be like:
"Help me! Am drowning! But dont you dare touch me nor come near me!"

User avatar
thagrol
Posts: 5221
Joined: Fri Jan 13, 2012 4:41 pm
Location: Darkest Somerset, UK
Contact: Website

Re: Mirror ext4 to NTFS HDD advice?

Wed Jun 16, 2021 11:06 am

There are two main issues with syncing ext4 to NTFS:
  • As already mentioned poor write performance. Compunded by everything needing to go through a PCIe x1 link twice.
  • Loss of all metadata - owner, group, permissions, etc as they are not supported by NTFS.
There's not a lot that can be done except live with them if you must use NTFS.
I'm a volunteer. Take me for granted or abuse my support and I will walk away

All advice given is based on my experience. it worked for me, it may not work for you.
Need help? https://github.com/thagrol/Guides

ejolson
Posts: 7540
Joined: Tue Mar 18, 2014 11:47 am

Re: Mirror ext4 to NTFS HDD advice?

Wed Jun 16, 2021 12:17 pm

thagrol wrote:
Wed Jun 16, 2021 11:06 am
There are two main issues with syncing ext4 to NTFS:
  • As already mentioned poor write performance. Compunded by everything needing to go through a PCIe x1 link twice.
  • Loss of all metadata - owner, group, permissions, etc as they are not supported by NTFS.
There's not a lot that can be done except live with them if you must use NTFS.
If you are backing up your home directory, loss of the metadata, file permissions and ownership would not be a problem. A possible work around is to store an ext4 image on the NTFS drive, mount it using loopback and then perform the rsync to the loopback device.

User avatar
thagrol
Posts: 5221
Joined: Fri Jan 13, 2012 4:41 pm
Location: Darkest Somerset, UK
Contact: Website

Re: Mirror ext4 to NTFS HDD advice?

Wed Jun 16, 2021 2:34 pm

ejolson wrote:
Wed Jun 16, 2021 12:17 pm
thagrol wrote:
Wed Jun 16, 2021 11:06 am
There are two main issues with syncing ext4 to NTFS:
  • As already mentioned poor write performance. Compunded by everything needing to go through a PCIe x1 link twice.
  • Loss of all metadata - owner, group, permissions, etc as they are not supported by NTFS.
There's not a lot that can be done except live with them if you must use NTFS.
If you are backing up your home directory, loss of the metadata, file permissions and ownership would not be a problem. A possible work around is to store an ext4 image on the NTFS drive, mount it using loopback and then perform the rsync to the loopback device.
Or use tar to an archive file instead of rsync to a directory.
I'm a volunteer. Take me for granted or abuse my support and I will walk away

All advice given is based on my experience. it worked for me, it may not work for you.
Need help? https://github.com/thagrol/Guides

tyrionbean
Posts: 15
Joined: Sat Jun 12, 2021 8:50 am

Re: Mirror ext4 to NTFS HDD advice?

Wed Jun 16, 2021 4:20 pm

Thanks! I was actually asking in part because I heard *not* to use the archive tag with NTFS. I guess I should give it a try anyway with a test directory?

After reading the answers, something tells me maybe I should just format it to something else. I just wish ext4 was easily recognized by Mac and also PC.
LTolledo wrote:
Wed Jun 16, 2021 10:22 am
the questionable result with NTFS formatted drive is lower transfer rate...

as I am trying now on my RPi4B-4GB, with 1 HDD formatted as EXT4 (source) and another HDD formatted as NTFS (target)

using the command

Code: Select all

rsync -auvh --progress /media/pi/MSL4TB/DBZ /media/pi/USB2TB/DBZ

am getting just about 20MB/s (below is an excerpt of that transfer)

Code: Select all

        259.02M 100%   20.57MB/s    0:00:12 (xfr#17, to-chk=287/316)

        257.81M 100%   20.40MB/s    0:00:12 (xfr#18, to-chk=286/316)
so you'll have to live with that kind of transfer rate....

User avatar
thagrol
Posts: 5221
Joined: Fri Jan 13, 2012 4:41 pm
Location: Darkest Somerset, UK
Contact: Website

Re: Mirror ext4 to NTFS HDD advice?

Wed Jun 16, 2021 7:36 pm

tyrionbean wrote:
Wed Jun 16, 2021 4:20 pm
Thanks! I was actually asking in part because I heard *not* to use the archive tag with NTFS. I guess I should give it a try anyway with a test directory?
If that relates to using a tar archive, you may be misunderstanding things. You can think of tar* as an older and uncompressed equivalent to zip.
After reading the answers, something tells me maybe I should just format it to something else. I just wish ext4 was easily recognized by Mac and also PC.
Pretty much any non linux native file system will have the lost metadata issue. There tools for windows that allow reading (and writing) to ext4 on windows but AFAIK the free ones do not support the journal and will mount an ext4 partition as ext2.

I've not used them and can't comment on how they handle the meta data nor on how reliable they are.

No idea about MacOS. Sorry.

*:tar is short for tape archive
I'm a volunteer. Take me for granted or abuse my support and I will walk away

All advice given is based on my experience. it worked for me, it may not work for you.
Need help? https://github.com/thagrol/Guides

Return to “Beginners”