jimjamz
Posts: 39
Joined: Sun Feb 09, 2014 12:37 am

Value too large for defined data type on files larger than 2GB

Thu Feb 22, 2018 9:54 pm

I'm trying to verify the integrity of some files I have stored on an external disk, which is connected to my Pi Model B:

Code: Select all

openssl sha1 file.tar
This works successfully on files that are less than 2GB in size. However, on files larger than 2GB in size I receive the following result:

Code: Select all

 Value too large for defined data type
I found the following information:
https://www.gnu.org/software/coreutils/ ... -data-type

At first, I thought maybe it has always been this way for my Pi and external disk setup but from what I can see, I have been having this issue since around October 2017. Prior to that, I was able to successfully generate checksum hashes on files larger than 2GB. I know this because every attempt at generating a hash for the file is output to another file.

Have GNU utilities in Raspbian (as of around October 2017) stopped being compiled with large file support?

Current Raspbian Lite OS version is:

Code: Select all

PRETTY_NAME="Raspbian GNU/Linux 9 (stretch)"
NAME="Raspbian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=raspbian
ID_LIKE=debian
Kernel is:

Code: Select all

Linux myPi 4.9.59+ #1047 Sun Oct 29 11:47:10 GMT 2017 armv6l GNU/Linux
Is the problem because I am using Raspbian Lite, instead of the full distribution of Raspbian?

danjperron
Posts: 3531
Joined: Thu Dec 27, 2012 4:05 am
Location: Québec, Canada

Re: Value too large for defined data type on files larger than 2GB

Fri Feb 23, 2018 3:45 am

Is your external drive format in fat! if yes than the maximum is 2GB per file.

You have two option. Format your drive in ext4 or ntfs. In NTFS you could read the external drive in windows but you need to mount using ntfs-3g to be able to write on it

User avatar
jahboater
Posts: 6522
Joined: Wed Feb 04, 2015 6:38 pm
Location: Wonderful West Dorset

Re: Value too large for defined data type on files larger than 2GB

Fri Feb 23, 2018 8:31 am

Not your problem, but for interest see also:

sha1sum file.tar

and my vote goes to ext4 on Linux.

jimjamz
Posts: 39
Joined: Sun Feb 09, 2014 12:37 am

Re: Value too large for defined data type on files larger than 2GB

Sun Feb 25, 2018 10:06 am

danjperron wrote:
Fri Feb 23, 2018 3:45 am
Is your external drive format in fat! if yes than the maximum is 2GB per file.
All drives are EXT4. I don't use FAT nor NTFS.

jimjamz
Posts: 39
Joined: Sun Feb 09, 2014 12:37 am

Re: Value too large for defined data type on files larger than 2GB

Sun Feb 25, 2018 10:13 am

jahboater wrote:
Fri Feb 23, 2018 8:31 am
Not your problem, but for interest see also:

sha1sum file.tar
I appreciate the suggestion of the alternative program, but I would like to get to the bottom of why this no longer continues to work.
The configuration of drives has not changed. The only change has been that Raspbian was re-installed around October 2017 and according to my logs it was around then when the issue started to occur.
I can't say for sure, but it may have been that I was using a full distribution of Raspbian OS before October. Although, I doubt this was the case.
It seems to be that the the GNU utilities have not been compiled with large file support. This is (at the very least) the situation for `openssl` on Raspbian Lite.

jimjamz
Posts: 39
Joined: Sun Feb 09, 2014 12:37 am

Re: Value too large for defined data type on files larger than 2GB

Wed Feb 28, 2018 2:18 pm

Just a quick update.

@jahboater - When I use the `sha1sum` program to calculate the hash on a file larger than 2GB, it works successfully.
When I use the `openssl sha1` program to calculate the hash on a file larger than 2GB, it fails with:

Code: Select all

Value too large for defined data type
When I use the `openssl sha1` program to calculate the hash on a file smaller than 2GB, it it works successfully.

This points to the `openssl` GNU utlities not being compiled with Large File Support:
https://www.gnu.org/software/coreutils/ ... -data-type

I have tested this only on Raspbian Stretch Lite.

jimjamz
Posts: 39
Joined: Sun Feb 09, 2014 12:37 am

Re: Value too large for defined data type on files larger than 2GB

Thu Mar 22, 2018 10:05 pm

This is a confirmed issue with all Debian Stretch versions for both variants of Raspbian OS ( both +Desktop and Lite).
I have tested the following fresh installs:
2017-04-10-raspbian-jessie
2017-07-05-raspbian-jessie => last Jessie version
2017-08-10-raspbian-stretch => first Stretch version
2018-03-13-raspbian-stretch => latest Stretch version

On the two Stretch versions above, I receive the "Value too large ..." error when running the `openssl sha1` command on files larger than 2GB.
I receive no error when running the `openssl sha1` command on the Jessie versions. It generates the SHA-1 hashes successfully for files larger than 2GB.

Additionally, I've performed the same test on a separate machine with an installation of Debian netinst (https://www.debian.org/CD/netinst/). This contains a minimal install of Debian and that also successfully works with the `openssl sha1`.

Please can someone tell me how to raise bugs with the Raspbian team as I have not had to do this before.

cotarelo
Posts: 1
Joined: Wed Apr 10, 2019 10:27 pm

Re: Value too large for defined data type on files larger than 2GB

Wed Apr 10, 2019 10:30 pm

Hello,

I am having the same error in Raspbian on Stretch using unrar-free with files larger than 2 GB. I use EXT4 in all partitions.

Is there solution for it yet?

danjperron
Posts: 3531
Joined: Thu Dec 27, 2012 4:05 am
Location: Québec, Canada

Re: Value too large for defined data type on files larger than 2GB

Thu Apr 11, 2019 11:11 am

On rar and unrar ,could you pipe the output to the split command ?

When I compress file using zip for example I used the split command.

ex: To image partition1 of an hard disk I used the command.

Code: Select all

dd if=/dev/sda1 | gzip -1 -c | split -b 1GB - sda1.img.gz.
and to restore

Code: Select all

cat sda1.img.gz.* | gzip -dc | dd of=/dev/sda1 

Daniel

Return to “Troubleshooting”