corrupted files on raspbian jessie 2017-07-05

[mouseclicker]

Hi,
I have downloaded and installed latest raspbian from https://downloads.raspberrypi.org/raspbian_latest. It seems that there are corrutped files in the package manager. I tried

Code: Select all

sudo apt-get update
sudo apt-get dist-upgrade

and ended with an error:

Code: Select all

Reading changelogs... Done
dpkg: unrecoverable fatal error, aborting:
 files list file for package 'qdbus' is missing final newline
E: Sub-process /usr/bin/dpkg returned an error code (2)

Indeed the file qdbus.list contains nothig but 0x0 characters.

Code: Select all

pi@pi3new:/var/lib/dpkg/info $ cat ./qdbus.list 
pi@pi3new:/var/lib/dpkg/info $ hexdump qdbus.list 
0000000 0000 0000 0000 0000 0000 0000 0000 0000
*
0000180

First I suspected my SD card but I tried a different one with the same result. I verfied the SHA1-checksum for the download and it is okay.

Any fix to this?

Thanks Jens

[DirkS]

Did you check the file *before* trying the update / upgrade?

[mouseclicker]

No I checked after the (failed) ugrade. But if it woud be caused by the update wouldn't it mean a broken repository then? This would not make things better or am I wrong?

[DirkS]

No I checked after the (failed) ugrade. But if it woud be caused by the update wouldn't it mean a broken repository then? This would not make things better or am I wrong?

The repo is fine.
More likely one of the files that was downloaded in the update stage got corrupted somehow.

[anubus]

I have found the exact same issue. I attempted to reinstall using:
$ sudo apt-get install qdbus --reinstall
but it revealed several other dependant file lists also corrupted so no help. I hope a new image is posted soon.
Oh, I dounloaded both the .zip and the torrent version...same issue. Also tried unzipping with WinZip and the Mac archive utiility...same issue. 2 different SD card manufacturers as well...same issue. I am now downloading the NOOBS image to see if that is also corrupted.

[anubus]

The slightly good news is that the NOOBS Raspbian img does not have this problem. Checked:
$ sudo cat /var/lib/dpkg/info/qdbus.list
is intacked and I successfully installed libusb-1.0 and libusb-dev which revealed this issue in the first place.

[LittlePea13]

I'm having the same problem after just installing Jessie. Only commands I have runned are apt-get update and then upgrade, returning the same error.

[mouseclicker]

Thanks Anubus,

this is good news, I will try the NOOBs package. Hopefully my 8GB card has enough capacity for NOOBS.

[clethro]

I also tried the suggested fixes but was not able to fix the problem. I concur that it appears that one or more files got corrupted in the building of the 07-05 version. Fortunately, I still have my copy of the 2017-03-02 version of Jessie. I loaded it on SD card and proceeded without any trouble. Of course, that first apt-get upgrade is going to take some time. There are over 130 packages that have updates.

[mouseclicker]

rechecked with a fresh sd card, just initial boot, resize fs, reboot. No raspbi-config, no network attached, still the file qdbus.list (and some others) contain no readable text, just hex values.

[DirkS]

rechecked with a fresh sd card, just initial boot, resize fs, reboot. No raspbi-config, no network attached, still the file qdbus.list (and some others) contain no readable text, just hex values.

Did you check if your downloaded zip file is OK?
Check SHA-1 checksum of the zip file with the one listed at https://www.raspberrypi.org/downloads/raspbian/
(NB: the checksum is for the zip file, not the image itself)

BTW: you don't have to run the filesystem resize manually; that's done automatically on first boot)

[mouseclicker]

Yes I checked hash of the zip (as mentioned in my initial post):

Code: Select all

openssl sha1 2017-07-05-raspbian-jessie.zip 
SHA1(2017-07-05-raspbian-jessie.zip)= e1edd4d26090b3e67a66939fa77eeb656de8a2c5

and sorry for the confusion, resize of the file system is of course done automatically, just wanted to be precise what actions happened before my check.

I also can confirm that NOOBS does not have this problem. So I am happy now.

IMHO this should be very easily reproducible for anyone. As others report this problem too, please check your distribution zip.

[nothlit]

I am having the exact same problem trying to run sudo apt-get upgrade on a fresh install of the 2017-07-05 image. Will give NOOBS a try.

Code: Select all

$ shasum ~/Downloads/2017-07-05-raspbian-jessie.zip
e1edd4d26090b3e67a66939fa77eeb656de8a2c5

[ShiftPlusOne]

Not able to reproduce this.

Code: Select all

pi@raspberrypi:~ $ cat /var/lib/dpkg/info/qdbus.list 
/.
/usr
/usr/lib
/usr/lib/arm-linux-gnueabihf
/usr/lib/arm-linux-gnueabihf/qt4
/usr/lib/arm-linux-gnueabihf/qt4/bin
/usr/lib/arm-linux-gnueabihf/qt4/bin/qdbus
/usr/share
/usr/share/doc
/usr/share/doc/qdbus
/usr/share/doc/qdbus/LGPL_EXCEPTION.txt
/usr/share/doc/qdbus/copyright
/usr/share/doc/qdbus/changelog.Debian.gz
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/qdbus.1.gz
pi@raspberrypi:~ $ sudo apt-get update
Get:1 http://archive.raspberrypi.org jessie InRelease [22.9 kB]
Get:2 http://archive.raspberrypi.org jessie/main armhf Packages [168 kB]   
Hit http://archive.raspberrypi.org jessie/ui armhf Packages                    
Get:3 http://mirrordirector.raspbian.org jessie InRelease [14.9 kB]            
Get:4 http://mirrordirector.raspbian.org jessie/main armhf Packages [9,533 kB] 
Ign http://archive.raspberrypi.org jessie/main Translation-en_GB               
Ign http://archive.raspberrypi.org jessie/main Translation-en                  
Ign http://archive.raspberrypi.org jessie/ui Translation-en_GB                 
Ign http://archive.raspberrypi.org jessie/ui Translation-en
Get:5 http://mirrordirector.raspbian.org jessie/contrib armhf Packages [43.3 kB]
Get:6 http://mirrordirector.raspbian.org jessie/non-free armhf Packages [84.2 kB]
Get:7 http://mirrordirector.raspbian.org jessie/rpi armhf Packages [1,356 B]
Ign http://mirrordirector.raspbian.org jessie/contrib Translation-en_GB
Ign http://mirrordirector.raspbian.org jessie/contrib Translation-en
Ign http://mirrordirector.raspbian.org jessie/main Translation-en_GB
Ign http://mirrordirector.raspbian.org jessie/main Translation-en
Ign http://mirrordirector.raspbian.org jessie/non-free Translation-en_GB
Ign http://mirrordirector.raspbian.org jessie/non-free Translation-en
Ign http://mirrordirector.raspbian.org jessie/rpi Translation-en_GB
Ign http://mirrordirector.raspbian.org jessie/rpi Translation-en
Fetched 9,868 kB in 13s (737 kB/s)                                             
Reading package lists... Done
pi@raspberrypi:~ $ sudo apt-get dist-upgrade
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
The following packages will be upgraded:
  bind9-host libbind9-90 libdns-export100 libdns100 libgcrypt20
  libirs-export91 libisc-export95 libisc95 libisccc90 libisccfg-export90
  libisccfg90 liblwres90 libtiff5
13 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Need to get 2,126 kB of archives.
After this operation, 7,168 B of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://mirrordirector.raspbian.org/raspbian/ jessie/main libgcrypt20 armhf 1.6.3-2+deb8u4 [334 kB]
Get:2 http://mirrordirector.raspbian.org/raspbian/ jessie/main libisc-export95 armhf 1:9.9.5.dfsg-9+deb8u12 [125 kB]
Get:3 http://mirrordirector.raspbian.org/raspbian/ jessie/main libdns-export100 armhf 1:9.9.5.dfsg-9+deb8u12 [398 kB]
Get:4 http://mirrordirector.raspbian.org/raspbian/ jessie/main libisccfg-export90 armhf 1:9.9.5.dfsg-9+deb8u12 [38.5 kB]
Get:5 http://mirrordirector.raspbian.org/raspbian/ jessie/main libirs-export91 armhf 1:9.9.5.dfsg-9+deb8u12 [37.1 kB]
Get:6 http://mirrordirector.raspbian.org/raspbian/ jessie/main libtiff5 armhf 4.0.3-12.3+deb8u4 [202 kB]
Get:7 http://mirrordirector.raspbian.org/raspbian/ jessie/main bind9-host armhf 1:9.9.5.dfsg-9+deb8u12 [66.1 kB]
Get:8 http://mirrordirector.raspbian.org/raspbian/ jessie/main libisc95 armhf 1:9.9.5.dfsg-9+deb8u12 [150 kB]
Get:9 http://mirrordirector.raspbian.org/raspbian/ jessie/main libdns100 armhf 1:9.9.5.dfsg-9+deb8u12 [599 kB]
Get:10 http://mirrordirector.raspbian.org/raspbian/ jessie/main libisccc90 armhf 1:9.9.5.dfsg-9+deb8u12 [34.8 kB]
Get:11 http://mirrordirector.raspbian.org/raspbian/ jessie/main libisccfg90 armhf 1:9.9.5.dfsg-9+deb8u12 [51.0 kB]
Get:12 http://mirrordirector.raspbian.org/raspbian/ jessie/main libbind9-90 armhf 1:9.9.5.dfsg-9+deb8u12 [41.7 kB]
Get:13 http://mirrordirector.raspbian.org/raspbian/ jessie/main liblwres90 armhf 1:9.9.5.dfsg-9+deb8u12 [48.2 kB]
Fetched 2,126 kB in 1s (1,134 kB/s)    
Reading changelogs... Done
(Reading database ... 115401 files and directories currently installed.)
Preparing to unpack .../libgcrypt20_1.6.3-2+deb8u4_armhf.deb ...
Unpacking libgcrypt20:armhf (1.6.3-2+deb8u4) over (1.6.3-2+deb8u3) ...
Setting up libgcrypt20:armhf (1.6.3-2+deb8u4) ...
Processing triggers for libc-bin (2.19-18+deb8u10) ...
(Reading database ... 115401 files and directories currently installed.)
Preparing to unpack .../libisc-export95_1%3a9.9.5.dfsg-9+deb8u12_armhf.deb ...
Unpacking libisc-export95 (1:9.9.5.dfsg-9+deb8u12) over (1:9.9.5.dfsg-9+deb8u11) ...
Preparing to unpack .../libdns-export100_1%3a9.9.5.dfsg-9+deb8u12_armhf.deb ...
Unpacking libdns-export100 (1:9.9.5.dfsg-9+deb8u12) over (1:9.9.5.dfsg-9+deb8u11) ...
Preparing to unpack .../libisccfg-export90_1%3a9.9.5.dfsg-9+deb8u12_armhf.deb ...
Unpacking libisccfg-export90 (1:9.9.5.dfsg-9+deb8u12) over (1:9.9.5.dfsg-9+deb8u11) ...
Preparing to unpack .../libirs-export91_1%3a9.9.5.dfsg-9+deb8u12_armhf.deb ...
Unpacking libirs-export91 (1:9.9.5.dfsg-9+deb8u12) over (1:9.9.5.dfsg-9+deb8u11) ...
Preparing to unpack .../libtiff5_4.0.3-12.3+deb8u4_armhf.deb ...
Unpacking libtiff5:armhf (4.0.3-12.3+deb8u4) over (4.0.3-12.3+deb8u3) ...
Preparing to unpack .../bind9-host_1%3a9.9.5.dfsg-9+deb8u12_armhf.deb ...
Unpacking bind9-host (1:9.9.5.dfsg-9+deb8u12) over (1:9.9.5.dfsg-9+deb8u11) ...
Preparing to unpack .../libisc95_1%3a9.9.5.dfsg-9+deb8u12_armhf.deb ...
Unpacking libisc95 (1:9.9.5.dfsg-9+deb8u12) over (1:9.9.5.dfsg-9+deb8u11) ...
Preparing to unpack .../libdns100_1%3a9.9.5.dfsg-9+deb8u12_armhf.deb ...
Unpacking libdns100 (1:9.9.5.dfsg-9+deb8u12) over (1:9.9.5.dfsg-9+deb8u11) ...
Preparing to unpack .../libisccc90_1%3a9.9.5.dfsg-9+deb8u12_armhf.deb ...
Unpacking libisccc90 (1:9.9.5.dfsg-9+deb8u12) over (1:9.9.5.dfsg-9+deb8u11) ...
Preparing to unpack .../libisccfg90_1%3a9.9.5.dfsg-9+deb8u12_armhf.deb ...
Unpacking libisccfg90 (1:9.9.5.dfsg-9+deb8u12) over (1:9.9.5.dfsg-9+deb8u11) ...
Preparing to unpack .../libbind9-90_1%3a9.9.5.dfsg-9+deb8u12_armhf.deb ...
Unpacking libbind9-90 (1:9.9.5.dfsg-9+deb8u12) over (1:9.9.5.dfsg-9+deb8u11) ...
Preparing to unpack .../liblwres90_1%3a9.9.5.dfsg-9+deb8u12_armhf.deb ...
Unpacking liblwres90 (1:9.9.5.dfsg-9+deb8u12) over (1:9.9.5.dfsg-9+deb8u11) ...
Processing triggers for man-db (2.7.5-1~bpo8+1) ...
Setting up libisc-export95 (1:9.9.5.dfsg-9+deb8u12) ...
Setting up libdns-export100 (1:9.9.5.dfsg-9+deb8u12) ...
Setting up libisccfg-export90 (1:9.9.5.dfsg-9+deb8u12) ...
Setting up libirs-export91 (1:9.9.5.dfsg-9+deb8u12) ...
Setting up libtiff5:armhf (4.0.3-12.3+deb8u4) ...
Setting up libisc95 (1:9.9.5.dfsg-9+deb8u12) ...
Setting up libdns100 (1:9.9.5.dfsg-9+deb8u12) ...
Setting up libisccc90 (1:9.9.5.dfsg-9+deb8u12) ...
Setting up libisccfg90 (1:9.9.5.dfsg-9+deb8u12) ...
Setting up libbind9-90 (1:9.9.5.dfsg-9+deb8u12) ...
Setting up liblwres90 (1:9.9.5.dfsg-9+deb8u12) ...
Setting up bind9-host (1:9.9.5.dfsg-9+deb8u12) ...
Processing triggers for libc-bin (2.19-18+deb8u10) ...
pi@raspberrypi:~ $ cat /var/lib/dpkg/info/qdbus.list 
/.
/usr
/usr/lib
/usr/lib/arm-linux-gnueabihf
/usr/lib/arm-linux-gnueabihf/qt4
/usr/lib/arm-linux-gnueabihf/qt4/bin
/usr/lib/arm-linux-gnueabihf/qt4/bin/qdbus
/usr/share
/usr/share/doc
/usr/share/doc/qdbus
/usr/share/doc/qdbus/LGPL_EXCEPTION.txt
/usr/share/doc/qdbus/copyright
/usr/share/doc/qdbus/changelog.Debian.gz
/usr/share/man
/usr/share/man/man1
/usr/share/man/man1/qdbus.1.gz
pi@raspberrypi:~ $ cat /etc/rpi-issue 
Raspberry Pi reference 2017-07-05
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, 234c4b79371dd2f4531482e1045752de9e183a68, stage5

[CasualRPiUser2]

Hi,

same problem here.

I have build my own solution:

Code: Select all

#!/usr/bin/env bash

# Description:
#   This simple script fixes the corrupt .list files
#   in the raspbian 2017-07-05 image.
#
# Author:   Patrick Neumann, not[dot]at[dot]home[at]gmx[dot]de
# Date:     2017-07-10
# Version:  1.0
# License:  GPL3
# Warranty: WITHOUT ANY WARRANTY

# Declare an empty array:
typeset -a PACKAGES

# Put explicit packages into the array:
PACKAGES[0]="qdbus"
PACKAGES[1]="qjackctl"
PACKAGES[2]="qtchooser"
PACKAGES[3]="qtcore4-l10n"
PACKAGES[4]="raspberrypi-artwork"
PACKAGES[5]="qttranslations5-l10n"
PACKAGES[6]="raspberrypi-bootloader"
PACKAGES[7]="raspberrypi-kernel"
PACKAGES[8]="raspberrypi-ui-mods"
PACKAGES[9]="raspi-config"
PACKAGES[10]="raspi-copies-and-fills"
PACKAGES[11]="python3.4"
PACKAGES[12]="python3"

# Make the array readonly:
typeset -r PACKAGES

# Iterate over the array and
#   make backups of the .list files of the packages:
for package in "${PACKAGES[@]}" ; do
  /bin/mv /var/lib/dpkg/info/"${package}".list{,_BACKUP}
done

# Reinstall the packages:
/usr/bin/apt-get install --reinstall --assume-yes "${PACKAGES[*]}"

exit 0

Because I have allready fixed my last RPi I have not choosen "find -exec", "file" and "grep".
Maybe someone else could do it!?

Just put the code with the Text Editor in a file, save it, open a terminal, cd to the directory with the new file and execute it with "sudo bash file".

Feel free to use, debug, modify and give feedback.

Regards
Patrick

[jojopi]

Looking at the 13 files mentioned above using debugfs blocks, I notice that all of them have extents of 1126147 and higher, in a filesystem of total block count 1126281. That cannot be a coincidence.

In fact the total size of the .img file is 4445.537MiB, and all the problem files are in that last 0.537MiB. (Despite the fact that there is plenty of free space within the image.)

I wonder if the people having the issue are using dd programs that discard the last fractional transfer instead of padding it. It could just be a coincidence that this is the first image where the tiny amount of lost data happens to be important enough to cause problems for apt-get.

[ShiftPlusOne]

Looking at the 13 files mentioned above using debugfs blocks, I notice that all of them have extents of 1126147 and higher, in a filesystem of total block count 1126281. That cannot be a coincidence.

In fact the total size of the .img file is 4445.537MiB, and all the problem files are in that last 0.537MiB. (Despite the fact that there is plenty of free space within the image.)

I wonder if the people having the issue are using dd programs that discard the last fractional transfer instead of padding it. It could just be a coincidence that this is the first image where the tiny amount of lost data happens to be important enough to cause problems for apt-get.

That was clever. Good thinking.

Also, I used to be under the impression that dd would sync before finishing, but that only happens if you give it the appropriate parameter. I wonder if people are yanking out the card without running 'sync' first.

[mouseclicker]

That's interesting!

Yes I have used dd (on MacOS, not sure if this does padding or not) and I did not call sync.

I see that this is mentioned on https://www.raspberrypi.org/documentati ... s/linux.md but rather hidden in the section for verifying the SD card. It is not mentioned on the Mac page https://www.raspberrypi.org/documentati ... ges/mac.md

You should put this to a more prominent place.Thanks for sharing this hint!

[mouseclicker]

and maybe for others useful: the necessary parameter to add to the dd command seems to be:

Code: Select all

 conv=sync

I did not try myself if this does the trick.

[ShiftPlusOne]

I don't have the dd manpage in front of me, but there are a few similar conv options and I don't think that's the one.

[DirkS]

I don't have the dd manpage in front of me, but there are a few similar conv options and I don't think that's the one.

I agree:

Code: Select all

       sync   pad every input block with NULs to ibs-size; when used with block or unblock, pad with spaces rather than NULs

       fdatasync
              physically write output file data before finishing

       fsync  likewise, but also write metadata

[TimG]

Code: Select all

dd bs=1M if=2017-07-05-raspbian-jessie.img of=/dev/sdX conv=fdatasync

[jfield]

Same issue here - checked the sha-1, tried different recommended unzip tools, different computers, different SD cards -- /var/lib/dpkg/info/qdbus.list (and others) were always corrupt. Can confirm it is an issue with usage of dd. In fact after the suggestions here I noticed the final output of my dd command (from the rpi docs) was:

Code: Select all

4445+1 records in
4445+0 records out

Signifying the last record was dropped because the input buffer didn't get filled. The solution is adding conv=sync like so:

Code: Select all

sudo dd bs=1m if=path_of_your_image.img of=/dev/rdiskn conv=sync

The resulting dd output was:

Code: Select all

4445+1 records in
4446+0 records out

Signifying that last input block got padded to full size and written to the SD card. Tested and everything works fine. I'm guessing this might be the first image that was written with blocks less than 1m or something? Or the first that had meaningful data in that last block?

In any case, it would be great if someone updated the docs to include the conv=sync argument: https://www.raspberrypi.org/documentati ... ges/mac.md

Thanks for everyone's input (no pun intended)!

[rainerstandke]

I can confirm that this worked for me on a Mac:

Code: Select all

sudo dd bs=1m if=/path/to/Downloads/2017-07-05-raspbian-jessie.img of=/dev/rdisk2 conv=sync

[WorldScott]

I too encountered this issue.

I found a solution here: https://ubuntuforums.org/showthread.php?t=1319791

It involves a small python script that Ubuntu forums user pokerbirch came up with:

Code: Select all

#!/usr/bin/python


# 8th November, 2009
# update manager failed, giving me the error:
#       'files list file for package 'xxx' is missing final newline' for every package.
# some Googling revealed that this problem was due to corrupt files(s) in /var/lib/dpkg/info/
# looping though those files revealed that some did not have a final new line
# this script will resolve that problem by appending a newline to all files that are missing it
# NOTE: you will need to run this script as root, e.g. sudo python newline_fixer.py

import os

dpkg_path = '/var/lib/dpkg/info/'
paths = os.listdir(dpkg_path)
for path in paths:
    path = dpkg_path + path
    f = open(path, 'a+')
    data = f.read()
    if len(data) > 1 and data[-1:] != '\n':
        f.write('\n')
        print 'added newline character to:', path
    f.close()  

One needs only copy that bit of code into a .py file and then sudo run it.

I named the file fileslistfix.py and ran it.

It fixed the EOF error not only on the problem file but on a whole list of additional files with the same issue.

-Scott