martinrowan
Posts: 9
Joined: Wed Aug 26, 2015 11:00 am

WARNING: node-rpi-ws281x-native causes file system/SD card corruption

Wed Jun 14, 2017 5:37 pm

I have what I consider a fairly serious issue which I've been able to reproduce on multiple brand new RPi3 with multiple different microSD cards, including several brand new ones.
Repro:
1. Format SD with SDFormatter.
2. Use Win32DiskImager to write 2017-04-10-raspbian-jessie.img
3. Configure my image with node and a couple of other minor tweaks.
4. Run my node service. Everything's happy no errors reported in dmesg system running smoothly.
5. run sudo apt-get update && sudo apt-get upgrade, followed by a reboot.
6. Depending on the card I'm seeing slugging performance to complete hangs within a few minutes.

I think this depends on how quickly the card gets enough write errors to throw a fit and make the file system read only.

In the attached dmesg log you'll see errors like:

Code: Select all

[  303.215569] mmc0: timeout waiting for hardware interrupt.
[  303.218188] mmcblk0: error -110 transferring data, sector 10372992, nr 8, cmd response 0x900, card status 0xc00
[  313.455603] mmc0: timeout waiting for hardware interrupt.
[  313.465817] mmcblk0: error -110 transferring data, sector 3767312, nr 8, cmd response 0x900, card status 0xc00
[  323.695659] mmc0: timeout waiting for hardware interrupt.
[  323.698214] mmcblk0: error -110 transferring data, sector 3767312, nr 8, cmd response 0x900, card status 0xc00
[  323.698359] blk_update_request: I/O error, dev mmcblk0, sector 3767312
[  323.698418] Aborting journal on device mmcblk0p2-8.
[  333.935677] mmc0: timeout waiting for hardware interrupt.
[  333.938401] mmcblk0: error -110 transferring data, sector 92160, nr 8, cmd response 0x900, card status 0xc00
[  344.175717] mmc0: timeout waiting for hardware interrupt.
[  344.178360] mmcblk0: error -110 transferring data, sector 92160, nr 8, cmd response 0x900, card status 0xc00
[  344.178480] blk_update_request: I/O error, dev mmcblk0, sector 92160
[  344.178502] Buffer I/O error on dev mmcblk0p2, logical block 0, lost sync page write
[  344.178552] EXT4-fs (mmcblk0p2): previous I/O error to superblock detected
[  344.179720] EXT4-fs error (device mmcblk0p2): ext4_journal_check_start:56: Detected aborted journal
[  344.179727] EXT4-fs error (device mmcblk0p2): ext4_journal_check_start:56: Detected aborted journal
[  344.179738] EXT4-fs (mmcblk0p2): Remounting filesystem read-only
[  344.180900] EXT4-fs (mmcblk0p2): ext4_writepages: jbd2_start: 6144 pages, ino 22151; err -30
I've reproduced this on:
* Sandisk 8GB - man:0x000003 oem:0x5344 name:SL08G hwrev:0x8 fwrev:0x0
* Samsung Pro 16GB - man:0x00001b oem:0x534d name:00000 hwrev:0x1 fwrev:0x0
* Integral 16GB - man:0x00009e oem:0x4245 name: hwrev:0x1 fwrev:0x0

I can't seem to upload the log, but it's linked to a github issue here: https://github.com/jamesbulpin/meshblu- ... /dmesg.txt

Ernst
Posts: 388
Joined: Sat Feb 04, 2017 9:39 am
Location: Germany

Re: File system going read-only since recent raspbian update

Wed Jun 14, 2017 5:51 pm

Please describe exactly how you rebooted the Pi.
My first computer was an ICT1500


User avatar
DougieLawson
Posts: 29790
Joined: Sun Jun 16, 2013 11:19 pm
Location: Basingstoke, UK
Contact: Website

Re: File system going read-only since recent raspbian update

Thu Jun 15, 2017 7:55 am

Have you considered that your SDCard may be worn out and needs to be replaced with a factory fresh one?
Microprocessor, Raspberry Pi & Arduino Hacker
Mainframe database troubleshooter
MQTT Evangelist
Twitter: @DougieLawson

Since 2012: 1B*5, 2B*2, B+, A+, Zero*2, 3B*3

Please post ALL technical questions on the forum. Do not send private messages.

B.Goode
Posts: 4406
Joined: Mon Sep 01, 2014 4:03 pm
Location: UK

Re: File system going read-only since recent raspbian update

Thu Jun 15, 2017 8:08 am

DougieLawson wrote:Have you considered that your SDCard may be worn out and needs to be replaced with a factory fresh one?
Martin wrote:
I've been able to reproduce on multiple brand new RPi3 with multiple different microSD cards, including several brand new ones.

Ernst
Posts: 388
Joined: Sat Feb 04, 2017 9:39 am
Location: Germany

Re: File system going read-only since recent raspbian update

Thu Jun 15, 2017 8:38 am

B.Goode wrote:
DougieLawson wrote:Have you considered that your SDCard may be worn out and needs to be replaced with a factory fresh one?
Martin wrote:
I've been able to reproduce on multiple brand new RPi3 with multiple different microSD cards, including several brand new ones.
If it is not the RPi3 and it is not the microSD card and it is not the software then it could be the power supply.
My first computer was an ICT1500

martinrowan
Posts: 9
Joined: Wed Aug 26, 2015 11:00 am

Re: File system going read-only since recent raspbian update

Thu Jun 15, 2017 9:35 am

I've tried with 2 PSU too. I've the official Raspberry Pi version (that came with the RPi3 Kit) and I've a 5V 10A PSU that I'm using to control a WS2812B LED Strip.
FWIW The LED Strip is connected to the RPi3 via GPIO18 (Pin 12) and sharing a common ground (Pin 6). Connected to a 74HCT125 level shifter and then a 470 ohm resistor to the data line of the WS2812b. Power for the LED Strip is supplied directly from the 10A PSU, and not being drawn from the GPIO.
Running https://www.npmjs.com/package/rpi-ws281x-native examples is sufficient to trigger warnings in dmesg and continuous use appears to trigger enough errors to set the file system readonly.

I've reformatted the card, re-write it with the original Jessie image from April and it runs just fine, no warnings. Only when I update the image to updates later than those included in the April image does the problem appear.

For example I left my node program running overnight, dmesg shows no new messages since it booted.

The current list of updates available from the April image is as follows, so I'm pretty convinced this is something in Debian/Raspbian.

Code: Select all

PROGRAM: login INSTALLED: 1:4.2-3+deb8u3 AVAILABLE: 1:4.2-3+deb8u4
PROGRAM: perl INSTALLED: 5.20.2-3+deb8u6 AVAILABLE: 5.20.2-3+deb8u7
PROGRAM: perl-base INSTALLED: 5.20.2-3+deb8u6 AVAILABLE: 5.20.2-3+deb8u7
PROGRAM: perl-modules INSTALLED: 5.20.2-3+deb8u6 AVAILABLE: 5.20.2-3+deb8u7
PROGRAM: libc6-dev INSTALLED: 2.19-18+deb8u7 AVAILABLE: 2.19-18+deb8u9
PROGRAM: libc-dev-bin INSTALLED: 2.19-18+deb8u7 AVAILABLE: 2.19-18+deb8u9
PROGRAM: rpcbind INSTALLED: 0.2.1-6+deb8u1 AVAILABLE: 0.2.1-6+deb8u2
PROGRAM: libc-bin INSTALLED: 2.19-18+deb8u7 AVAILABLE: 2.19-18+deb8u9
PROGRAM: libc6-dbg INSTALLED: 2.19-18+deb8u7 AVAILABLE: 2.19-18+deb8u9
PROGRAM: libc6 INSTALLED: 2.19-18+deb8u7 AVAILABLE: 2.19-18+deb8u9
PROGRAM: libtirpc1 INSTALLED: 0.2.5-1 AVAILABLE: 0.2.5-1+deb8u1
PROGRAM: libudev1 INSTALLED: 215-17+deb8u6 AVAILABLE: 215-17+deb8u7
PROGRAM: udev INSTALLED: 215-17+deb8u6 AVAILABLE: 215-17+deb8u7
PROGRAM: initramfs-tools INSTALLED: 0.120+deb8u2 AVAILABLE: 0.120+deb8u3
PROGRAM: libsystemd0 INSTALLED: 215-17+deb8u6 AVAILABLE: 215-17+deb8u7
PROGRAM: libpam-systemd INSTALLED: 215-17+deb8u6 AVAILABLE: 215-17+deb8u7
PROGRAM: systemd INSTALLED: 215-17+deb8u6 AVAILABLE: 215-17+deb8u7
PROGRAM: systemd-sysv INSTALLED: 215-17+deb8u6 AVAILABLE: 215-17+deb8u7
PROGRAM: libgnutls-openssl27 INSTALLED: 3.3.8-6+deb8u4 AVAILABLE: 3.3.8-6+deb8u5
PROGRAM: libtasn1-6 INSTALLED: 4.2-3+deb8u2 AVAILABLE: 4.2-3+deb8u3
PROGRAM: libgnutls-deb0-28 INSTALLED: 3.3.8-6+deb8u4 AVAILABLE: 3.3.8-6+deb8u5
PROGRAM: libldap-2.4-2 INSTALLED: 2.4.40+dfsg-1 AVAILABLE: 2.4.40+dfsg-1+deb8u2
PROGRAM: libavutil54 INSTALLED: 6:11.8-1~deb8u1+rpi1 AVAILABLE: 6:11.9-1~deb8u1+rpi1
PROGRAM: libavresample2 INSTALLED: 6:11.8-1~deb8u1+rpi1 AVAILABLE: 6:11.9-1~deb8u1+rpi1
PROGRAM: libavcodec56 INSTALLED: 6:11.8-1~deb8u1+rpi1 AVAILABLE: 6:11.9-1~deb8u1+rpi1
PROGRAM: librtmp1 INSTALLED: 2.4+20150115.gita107cef-1 AVAILABLE: 2.4+20150115.gita107cef-1+deb8u1
PROGRAM: libavformat56 INSTALLED: 6:11.8-1~deb8u1+rpi1 AVAILABLE: 6:11.9-1~deb8u1+rpi1
PROGRAM: libisc-export95 INSTALLED: 1:9.9.5.dfsg-9+deb8u10 AVAILABLE: 1:9.9.5.dfsg-9+deb8u11
PROGRAM: libdns-export100 INSTALLED: 1:9.9.5.dfsg-9+deb8u10 AVAILABLE: 1:9.9.5.dfsg-9+deb8u11
PROGRAM: libgudev-1.0-0 INSTALLED: 215-17+deb8u6 AVAILABLE: 215-17+deb8u7
PROGRAM: libicu52 INSTALLED: 52.1-8+deb8u4 AVAILABLE: 52.1-8+deb8u5
PROGRAM: libisccfg-export90 INSTALLED: 1:9.9.5.dfsg-9+deb8u10 AVAILABLE: 1:9.9.5.dfsg-9+deb8u11
PROGRAM: libirs-export91 INSTALLED: 1:9.9.5.dfsg-9+deb8u10 AVAILABLE: 1:9.9.5.dfsg-9+deb8u11
PROGRAM: libnss3 INSTALLED: 2:3.26-1+debu8u1 AVAILABLE: 2:3.26-1+debu8u2
PROGRAM: libwbclient0 INSTALLED: 2:4.2.14+dfsg-0+deb8u5 AVAILABLE: 2:4.2.14+dfsg-0+deb8u6
PROGRAM: libsmbclient INSTALLED: 2:4.2.14+dfsg-0+deb8u5 AVAILABLE: 2:4.2.14+dfsg-0+deb8u6
PROGRAM: samba-libs INSTALLED: 2:4.2.14+dfsg-0+deb8u5 AVAILABLE: 2:4.2.14+dfsg-0+deb8u6
PROGRAM: libswscale3 INSTALLED: 6:11.8-1~deb8u1+rpi1 AVAILABLE: 6:11.9-1~deb8u1+rpi1
PROGRAM: libtiff5 INSTALLED: 4.0.3-12.3+deb8u2 AVAILABLE: 4.0.3-12.3+deb8u3
PROGRAM: libxslt1.1 INSTALLED: 1.1.28-2+deb8u2 AVAILABLE: 1.1.28-2+deb8u3
PROGRAM: samba-common INSTALLED: 2:4.2.14+dfsg-0+deb8u5 AVAILABLE: 2:4.2.14+dfsg-0+deb8u6
PROGRAM: multiarch-support INSTALLED: 2.19-18+deb8u7 AVAILABLE: 2.19-18+deb8u9
PROGRAM: passwd INSTALLED: 1:4.2-3+deb8u3 AVAILABLE: 1:4.2-3+deb8u4
PROGRAM: tzdata INSTALLED: 2016j-0+deb8u1 AVAILABLE: 2017b-0+deb8u1
PROGRAM: vim-tiny INSTALLED: 2:7.4.488-7+deb8u2 AVAILABLE: 2:7.4.488-7+deb8u3
PROGRAM: vim-common INSTALLED: 2:7.4.488-7+deb8u2 AVAILABLE: 2:7.4.488-7+deb8u3
PROGRAM: wget INSTALLED: 1.16-1+deb8u1 AVAILABLE: 1.16-1+deb8u2
PROGRAM: bind9-host INSTALLED: 1:9.9.5.dfsg-9+deb8u10 AVAILABLE: 1:9.9.5.dfsg-9+deb8u11
PROGRAM: libisc95 INSTALLED: 1:9.9.5.dfsg-9+deb8u10 AVAILABLE: 1:9.9.5.dfsg-9+deb8u11
PROGRAM: libdns100 INSTALLED: 1:9.9.5.dfsg-9+deb8u10 AVAILABLE: 1:9.9.5.dfsg-9+deb8u11
PROGRAM: libisccc90 INSTALLED: 1:9.9.5.dfsg-9+deb8u10 AVAILABLE: 1:9.9.5.dfsg-9+deb8u11
PROGRAM: libisccfg90 INSTALLED: 1:9.9.5.dfsg-9+deb8u10 AVAILABLE: 1:9.9.5.dfsg-9+deb8u11
PROGRAM: libbind9-90 INSTALLED: 1:9.9.5.dfsg-9+deb8u10 AVAILABLE: 1:9.9.5.dfsg-9+deb8u11
PROGRAM: liblwres90 INSTALLED: 1:9.9.5.dfsg-9+deb8u10 AVAILABLE: 1:9.9.5.dfsg-9+deb8u11
PROGRAM: locales INSTALLED: 2.19-18+deb8u7 AVAILABLE: 2.19-18+deb8u9
PROGRAM: binutils INSTALLED: 2.25-5 AVAILABLE: 2.25-5+deb8u1
PROGRAM: ca-certificates INSTALLED: 20141019+deb8u2 AVAILABLE: 20141019+deb8u3
PROGRAM: git-man INSTALLED: 1:2.1.4-2.1+deb8u2 AVAILABLE: 1:2.1.4-2.1+deb8u3
PROGRAM: git INSTALLED: 1:2.1.4-2.1+deb8u2 AVAILABLE: 1:2.1.4-2.1+deb8u3
PROGRAM: git-core INSTALLED: 1:2.1.4-2.1+deb8u2 AVAILABLE: 1:2.1.4-2.1+deb8u3
PROGRAM: libblas-common INSTALLED: 1.2.20110419-10 AVAILABLE: 3.7.0-1~bpo8+1
PROGRAM: libblas3 INSTALLED: 1.2.20110419-10 AVAILABLE: 3.7.0-1~bpo8+1
PROGRAM: libbluray1 INSTALLED: 1:0.6.2-1 AVAILABLE: 1:0.9.3-3
PROGRAM: libcupsfilters1 INSTALLED: 1.0.61-5+deb8u3 AVAILABLE: 1.0.61-5+deb8u3+rpi1
PROGRAM: liblapack3 INSTALLED: 3.5.0-4 AVAILABLE: 3.7.0-1~bpo8+1
PROGRAM: libmwaw-0.3-3 INSTALLED: 0.3.1-2 AVAILABLE: 0.3.1-2+deb8u1
PROGRAM: raspberrypi-sys-mods INSTALLED: 20170313 AVAILABLE: 20170519
PROGRAM: raspi-config INSTALLED: 20170307 AVAILABLE: 20170503
PROGRAM: rc-gui INSTALLED: 1.7 AVAILABLE: 1.9
PROGRAM: realvnc-vnc-server INSTALLED: 6.0.3.27095 AVAILABLE: 6.1.1.28093
PROGRAM: realvnc-vnc-viewer INSTALLED: 6.0.3.27095 AVAILABLE: 6.1.1.28093
PROGRAM: rpi-chromium-mods INSTALLED: 20170317 AVAILABLE: 20170614
PROGRAM: sudo INSTALLED: 1.8.10p3-1+deb8u3 AVAILABLE: 1.8.10p3-1+deb8u4
PROGRAM: unzip INSTALLED: 6.0-16+deb8u2 AVAILABLE: 6.0-16+deb8u3
PROGRAM: libraspberrypi-dev INSTALLED: 1.20170405-1 AVAILABLE: 1.20170427-1
PROGRAM: libraspberrypi-doc INSTALLED: 1.20170405-1 AVAILABLE: 1.20170427-1
PROGRAM: raspberrypi-kernel INSTALLED: 1.20170405-1 AVAILABLE: 1.20170427-1
PROGRAM: libraspberrypi-bin INSTALLED: 1.20170405-1 AVAILABLE: 1.20170427-1
PROGRAM: libraspberrypi0 INSTALLED: 1.20170405-1 AVAILABLE: 1.20170427-1
PROGRAM: raspberrypi-bootloader INSTALLED: 1.20170405-1 AVAILABLE: 1.20170427-1
PROGRAM: raspberrypi-ui-mods INSTALLED: 1.20170315 AVAILABLE: 1.20170503

Ernst
Posts: 388
Joined: Sat Feb 04, 2017 9:39 am
Location: Germany

Re: File system going read-only since recent raspbian update

Thu Jun 15, 2017 10:23 am

martinrowan wrote: Running https://www.npmjs.com/package/rpi-ws281x-native examples is sufficient to trigger warnings in dmesg and continuous use appears to trigger enough errors to set the file system readonly.
Over many years analyzing problems in IT I have learned that it is important to pay attention to any small detail as it can become a big problem later.
IMHO this is your problem.

I looked at the information on web site above and I found a few things that are suspect:
This module provides native bindings to the rpi_ws281x library by Jeremy Garff to provide a very basic set of functions to write data to a strip of ws2811/ws2812 LEDs. this will only run on the Raspberry Pi.
What Pi ? There are a number of different models but what model ?
As the native part of this module needs to directly interface with the physical memory of the raspberry-pi (which is required in order to configure the PWM and DMA-modules), it always has to run with root-privileges (there are probably ways around this requirement, but that doesn't change the fact that the node-process running the LEDs needs access to the raw physical memory – a thing you should never allow to any user other than root).
Same here what Pi and what memory ? There are 256MB, 512MB, 1024MB models, and the Pi3 afaik has a different layout compared to the other models.

It may well be possible that your problem above is what causing the problems after upgrade.
Last edited by Ernst on Thu Jun 15, 2017 10:40 am, edited 1 time in total.
My first computer was an ICT1500

martinrowan
Posts: 9
Joined: Wed Aug 26, 2015 11:00 am

Re: File system going read-only since recent raspbian update

Thu Jun 15, 2017 10:31 am

Sorry perhaps by explaining my exact setup I've added more confusion. To try and clarify and/or narrow this down, I will try and more simple repro that doesn't involve running more than just the bare Raspbian and Pixel. No Node, nothing connected to the GPIO pins and report back tomorrow.
Thanks for the replies so far.
Martin

martinrowan
Posts: 9
Joined: Wed Aug 26, 2015 11:00 am

Re: File system going read-only since recent raspbian update

Fri Jun 16, 2017 12:45 pm

Ok thanks for the pointers, the problem appears to be caused only when running https://github.com/beyondscreen/node-rpi-ws281x-native examples or other apps based on it. This consumes a 2 year old fork of https://github.com/jgarff/rpi_ws281x
Running Python examples from rpi_ws281x also triggers the warnings in dmesg though with a lot lower frequency, talking about 10 minutes before it triggered a read only file system yet. But it does look like something in the low level control is triggering this issue in combination with the latest raspbian updates.

I imagine this ws281x library is used fairly extentisvely as it's the reference given for interfacing these LEDs with the RPi: https://learn.adafruit.com/neopixels-on ... i/software

Thanks for your help. Time to raise an issue with the project owner(s), or see if I can figure out what's going on.

seblagaffe
Posts: 1
Joined: Sun Aug 13, 2017 11:38 am

Re: File system going read-only since recent raspbian update

Sun Aug 13, 2017 11:43 am

martinrowan wrote:
Fri Jun 16, 2017 12:45 pm
Thanks for your help. Time to raise an issue with the project owner(s), or see if I can figure out what's going on.
Hi martin, I have a similar issue (i use hyperion server to ambilight clone solution with pi zero W and WS2813). Did you resolve your issue ?

martinrowan
Posts: 9
Joined: Wed Aug 26, 2015 11:00 am

Re: File system going read-only since recent raspbian updates

Sun Aug 13, 2017 1:00 pm

Yes, it caused by a conflict with DMA 5. see: https://github.com/beyondscreen/node-rp ... /issues/61

The fix is to modify whatever library you are using to use DMA 10 that has fixed it for me and others.

Heater
Posts: 7755
Joined: Tue Jul 17, 2012 3:02 pm

Re: File system going read-only since recent raspbian updates

Sun Aug 13, 2017 2:38 pm

It would be nice if the title of this thread were changed to something like:

"WARNING: node-rpi-ws281x-native causes file system/SD card corruption"

martinrowan
Posts: 9
Joined: Wed Aug 26, 2015 11:00 am

Re: File system going read-only since recent raspbian updates

Sun Aug 13, 2017 2:52 pm

Heater wrote:
Sun Aug 13, 2017 2:38 pm
It would be nice if the title of this thread were changed to something like:

"WARNING: node-rpi-ws281x-native causes file system/SD card corruption"
To be clear, It's not just node-rpi-ws281x-native it stems from this project rpi-ws281x https://github.com/jgarff/rpi_ws281x/issues/188 and anything which consumes that.

But agree, given what I have learned about the issue it would be good to update the title to: "File System going read-only due to DMA conflict in rpi-ws281x with recent raspbian updates".

However I don't appear to be able to change the thread title. Maybe a moderator can?

Heater
Posts: 7755
Joined: Tue Jul 17, 2012 3:02 pm

Re: File system going read-only since recent raspbian updates

Sun Aug 13, 2017 5:06 pm

So the problem is giving some random program privileged access to memory and hardware and having it wreck the system.

That project should have a big red warning on the README.

Return to “Raspbian”

Who is online

Users browsing this forum: No registered users and 21 guests