PhatFil
Posts: 517
Joined: Thu Apr 13, 2017 3:55 pm

Node red persistent data json file keeps disappearing

Wed Sep 19, 2018 12:18 am

Im learning Node Red with a fairly basic project that displays 7 sets of data in 4 graphs running on a headless pi-0w.

Using a named persistent data store (json file) for each chart, set to 'flush its buffer' (write to the file) every hour to minimise sd writes.
the buffer should generally contain less than 10 new data points in an hour. though during development/testing episods that could increase by a factor of 10..

all 4 files live in the home dir of the pi user, one file repeatedly empties/gets deleted? unexpectedly losing all the data?

on the original pi-zero this started happening on i tried to get vnc server up for a gui access, in doing so i messed up the system such that it failed to boot (dodgy sd card??) - anyway i moved development onto a new pi-0w with a freshly installed full version upto date of stretch and installed only my key tools onto its brand new 16gb sandisk ultra micro sd card bought from sandisk via amazon.

And I am again loosing the contents of the same file after unexpected power offs.. with the file being written to once an hour i am finding it difficult to comprehend that my pi always gets a hard reset when that file is accessed.. While not ideal the pi-0 is powered via a network AP's usb port and the AP gets a reboot every now and again when its signal is weak

The file name is PlantEC1.json in the pi user home dir. I have renamed the file once already.
meanwhile the data stores for the other 3 charts have survived intact on both pi's that have hosted this system under development.

any ideas as to WTF? or how to counter the problem, a cron job to copy the file to a backup location every hour If its larger then the one already backed up?

Thanks in advance

Heater
Posts: 10003
Joined: Tue Jul 17, 2012 3:02 pm

Re: Node red persistent data json file keeps disappearing

Thu Sep 20, 2018 7:23 am

PhatFil,
And I am again loosing the contents of the same file after unexpected power offs.
Be aware that when you write to a file in Linux the data is most likely not immediately written to the disk. Rather it is kept in buffers in RAM and will be written some time later.

You can alleviate this problem a little by always closing the file when you have updated it. Also you can use "sync" to tell Linux to synchronize the data to disk.

But, you main problem is that you have unexpected power outages and no clean shutdown. In those circumstances we can expect your SD cards files to be corrupted eventually. Either the file you are writing or some part of the operating system, which will cause your system to not boot.

epoch1970
Posts: 2075
Joined: Thu May 05, 2016 9:33 am
Location: Paris, France

Re: Node red persistent data json file keeps disappearing

Thu Sep 20, 2018 9:54 am

In agreement with Heater.
To improve data safety, you could:
a. keep the data on a separate partition, that gets mounted before updates and dismounted after updates. Use the "sync" mount option.
b. use 2 partitions and do everything twice, sequentially. In case of mishap you should still have a (possibly slightly older) copy available.

For OS safety, see ejolson's thread on making Raspbian run with read-only rootfs, or this solution (much lighter IMHO) or have a look at piCore, a linux distribution designed to run off RAM.

(And if you do all that, still an SD might die or turn read-only if power is cut at the wrong moment, it is in the nature of solid-state storage devices.)
"S'il n'y a pas de solution, c'est qu'il n'y a pas de problème." Les Shadoks, J. Rouxel

PhatFil
Posts: 517
Joined: Thu Apr 13, 2017 3:55 pm

Re: Node red persistent data json file keeps disappearing

Thu Sep 20, 2018 1:41 pm

Thanks for the insights, Sorry i didnt make myself clearer. I put the current Pi-zero system onto the unreliable power supply (router that gets rebooted when wifi plays up) in order to replicate and reproduce the problem after it occurred the first time on the 2nd system. Not an ideal setup but it does replicate the problem rapidly

My headless Pi-Zero systems have been getting more reliable since i have reduced the random s/w installed on them to look see and have only been installing the key s/w onto the base system that is getting used. but even so they do still on occasion drop the wifi connection and refuse to pick it back up again requiering the occasional restart (gpio triggered shutdown/restart should perhaps be my next priority,)

I have no idea how node red handles its persistent data stores, going by the gui settings for the node.
Usingt the interval setting which i have set to 3600 (ihour)

Code: Select all

 Interval integer
the minimum buffer flush interval in seconds.    
I was working under the assumption that this would open the data store file every hour and write the data buffered (within the node red chart node to the the file and then close it.

whatever the mechanism is that indexes system file write/commits and queues them up something about this particular file must be relegating it to last position in that list for it to be consistently the only file regardless of file name that gets effected.

So I will look into issuing a sync command from within node red, im also considering a cron job to copy the file as long as its bigger than the last file copied/backed up.

Or look at using a dbs system to record the data ..

Heater
Posts: 10003
Joined: Tue Jul 17, 2012 3:02 pm

Re: Node red persistent data json file keeps disappearing

Fri Sep 21, 2018 7:00 am

I know nothing much about what is available in Node Red, I prefer to write my Javascript by hand, but perhaps it would help if you told us what Node red module you are using as a persistent store.

json-db-node-red perhaps? Something else?

PhatFil
Posts: 517
Joined: Thu Apr 13, 2017 3:55 pm

Re: Node red persistent data json file keeps disappearing

Fri Sep 21, 2018 8:44 pm

node-red-contrib-persist V1.03 installed via the node red Palette Manager I am using both the persist in and persist out nodes to save and repopulate the chart node data. Nod red itself was installed/updated via the script on the Node Red Pi Page.
Image
the persist nodes are the turquoise ones..

each pair of in/out nodes is serviced by a discrete config node pointing to a json file for each charts data set. that is the node configured to 'flush' its buffer after 3600 secs..

I could try pointing all the charts at a signle data store (json file)/config node? i assume the json formatting will take care of loading if i do make that change but it seems more sensible to devote a file to each chart?? perhaps using more than one data store is the problem?

Return to “General programming discussion”