User avatar
rpdom
Posts: 19037
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Zip keeps running out of memory

Wed Aug 04, 2021 10:21 am

That free output looks normal. Where do you get
kcajminer2312 wrote:
Wed Aug 04, 2021 10:02 am
kswapd0 seems to think the pi only has 500MB
from?
Unreadable squiggle

kcajminer2312
Posts: 303
Joined: Sun Jan 31, 2021 12:37 pm

Re: Zip keeps running out of memory

Wed Aug 04, 2021 10:44 am

found this
In case someone faces the same issues - the reason was malware: Multios.Coinminer.Miner.

kswapd0 was a binary file located in /root/.configrc/a/kswapd0. What you need to do is:

Clear crontab jobs referring to /root/.configrc
Clear ssh keys
Delete /root/.configrc

kcajminer2312
Posts: 303
Joined: Sun Jan 31, 2021 12:37 pm

Re: Zip keeps running out of memory

Wed Aug 04, 2021 10:47 am

rpdom wrote:
Wed Aug 04, 2021 10:21 am
That free output looks normal. Where do you get
kcajminer2312 wrote:
Wed Aug 04, 2021 10:02 am
kswapd0 seems to think the pi only has 500MB
from?
Task Manager and conky
it is rarely using over 500MB when unzipping a file

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 29342
Joined: Sat Jul 30, 2011 7:41 pm

Re: Zip keeps running out of memory

Wed Aug 04, 2021 11:38 am

kcajminer2312 wrote:
Wed Aug 04, 2021 10:44 am
found this
In case someone faces the same issues - the reason was malware: Multios.Coinminer.Miner.

kswapd0 was a binary file located in /root/.configrc/a/kswapd0. What you need to do is:

Clear crontab jobs referring to /root/.configrc
Clear ssh keys
Delete /root/.configrc
And does your system have this on it?
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Working in the Applications Team.

kcajminer2312
Posts: 303
Joined: Sun Jan 31, 2021 12:37 pm

Re: Zip keeps running out of memory

Wed Aug 04, 2021 11:40 am

write caching
that's why kswapd0 is going mad

but write caching is not supported by the backup disk
Last edited by kcajminer2312 on Wed Aug 04, 2021 11:53 am, edited 1 time in total.

brainsys
Posts: 172
Joined: Fri Jun 29, 2012 7:16 pm

Re: Zip keeps running out of memory

Wed Aug 04, 2021 11:53 am

kcajminer2312 wrote:
Wed Aug 04, 2021 10:44 am
found this
In case someone faces the same issues - the reason was malware: Multios.Coinminer.Miner.

kswapd0 was a binary file located in /root/.configrc/a/kswapd0. What you need to do is:

Clear crontab jobs referring to /root/.configrc
Clear ssh keys
Delete /root/.configrc
This would explain it - and why we were all going around in circles chasing the wrong piece of software.

If you haven't deleted everything yet - you should be able to deduce the time/date of infection and using that maybe the syslog will show the point and method of entry.

Good to know.

kcajminer2312
Posts: 303
Joined: Sun Jan 31, 2021 12:37 pm

Re: Zip keeps running out of memory

Wed Aug 04, 2021 11:54 am

brainsys wrote:
Wed Aug 04, 2021 11:53 am
kcajminer2312 wrote:
Wed Aug 04, 2021 10:44 am
found this
In case someone faces the same issues - the reason was malware: Multios.Coinminer.Miner.

kswapd0 was a binary file located in /root/.configrc/a/kswapd0. What you need to do is:

Clear crontab jobs referring to /root/.configrc
Clear ssh keys
Delete /root/.configrc
This would explain it - and why we were all going around in circles chasing the wrong piece of software.

If you haven't deleted everything yet - you should be able to deduce the time/date of infection and using that maybe the syslog will show the point and method of entry.

Good to know.
there is no /root/.configrc on my system maybe its on the user pi

kcajminer2312
Posts: 303
Joined: Sun Jan 31, 2021 12:37 pm

Re: Zip keeps running out of memory

Wed Aug 04, 2021 11:57 am

read caching maybe

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 29342
Joined: Sat Jul 30, 2011 7:41 pm

Re: Zip keeps running out of memory

Wed Aug 04, 2021 12:04 pm

Check the cron jobs to see if its started from there. Do a diskwide search for kswapd? Can you track down the actual executable running (i.e. full path) from the ps display?
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Working in the Applications Team.

kcajminer2312
Posts: 303
Joined: Sun Jan 31, 2021 12:37 pm

Re: Zip keeps running out of memory

Wed Aug 04, 2021 12:17 pm

jamesh wrote:
Wed Aug 04, 2021 12:04 pm
Check the cron jobs to see if its started from there. Do a diskwide search for kswapd? Can you track down the actual executable running (i.e. full path) from the ps display?
nothing in root cron
nothing in pi cron
I used angrysearch as well it found nothing for kswapd

so it is some sort of disk cache

bjtheone
Posts: 1480
Joined: Mon May 20, 2019 11:28 pm
Location: The Frozen North (AKA Canada)

Re: Zip keeps running out of memory

Wed Aug 04, 2021 12:41 pm

rpdom wrote:
Wed Aug 04, 2021 10:00 am
kcajminer2312 wrote:
Wed Aug 04, 2021 9:36 am
dist-upgrade updates the OS, I dont think full-upgrade does that
dist-upgrade and full upgrade are just different names for the same thing.
Further to that:

Code: Select all

 full-upgrade (apt-get(8))
           full-upgrade performs the function of upgrade but will remove currently installed
           packages if this is needed to upgrade the system as a whole.
Well explained by https://superuser.com/questions/1554163 ... pt-upgrade

---
apt upgrade will upgrade all packages that can be upgraded without the need to install additional packages or remove any conflicting installed packages. Basically it will apply all package upgrades that do not include changed dependencies.

apt full-upgrade (the correct equivalent for apt-get dist-upgrade) applies package upgrades as well if they require either the install of new packages or the removal of conflicting installed packages. Basically it will apply all package upgrades including those with changed dependencies.

On stable distro suites, when not adding new APT sources, one will face changed dependencies rarely, hence usually apt full-upgrade is not required or does not apply any additional upgrades compared to apt upgrade, except for Linux image package upgrades. Linux image packages are usually shipped as meta packages, e.g. linux-image-amd64, which then depends on the actual versioned package, e.g. linux-image-4.19.0-9-amd64. When one uses apt upgrade, the Linux image meta package will usually not be upgraded, since it would required to install a new versioned Linux image package. Running apt full-upgrade on the other hand will do the upgrade, furthermore it might remove the old versioned package, if the new one has a related conflict defined (which is usually not the case for Linux image packages, just as example!).

Due to the nature that installed packages can be removed by apt full-upgrade, it should not be called non-interactively. I personally ran into situations while testing the new Raspbian testing suite, where due to unresolved dependencies or conflicting repository updates the whole webserver stack would have been removed. I needed to wait until that repository inconsistency was resolved, before I was able to do the upgrades without causing havoc. However on stable suites I never faced such issue.
---

Personally I would never run anything that might remove things, non-interactively. There is a hierarchy of software, that should get the "most important" packages sorted at the expense of less important packages, in the case of a depancy conflict. However, you still may end up packages unexpected removed.

The value of staying totally up to date is not worth the risk, in my opinion. Dropping down to just "apt upgrade" will update everything possible without removing packages. Even then, I always run updates interactively.

kcajminer2312
Posts: 303
Joined: Sun Jan 31, 2021 12:37 pm

Re: Zip keeps running out of memory

Wed Aug 04, 2021 12:44 pm

bjtheone wrote:
Wed Aug 04, 2021 12:41 pm
rpdom wrote:
Wed Aug 04, 2021 10:00 am
kcajminer2312 wrote:
Wed Aug 04, 2021 9:36 am
dist-upgrade updates the OS, I dont think full-upgrade does that
dist-upgrade and full upgrade are just different names for the same thing.
Further to that:

Code: Select all

 full-upgrade (apt-get(8))
           full-upgrade performs the function of upgrade but will remove currently installed
           packages if this is needed to upgrade the system as a whole.
Well explained by https://superuser.com/questions/1554163 ... pt-upgrade

---
apt upgrade will upgrade all packages that can be upgraded without the need to install additional packages or remove any conflicting installed packages. Basically it will apply all package upgrades that do not include changed dependencies.

apt full-upgrade (the correct equivalent for apt-get dist-upgrade) applies package upgrades as well if they require either the install of new packages or the removal of conflicting installed packages. Basically it will apply all package upgrades including those with changed dependencies.

On stable distro suites, when not adding new APT sources, one will face changed dependencies rarely, hence usually apt full-upgrade is not required or does not apply any additional upgrades compared to apt upgrade, except for Linux image package upgrades. Linux image packages are usually shipped as meta packages, e.g. linux-image-amd64, which then depends on the actual versioned package, e.g. linux-image-4.19.0-9-amd64. When one uses apt upgrade, the Linux image meta package will usually not be upgraded, since it would required to install a new versioned Linux image package. Running apt full-upgrade on the other hand will do the upgrade, furthermore it might remove the old versioned package, if the new one has a related conflict defined (which is usually not the case for Linux image packages, just as example!).

Due to the nature that installed packages can be removed by apt full-upgrade, it should not be called non-interactively. I personally ran into situations while testing the new Raspbian testing suite, where due to unresolved dependencies or conflicting repository updates the whole webserver stack would have been removed. I needed to wait until that repository inconsistency was resolved, before I was able to do the upgrades without causing havoc. However on stable suites I never faced such issue.
---

Personally I would never run anything that might remove things, non-interactively. There is a hierarchy of software, that should get the "most important" packages sorted at the expense of less important packages, in the case of a depancy conflict. However, you still may end up packages unexpected removed.

The value of staying totally up to date is not worth the risk, in my opinion. Dropping down to just "apt upgrade" will update everything possible without removing packages. Even then, I always run updates interactively.
Thanks for the explanation but this isn't fixing my issue

kswapd0 is still going mad

I have had this issue over 4 days.
every time the backup starts the system hangs, then nobody in the house has internet because that raspberry pi runs pihole

brainsys
Posts: 172
Joined: Fri Jun 29, 2012 7:16 pm

Re: Zip keeps running out of memory

Wed Aug 04, 2021 1:40 pm

Hang on - - this is about a backup induced issue but you have no backup for pi-hole?

As much as I love RaspberryPis I wouldn't have one as a single point of failure. And losing DNS is about as catastrophic as you can get. As a minimum you should have a plan to revert to the ISP Router DHCP/DNS. Personally I wouldn't share the Pi-hole server with much else because you increase the possibility of something else bringing it down - or certainty in your case. My pi-hole is cloned on a backup card should the original go corrupt (it has done). Also pi-hole should run happiliy on an original 256MB RaspberryPi you may have at the bottom of the drawer. Do you have a spare?

I'd sort that out before thinking of sorting the compromised system. A fresh pi-hole install (provided you do have DNS) only takes minutes. At the very least it may keep the rest of the household quiet while you sort.

User avatar
rpdom
Posts: 19037
Joined: Sun May 06, 2012 5:17 am
Location: Chelmsford, Essex, UK

Re: Zip keeps running out of memory

Wed Aug 04, 2021 1:48 pm

kcajminer2312 wrote:
Wed Aug 04, 2021 11:54 am
there is no /root/.configrc on my system maybe its on the user pi
Filenames starting with a dot "." are hidden by default. Have you checked with "sudo ls -a /root" ?
Unreadable squiggle

kcajminer2312
Posts: 303
Joined: Sun Jan 31, 2021 12:37 pm

Re: Zip keeps running out of memory

Wed Aug 04, 2021 1:54 pm

rpdom wrote:
Wed Aug 04, 2021 1:48 pm
kcajminer2312 wrote:
Wed Aug 04, 2021 11:54 am
there is no /root/.configrc on my system maybe its on the user pi
Filenames starting with a dot "." are hidden by default. Have you checked with "sudo ls -a /root" ?
I have hidden files on
thats clearly not the issue

for the third time
its some sort of cache I think.
read maybe
because write is not supported

kcajminer2312
Posts: 303
Joined: Sun Jan 31, 2021 12:37 pm

Re: Zip keeps running out of memory

Wed Aug 04, 2021 1:57 pm

brainsys wrote:
Wed Aug 04, 2021 1:40 pm
Hang on - - this is about a backup induced issue but you have no backup for pi-hole?

As much as I love RaspberryPis I wouldn't have one as a single point of failure. And losing DNS is about as catastrophic as you can get. As a minimum you should have a plan to revert to the ISP Router DHCP/DNS. Personally I wouldn't share the Pi-hole server with much else because you increase the possibility of something else bringing it down - or certainty in your case. My pi-hole is cloned on a backup card should the original go corrupt (it has done). Also pi-hole should run happiliy on an original 256MB RaspberryPi you may have at the bottom of the drawer. Do you have a spare?

I'd sort that out before thinking of sorting the compromised system. A fresh pi-hole install (provided you do have DNS) only takes minutes. At the very least it may keep the rest of the household quiet while you sort.
pihole is backed up under etc and I have had no issues with it
the issue is kswap0 using 90+% load
causing the pihole-FTL to slow down

brainsys
Posts: 172
Joined: Fri Jun 29, 2012 7:16 pm

Re: Zip keeps running out of memory

Wed Aug 04, 2021 2:00 pm

I don't know angrysearch but does 'sudo find / -name kswap*' return nothing?
And when the system is being hammered does the top command show kswap0 and, if so, which user?

kcajminer2312
Posts: 303
Joined: Sun Jan 31, 2021 12:37 pm

Re: Zip keeps running out of memory

Wed Aug 04, 2021 2:03 pm

brainsys wrote:
Wed Aug 04, 2021 2:00 pm
I don't know angrysearch but does 'sudo find / -name kswap*' return nothing?
And when the system is being hammered does the top command show kswap0 and, if so, which user?
I have said that's not my issue
so please listen to me
I am running it anyway

top says
kswap0 and the user is root

brainsys
Posts: 172
Joined: Fri Jun 29, 2012 7:16 pm

Re: Zip keeps running out of memory

Wed Aug 04, 2021 2:21 pm

OK - sorry if you think I'm confused - but I am.

You thought you had solved the issue in that some mining software had let loose masquerading as kswap0. But you cannot find the file. So no file exists? Hence how do we know if there really is mining software or not - or, as we began, the genuine kswap0 goes wild?

Can you explain your line of thinking to conclude it is a cache problem?

Please remember it is hard diagnosing a problem from afar where you are the only medium. Checking and re-checking facts and assumptions (and the difference) is the only way we can really help.

kcajminer2312
Posts: 303
Joined: Sun Jan 31, 2021 12:37 pm

Re: Zip keeps running out of memory

Wed Aug 04, 2021 2:27 pm

brainsys wrote:
Wed Aug 04, 2021 2:21 pm
OK - sorry if you think I'm confused - but I am.

You thought you had solved the issue in that some mining software had let loose masquerading as kswap0. But you cannot find the file. So no file exists? Hence how do we know if there really is mining software or not - or, as we began, the genuine kswap0 goes wild?

Can you explain your line of thinking to conclude it is a cache problem?

Please remember it is hard diagnosing a problem from afar where you are the only medium. Checking and re-checking facts and assumptions (and the difference) is the only way we can really help.
For what I have read the genuine kswap0 deals with memory management, swap management and caches
so disk caches.

that makes sense, only when it is writing or reading from the disk kswap0 goes wild

bjtheone
Posts: 1480
Joined: Mon May 20, 2019 11:28 pm
Location: The Frozen North (AKA Canada)

Re: Zip keeps running out of memory

Wed Aug 04, 2021 2:30 pm

I would strongly recommend getting and building a new/fresh SD card. Building a card takes less than 30 minutes and setting up PiHole takes another say 10 minutes. At that point you should have a sane Pi, and functioning internet for the house. If you think you have a malware issue you should ensure the Pi is appropriately locked down before it ever sees the internet.

At that point clone the SD card and carefully start adding in the rest of the stuff the Pi was doing. If you suspect malware, I would be very cautious about copying stuff over form the suspect SD card and try and reload from source repos where ever possible. If the Pi breaks at in the same way, you know what package or configuration step you have issues with.

kcajminer2312
Posts: 303
Joined: Sun Jan 31, 2021 12:37 pm

Re: Zip keeps running out of memory

Wed Aug 04, 2021 2:44 pm

bjtheone wrote:
Wed Aug 04, 2021 2:30 pm
I would strongly recommend getting and building a new/fresh SD card. Building a card takes less than 30 minutes and setting up PiHole takes another say 10 minutes. At that point you should have a sane Pi, and functioning internet for the house. If you think you have a malware issue you should ensure the Pi is appropriately locked down before it ever sees the internet.

At that point clone the SD card and carefully start adding in the rest of the stuff the Pi was doing. If you suspect malware, I would be very cautious about copying stuff over form the suspect SD card and try and reload from source repos where ever possible. If the Pi breaks at in the same way, you know what package or configuration step you have issues with.
Right I have said its not malware now
SO LISTEN TO ME
From what I have read the genuine kswap0 deals with memory management, swap management and caches
so disk caches.

that makes sense, only when it is writing or reading from the disk kswap0 goes wild

in fact it could not be malware because it doesn't happen all the time, only when reading or writing to the BACKUP disk

clearly kswapd0 is the genuine one because it is run by root and a PID of 65, so one of the first things that started

User avatar
lpsw
Posts: 303
Joined: Thu Jun 25, 2015 4:45 pm
Location: USA

Re: Zip keeps running out of memory

Wed Aug 04, 2021 3:22 pm

kcajminer2312 wrote:
Wed Aug 04, 2021 2:44 pm

Right I have said its not malware now
SO LISTEN TO ME
Where?
viewtopic.php?f=63&t=316693&start=50#p1897560
Self-education is, I firmly believe, the only kind of education there is - Isaac Asimov

jamesh
Raspberry Pi Engineer & Forum Moderator
Raspberry Pi Engineer & Forum Moderator
Posts: 29342
Joined: Sat Jul 30, 2011 7:41 pm

Re: Zip keeps running out of memory

Wed Aug 04, 2021 3:25 pm

kcajminer2312 wrote:
Wed Aug 04, 2021 2:44 pm
bjtheone wrote:
Wed Aug 04, 2021 2:30 pm
I would strongly recommend getting and building a new/fresh SD card. Building a card takes less than 30 minutes and setting up PiHole takes another say 10 minutes. At that point you should have a sane Pi, and functioning internet for the house. If you think you have a malware issue you should ensure the Pi is appropriately locked down before it ever sees the internet.

At that point clone the SD card and carefully start adding in the rest of the stuff the Pi was doing. If you suspect malware, I would be very cautious about copying stuff over form the suspect SD card and try and reload from source repos where ever possible. If the Pi breaks at in the same way, you know what package or configuration step you have issues with.
Right I have said its not malware now
SO LISTEN TO ME
From what I have read the genuine kswap0 deals with memory management, swap management and caches
so disk caches.

that makes sense, only when it is writing or reading from the disk kswap0 goes wild

in fact it could not be malware because it doesn't happen all the time, only when reading or writing to the BACKUP disk

clearly kswapd0 is the genuine one because it is run by root and a PID of 65, so one of the first things that started
You have said many things, but we are still no closer to determining what the problem might be. kswapd seem to be taking lots of CPU time is the only diagnostic you have given us so far.

This doesn't happen for most people, and since this is a standard kernel features (i.e. nothing we have changed specifically for the Pi) its vanishingly unlikely there is a technical problem with it. So that leaves the setup or perhaps some runaway process that is taking up all the CPU time.

So, try a new SD card with basic Raspberry Pi OS on it, then run your script (with the appropriate fixes as people have suggested above). Don't install anything else, just try the very basics.

Get back with the results once you have them.
Principal Software Engineer at Raspberry Pi (Trading) Ltd.
Working in the Applications Team.

kcajminer2312
Posts: 303
Joined: Sun Jan 31, 2021 12:37 pm

Re: Zip keeps running out of memory

Wed Aug 04, 2021 3:34 pm

lpsw wrote:
Wed Aug 04, 2021 3:22 pm
kcajminer2312 wrote:
Wed Aug 04, 2021 2:44 pm

Right I have said its not malware now
SO LISTEN TO ME
Where?
viewtopic.php?f=63&t=316693&start=50#p1897560
three times

Return to “General discussion”