I’ve misplaced a user cron job (and it is still running)


16 posts
by whynot » Mon Mar 18, 2013 10:23 am
A few months ago, I set up a user cron job. The job was delete webcam (motion) files older than a few days. For some reason, I created the cron job in a non standard location (one of the user areas, I think). The problem is, I have altered the web cam directory and the cron job is sending me warning emails, every day at 12 hrs. I would like to find where I put this cron job.

Obviously, I have not stored it in
Code: Select all
crontab –e
. Nor,
Code: Select all
sudo crontab –u pi – l
(or the other generic accounts). The only clue that I have is that part of the command is
Code: Select all
find /mnt/share/webcam/* -mtime +2 -exec rm {} \;
Is it possible to do a wildcard search through all of the files to find this string (or even part of it)?
Posts: 8
Joined: Tue Dec 25, 2012 9:49 pm
by azeam » Mon Mar 18, 2013 12:09 pm
Check in the /etc/cron* folders (probably /etc/cron.daily) and see if you can find it there. Otherwise you can search for it like this
Code: Select all
grep -r "find /mnt/share/webcam/* -mtime +2 -exec rm {} \;" /
User avatar
Posts: 192
Joined: Fri Oct 26, 2012 11:13 pm
by whynot » Sat Mar 23, 2013 12:49 am
Thanks for the reply. One small problem. When I run the command it causes the Pi to crash. But, thanks for the tip on /etc/cron.daily. I'll check those files.
Posts: 8
Joined: Tue Dec 25, 2012 9:49 pm
by Joe Schmoe » Sat Mar 23, 2013 12:58 am
I would do the search something like:

find / -xdev -type f -size -5000c -print0 | xargs -0 grep "find.*mnt.share.webcam"

Yes, those are zeroes, not Ohs.
Never answer the question you are asked. Rather, answer the question you wish you had been asked.

- Robert S. McNamara - quoted in "Fog of War" -
Posts: 2250
Joined: Sun Jan 15, 2012 1:11 pm
by azeam » Sat Mar 23, 2013 2:42 pm
whynot wrote:Thanks for the reply. One small problem. When I run the command it causes the Pi to crash. But, thanks for the tip on /etc/cron.daily. I'll check those files.


Joe's command is more efficient but the one I posted shouldn't crash the pi. It's very time consuming to search through all files however (guessing it would take a couple of hours) and nothing will be displayed unless it finds a file matching the search string so it might seem as if it crashed, you should be able to cancel the command with "ctrl + c".
User avatar
Posts: 192
Joined: Fri Oct 26, 2012 11:13 pm
by Joe Schmoe » Sat Mar 23, 2013 3:02 pm
azeam wrote:
whynot wrote:Thanks for the reply. One small problem. When I run the command it causes the Pi to crash. But, thanks for the tip on /etc/cron.daily. I'll check those files.


Joe's command is more efficient but the one I posted shouldn't crash the pi. It's very time consuming to search through all files however (guessing it would take a couple of hours) and nothing will be displayed unless it finds a file matching the search string so it might seem as if it crashed, you should be able to cancel the command with "ctrl + c".


Comments:
1) You are right to be skeptical of the claim "it crashed my system". Question for the OP: Why do you think it crashed your system? What did you try/test?

Note that many times we hear of people complaining that the "dd" process of generating the bootable SD card crashed their system, because it runs for a long time and gives no visual feedback.

2) In the above "find" command, the goal wasn't so much "efficiency" as "not grepping files you shouldn't be grepping". This includes binary files, files in /proc and files in /dev. I think there are files in /proc that if you try to grep them, bad things happen (even as root). And of course, that goes double for files in /dev.

FWIW, I think grepping a file in /dev very well *could* "crash" the system.
Never answer the question you are asked. Rather, answer the question you wish you had been asked.

- Robert S. McNamara - quoted in "Fog of War" -
Posts: 2250
Joined: Sun Jan 15, 2012 1:11 pm
by whynot » Sat Mar 30, 2013 5:27 am
I must admit I do have a lot to learn about Linux, so I could have miss interpreted a crash. I usually connect to the PI via SSH. Usually, I have three or four SSH running concurrently (e.g. one with a man page open, another with a top page, another SSH open as a spare command prompt, etc). As I connect via Windows 7, I usually run sessions in screen. So that when my PC goes to sleep and drops the SSH, I can pick it up where I left of.

I was aware that the search could take a long time. From memory, I started a screen session, then ran the suggested command. As the command runs, it does write to the screen. After a few minutes, it stopped dumping to the screen. At the same time, my other SSH sessions became unresponsive. There is also a Python session running in the background that uploads the readings from my PV system to pvoutput.org. That stopped as well. The little red lights on the RS-485 interface stopped flashing (indicating polling of that RS-485 bus had stopped as well). The PI also stopped accepting new SSH. And, a peak at the router indicated that the PI had stopped sending network traffic. If there was a process still running deep inside the PI by this stage, I certainly don’t have the skills to resuscitate it. I drew the conclusion that the PI had crashed. The PI would partially restart after a hard power cycle. But, to get it going again, I had to physically recover it. Hook it up to a keyboard and monitor, and then run fdisk on it. Two other attempts to run the command caused exactly the same problem (and another two cycles of recovering the PI and running fdisk).


I didn’t want to make a fuss about the suggested command, as someone was trying to be genuinely helpful. But, I did feel it was important to make a note of the issue in case someone else was having the same problem.


For now, the problem remains unresolved. I still have a mystery cron command running on the PI. I am sure I will find it, once I have enough free time to hunt it down.

Edit: I should add this this has been a jolly good learning experience. I have learnt more about using Unix in the last few months with the PI than my attempts over the previous fifteen years. Even having "problems" generates more learning and understanding. And, I feel a lot more comfortable about accidently trashing a PI than one of the mainframes at work.
Posts: 8
Joined: Tue Dec 25, 2012 9:49 pm
by Joe Schmoe » Sat Mar 30, 2013 1:43 pm
whynot wrote:I didn’t want to make a fuss about the suggested command, as someone was trying to be genuinely helpful. But, I did feel it was important to make a note of the issue in case someone else was having the same problem.


For now, the problem remains unresolved. I still have a mystery cron command running on the PI. I am sure I will find it, once I have enough free time to hunt it down.

Did you try my suggested command (with find, xargs, grep)? That should definitely work. And, the more I think about it, the more I think that the reason it crashes (and yes, I am convinced that you were right in your diagnosis) was because you tried to grep stuff in /dev.

P.S. You mentioned having to run "fdisk". ITYM "fsck".
Never answer the question you are asked. Rather, answer the question you wish you had been asked.

- Robert S. McNamara - quoted in "Fog of War" -
Posts: 2250
Joined: Sun Jan 15, 2012 1:11 pm
by whynot » Sat Mar 30, 2013 11:35 pm
Ok, after backing up the Pi, I ran your suggested code. I am afraid no luck.
Posts: 8
Joined: Tue Dec 25, 2012 9:49 pm
by jojopi » Sun Mar 31, 2013 12:26 am
What is your evidence that a cron job is running, and why do you not know where it is running from? Compare /var/log/syslog, for instance.

The only places cron runs jobs from are /etc/crontab, /etc/cron.d/*, and /var/spool/cron/crontabs/*. Plus /etc/crontab indirectly includes /etc/cron.{daily,weekly,monthly}/*.

Searching the whole filesystem is a lazy solution. Lazy solutions are often great, especially for suggesting over the internet where it could take ages to get a reply. But they can only work if the original premises are correct.
User avatar
Posts: 1861
Joined: Tue Oct 11, 2011 8:38 pm
by Joe Schmoe » Sun Mar 31, 2013 12:47 am
jojopi wrote:What is your evidence that a cron job is running, and why do you not know where it is running from? Compare /var/log/syslog, for instance.

The only places cron runs jobs from are /etc/crontab, /etc/cron.d/*, and /var/spool/cron/crontabs/*. Plus /etc/crontab indirectly includes /etc/cron.{daily,weekly,monthly}/*.

Searching the whole filesystem is a lazy solution. Lazy solutions are often great, especially for suggesting over the internet where it could take ages to get a reply. But they can only work if the original premises are correct.


Agreed. Note also that /var/spool/cron/crontabs is normally a protected directory, so you'd have to run my "lazy" command line as root. Or, you could just (as root) do an ls of that directory.
Never answer the question you are asked. Rather, answer the question you wish you had been asked.

- Robert S. McNamara - quoted in "Fog of War" -
Posts: 2250
Joined: Sun Jan 15, 2012 1:11 pm
by whynot » Sun Mar 31, 2013 8:36 am
jojopi wrote:What is your evidence that a cron job is running, and why do you not know where it is running from? Compare /var/log/syslog, for instance.

The only places cron runs jobs from are /etc/crontab, /etc/cron.d/*, and /var/spool/cron/crontabs/*. Plus /etc/crontab indirectly includes /etc/cron.{daily,weekly,monthly}/*.

Searching the whole filesystem is a lazy solution. Lazy solutions are often great, especially for suggesting over the internet where it could take ages to get a reply. But they can only work if the original premises are correct.


Because, every day, precisely at 12:00hrs, I get an email message from Cron <root@raspberrypi> stating that it has run my command and there is an error in it. Even better, the cron job is doing what it is suppose to do! Clean out files in a certain directory older than two days.

I changed part of the directory structure. Now, I would like to change the cron job to suit.

Yes, silly me for misplacing the file.

I have checked through the directories suggested and the cron job is not there.

Speaking of lazy solutions … if I reinstalled cron, would it delete all existing jobs?

BTW $crontab – l returns the comment “no crontab for pi”. Go figure …
Posts: 8
Joined: Tue Dec 25, 2012 9:49 pm
by jojopi » Sun Mar 31, 2013 9:25 am
whynot wrote:Because, every day, precisely at 12:00hrs, I get an email message from Cron <root@raspberrypi> stating that it has run my command and there is an error in it.
The Subject header of the mail gives the precise command being executed. The X-Cron-Env header for LOGNAME will tell you which user it is running as. According to your original post, you may not have checked the crontab for root ("sudo crontab -e").

(No, uninstalling and reinstalling cron will not delete your cron jobs. Even if it did, it could only delete them from the places that it looks. So just checking those places yourself would be fine.)
User avatar
Posts: 1861
Joined: Tue Oct 11, 2011 8:38 pm
by whynot » Sun Mar 31, 2013 12:02 pm
jojopi wrote:The Subject header of the mail gives the precise command being executed. The X-Cron-Env header for LOGNAME will tell you which user it is running as. According to your original post, you may not have checked the crontab for root ("sudo crontab -e").

(No, uninstalling and reinstalling cron will not delete your cron jobs. Even if it did, it could only delete them from the places that it looks. So just checking those places yourself would be fine.)


Thanks. sudo crontab -e was the first place I went looking.

Hhhaarrr ... the email headers. I never thought of looking there.

Below is part of the header

Received: by localhost (sSMTP sendmail emulation); Sun, 31 Mar 2013 12:00:01 +1000
Date: Sun, 31 Mar 2013 12:00:01 +1000
...
To: root
Subject: Cron <root@raspberrypi> find /mnt/share/webcam/* -mtime +2 -exec rm {} \;
Content-Type: text/plain; charset=UTF-8
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin>
X-Cron-Env: <HOME=/root>
X-Cron-Env: <LOGNAME=root>

rm: cannot remove `/mnt/share/webcam/archive': Is a directory
Posts: 8
Joined: Tue Dec 25, 2012 9:49 pm
by jojopi » Sun Mar 31, 2013 4:02 pm
whynot wrote:X-Cron-Env: <LOGNAME=root>
So the job is either in root's crontab, or in one of the system crontabs and with "root" in the username field. It cannot be in any other user's crontab.

We can rule out cron.{hourly,daily,weekly,monthly}, because runparts sends a single summary email for each of those directories, not a separate message for each job.

This only leaves /etc/crontab and /etc/cron.d/*. If you were deliberately hiding the job, you may have created a file in /etc/cron.d whose name starts with a dot, or set the permissions so that pi cannot read it. Otherwise all you need is:
Code: Select all
grep webcam /etc/crontab /etc/cron.d/*
User avatar
Posts: 1861
Joined: Tue Oct 11, 2011 8:38 pm
by whynot » Sun Mar 31, 2013 9:22 pm
jojopi wrote:This only leaves /etc/crontab and /etc/cron.d/*. If you were deliberately hiding the job, you may have created a file in /etc/cron.d whose name starts with a dot, or set the permissions so that pi cannot read it. Otherwise all you need is:
Code: Select all
grep webcam /etc/crontab /etc/cron.d/*


Success!

grep webcam /etc/crontab returns the result

Code: Select all
0 12 * * *  root find /mnt/share/webcam/* -mtime +2 -ecex rm {} \;


Thanks for helping me find it.
Posts: 8
Joined: Tue Dec 25, 2012 9:49 pm