Cron: Schedule Task with Log Housekeeping
Many Linux servers run scheduled tasks using Cron Jobs as a matter of routine. According to the job definition, Cron will run the task (usually a command or a group of commands in a shell script) at specific times (e.g. daily, 1st of every month, every 20 minutes, etc.).
With this powerful utility, we can easily deploy a job that is activated periodically with log housekeeping. Let's see how it is done with a simple example.
Create a Schedule Task With Logging
To get started, create a new directory and a dummy task that should run every minute.
cd Documnets
mkdir cronDemo
cd cronDemo
vim dummyTask.sh
Inside dummyTask.sh, put commands that print out the current timestamp and indicate the job is activated:
date
echo "Dummy task run"
Then, change the permission of dummyTask.sh so that we and Cron can execute the script on our behalf.
# rwxr--r--
chmod 744 dummyTask.sh
Try to run it to see if it is ready.
./dummyTask.sh
# Example result:
# Mon Jan 23 16:35:14 HKT 2023
# Dummy task run
As the CronJob output is "invisible" to us, we need to redirect the output to some log file (or just txt, as you like). We can create a log directory and place all the output there.
mkdir log
# This will redirect the output to the dummy.log in append mode
./dummyTask.sh >> log/dummy.log
Print the output file for checking.
cat log/dummy.log
# Example result:
# Mon Jan 23 16:41:33 HKT 2023
# Dummy task run
We are now ready to add it as a Cron Job. Firstly, open the crontab entries in edit mode:
crontab -e
Be careful not to mistype it as
crontab -r. Otherwise, all existing cron jobs will be cleared!
Put the last line into the list:
# Example of job definition:
# .---------------- minute (0 - 59)
# | .------------- hour (0 - 23)
# | | .---------- day of month (1 - 31)
# | | | .------- month (1 - 12)
# | | | | .---- day of week (0 - 6) (Sunday=0 or 7)
# | | | | |
# * * * * * command to be executed
# remember to use absoulte path or Cron may fail to locate the file
* * * * * /Users/maisytse/Documents/cronDemo/dummyTask.sh >> /Users/maisytse/Documents/cronDemo/log/dummy.log 2>&1
Read through some tutorials like https://www.tutorialspoint.com/unix_commands/crontab.htm for details.
Now you should able to see the job run once a minute by tracking the log file.
If you are using Mac, you may find the output file remains empty. Check out this post to fix it.
Log Rotate
From time to time, we may want to check for the log of a specific day. It will be hard to locate the log entries if all of them are in the same file. Therefore, it is a good idea to keep each day's log in separated files. By convention, the file will be named with date.
The easiest way is to change the crontab to this:
* * * * * /Users/maisytse/Documents/cronDemo/dummyTask.sh >> /Users/maisytse/Documents/cronDemo/log/dummy.log.`/bin/date +\%Y\%m\%d` 2>&1
As a result, the output will now go to a file like this:
-rw-r--r-- 1 <usename> <usergroup> 44B Jan 23 17:59 dummy.log.20230123
Another way is to keep today's log in the file without the date suffix so that it can be found quickly:
0 0 * * * mv /Users/maisytse/Documents/cronDemo/log/dummy.log /Users/maisytse/Documents/cronDemo/log/dummy.log.`/bin/date -d "yesterday" +\%Y\%m\%d`
* * * * * /Users/maisytse/Documents/cronDemo/dummyTask.sh >> /Users/maisytse/Documents/cronDemo/log/dummy.log 2>&1
The first Cron Job moves the current log to the log file with yesterday's date suffix each day at 00:00 am while the second one allows us to access the current log by the name dummy.log.
Housekeeping
On the other hand, if the scheduled task keeps generating lots of output, the log file's size may grow rapidly and use up the storage. That's why we would like some housekeeping to remove old logs.
One method is to make use of this command:
find . -type f -mtime +30 -name "*.log*" -delete
# . : under current directory
# -type f : all files
# -mtime +30 : with last modified date more than 30 days
# -name "*.log*" : with name match the pattern *.log
# -delete : delete the matched files
Let's put this into the crontab.
0 0 * * * find /Users/maisytse/Documents/cronDemo/log -type f -mtime +30 -name "*.log*" -delete
You can use touch to create files with last modified date in the past (e.g. touch -t 202102151209 dummy.log.20210215) to test out the housekeeping command.
Another common practice is to zip the old log to save space.
1 0 * * * gzip /Users/maisytse/Documents/cronDemo/log/dummy.log.*
Make use of
zcatandviewto read the zipped log (e.g.zcat < dummy.log.20230123.gz | view -).
We did it! You are now ready to build your own Cron Job!