Cron Runs Job One Time Then Never Again

The cron task scheduler has been arranged with unix and linux operating systems for decades. It'due south widely used and unremarkably very reliable. This guide volition walk you lot through locating cron jobs on your server, validating their schedules, and debugging common problems.

In this guide

1. The common reasons cron jobs fail
2. What to do if your cron task isn't running when expected
3. What to practice if your cron chore fails unexpectedly
4. What to practise if nothing else works

The common reasons cron jobs fail

Cron jobs tin can neglect for an infinite variety of reasons but at Cronitor nosotros have observed common failure patterns that can provide a useful starting point for your debugging process. In the rest of this guide, nosotros will dive into these common failures with suggestions and solutions.

If you're adding a new cron job and it is not working...

• Schedule errors are easy to make

  Cron job schedule expressions are quirky and hard to write. If your job didn't run when y'all expected it to, the easiest thing to rule out is a fault with the cron expression.

• Cron has subtle environment differences

  A common experience is to have a job that works flawlessly when run at the control line simply fails whenever it's run past cron. When this happens, eliminate these common failures:

  1. The command has an unresovable relative path like ../scripts. (Endeavor an accented path)
  2. The task uses environment variables. (Cron does not load .bashrc and similar files)
  3. The command uses advanced bash features (cron uses /bin/sh past default)

  Tip: Our free software, CronitorCLI, includes a shell command to exam run any chore the mode cron does.

• There is a problem with permissions

  Invalid permissions tin cause your cron jobs to neglect in at least 3 ways. This guide will comprehend them all.

If your cron job stopped working of a sudden...

• System resources have been depleted

  The almost stable cron task is no match for a disk that is full or an OS that can't spawn new threads. Check all the usual graphs to rule this out.

• You lot've reached an inflection bespeak

  Cron jobs are often used for batch processing and other data-intensive tasks that tin can reveal the constraints of your stack. Jobs often work fine until your data size grows to a point where queries get-go timing-out or file transfers are too slow.

• Infrastructure drift occurs

  When app configuration and code changes are deployed information technology tin can be easy to overlook the cron jobs on each server. This causes infrastructure drift where hosts are retired or credentials modify that intermission the forgotten cron jobs.

• Jobs accept begun to overlap themselves

  Cron is a very simple scheduler that starts a task at the scheduled fourth dimension, even if the previous invocation is nevertheless running. A small slow-down can pb to a pile-up of overlapped jobs sucking up bachelor resources.

• A bug has been introduced

  Sometimes a failure has nil to do with cron. It tin can exist difficult to thoroughly test cron jobs in a evolution environment and a bug might be merely in production.

What to do if your cron job doesn't run when expected

You may find that you have scheduled a cron job that is just not running when expected. Follow these steps to locate your scheduled job and diagnose the problem.

1. Locate the scheduled job

   Tip: If you know where your task is scheduled, skip this step

   Cron jobs are run by a system daemon called crond that watches several locations for scheduled jobs. The first footstep to understanding why your job didn't showtime when expected is to find where your job is scheduled.

   Search manually for cron jobs on your server

   • Check your user crontab with crontab -l

     dev01: ~ $ crontab -fifty # Edit this file to innovate tasks to be run past cron. # chiliad h  dom mon dow   command 5 4 * * *      /var/cronitor/bin/database-fill-in.sh

   • Jobs are usually created by adding a crontab file in /etc/cron.d/
   • System-level cron jobs can too exist added as a line in /etc/crontab
   • Sometimes for easy scheduling, jobs are added to /etc/cron.hourly/, /etc/cron.daily/, /etc/cron.weekly/ or /etc/cron.monthly/
   • Information technology's possible that the job was created in the crontab of another user. Go through each user's crontab using crontab -u username -l

   Or, scan for cron jobs automatically with CronitorCLI

   • Install CronitorCLI for costless
   • Run cronitor listing to browse your system for cron jobs:
     

   If you tin can't detect your job merely believe information technology was previously scheduled double cheque that you are on the correct server.

2. Validate your task schedule

   One time you lot accept constitute your job, verify that it'south scheduled correctly. Cron schedules are usually used because they are expressive and powerful, but like regular expressions can sometimes be difficult to read. Nosotros advise using Crontab Guru to validate your schedule.

   • Paste the schedule expression from your crontab into the text field on Crontab Guru
   • Verify that the plaintext translation of your schedule is right, and that the adjacent scheduled execution times match your expectations
   • Check that the effective server timezone matches your expectation. In improver to checking system fourth dimension using date, check your crontab file for TZ or CRON_TZ timezone declarations that would override system settings. For example, CRON_TZ=America/New_York

3. Bank check your permissions

   Invalid permissions tin cause your cron jobs to fail in at to the lowest degree 3 ways:

   1. Jobs added as files in a /etc/cron.*/ directory must be owned past root. Files owned past other users will be ignored and you lot may encounter a message like to WRONG FILE Owner in your syslog.
   2. The command must be executable by the user that cron is running your job as. For example if your ubuntu user crontab invokes a script like database-fill-in.sh, ubuntu must have permission to execute the script. The most direct manner is to ensure that the ubuntu user owns the file and and then ensure execute permissions are available using chmod +x database-backup.sh.
   3. The user business relationship must be allowed to utilize cron. First, if a /etc/cron.let file exists, the user must exist listed. Separately, the user cannot be in a /etc/cron.deny list.

   Related to the permissions trouble, ensure that if your command string contains a % that it is escaped with a backslash.

4. Check that your cron job is running past finding the attempted execution in syslog

   When cron attempts to run a command, it logs it in syslog. By grepping syslog for the name of the command you constitute in a crontab file yous can validate that your chore is scheduled correctly and cron is running.

   • Begin by grepping for the command in /var/log/syslog (You will probably need root or sudo access.)

     dev01: ~ $ grep database-fill-in.sh /var/log/syslog Aug  5 4:05:01 dev01 CRON[2128]: (ubuntu) CMD (/var/cronitor/bin/database-backup.sh)

   • If you can't observe your control in the syslog information technology could exist that the log has been rotated or cleared since your job ran. If possible, rule that out by updating the chore to run every minute by irresolute its schedule to * * * * *.
   • If your command doesn't announced as an entry in syslog within ii minutes the trouble could be with the underlying cron daemon known as crond. Rule this out quickly by verifying that cron is running by looking up its procedure ID. If cron is not running no procedure ID will be returned.

     dev01: ~ $ pgrep cron 323

   • If you've located your job in a crontab file but persistently cannot find it referenced in syslog, double check that crond has correctly loaded your crontab file. The easiest manner to exercise this is to force a reparse of your crontab by running EDITOR=true crontab -due east from your command prompt. If everything is up to engagement you will see a message like No modification made. Any other message indicates that your crontab file was not reloaded after a previous update merely has at present been updated. This volition also ensure that your crontab file is free of syntax errors.

If you tin can come across in syslog that your job was scheduled and attempted to run correctly but still did not produce the expected issue you lot can assume there is a problem with the command you are trying to run.

What to do if your cron job fails unexpectedly

Your cron jobs, like every other office of your organization, will eventually fail. This department will walk you through testing your job and spotting common failures.

1. Exam run your command like cron does

   When cron runs your command the surroundings is unlike from your normal command prompt in subtle but important ways. The offset footstep to troubleshooting is to simulate the cron environment and run your control in an interactive beat.

   Run whatsoever control similar cron does with CronitorCLI

   • Install CronitorCLI for free
   • To force a scheduled cron job to run immediately, use cronitor select to browse your system and present a list of jobs to choose from.
   • To simulate running whatever command the way cron does, utilize cronitor shell:
     

   Or, manually test run a command like cron does

   • If you are parsing a file in /etc/cron.d/ or /etc/crontab each line is immune to have an effective "run as" username later the schedule and before the command itself. If this applies to your job, or if your job is in some other user's crontab, brainstorm by opening a bash prompt as that user sudo -u username bash
   • Past default, cron will run your command using /bin/sh, not the fustigate or zsh prompt you are familiar with. Double check your crontab file for an optional SHELL=/bin/fustigate declaration. If using the default /bin/sh beat, sure features that work in bash similar [[control]] syntax volition crusade syntax errors under cron.
   • Unlike your interactive shell, cron doesn't load your bashrc or bash_profile and so any environment variables divers there are unavailable in your cron jobs. This is true fifty-fifty if you have a Beat out=/bin/bash declaration. To simulate this, create a command prompt with a clean environment.

     dev01: ~ $ env -i /bin/sh $

   • By default, cron volition run commands with your home directory every bit the electric current working directory. To ensure you are running the command like cron does, run cd ~ at your prompt.
   • Paste the control to run (everything after the schedule or declared username) into the command prompt. If crontab is unable to run your command, this should fail too and volition hopefully comprise a useful fault message. Common errors include invalid permissions, command not found, and command line syntax errors.
   • If no useful mistake message is available, double bank check any awarding logs your job is expected to produce, and ensure that you are not redirecting log and error messages. In linux, control >> /path/to/file volition redirect console log messages to the specified file and command >> /path/to/file ii>&one will redirect both the console log and error letters. Make up one's mind if your control has a verbose output or debug log flag that can be added to see additional details at runtime.
   • Ideally, if your task is failing under cron information technology volition fail here as well and you lot will see a useful fault message, giving you a risk to alter and debug equally needed. Common errors include file not found, misconfigured permissions, and commandline syntax errors.

2. Check for overlapping jobs

   At Cronitor our information shows that runtime durations increase over time for a large percentage of cron jobs. Equally your dataset and user base grows it'due south normal to find yourself in a situation where cron starts an instance of your job before the previous one has finished. Depending on the nature of your chore this might not exist a problem, just several undesired side effects are possible:

   • Unexpected server or database load could bear on other users
   • Locking of shared resources could crusade deadlocks and foreclose your jobs from ever completing successfully
   • Creation of an unanticipated race condition that might result in records existence candy multiple times, amplifying server load and possibly impacting customers

   To verify if any instances of a job are running on your server presently, grep your process list. In this case, iii job invocations are running simultaneously:

   dev01: ~ $ ps aux | grep database-fill-in.sh ubuntu           1343   0.0  0.1  2585948  12004   ??  S    31Jul18   ane:04.15 /var/cronitor/bin/database-backup.sh ubuntu           3659   0.0  0.1  2544664    952   ??  S     1Aug18   0:34.35 /var/cronitor/bin/database-backup.sh ubuntu           7309   0.0  0.i  2544664   8012   ??  Southward     2Aug18   0:18.01 /var/cronitor/bin/database-backup.sh

   To quickly recover from this, first kill the overlapping jobs and then lookout man closely when your command is next scheduled to run. It'southward possible that a one fourth dimension failure cascaded into several overlapping instances. If it becomes clear that the job often takes longer than the interval between chore invocations you may need to accept additional steps, e.1000.:

   • Increase the duration between invocations of your job. For example if your job runs every minute at present, consider running information technology every other infinitesimal.
   • Utilize a tool like flock to ensure that only a single case of your command is running at any given fourth dimension. Using flock is piece of cake. After installing from apt-go or yum you simply need to prefix the command in your crontab:

   dev01: ~ $ crontab -l # Edit this file to introduce tasks to be run by cron. # m h  dom mon dow   control * * * * *      flock -w 0 /var/cronitor/bin/database-backup.sh

   Read more nearly flock from the man page.

What to do if cypher else works

Here are a few things you can try if you lot've followed this guide and find that your job works flawlessly when run from the cron-like command prompt just fails to complete successful under crontab.

• Commencement go the most basic cron job working with a command similar date >> /tmp/cronlog. This command volition simply echo the execution fourth dimension to the log file each time it runs. Schedule this to run every minute and tail the logfile for results.

• If your bones command works, replace it with your command. Every bit a sanity cheque, verify if it works.

• If your control works by invoking a runtime like python some-command.py perform a few checks to determine that the runtime version and environment is correct. Each language runtime has quirks that tin can cause unexpected behavior under crontab.

  • For python you lot might notice that your web app is using a virtual environment you need to invoke in your crontab.
  • For node a common trouble is falling back to a much older version bundled with the distribution.
  • When using php you might meet the event that custom settings or extensions that work in your web app are not working under cron or commandline because a unlike php.ini is loaded.

  If you are using a runtime to invoke your command double check that information technology's pointing to the expected version from within the cron environment.

• If nothing else works, restart the cron daemon and hope it helps, stranger things have happened. The manner to practise this varies from distro to distro so it's best to inquire google.

Related Guides

---

Cronitor automatically finds and monitors cron jobs.  Complete your setup in minutes.

Get Started Free

westcalt1986.blogspot.com

Source: https://cronitor.io/cron-reference/cron-troubleshooting-guide

Share this post