Build a Self-Healing Content Pipeline with macOS LaunchAgents

Why This Matters

Cron-based pipelines are ubiquitous in content workflows, but they have a critical flaw: silent failures. When a script fails, cron doesn’t alert you. The first sign of trouble is often a gap in your data or content. Worse, LaunchAgents can unload silently, leaving you unaware that your automation has stopped.

This tutorial shows you how to build a self-healing content pipeline using macOS launchd and LaunchAgents. You’ll learn to:

1. Create and manage LaunchAgents.
2. Implement a watchdog pattern to monitor and restart agents.
3. Ensure resilience with auto-recovery and idempotent scripts.
4. Monitor your pipeline effectively.

By the end, you’ll have a robust system that minimizes downtime and alerts you to issues.

Prerequisites

• macOS: This tutorial assumes you’re using macOS, which uses launchd for managing daemons and agents.
• Scripts: You should have scripts you want to run on a schedule (e.g., Python, Bash, Ruby).
• Basic Shell: Familiarity with shell scripting and command-line tools.

Step 1: The Naive Version

Start with a simple LaunchAgent that runs a script daily.

1.1. Create a Plist File

Create a .plist file in ~/Library/LaunchAgents/. Here’s a template using StartCalendarInterval:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" 
    "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.example.site-cron</string>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/bin/python3</string>
        <string>/Users/username/scripts/site-cron.py</string>
    </array>
    <key>StartCalendarInterval</key>
    <dict>
        <key>Hour</key>
        <integer>9</integer>
        <key>Minute</integer>
        <integer>0</integer>
    </dict>
    <key>StandardOutPath</key>
    <string>/tmp/site-cron.out</string>
    <key>StandardErrorPath</key>
    <string>/tmp/site-cron.err</string>
</dict>
</plist>

Key Points:

• Label: Unique identifier for the agent.
• ProgramArguments: Command and arguments to execute.
• StartCalendarInterval: Schedule in cron-like format.
• StandardOutPath and StandardErrorPath: Log file paths.

1.2. Load the LaunchAgent

Use launchctl to load the agent:

launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.example.site-cron.plist

Note: gui/$(id -u) specifies the user domain.

1.3. Verify It’s Loaded

launchctl list | grep com.example.site-cron

Step 2: What Goes Wrong

A naive setup fails silently when things go wrong.

2.1. Example Failure

Consider a script that crashes due to a ModuleNotFoundError:

# site-cron.py
import nonexistent
print("Site cron running")

2.2. Silent Failure

The agent is “loaded” but not doing anything. You won’t see errors unless you check the logs.

2.3. Debugging

Use launchctl print to inspect the agent:

launchctl print gui/$(id -u)/com.example.site-cron

Redirected Logs:

Check site-cron.out and site-cron.err in /tmp/:

cat /tmp/site-cron.err

2.4. Reloading After Fixes

Even after fixing the script, the agent won’t restart unless you reload it:

launchctl unload gui/$(id -u)/com.example.site-cron
launchctl load ~/Library/LaunchAgents/com.example.site-cron.plist

Step 3: The Watchdog Pattern

To mitigate silent failures, implement a watchdog that monitors and restarts agents.

3.1. Create the Watchdog Script

Create site-cron-watchdog.sh:

#!/bin/bash

WATCHDOG_LABEL="com.example.site-cron-watchdog"
PRIMARY_LABEL="com.example.site-cron"
LOGFILE="/tmp/site-cron-watchdog.log"

function log {
    echo "$(date '+%Y-%m-%d %H:%M:%S') $1" >> "$LOGFILE"
}

function check_agent {
    if ! launchctl list | grep -q "$1"; then
        log "Agent $1 is missing. Reloading."
        launchctl load ~/Library/LaunchAgents/"$1".plist
    fi
}

log "Watchdog started."

while true; do
    check_agent "$PRIMARY_LABEL"
    sleep 600  # Check every 10 minutes
done

3.2. Make the Script Executable

chmod +x site-cron-watchdog.sh

3.3. Create the Watchdog Plist

Create com.example.site-cron-watchdog.plist:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" 
    "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.example.site-cron-watchdog</string>
    <key>ProgramArguments</key>
    <array>
        <string>/Users/username/scripts/site-cron-watchdog.sh</string>
    </array>
    <key>StartInterval</key>
    <integer>600</integer>
    <key>RunAtLoad</key>
    <true/>
    <key>StandardOutPath</key>
    <string>/tmp/site-cron-watchdog.out</string>
    <key>StandardErrorPath</key>
    <string>/tmp/site-cron-watchdog.err</string>
</dict>
</plist>

3.4. Load the Watchdog Agent

launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/com.example.site-cron-watchdog.plist

3.5. Explanation

• Watchdog Label: Unique identifier for the watchdog.
• Primary Label: Identifier of the agent to monitor.
• Logfile: Path to the watchdog log.
• check_agent Function: Checks if the primary agent is loaded; if not, reloads it.
• Loop: Runs every 10 minutes.

Step 4: Auto-Recovery on Reboot

Ensure your agents restart after a reboot.

4.1. Set RunAtLoad

Add RunAtLoad to every plist:

<key>RunAtLoad</key>
<true/>

4.2. Trade-Offs

• Pros: Agents restart after reboot.
• Cons: If a script depends on a service that’s not ready at boot, it may fail. However, the watchdog will attempt to reload it.

Step 5: The Watchdog-of-Watchdog

The watchdog itself can fail, so it needs to monitor itself.

5.1. Modify the Watchdog Script

Update site-cron-watchdog.sh to check its own label:

function check_self {
    if ! launchctl list | grep -q "$WATCHDOG_LABEL"; then
        log "Watchdog is missing. Reloading."
        launchctl load ~/Library/LaunchAgents/"$WATCHDOG_LABEL".plist
    fi
}

# Inside the loop
check_self
check_agent "$PRIMARY_LABEL"

5.2. Explanation

• check_self Function: Checks if the watchdog is loaded; if not, reloads it.

Step 6: Resilience Patterns

Ensure your scripts are idempotent to handle re-runs.

6.1. Idempotency Techniques

• File Locks: Use lock files to prevent concurrent executions.
• Atomic Renames: Use temporary files and atomic rename operations.
• SQLite UPSERT: Use database transactions to avoid duplicate entries.

6.2. Example: File Lock

import fcntl

lockfile = open('site-cron.lock', 'w')
fcntl.flock(lockfile, fcntl.LOCK_EX | fcntl.LOCK_NB)
try:
    # Your code here
finally:
    fcntl.flock(lockfile, fcntl.LOCK_UN)
    lockfile.close()

6.3. Explanation

• File Locks: Prevents multiple instances of the script from running simultaneously.
• Exception Handling: Ensures the lock is released even if the script crashes.

Step 7: What to Monitor

Implement monitoring to verify your pipeline’s health.

7.1. Log Tailing

Tail logs to catch issues early:

tail -F /tmp/site-cron.out /tmp/site-cron.err /tmp/site-cron-watchdog.out /tmp/site-cron-watchdog.err

7.2. Daily Verification

Set up a daily cron job to verify output:

0 0 * * * /Users/username/scripts/site-cron-verify.sh

7.3. Example Verification Script

Create site-cron-verify.sh:

#!/bin/bash

EXPECTED_OUTPUT="/path/to/expected/output"
ACTUAL_OUTPUT="/path/to/actual/output"

if diff "$EXPECTED_OUTPUT" "$ACTUAL_OUTPUT" >/dev/null; then
    echo "Verification succeeded."
else
    echo "Verification failed!" | telegram-send
fi

7.4. Explanation

• diff: Compares expected and actual output.
• telegram-send: Sends a message via Telegram if verification fails.

What’s Next

For a complete, working example, visit the source code on GitHub. The site-cron-watchdog.sh script there includes additional features like:

• Logging: Enhanced logging with timestamps and error codes.
• Configuration: External configuration files for easier management.
• Notifications: Email and Slack notifications for critical failures.

By implementing these strategies, you’ll create a self-healing content pipeline that minimizes downtime and keeps your data and content flowing smoothly.

This tutorial provides a solid foundation for building robust automation on macOS. Adapt the examples to fit your specific use case and expand upon the concepts to suit more complex workflows.

Summary Table

Additional Resources

• launchd documentation
• LaunchAgent Examples
• Python File Locking

By following this guide, you’ll be well on your way to creating a resilient, self-healing content pipeline that ensures your automation tasks run smoothly and reliably.