shell-scripting

---
name: shell-scripting
description: Write robust, portable shell scripts. Use when parsing arguments, handling errors properly, writing POSIX-compatible scripts, managing temp files, running commands in parallel, managing background processes, or adding --help to scripts.
metadata: {"clawdbot":{"emoji":"🐚","requires":{"bins":["bash"]},"os":["linux","darwin","win32"]}}
---

# Shell Scripting

Write reliable, maintainable bash scripts. Covers argument parsing, error handling, portability, temp files, parallel execution, process management, and self-documenting scripts.

## When to Use

- Writing scripts that others (or future you) will run
- Automating multi-step workflows
- Parsing command-line arguments with flags and options
- Handling errors and cleanup properly
- Running tasks in parallel
- Making scripts portable across Linux and macOS
- Wrapping complex commands with a simpler interface

## Script Template

```bash
#!/usr/bin/env bash
set -euo pipefail

# Description: What this script does (one line)
# Usage: script.sh [options] <required-arg>

readonly SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
readonly SCRIPT_NAME="$(basename "$0")"

# Defaults
VERBOSE=false
OUTPUT_DIR="./output"

usage() {
    cat <<EOF
Usage: $SCRIPT_NAME [options] <input-file>

Description:
  Process the input file and generate output.

Options:
  -o, --output DIR    Output directory (default: $OUTPUT_DIR)
  -v, --verbose       Enable verbose output
  -h, --help          Show this help message

Examples:
  $SCRIPT_NAME data.csv
  $SCRIPT_NAME -v -o /tmp/results data.csv
EOF
}

log() { echo "[$(date '+%H:%M:%S')] $*" >&2; }
debug() { $VERBOSE && log "DEBUG: $*" || true; }
die() { log "ERROR: $*"; exit 1; }

# Parse arguments
while [[ $# -gt 0 ]]; do
    case "$1" in
        -o|--output) OUTPUT_DIR="$2"; shift 2 ;;
        -v|--verbose) VERBOSE=true; shift ;;
        -h|--help) usage; exit 0 ;;
        --) shift; break ;;
        -*) die "Unknown option: $1" ;;
        *) break ;;
    esac
done

INPUT_FILE="${1:?$(usage >&2; echo "Error: input file required")}"
[[ -f "$INPUT_FILE" ]] || die "File not found: $INPUT_FILE"

# Main logic
main() {
    debug "Input: $INPUT_FILE"
    debug "Output: $OUTPUT_DIR"
    mkdir -p "$OUTPUT_DIR"

    log "Processing $INPUT_FILE..."
    # ... do work ...
    log "Done. Output in $OUTPUT_DIR"
}

main "$@"
```

## Error Handling

### set flags

```bash
set -e          # Exit on any command failure
set -u          # Error on undefined variables
set -o pipefail # Pipe fails if any command in the pipe fails
set -x          # Debug: print each command before executing (noisy)

# Combined (use this in every script)
set -euo pipefail

# Temporarily disable for commands that are allowed to fail
set +e
some_command_that_might_fail
exit_code=$?
set -e
```

### Trap for cleanup

```bash
# Cleanup on exit (any exit: success, failure, or signal)
TMPDIR=""
cleanup() {
    [[ -n "$TMPDIR" ]] && rm -rf "$TMPDIR"
}
trap cleanup EXIT

TMPDIR=$(mktemp -d)
# Use $TMPDIR freely — it's cleaned up automatically

# Trap specific signals
trap 'echo "Interrupted"; exit 130' INT    # Ctrl+C
trap 'echo "Terminated"; exit 143' TERM    # kill
```

### Error handling patterns

```bash
# Check command exists before using it
command -v jq >/dev/null 2>&1 || die "jq is required but not installed"

# Provide default values
NAME="${NAME:-default_value}"

# Required variable (fail if unset)
: "${API_KEY:?Error: API_KEY environment variable is required}"

# Retry a command
retry() {
    local max_attempts=$1
    shift
    local attempt=1
    while [[ $attempt -le $max_attempts ]]; do
        "$@" && return 0
        log "Attempt $attempt/$max_attempts failed. Retrying..."
        ((attempt++))
        sleep $((attempt * 2))
    done
    die "Command failed after $max_attempts attempts: $*"
}

retry 3 curl -sf https://api.example.com/health
```

## Argument Parsing

### Simple: positional + flags

```bash
# Manual parsing (no dependencies)
FORCE=false
DRY_RUN=false

while [[ $# -gt 0 ]]; do
    case "$1" in
        -f|--force) FORCE=true; shift ;;
        -n|--dry-run) DRY_RUN=true; shift ;;
        -o|--output)
            [[ -n "${2:-}" ]] || die "--output requires a value"
            OUTPUT="$2"; shift 2 ;;
        --output=*)
            OUTPUT="${1#*=}"; shift ;;
        -h|--help) usage; exit 0 ;;
        --) shift; break ;;  # End of options
        -*) die "Unknown option: $1" ;;
        *) break ;;  # Start of positional args
    esac
done

# Remaining args are positional
FILES=("$@")
[[ ${#FILES[@]} -gt 0 ]] || die "At least one file is required"
```

### getopts (POSIX, short options only)

```bash
while getopts ":o:vhf" opt; do
    case "$opt" in
        o) OUTPUT="$OPTARG" ;;
        v) VERBOSE=true ;;
        f) FORCE=true ;;
        h) usage; exit 0 ;;
        :) die "Option -$OPTARG requires an argument" ;;
        ?) die "Unknown option: -$OPTARG" ;;
    esac
done
shift $((OPTIND - 1))
```

## Temp Files and Directories

```bash
# Create temp file (automatically unique)
TMPFILE=$(mktemp)
echo "data" > "$TMPFILE"

# Create temp directory
TMPDIR=$(mktemp -d)

# Create temp with custom prefix/suffix
TMPFILE=$(mktemp /tmp/myapp.XXXXXX)
TMPFILE=$(mktemp --suffix=.json)  # GNU only

# Always clean up with trap
trap 'rm -f "$TMPFILE"' EXIT

# Portable pattern (works on macOS and Linux)
TMPDIR=$(mktemp -d 2>/dev/null || mktemp -d -t 'myapp')
trap 'rm -rf "$TMPDIR"' EXIT
```

## Parallel Execution

### xargs -P

```bash
# Run 4 commands in parallel
cat urls.txt | xargs -P 4 -I {} curl -sO {}

# Process files in parallel (4 at a time)
find . -name "*.csv" | xargs -P 4 -I {} ./process.sh {}

# Parallel with progress indicator
find . -name "*.jpg" | xargs -P 8 -I {} sh -c 'convert {} -resize 800x600 resized/{} && echo "Done: {}"'
```

### Background jobs + wait

```bash
# Run tasks in background, wait for all
pids=()
for file in data/*.csv; do
    process_file "$file" &
    pids+=($!)
done

# Wait for all and check results
failed=0
for pid in "${pids[@]}"; do
    wait "$pid" || ((failed++))
done
[[ $failed -eq 0 ]] || die "$failed jobs failed"
```

### GNU Parallel (if available)

```bash
# Process files with 8 parallel jobs
parallel -j 8 ./process.sh {} ::: data/*.csv

# With progress bar
parallel --bar -j 4 convert {} -resize 800x600 resized/{/} ::: *.jpg

# Pipe input lines
cat urls.txt | parallel -j 10 curl -sO {}
```

## Process Management

### Background processes

```bash
# Start in background
long_running_command &
BG_PID=$!

# Check if still running
kill -0 $BG_PID 2>/dev/null && echo "Running" || echo "Stopped"

# Wait for it
wait $BG_PID
echo "Exit code: $?"

# Kill on script exit
trap 'kill $BG_PID 2>/dev/null' EXIT
```

### Process supervision

```bash
# Run a command, restart if it dies
run_with_restart() {
    local cmd=("$@")
    while true; do
        "${cmd[@]}" &
        local pid=$!
        log "Started PID $pid"
        wait $pid
        local exit_code=$?
        log "Process exited with code $exit_code. Restarting in 5s..."
        sleep 5
    done
}

run_with_restart ./my-server --port 8080
```

### Timeout

```bash
# Kill command after 30 seconds
timeout 30 long_running_command

# With custom signal (SIGKILL after SIGTERM fails)
timeout --signal=TERM --kill-after=10 30 long_running_command

# Portable (no timeout command)
( sleep 30; kill $$ 2>/dev/null ) &
TIMER_PID=$!
long_running_command
kill $TIMER_PID 2>/dev/null
```

## Portability (Linux vs macOS)

### Common differences

```bash
# sed: macOS requires -i '' (empty backup extension)
# Linux:
sed -i 's/old/new/g' file.txt
# macOS:
sed -i '' 's/old/new/g' file.txt
# Portable:
sed -i.bak 's/old/new/g' file.txt && rm file.txt.bak

# date: different flags
# GNU (Linux):
date -d '2026-02-03' '+%s'
# BSD (macOS):
date -j -f '%Y-%m-%d' '2026-02-03' '+%s'

# readlink -f: doesn't exist on macOS
# Portable alternative:
real_path() { cd "$(dirname "$1")" && echo "$(pwd)/$(basename "$1")"; }

# stat: different syntax
# GNU: stat -c '%s' file
# BSD: stat -f '%z' file

# grep -P: not available on macOS by default
# Use grep -E instead, or install GNU grep
```

### POSIX-safe patterns

```bash
# Use printf instead of echo -e (echo behavior varies)
printf "Line 1\nLine 2\n"

# Use $() instead of backticks
result=$(command)   # Good
result=`command`    # Bad (deprecated, nesting issues)

# Use [[ ]] for tests (bash), [ ] for POSIX sh
[[ -f "$file" ]]   # Bash (safer, no word splitting)
[ -f "$file" ]     # POSIX sh

# Array check (bash only, not POSIX)
if [[ ${#array[@]} -gt 0 ]]; then
    echo "Array has elements"
fi
```

## Config File Parsing

### Source a config file

```bash
# Simple: source a key=value file
# config.env:
# DB_HOST=localhost
# DB_PORT=5432

# Validate before sourcing (security: check for commands)
if grep -qP '^[A-Z_]+=.*[;\`\$\(]' config.env; then
    die "Config file contains unsafe characters"
fi
source config.env
```

### Parse INI-style config

```bash
# config.ini:
# [database]
# host = localhost
# port = 5432
# [app]
# debug = true

parse_ini() {
    local file="$1" section=""
    while IFS='= ' read -r key value; do
        [[ -z "$key" || "$key" =~ ^[#\;] ]] && continue
        if [[ "$key" =~ ^\[(.+)\]$ ]]; then
            section="${BASH_REMATCH[1]}"
            continue
        fi
        value="${value%%#*}"     # Strip inline comments
        value="${value%"${value##*[![:space:]]}"}"  # Trim trailing whitespace
        printf -v "${section}_${key}" '%s' "$value"
    done < "$file"
}

parse_ini config.ini
echo "$database_host"  # localhost
echo "$app_debug"      # true
```

## Useful Patterns

### Confirm before destructive action

```bash
confirm() {
    local prompt="${1:-Are you sure?}"
    read -rp "$prompt [y/N] " response
    [[ "$response" =~ ^[Yy]$ ]]
}

confirm "Delete all files in /tmp/data?" || die "Aborted"
rm -rf /tmp/data/*
```

### Progress indicator

```bash
# Simple counter
total=$(wc -l < file_list.txt)
count=0
while IFS= read -r file; do
    ((count++))
    printf "\rProcessing %d/%d..." "$count" "$total" >&2
    process "$file"
done < file_list.txt
echo "" >&2
```

### Lock file (prevent concurrent runs)

```bash
LOCKFILE="/tmp/${SCRIPT_NAME}.lock"

acquire_lock() {
    if ! mkdir "$LOCKFILE" 2>/dev/null; then
        die "Another instance is running (lock: $LOCKFILE)"
    fi
    trap 'rm -rf "$LOCKFILE"' EXIT
}

acquire_lock
# ... safe to proceed, only one instance runs ...
```

### Stdin or file argument

```bash
# Read from file argument or stdin
input="${1:--}"   # Default to "-" (stdin)
if [[ "$input" == "-" ]]; then
    cat
else
    cat "$input"
fi | while IFS= read -r line; do
    process "$line"
done
```

## Tips

- Always start with `set -euo pipefail`. It catches 80% of silent bugs.
- Always use `trap cleanup EXIT` for temp files. Never rely on reaching the cleanup code at the end.
- Quote all variable expansions: `"$var"` not `$var`. Unquoted variables break on spaces and globs.
- Use `[[ ]]` instead of `[ ]` in bash. It handles empty strings, spaces, and pattern matching better.
- `shellcheck` is the best linter for shell scripts. Run it: `shellcheck myscript.sh`. Install it if available.
- `readonly` for constants prevents accidental overwrite: `readonly DB_HOST="localhost"`.
- Write a `usage()` function and call it on `-h`/`--help` and on missing required arguments. Future users (including you) will thank you.
- Prefer `printf` over `echo` for anything that might contain special characters or needs formatting.
- Test scripts with `bash -n script.sh` (syntax check) before running.