You are the **lead background-processing engineer and reliability implementation owner**.

Perform a comprehensive robustness pass on this website’s cron jobs, workers, scheduled tasks, batch processing, imports, and asynchronous jobs.

The project is typically built with:

- PHP;
- MySQL;
- shell/cron scheduling;
- existing project utilities.

Do not introduce frameworks, queue platforms, or unnecessary third-party libraries.

Primary question:

> **What can cause background work to run twice, overlap incorrectly, get stuck, silently disappear, partially complete, process stale data, retry unsafely, or stop making progress?**

Then fix those problems.

Use:

> **map jobs → trace lifecycle → attack failure paths → implement → validate → stress again**

This is an implementation task, not merely an audit.

## Scope

Inspect only systems that exist:

- cron scripts;
- long-running workers;
- scheduled jobs;
- queues;
- batch processing;
- imports/exports;
- notification jobs;
- email jobs;
- API polling;
- synchronization;
- cleanup tasks;
- aggregation;
- maintenance tasks;
- retries;
- locking;
- cursors/checkpoints;
- job status;
- worker heartbeats;
- database-backed work claiming.

Do not redesign the application, migrate to a queue framework, or build distributed infrastructure unless a confirmed severe defect requires it.

## Core Invariants

Protect invariants such as:

- one exclusive job is not processed twice unintentionally;
- repeated execution is safe where possible;
- overlapping cron runs do not corrupt data;
- failures do not masquerade as success;
- partial batches can resume safely;
- stale workers do not permanently lock work;
- retries do not duplicate side effects;
- one bad item does not unnecessarily kill all progress;
- completed work is not silently reprocessed forever;
- failed work remains diagnosable.

## Priority

**P0:** duplication, corruption, destructive repeated work, runaway workers, lost jobs.

**P1:** overlapping execution, stale locks, retry duplication, jobs permanently stuck, silent failures.

**P2:** weak batching, poor recovery, inefficient claiming, incomplete logging, fragile cursors.

**P3:** cleanup, maintainability, minor operational polish.

Fix P0/P1 first.

## Establish the Job Model

Map each job:

- trigger;
- frequency;
- entry point;
- work source;
- claim mechanism;
- batch size;
- success condition;
- retry behavior;
- failure state;
- progress/checkpoint;
- locking;
- timeout;
- completion.

Trace:

> **schedule → start → discover work → claim → process → persist result → mark complete → next work**

Identify where that lifecycle can break.

## Overlap Attack

Assume a cron job starts again before the previous run finishes.

Ask:

> **What happens if two copies run simultaneously?**

Inspect:

- duplicate processing;
- conflicting updates;
- repeated notifications;
- repeated API calls;
- competing cleanup;
- stale cursor writes.

Use appropriate protection such as:

- `flock`;
- database locks;
- atomic claiming;
- unique job ownership.

Do not add locks where concurrent execution is intentionally safe.

## Work Claiming

For shared workers, claiming must be race-safe.

Dangerous pattern:

> select unprocessed row → later mark processing

Two workers may select the same row.

Prefer an atomic claim or transaction compatible with the existing architecture.

Verify:

- only intended workers can claim;
- expired claims can recover;
- successful work cannot be accidentally reclaimed.

## Stale Locks / Leases

If locks or leases exist, inspect:

- expiry;
- worker crash;
- server reboot;
- timeout;
- cleanup.

A crashed worker must not permanently strand work.

But do not expire legitimate long-running work so aggressively that another worker duplicates it.

## Retry Safety

Ask:

> **What happens if this job executes twice?**

Especially inspect:

- email;
- notifications;
- payments;
- rewards;
- imports;
- external API mutations;
- file generation.

Differentiate:

- naturally idempotent state replacement;
- side-effecting operations.

Use appropriate idempotency, unique records, status checks, or transaction boundaries.

Do not blindly retry side effects.

## Partial Failure

For batches ask:

> **What happens if item 37 of 100 fails?**

Avoid designs where:

- first failure loses remaining work;
- entire successful batch is repeated;
- progress advances past failed work silently;
- partial state is marked complete.

Handle failures according to consequence.

One malformed record should not necessarily halt unrelated processing.

## Cursor / Checkpoint Safety

If jobs use:

- IDs;
- timestamps;
- offsets;
- cursors;
- last-processed values;

inspect whether progress can:

- skip records;
- repeat records;
- move backward;
- advance before durable success.

Update checkpoints only when their meaning is actually satisfied.

Do not assume IDs are perfectly contiguous.

## Batch Processing

Inspect batch sizes for:

- memory growth;
- execution timeout;
- huge transactions;
- long locks;
- excessive API calls.

Prefer bounded batches with reliable progress.

Do not solve every problem by shrinking batches to one item if that destroys throughput.

## Long-Running Workers

If workers stay alive, inspect:

- memory growth;
- stale DB connections;
- stale configuration/state;
- exception recovery;
- idle behavior;
- graceful shutdown;
- reconnect behavior.

A single failed job should not silently kill the worker unless that is explicitly intended.

## Cron Environment

Cron environments often differ from interactive shells.

Inspect:

- PHP binary/version;
- working directory;
- relative paths;
- environment variables;
- PATH;
- permissions;
- user account;
- required extensions.

Use reliable absolute paths where appropriate.

Do not assume a script that works manually will work from cron.

## Timeouts

Inspect:

- PHP execution limits;
- HTTP timeouts;
- database timeouts;
- external API timeouts;
- worker time budgets.

External requests should not hang indefinitely.

Timeout does not always mean the external action failed, so retry side effects carefully.

## External API Jobs

For polling/sync jobs inspect:

- rate limits;
- pagination;
- cursors;
- transient failure;
- partial response;
- retries;
- duplicate data;
- backoff.

Do not hammer external services during failure loops.

Preserve existing API contracts.

## Database Transactions

For multi-step job mutations ask:

> **Which changes must commit together?**

Use transactions where partial completion would corrupt state.

Avoid holding transactions open during slow external network calls unless architecture clearly requires it.

## Status State Machines

If jobs use states such as:

- pending;
- processing;
- completed;
- failed;

verify transitions are coherent.

Look for:

- completed → processing unexpectedly;
- processing forever;
- failure marked complete;
- duplicate completion;
- invalid reset behavior.

Keep one clear source of truth for job state.

## Logging & Diagnostics

Failures should be diagnosable.

Log useful context such as:

- job;
- record ID;
- stage;
- error class/message;
- retry decision.

Avoid logging:

- passwords;
- tokens;
- secrets;
- massive payloads.

Do not flood logs with routine successful iterations unless operationally useful.

## Silent Failure Hunt

Search for:

- swallowed exceptions;
- ignored return values;
- empty catches;
- failed SQL treated as success;
- command exit codes ignored;
- HTTP failures ignored.

Make failures visible enough to diagnose.

## Notifications / Email

If background jobs send messages, verify:

- correct recipient;
- duplicate prevention;
- retry semantics;
- failure recording;
- stale notifications;
- successful send marking.

Do not mark sent before the send operation has reached the project’s intended success point.

## Cleanup Jobs

Inspect deletion/cleanup for:

- wrong age calculation;
- overly broad queries;
- missing ownership/type filters;
- repeated deletion;
- deleting active data;
- huge unbounded operations.

Treat destructive jobs carefully.

## Pattern Hunt

When one problem appears, inspect sibling jobs.

Examples:

- one unsafe claim → inspect other queues;
- one retry duplicates email → inspect other notifications;
- one stale lock never expires → inspect other leases;
- one cursor advances too early → inspect other incremental jobs.

Fix repeated patterns where justified.

## Implementation Standard

For each meaningful issue:

1. identify the job;
2. define the intended invariant;
3. reproduce or prove the failure;
4. identify root cause;
5. implement the smallest robust fix;
6. test normal execution;
7. test overlap/retry/failure where relevant;
8. test restart/recovery where relevant;
9. inspect sibling jobs;
10. add focused regression coverage where practical.

## Productive Loop

After each wave ask:

- What can still run twice?
- What can get stuck forever?
- What can silently disappear?
- Which retry can duplicate a side effect?
- Which cursor can skip work?
- Which lock can become stale?
- Which failure stops unrelated progress?
- Which worker can die without anyone knowing?

Continue while material, safely actionable reliability issues remain.

## Anti-Overengineering

Do not introduce:

- Redis;
- RabbitMQ;
- Kafka;
- distributed schedulers;
- generic queue frameworks;
- complicated lease systems;

unless the actual application already needs them.

Prefer understandable PHP, MySQL, cron, `flock`, transactions, and atomic state changes where sufficient.

## Completion

Finish when:

- important jobs have clear lifecycle;
- dangerous overlap is controlled;
- shared work is claimed safely;
- stale locks can recover;
- consequential retries resist duplication;
- partial failures do not silently lose work;
- cursors/checkpoints are safe;
- worker failures are diagnosable;
- no discovered P0/P1 background-processing issue remains without a blocker.

## Final Report

Provide:

1. **Overall result**
2. **Critical worker/cron fixes**
3. **Overlap and locking**
4. **Claiming and concurrency**
5. **Retries/idempotency**
6. **Batching/cursors/recovery**
7. **External API/background integrations**
8. **Logging/failure handling**
9. **Exact validation performed**
10. **Deliberately unchanged behavior**
11. **Remaining risks with severity and next action**

# Final Directive

Treat every background process as something that may:

- overlap;
- crash;
- restart;
- time out;
- execute twice;
- stop halfway.

Repeatedly ask:

> **What happens if another copy starts now?**

> **What happens if this dies after the side effect but before completion is recorded?**

> **Can this work be claimed twice?**

> **Can a stale lock block progress forever?**

> **Can retry duplicate the result?**

Leave behind the **same website**, but make its cron jobs, workers, and background processing substantially safer, more recoverable, concurrency-safe, and operationally reliable.