How daemon-managed Agent tasks are scheduled, timed out, reported, retried, and recovered.

Daemon Task Execution

This page describes the lifecycle of one Agent task after a daemon receives a dispatch notification. For daemon installation, runtime registration, and CLI lifecycle commands, see What Is the Daemon.

Lifecycle and Reporting

The daemon stops at every meaningful state boundary before proceeding:

A poller reserves a daemon concurrency slot and claims a queued task for its runtime.
The scheduler prepares the task work directory and any required repository state.
The daemon marks the claimed task as running, then starts one Provider CLI process.
Provider output, tool activity, usage checkpoints, and the terminal result are persisted locally before being reported to the server.
The daemon retries undelivered records after reconnecting, at startup, and before subsequent claims. It releases the local slot only after task settlement has finished.

WebSocket notifications only wake the scheduler quickly. They are not the source of truth: a reconnecting daemon reloads pending task interactions and replays durable event records. A temporary server error therefore does not by itself turn a locally completed task into a generic failure.

Concurrency

Two limits apply together. A task needs capacity at both levels before it can run.

Level	Setting	Default	Scope
Agent	`maxConcurrentAgentTasks`	6	One Agent
Daemon	`MOPHEUS_DAEMON_MAX_CONCURRENT_AGENT_TASKS`	20	All runtimes handled by one daemon

The server prevents an Agent from exceeding its own limit. The daemon also uses one global slot pool across all its runtimes, so a busy Provider or workspace cannot exceed the host-level limit. When either limit is full, tasks remain queued and are claimed later.

Scheduling and comment coalescing

The daemon poller only reserves a concurrency slot and calls the task scheduler for one runtime. The scheduler then claims the task, prepares its local working directory and repositories, starts it on the server, runs one provider process for that task, and releases local resources when it ends.

Provider output, tool records, usage checkpoints, and the final completion or failure intent are written to a durable local queue before they are sent to the server. If the server is temporarily unavailable, the daemon retries those records after reconnecting, on startup, and before later claims. A task that was already completed locally is not changed into a generic failed task merely because the final HTTP request failed.

Messages sent to a running task are also durable. WebSocket notifications wake the scheduler quickly, but the scheduler reloads undelivered messages from the server after reconnecting or retrying. Completion waits until those messages have been delivered or withdrawn.

Before a task starts running, the server coalesces compatible comment triggers for the same agent and ticket. Multiple top-level comments with no replies become one pending task. A member reply in a thread with an active task is dispatched even before the agent replies: if the active task is running, that reply queues a successor, and later replies merge into the pending successor. Once any root in the shared pool receives a reply, the pool is bound to that thread and later top-level comments do not join it. The merged task always keeps the highest priority. After a task starts, the daemon runs from its fixed comment snapshot; later comments queue a successor, and tasks for the same agent and ticket are not claimed concurrently.

Timeouts

timeoutSeconds on an Agent is the shared task deadline when it is set. The daemon enforces it during execution and the server sweeper independently detects tasks that remain active beyond the same deadline.

Condition	Daemon behavior	Server behavior
Agent `timeoutSeconds` is set	Uses the task value as the execution deadline.	Stores the same value and sweeps against it.
Agent `timeoutSeconds` is unset	Uses `MOPHEUS_AGENT_TIMEOUT`; `0` disables the daemon deadline.	Uses `agent_task_timeout`, defaulting to 24 hours.

The server also detects tasks stuck in intermediate states: queued tasks have a queue TTL, and dispatched tasks must start within the dispatch timeout. MOPHEUS_AGENT_IDLE_WATCHDOG is separate from the total deadline: it stops a Provider that produces no output for the configured interval. Set it to 0 to disable it.

The timeout model has two parallel resolution paths. The daemon path falls back from the agent value to MOPHEUS_AGENT_TIMEOUT; the server path falls back from the same agent value to the system agent_task_timeout setting. When the agent value is set, both paths use it; when it is unset, the two fallback settings are independent. The server sweep runs about once per minute and catches tasks the daemon failed to clean up. Queued and dispatched tasks use their separate 2h and 5min stale windows.

Failure and Retry

Retries consume the task retry budget and happen only while attempt < max_attempts. Only timeout and restart or runtime-recovery failures are automatically retried.

Failure reason	Meaning	Automatic retry
`run_timeout`	The task execution deadline elapsed.	Yes
`dispatch_timeout`	A claimed task did not enter running state in time.	Yes
`daemon_shutdown`	The daemon was force-stopped or restarted while processing the task.	Yes
`runtime_offline`, `runtime_recovery`	The runtime became unavailable and the server recovered its work.	Yes
`task_cancel`	A user or server cancellation request.	No
`idle_watchdog`, `guard_tripped`	Idle or resource protection stopped the task.	No
`queue_ttl_exceeded`	The task waited in the queue too long.	No
`setup`, `execute`, `failed`	Preparation, Provider, or ordinary execution failure.	No

The daemon records the task ID, error, failure reason, and final status when it reports a failure. These fields are the primary starting point when investigating a task that did not retry.

Stop and Restart

mopheus daemon stop and mopheus daemon restart first stop claiming new work, then allow in-flight tasks to finish. During this drain, the daemon continues reporting durable events, usage, and the terminal state.

With --force, remaining executions are cancelled and settled as daemon_shutdown. This is retryable, so the server requeues the task when retry attempts remain. After a daemon restart, the server also recovers work left in dispatched or running; providers that support session reuse can continue with the previous session and work directory.

If a claim becomes stale before the daemon starts it, the start request is rejected as a state conflict rather than reported as a new Provider failure. The requeued or already-settled task is then handled by its normal lifecycle.

While waiting for tasks to finish, mopheus daemon status and the local health endpoint report the remaining active-task count. The drain buffer is only an internal grace period for final state reporting. When timeoutSeconds is set, the daemon execution timeout and server sweep are two detection paths for the same deadline; when it is unset, the daemon uses MOPHEUS_AGENT_TIMEOUT and the server uses agent_task_timeout independently.

What Is the Daemon for installation, runtime registration, and daemon commands.
Runtime Resource Guard for resource and token protection.
Work Directories and Cleanup for task filesystem lifecycle.