feat(worker): isolate run_command children behind a uid boundary #348

Merged

rwachters merged 6 commits from feat/worker-command-uid-isolation into master

2026-10-20 10:43:49 +00:00

rwachters commented

2026-10-20 10:41:37 +00:00

(Migrated from github.com)

Summary

Built-in tool commands previously ran as the same OS account as the worker, so an agent-chosen command could read secrets.json, the service-token cache and the worker's /proc entries, and could signal worker processes. This change runs commands spawned by run_command as a separate unprivileged agent account, so ordinary file permissions and /proc isolation protect the worker identity, without adding capabilities, seccomp changes, namespaces or mounts to the container. Isolation is opt-in (worker.commandIsolation.enabled = false is the shipped default and logs a WARN; when enabled the worker refuses to start if the privilege drop cannot be provided), and the shipped worker image installs the account and the helper.

Changes

Added the worker.commandIsolation configuration group (enabled, user, helperPath, nested child environment) with defaults, validation, documentation in every shipped config layer, and preservation across the setup and trusted-signer config rewrites.
Added the isolation runtime: launcher and resolution, a startup detector with a helper probe, an allowlist-based child environment provider, the process-tree terminator, and a helper signal relay.
Adopted it in run_command, which now spawns through the resolved launcher with an explicitly built environment, and resolves isolation once at startup; the tool's in-process tree teardown was replaced by the terminator so a timed-out or cancelled command has its process group signalled through the helper.
Deployment: added the run-as-agent setuid helper (exec mode and signal mode, compiled in a builder stage), created the agent/shared accounts, tightened the secret-bearing directories, made the workspace and caches group-writable with setgid, and normalized accounts and permissions on every container start.
Enabled the switch in both compose examples with init: true, and documented the accounts, permissions, log lines, termination behaviour and upgrade steps.
Hardened the run-command test timings (named constants; pipe-drain command timeout 60s with a strictly larger outer test deadline) so the suite stays stable under a loaded parallel runner.

Implementation notes

CommandIsolationResolution is the single place that knows whether the privilege drop is active, which account a child runs as and how its tree is terminated; the launcher, environment provider and terminator all consume it, so the inconsistent states the earlier draft admitted are unrepresentable.
Children receive an explicitly assigned environment rather than the inherited one: while the drop is active they get the allowlist-derived map, otherwise the worker's own environment, so no behaviour changes on a host without the mechanism.
The helper is the only privileged component; it verifies its own credential drop before acting, takes no UID from argv, and in signal mode accepts only TERM/KILL for a positive process-group id, treating a non-existent group as success.
Git refuses the worker-owned workspace once commands run as another account; the default child environment therefore carries the safe.directory exemption for isolated commands only, so a git the worker itself runs keeps refusing such repositories.

Validation

./gradlew :worker:classes :worker:testClasses :worker:test

62 suites / 567 tests / 0 failures.

Not verified in this environment (no gcc, no docker): the helper compile, the image build and the container steps. The recorded Docker-host checks are the isolation-active startup log line, the drop of the helper to agent, and the unreadability of secrets.json and /proc/<jvm>/environ from an isolated command.

Deployment owners must rebuild the worker image before enabling the switch, since the accounts and the helper only exist in the new image.

Risks and follow-up

enabled = false is the shipped default, so an installation that does not set the switch keeps running commands as the worker account; the worker logs a WARN naming that consequence.
The entrypoint runs as root on every start and re-applies ownership and modes to /app/config, /app/data, /app/logs, /app/workspace and /app/cache, which are commonly bind-mounted; a deployment that intentionally leaves different modes on those host directories will see them reset.
A custom worker.workspace.path, an agent account whose home is not /home/<user>, and environment allowlist entries that are not valid variable names are known gaps recorded as follow-ups rather than part of this change.

## Summary Built-in tool commands previously ran as the same OS account as the worker, so an agent-chosen command could read `secrets.json`, the service-token cache and the worker's `/proc` entries, and could signal worker processes. This change runs commands spawned by `run_command` as a separate unprivileged `agent` account, so ordinary file permissions and `/proc` isolation protect the worker identity, without adding capabilities, seccomp changes, namespaces or mounts to the container. Isolation is opt-in (`worker.commandIsolation.enabled = false` is the shipped default and logs a WARN; when enabled the worker refuses to start if the privilege drop cannot be provided), and the shipped worker image installs the account and the helper. ## Changes - Added the `worker.commandIsolation` configuration group (enabled, user, helperPath, nested child environment) with defaults, validation, documentation in every shipped config layer, and preservation across the setup and trusted-signer config rewrites. - Added the isolation runtime: launcher and resolution, a startup detector with a helper probe, an allowlist-based child environment provider, the process-tree terminator, and a helper signal relay. - Adopted it in `run_command`, which now spawns through the resolved launcher with an explicitly built environment, and resolves isolation once at startup; the tool's in-process tree teardown was replaced by the terminator so a timed-out or cancelled command has its process group signalled through the helper. - Deployment: added the `run-as-agent` setuid helper (exec mode and signal mode, compiled in a builder stage), created the `agent`/`shared` accounts, tightened the secret-bearing directories, made the workspace and caches group-writable with setgid, and normalized accounts and permissions on every container start. - Enabled the switch in both compose examples with `init: true`, and documented the accounts, permissions, log lines, termination behaviour and upgrade steps. - Hardened the run-command test timings (named constants; pipe-drain command timeout 60s with a strictly larger outer test deadline) so the suite stays stable under a loaded parallel runner. ## Implementation notes - `CommandIsolationResolution` is the single place that knows whether the privilege drop is active, which account a child runs as and how its tree is terminated; the launcher, environment provider and terminator all consume it, so the inconsistent states the earlier draft admitted are unrepresentable. - Children receive an explicitly assigned environment rather than the inherited one: while the drop is active they get the allowlist-derived map, otherwise the worker's own environment, so no behaviour changes on a host without the mechanism. - The helper is the only privileged component; it verifies its own credential drop before acting, takes no UID from argv, and in signal mode accepts only `TERM`/`KILL` for a positive process-group id, treating a non-existent group as success. - Git refuses the worker-owned workspace once commands run as another account; the default child environment therefore carries the `safe.directory` exemption for isolated commands only, so a git the worker itself runs keeps refusing such repositories. ## Validation ``` ./gradlew :worker:classes :worker:testClasses :worker:test ``` 62 suites / 567 tests / 0 failures. Not verified in this environment (no `gcc`, no `docker`): the helper compile, the image build and the container steps. The recorded Docker-host checks are the isolation-active startup log line, the drop of the helper to `agent`, and the unreadability of `secrets.json` and `/proc/<jvm>/environ` from an isolated command. Deployment owners must rebuild the worker image before enabling the switch, since the accounts and the helper only exist in the new image. ## Risks and follow-up - `enabled = false` is the shipped default, so an installation that does not set the switch keeps running commands as the worker account; the worker logs a WARN naming that consequence. - The entrypoint runs as root on every start and re-applies ownership and modes to `/app/config`, `/app/data`, `/app/logs`, `/app/workspace` and `/app/cache`, which are commonly bind-mounted; a deployment that intentionally leaves different modes on those host directories will see them reset. - A custom `worker.workspace.path`, an `agent` account whose home is not `/home/<user>`, and environment allowlist entries that are not valid variable names are known gaps recorded as follow-ups rather than part of this change.