Skip to the content.

Home · Agents · Reference · Design · Security · Sandbox

Design

How qex operates

qex is one program with three roles.

The CLI reads your command. It calculates the full job specification at this moment, because the coordinator can start many hours before from a different shell. If no coordinator operates, the CLI starts one. A lock file lets one CLI process only start the coordinator, so twenty commands at the same time give one coordinator.

The coordinator holds the queue and starts each job when the machine has capacity. It stops one hour after the last job and the last command, so it uses no memory between your tasks.

A supervisor controls one job. The coordinator gives the supervisor a session. The supervisor gives the job a process group. The supervisor writes the two output files and records the result.

The supervisor is a separate process for one reason: the coordinator can stop, fail or restart, and the job must continue and must still record its result. A kill -9 on the coordinator loses no job and no exit code.

Files

~/.config/qex.toml                  the config file
~/.local/state/qex/jobs/<uuid>/
    spec.json      the command, the environment and the claims (mode 0600)
    status.json    the state, the exit code, the times and the true use
    stdout.log
    stderr.log
    hook.ran       the record that the stop hook of this job ran (mode 0600)
    hook.log       the output of that hook (mode 0600)

hook.ran stops a second run of the hook for one job. Several processes can make a job terminal: the supervisor writes the usual result, and the coordinator writes cancelled, skipped and the failure of a job whose supervisor stopped. Each of them makes this file with create_new before it starts the hook, and that operation succeeds for one process only. A person thus receives one message, also when the coordinator stops and starts again while the job runs. qex makes this file BEFORE it runs the hook, so a process that stops between those two steps loses that message. The other order would give a second message in the same condition, and a message that arrives two times is worse.

status.json is the primary record. It records the command, the directory, the state, the exit code, the times and the measured use. The supervisor writes it in one operation, so a reader sees the old contents or the new contents, and never a part of them. qex wait reads this file directly when no coordinator operates.

A command pays for what it asks for

The coordinator holds the full record of each job that waits or operates. A job that stopped leaves that map one turn of the scheduler after its record on the disk says so, and the coordinator then keeps a short entry for it: the line of qex list, and what a filter, a dependency and a dedupe key read. A request for the full record of such a job reads status.json. The cost of a turn of the scheduler thus follows the queue, and not the history.

The coordinator keeps an index of every job by id, by name and by pipeline. A command that names a job sends the text that the person wrote, and the coordinator reads it through the index with one set of rules. qex cancel and qex kill send every id in one request. qex list sends its options, and the coordinator gives the short rows of the jobs that they select. The lock of the coordinator covers a copy of one pointer for each job of an answer; the text of the answer comes after the lock is free.

Each of these requests is a capability (query, resolve, stop-many). A command that meets a coordinator of an earlier version sends the request for the whole list in their place. The answer is the same, and only slower.

After the machine restarts

The state directory is ~/.local/state/qex, or $XDG_STATE_HOME/qex. It is not a temporary file system, so every record stays through a restart of the machine. The next coordinator reads each record and corrects it:

Only a coordinator makes these changes, and no coordinator starts by itself after a restart. Until one starts, status.json still says running for a job that no process runs. qex wait and qex top start no coordinator and read that file, so they show that state, and qex wait continues to wait. Run qex list when you come back after a restart: it starts a coordinator, the coordinator corrects every record, and the queue operates again. `qex status

` and `qex info` do the same. ### qex removes no record by itself The coordinator has no timer that deletes a record. The directory of every job stays until somebody runs `qex clean` or `qex gc`. `[gc] keep = "1d"` in the config file is the age that `qex gc` USES when you run it, and it is not a rule that qex applies for you. What grows is `jobs//` for each job: the two small JSON files, and `stdout.log` and `stderr.log`. `[logs] max_bytes` limits each log file, and the default is 32MB, so the two logs of one job hold about 64MB at most with the default. A queue that an agent drives for days, with no person who watches it, is the case where nobody runs `qex gc`, and the first sign is then a full disk. ```sh qex du # how much the state directory holds, and the largest jobs qex gc --dry-run # what `qex gc` would delete qex gc # delete every record that stopped more than `[gc] keep` ago ``` ### Several users Each coordinator writes its claims to `/tmp/qex`, and it reads the records of the other users before it starts a job. This method needs no administrator rights. The method is cooperative, and a different user can write an incorrect value. qex also tests the free memory of the machine, so it finds a load that no coordinator reports. ## Limits **qex does not limit the CPU of a job.** The `cpu` controller of cgroup v2 is not available to a user on a usual Linux system, and macOS has no equivalent. The queue controls the number of cores instead: qex does not start more work than the budget permits. **qex does not limit the memory of a job.** A claim controls the queue only, and the behaviour is the same on Linux and on macOS. Nothing holds a job to its claim after the job starts. **A claim is a promise, and not a measurement.** A job that claims 2GB and uses 20GB can still fill the machine. qex tests the free memory of the machine before each start, which limits the damage, but an accurate claim is better. ## The property that an agent needs most The job is not a child of your shell, and it is not a child of your agent. The supervisor holds the job in the session of the supervisor, and the record of the job is on the disk. Somebody can therefore stop your agent at any moment. The job continues, it writes its result, and a later session attaches to the same job with the id: ```sh qex status "$(cat build.id)" --wait ``` A monitor script cannot do this, because a monitor holds the answer in its own memory. Stop the monitor, and the answer is gone. ## The promise about the format on the socket The CLI and the coordinator speak JSON on a Unix socket. A coordinator can operate for hours, and a new build can replace the program in that time, so a mixture of versions is normal and qex must say what it promises. **The format is additive only.** - A new version can add a field to a request or to a response. - A new version can add a request name. - A new version does not change the meaning of a field that exists, and it does not change the meaning of a request name that exists. - A change that cannot obey those rules takes a new name, and a capability gates it. Each side thus ignores what it does not know, and that is correct: a field that a program does not know is a field that did not exist when somebody made that program, so no earlier behaviour depends on it. **A test holds this promise, and not a habit.** The rule is easy to break by accident: one `deny_unknown_fields`, added to a message for tidiness, turns every later addition into a fault on every machine whose CLI is older. `the_wire_format_stays_additive` in `src/proto.rs` speaks as a coordinator of a later version — it sends the fields of today with one field that does not exist yet — and the CLI of today must read it. The same test sends a request from a later CLI to the coordinator of today. **The window of a mixture is as long as the longest job.** A coordinator keeps the code that it started with until the last of its jobs stops, so the mixture is not a moment during an upgrade: a job of two hours holds an earlier coordinator for two hours, and a machine whose queue never empties holds one with no end. That is why the promise above is a promise, and not an expectation. **A newer coordinator is therefore safe, and an older coordinator is not.** An early CLI sends the fields that it knows, and a new coordinator understands each of them. In the opposite direction, a new CLI can send an option that an old coordinator ignores in silence — and a job that runs without the lock that you asked for is worse than a job that does not run. qex thus asks the coordinator what it can do, and it refuses the job with the option and the remedy. ## The capability floor A coordinator says what it can do from the first published version and above. One below that number gives no answer, so the CLI cannot learn which options it obeys, and it must not let a user believe a rule holds when it may not. It refuses such a coordinator and says how to replace it: ``` the coordinator (pid 4321) is version 0.5.2, and a coordinator says what it can do from 0.6.0 and above. ... kill 4321 The jobs that operate now continue; a new coordinator reads the same records. ``` The jobs are safe in that operation. A coordinator holds no result: the record of each job is on the disk, and a new coordinator reads the same records. ### A development build gets a warning, and not a refusal A build that a person makes reports a version such as `0.0.0-dev+g98513e2`, which names the commit that it holds. qex names such a version a development build, and a coordinator from one gets a warning only. The floor is a backstop for a coordinator so early that it came before the capability handshake: such a build cannot say what it cannot do, so the only safe answer is to refuse it. A development build is not that. It answers the `Capabilities` request like every other build, so qex still refuses each option that it cannot obey, and it names the option. The floor is the coarse gate, and the capability test is the exact one. A refusal would make every build that a person makes unusable by its own CLI, which is a worse fault than the fault that the floor guards against. ## Three questions that a user asked, with the answers ### Does the memory test understand swap and reclaimable cache? **Reclaimable cache: yes. Swap: deliberately not.** On Linux the number is `MemAvailable`, which the kernel calculates. It counts the page cache that the kernel can reclaim, so a machine with 12GB of cache does not look full. On macOS the number is the free pages, the inactive pages, the purgeable pages and the speculative pages, which is the nearest equivalent. Memory that the kernel wrote to swap is **not** in that number, and this is the case to know about: ``` MemAvailable 1.6GB, 10GB in swap, no page operations, the machine is healthy ``` A job that claims 6.5GB waits there, although the machine can supply the memory by taking it back from swap. qex now says so in the reason: when the pressure is low, the reason states that the machine is not short of memory, that the number does not count the memory in swap, and that a smaller claim or a lower `reserve_mem` corrects it. The pressure, and not the free memory, is the measurement that separates a machine that is full from a machine that parked memory nobody wants. qex reads it (`/proc/pressure/memory`, Linux only) as a second test with `max_pressure`. **qex does not yet use the pressure to RELEASE a job that the free-memory test holds.** That is the correct change for this case, and it needs a machine in that state to test it, so it waits. ### Does the learned claim record the peak memory? **Yes. The peak, and the largest peak of the samples.** The measurement is `max_rss` from `getrusage(RUSAGE_CHILDREN)`, which is the highest memory that the job reached, and not its memory at the end and not an average. qex keeps several samples and uses the **largest** of them with a margin. This is deliberate, and the reason is the job that climbs for hours: a claim that is too small stops the job, and a claim that is a little large costs capacity only. qex records a job that COMPLETED, and nothing else. A job that somebody stopped, that reached its time limit, or that the kernel stopped for memory shows the memory that it REACHED and not the memory that it needs, and that number would teach qex the wrong size. ### Is there fairness between agents? **No. It is capacity, and then the order of submission.** Each user has one coordinator, and each coordinator holds its own queue in the order of submission, with the priority first. Between users, the coordinators publish their claims to `/tmp/qex` and each one subtracts the claims of the others from its budget. That controls the total load, and it does NOT divide the load. A fan-out of 11 jobs from one agent takes the capacity, and the 2 jobs of the other agent wait until it ends. Nothing gives a share to each agent, and nothing reserves a slot for the agent that arrives second. Use `--priority` inside one queue. Between agents there is no answer in qex today, and a user who needs one must divide the budget by hand, with `[budget]` in the configuration of each user. ## The boundary: one machine qex controls one machine. Two machines that farm work to each other are not coordinated, and the accounting on each machine sees its own jobs and the jobs of the other users OF THAT MACHINE only.