For developers · App internals
Logging
On this page
Gitoodo writes a structured log to help explain what happened on a user's machine (for example a
token check that failed, a worker that did not start, or a sync that keeps failing). The app and
every background service (classifier, dispatcher with its worker supervision, satellite) use the
same module (crates/core/src/logging) and schema. The log never contains tokens, Authorization
headers, response bodies, or URL paths and queries; servers are logged as host[:port] only.
Location#
One folder per user, shared by the app and the services:
| System | Folder |
|---|---|
| macOS | ~/Library/Logs/de.gitoodo.app/ |
| Windows | %LOCALAPPDATA%\de.gitoodo.app\logs\ |
| Linux | $XDG_STATE_HOME/gitoodo/logs/ (~/.local/state/...) |
GITOODO_LOG_DIR overrides the folder; with GITOODO_STATE_DIR set (tests, isolated runs) it is
<state>/logs. The test-access build keeps its own folder under its identifier.
| File | Writer |
|---|---|
gitoodo.log | the app |
classifier.jsonl | gitoodo classifier (also when the app starts it) |
dispatcher.jsonl | gitoodo dispatcher, including the workers it supervises |
satellite.jsonl | gitoodo satellite (not the --stdio bridge) |
<service>.stderr.log | stderr of a service (bootstrap errors, a crash before logging started) |
runs/<run_id>/… | terminal transcript of one worker run (see Worker transcripts) |
Each process owns its file through a sidecar lock (<file>.lock). A second process of the same
kind (for example a classifier service waiting for the app's classifier) writes
<name>-<pid>.<ext> instead, so two processes never rotate one file.
Each line is one JSON object (schema v 2):
{
"v": 2,
"ts": "2026-11-16T11:00:11.175Z",
"seq": 12,
"level": "warn",
"proc": "dispatcher",
"pid": 4711,
"component": "dispatch",
"event": "dispatch.host_wait",
"cid": null,
"run_id": null,
"issue": "owner/repo#42",
"span": null,
"audit": false,
"msg": "waiting for a host; no matching host can take it",
"ctx": { "host": "mac-mini", "reason": "offline" }
}ts: UTC, RFC 3339 with milliseconds.seq: line number within the process, so gaps show dropped lines.proc:app,classifier,dispatcherorsatellite;pidthe process ID.event: stable name; service events are dotted (<area>.<what>), app events keep their earlier names.msgis human text, never parsed.cidis a correlation ID: every line of one sync run (or one command) shares it. Acidfield on an event (a classifier decision the dispatcher refers to) is lifted out ofctx.run_id,issueandspanare top-level for every line (nullwhen unknown): a field of that name on an event is lifted out ofctx.run_idandspanare reserved for timed worker runs (see Runs and steps).ctxholds the remaining typed fields (at most 32), each redacted.
Levels#
| Level | Meaning |
|---|---|
error | Unrecoverable failure the user relies on (token check failed, keychain write failed) |
warn | Failed, but retried or carried on (sync failed, integration error, keychain read failed) |
info | State changes (sync finished, settings changed, audit events) |
debug | Detail (sync started, every keychain access) |
Dependencies log only warn and error.
Rotation, budget and retention#
- Per file: 5 MiB, then the file becomes
<file>.1(older generations shift, beyond.3they are deleted): at most 4 files and 20 MiB per process. A line that would not fit after a failed rotation is dropped rather than written past the cap (on Windows a file held open by a viewer cannot be renamed; Gitoodo then copies and empties it, and if that fails too it drops lines and retries after 30 s). The next written line reports the count asdropped_line. - Whole folder: a disk budget, default 75 MiB (setting
log_budget_mb, 20–500), covering every log and stderr file in the folder and one level of subfolders. Files older than 14 days (settinglog_max_age_days, 1–90) are deleted. When the folder is over budget, the oldest files that no running process writes are deleted first.*.stderr.logfiles are emptied above 1 MiB. - When: at the start of every process and every 6 h after (
log.cleanupnames what was removed). Files of a running process are never deleted; their size is bounded by the per-file cap.
The status window shows the folder's size against the budget.
Service events#
| Area | Events |
|---|---|
| every service | service.started (version), service.stopped, service.fatal (error), panic (location), log.cleanup |
| classifier (polls) | classifier.cycle (duration_ms, accounts, repos, errors, forge_requests, forge_failures, forge_rate_limited, forge_retry_wait_ms, forge_p50_ms, forge_p95_ms, forge_max_ms) after every poll |
| dispatcher (forge) | dispatch.forge_summary (the same forge_* fields) every 5 min when it made requests |
| dispatcher (control) | control.received (cmd, action, issue), control.applied (result, duration_ms, error): park, stop, restart, attach and worker requests at info, queries at debug; never texts or capabilities |
| workers | worker.transcript (path), worker.transcript_failed, worker.transcripts_recovered (count) |
| classifier | classifier.started, .lock_wait, .lock_taken, .handover, .parent_gone, .poll_error, .account_error, .reconciled |
| classifier (runs) | classifier.ready (cid, issue, previous): an issue became ready |
| dispatcher | dispatch.started, .worker_started, .worker_exit, .start_failed, .host_wait (issue, reason, since, detail), .wait (reason, since), .hung, .merged, .merge_failed, .lease_*, .cleanup, .shutdown |
| runs | run.start, run.end, span.start, span.end, satellite.clock (see below) |
| automations (in dispatcher) | automation.started, .postponed, .rate_limited, .gave_up, .config_error |
| satellite | satellite.listening, .worker_started, .worker_exit (run_id, code, duration_ms), .spawn_failed, .kill, .delivery_failed, .subscription_lost, .shutdown |
Fields used across events: issue (issue key), host, repo, error (redacted text), reason.
Runs and steps#
One attempt of the dispatcher at an issue is a run with its own ID, r- and 10 lowercase
Crockford base32 characters (r-7k2m9q4xzt). Every line of the attempt carries it in run_id,
whichever process writes it:
- The classifier logs
classifier.readywith a freshcidwhen an issue becomes ready and keeps that decision (cid, since when) in the read model while the issue stays ready. - The dispatcher creates the run ID when it picks the issue up and logs
run.startwith thatcid,host,workerandagent, so the run is linked to the decision. - The ID is stored in the worktree lease, sent to a satellite in the
spawnrequest and given to the worker asGITOODO_RUN_ID. Satellite lines of the worker (its worktree, spawn andsatellite.worker_exit) carry it, and so does the main host's delivery (pr.open) of a satellite branch. The pull request itself does not carry it. run.endcloses the run withduration_msandresult:merged,stalled,needs_human,parked,paused,user_stop,start_failed,lease_refused,host_goneorskipped.
Each step is a span. A step that runs within one call is logged as span.start and span.end;
a step that already happened (queue.wait, host.select, merge) as one span.end. span.end
always has duration_ms and result (ok, error, cancelled, timeout); otherwise also
error_kind and, where useful, error (redacted). A run that ends with steps still open ends them
with the run's outcome.
| Step | From → to | Error kinds |
|---|---|---|
queue.wait | the classifier's ready decision → the dispatcher's pickup (since) | – |
host.select | choosing the host among the ready issues of the repo | – |
clone.ensure | finding the working copy, cloning it when missing, fetching the base (local hosts) | clone_failed |
worktree.create | cutting the worktree (on the dispatcher or the satellite) | worktree_failed |
worker.spawn | starting the worker (for a satellite: the whole spawn request) | spawn_failed, satellite_refused, …_unreachable |
worker.run | worker start → its exit (restarts reopen it) | exit_code, hung, the run's result |
pr.open | gitoodo worker pr: push and pull request | delivery_failed |
checks.wait | pull request found → checks decided | checks_failed, the run's result |
merge | the merge call (or enabling auto-merge) | merge_failed |
merge.wait | auto-merge or a human merge → merged | – |
cleanup | removing the worktree and the branch after the merge (local hosts) | cleanup_failed |
Waits are logged once per change, not on every pass: dispatch.wait (the dispatcher starts no
worker: paused, no_dispatching_repo, classifier_unreachable) and dispatch.host_wait (an
issue's os:/host: labels match no free host, with the concrete reason of #750:
unknown_host, os_mismatch, offline, agent_missing, usage_gate, low_memory,
at_capacity, global_limit; detail names the wanted host=/os= and the sentence with this
machine's names and the satellites) carry reason and since; a new reason logs again with previous and
previous_ms. The end of a wait is dispatch.wait_end / dispatch.host_wait_end with
waited_for and waited_ms.
Satellite clocks: every satellite report carries the satellite's time. When the difference to the
main host changes by more than 0.5 s, the dispatcher logs satellite.clock with host,
skew_ms (satellite minus main host), satellite_ts and received_ts (the receive time
includes the round trip). gitoodo trace moves the satellite's lines by that skew.
Reading runs#
Both commands only read the log folder, never write, and redact every printed text once more.
gitoodo trace <owner/repo#n | run-id> [--local] [--output]prints the timeline of one issue (all its runs and waits) or one run: time, offset from the first line, process (satellite@<host>for remote lines), step, result, duration and details, then a summary per run with the duration of each step, open steps, the total and the longest step. It also asks every configured satellite for its lines of the run (trace_linesrequest, at most 2000 lines and 4 MiB, newest kept); a satellite that does not answer is named and skipped.--localreads this machine only. It lists the run's worker transcripts;--outputprints them.gitoodo diagnose [--hours <n>]summarises the last 24 hours (orn): runs started and their results, per step the count, results and p50/p95/max duration, the most frequent wait reasons, errors by step or event and kind, and the log folder against its budget. A "cleanup" section shows the dispatcher's last repo-hygiene round (#771,<state dir>/hygiene.json): "clean" or the leftovers (host, repo#number or path, branch, reason, unsaved commits and files), the disk guard's reading and the round's errors. Further sections: classifier polls (count, p50/p95/max) with the forge requests of classifier and dispatcher (failures, rate limits, time waited before repeats, worst p95); per service whether it runs now, its last start (version), last end, panics and last warning or error; the size of the worker transcripts. A "questions" section counts the steps of the question flow (see Questions) and lists the issues whose last step is notanswer_applied, newest first, with that step and its detail.
The status window's "Run" section (#749) reads the same files for the last 24 hours through the
app commands run_overview and run_output (gitoodo_core::logging::overview): the
classifier's last poll (time, duration, repos, errors, forge requests and rate limits) with the
poll percentiles, the dispatcher's state (running since, current wait and its reason, forge
requests, last warning), the newest 20 worker runs with result, failed step and whether a
transcript is kept, and the newest 20 question steps. "Show output" opens the end of a run's
transcript (at most 256 KiB, from a whole line), redacted line by line once more on reading.
Questions#
Every step of a question is logged with the issue (account/owner/repo#n) and IDs only, never
question or answer texts, so "my answer did not land" can be followed step by step:
| Event | Process | Fields (in ctx) |
|---|---|---|
question.posted | classifier, dispatcher, satellite | question_id, kind, via (classifier intake or worker) |
question.post_failed | same | question_id, kind, via, error (redacted) |
question.answer_posted | app | question_id, comment_id, answers (count) |
question.answer_failed | app | question_id, category |
question.comment_posted | app | comment_id |
question.answer_seen | classifier | question_id (intake) or comment_id (detection), via, verdict |
question.answer_applied | classifier | question_id or comment_id, via, outcome (ready, review, closed, …) |
question.answer_pending | classifier | comment_id, reason: why the newest human reply does not count (once per reason) |
answer_pending reasons: detection_off (answer detection is off in the rules),
question_open (a question of the block is still unanswered), not_asked (no open question is
known), declined (the asked person said no to an approval the intake no longer reads, so the
issue keeps waiting). Without a human or a routing rule the login in a card's header is the person
asked, so such a reply counts.
Comments written by Gitoodo itself are not considered.
Worker transcripts#
Every worker run on the main host or a satellite writes its terminal output to
runs/<run_id>/run.log (a restarted run: run.2.log, …). The text is stripped of terminal
control sequences, cut at 8 KiB per line and redacted line by line like every log field. Per run
at most 2 MiB: the first 256 KiB are kept, then only the two newest tail chunks
(run.tail-N.log), with a gap marker in between. When the worker ends the head is compressed to
run.log.gz (worker.transcript); a service that starts finishes transcripts a crashed
predecessor left (worker.transcripts_recovered). Transcripts count against the log folder
budget and age limit like every other file.
A service panic writes one redacted line (and a backtrace only with RUST_BACKTRACE) to its
stderr file, never the raw panic text.
On macOS launchd sends a service's stderr to <service>.stderr.log. On Windows the Task
Scheduler redirects nothing, so the per-user scheduled task starts gitoodo classifier --service: the CLI removes the flag, treats itself as run by a service manager and points its
standard error at the same file before anything else runs (SetStdHandle, no admin rights). A
failed redirect is logged as service.stderr_failed.
time + process run event / step result took detail
10:00:00.000 +0 ms classifier - classifier.ready cid=ready-1
10:00:05.000 +5.0 s dispatcher r-7k2m9q4xzt run.start cid=ready-1 host=mini
10:00:05.000 +5.0 s dispatcher r-7k2m9q4xzt queue.wait ok 5.0 s
10:00:06.210 +6.2 s dispatcher r-7k2m9q4xzt worker.spawn ok 1.2 s host=mini
…
10:06:45.000 +6m45s dispatcher r-7k2m9q4xzt run.end merged 6m40sEvent types#
| Event | Level | Fields (in ctx) |
|---|---|---|
token_check | info / error | host, forge, category, status (HTTP, if any), missing_scopes / missing_scope |
account_add_failed | error | error |
sync_started | debug | account, host |
sync_finished | info | account, host, duration_ms, questions |
sync_failed | warn | account, host, category, error, duration_ms, consecutive_failures, retry_in_ms, repeats |
sync_recovered | info | account, host, suppressed_failures |
keychain_access | debug / warn / error | op (get/set/delete), account, result, error (never the content) |
settings_changed | info | keys (names of changed settings, never values) |
integration_error | warn | integration (autostart, shortcut, skill, open_url), error, host for open_url |
chat_notify | info / warn | host, kind, sent, skipped_old, dropped; on failure result, reason (category), repeats (never the URL) |
chat_webhook_set | info (audit) | none; chat_webhook_cleared likewise. The URL is never logged |
link_shared | info | via (copy/share), source (question, board, release), host (public forge name, else private; never the URL) |
app_restart | info | none; "Restart app" in the help menu |
Token check categories: unauthorized, forbidden, not_found, rate_limited, unreachable,
tls, server_error, decode, invalid_url. Together with status and host they explain
why a token was rejected.
Audit events#
Marked with "audit": true, written on the gitoodo::audit target. They bypass the level filter,
so they cannot be switched off in production: account_added, account_removed,
account_renewed, token_deleted (with reason).
Throttling#
Identical repeated errors (same key, e.g. the same sync error text) are written once per 10
minutes. The next line that gets through carries repeats, the number of suppressed occurrences;
sync_recovered reports suppressed_failures. Failed keychain reads are throttled the same way.
Debugging#
- Set
GITOODO_LOG=debug(overrides thelog_levelsetting) or change the log level in settings. - Follow one run:
jq 'select(.cid=="sync-1a2b3c")' gitoodo.log - One issue across the services:
gitoodo trace owner/repo#42, orjq 'select(.issue=="owner/repo#42")' *.jsonl - What is slow or failing lately:
gitoodo diagnose - All problems:
jq 'select(.level=="warn" or .level=="error")' gitoodo.log - Audit trail:
jq 'select(.audit)' gitoodo.log