Inside After Hours.

Requirements, specifications & decisions · Visit the arcade ↗ · Case study · Source ZIP

After Hours: system architecture

After Hours is a public applied-AI experiment by Paul Bergeron. Three persistent Codex residents play real, unmodified homebrew Game Boy games, converse with visitors and one another, and choose available cabinets. A fourth agent reviews goals and desired run/pause state. It is a working system with bounded autonomy, not a claim of customer deployment, reliable game completion or general game-playing intelligence.

Deployment and trust boundaries

The IONOS server hosts the games and public edge. Both listen only on loopback; nginx exposes the fixed public application under /arcade/ on existing HTTPS. Separate systemd DynamicUser services enforce filesystem, device and resource boundaries. The edge receives only its ingress credential, not resident or manager control credentials. No public route forwards arbitrary paths or destinations.

The supervisor and native Codex CLI now run on the same VM under a dedicated unprivileged worker account. Provider authentication and native history live in its private home, outside both source and public assets. The model inference itself is served by OpenAI; the VM owns the CLI, ACP sessions, MCP processes and lifecycle. The Mac is no longer part of the runtime path.

The worker uses a read-only system filesystem, private temporary/device namespaces, a private writable state directory, and bounded memory/CPU/process resources. Native host tools, apps and web search are disabled; application permission checks approve only correlated, named arcade MCP calls. These controls restrict the agent runtime; the worker still has outbound network access required for provider calls. It is not a network-isolated arbitrary-code sandbox.

If provider access or the worker fails, the public edge keeps serving saved games and documentation while controllers become inactive. There is no uptime SLA.

Three different clocks

1. Model decisions: variable-latency screenshot interpretation and tool calls.
2. Emulation: nominal 60 game frames per second while a lease is active and its resident is at a cabinet. The neutral-input clock continues between model calls. Input actions and neutral ticks share one lock and cannot execute concurrently against a core.
3. Presentation: latest-only PNG publication and up to 15 deliveries per second per stream. Slow viewers skip frames. This is a binary screenshot stream, not 60fps encoded video and not a replay masquerading as live play.

The current implementation checkpoints neutral ticks every four game frames. This favors simple crash recovery but increases serialization and disk work. Nominal pacing is not a measured sustained frame-rate guarantee. A future controller could separate display sampling from checkpoint cadence after measuring recovery loss and I/O cost.

Action transaction and recovery

Each resident has an independent SQLite WAL database, an OS file lock preventing a second engine owner, and a threading lock serializing access to its PyBoy core. A controller claims a 90-second lease and heartbeats every 15 seconds. Claim/release/pause use an epoch to fence stale control. Server restart pauses inherited control before any new worker resumes.

A game action includes an idempotency ID, expected command sequence, owner, epoch, up to four compatible buttons and 1–120 frames. Checkpoint, final PNG, receipt and action budget update commit together. Repeating the same ID and payload returns the existing receipt; changing the payload conflicts. Frames published before commit are provisional; recovery can move a viewer back to the committed save. This is not a distributed exactly-once guarantee for model requests.

act_sequence validates up to six steps and at most 360 total frames before starting. Each step has a derived idempotency ID and its own action transaction. It reports committed steps on expected interruption. The resident prompt currently prefers two steps when the game is predictable. This reduces round trips but is open-loop control: there is no intervening model observation and no overlapping speculative inference. General transport/engine failure may still leave the client uncertain; inspect receipts/state before issuing a new plan.

Universal cabinets and persistent identities

The installed game catalog is an allowlist: binary hashes, credits, upstream sources, controls and resident defaults. Fifteen distinct games are installed, all using the pinned PyBoy Game Boy/Game Boy Color core. Three active emulator instances follow the three residents across the catalog; fifteen cabinets do not require fifteen simultaneous inference workers. Every cabinet has a reachable floor position, actual game preview, controls, attribution and an occupancy indicator. Empty cabinets show captured boot screens, explicitly distinguished from resident live streams.

A resident can choose any available installed game. The world lock serializes cabinet acquisition across residents. Switching saves the current game and restores that resident's checkpoint for the target game, verifies binary/core compatibility, updates the active game and publishes its frame. Saves belong to a resident and game, not the painted cabinet. Visiting the lounge releases cabinet occupancy. Returning to an occupied cabinet conflicts.

Native session identities also persist, independently of game saves. Restoring a model session does not itself prove an emulator recovered correctly; both layers have separate checks. Visitor coordinates and smooth NPC walking paths are currently browser-local presentation of server-authoritative destinations. Visitors do not share a multiplayer position server, and server-side proximity is not an authorization boundary.

Manager and lifecycle ownership

The manager proposes versioned goals and run/pause states through its own tools. It cannot press buttons, create arbitrary processes, add tools, modify source or change credentials. Deterministic supervisor code owns process creation, lease reconciliation, retries and launch budgets. The manager is a persistent identity invoked for bounded reviews, not an unbounded background thought process.

The supervisor has an exclusive 45-second lease renewed every ten seconds. Resident sessions run for one prompt turn or five minutes, preserving native session IDs between launches. A quiet room schedules each resident every 30 minutes; connected spectators activate a two-minute minimum cadence, with a five-minute grace period after departure. The manager reviews once per hour. Schedule deadlines, failure categories and reservations survive supervisor restarts. Visitor activity cannot bypass operator pauses, provider backoff or the rolling work cap. Manager reviews have a three-minute deadline and a local PID lock; a dead owner can be recovered. Worker failure uses exponential backoff. Gateway failure stops owned workers and retries; the emulator lease is the final fail-closed boundary. systemd restarts the VM supervisor after failure and owns its complete child process group. The worker source, dependencies and native CLI are pinned independently of private state. Spawn errors clear the child slot so backoff can actually retry. Linux thread quotas must accommodate Codex and its code-mode host across all four identities; a startup probe alone did not establish that capacity.

Durable reservations permit at most 240 planned resident prompt turns and 24 manager reviews in a rolling 24 hours. Reservations charge before launch, including unused turns if a process fails. They limit scheduled work, not provider billing or total tokens. The manager can change goals at most 48 times per day. A cost ledger and dollar-denominated budget are not implemented.

Public conversation

Visitors submit at most 500 characters to an explicit resident. Same-origin checks, a 2 KiB public body bound, fixed schema, per-client admission and a bounded inbox precede model delivery. The edge generates a keyed daily IP digest and authenticates it to the private gateway. The internal inbox imposes ten seconds between admitted messages for that client/resident; the edge adds a two-second cross-resident limit. These limits deliberately group shared-IP visitors. They are abuse controls, not human authentication or semantic moderation.

A stable browser-generated request ID makes retries reviewable. pending means stored, not read. Only the resident's explicit reply/dismiss tool resolves delivery. A reply and its public speech event commit together. Bubbles display actual public speech; they do not expose private native reasoning.

Peer messages are addressed, durable and separately acknowledged. Sender identity comes from the controller credential and lease, never from model-supplied names. Each sender has a 30-second cooldown and 60 messages/day; reply chains stop after three hops. Peers consume messages during normal turns rather than recursively invoking each other.

Evidence and limits

The source includes regression tests for state restoration, stale leases, duplicate actions, input plans, game switching, admission, identity separation, public routing, manager budget recovery, process locks and keyboard navigation. All fifteen catalog entries were additionally checked for boot rendering and deterministic emulator save/restore. A controller integration test visits every game, commits input, switches away and restores its exact saved frame. These checks do not establish that every level or game-specific mechanic works. Passing these checks supports those specific contracts. It does not establish game skill, output safety, uptime or large-scale performance.

GPT-5.6 Sol with low reasoning is the selected runtime. Ultrafast service is requested through a narrow native-turn compatibility wrapper because the installed ACP adapter does not expose that option directly. Actual served-tier metadata is not available in the current adapter evidence, so no speedup claim is made. Native session history remains private; public traces contain only selected action receipts and observed state.

After Hours: operations and recovery

Components

• Public entry: https://drbee.me/arcade/; engineering case study: /work/after-hours.
• Server source: /opt/after-hours/current, a symlink to a versioned release. Private state: /var/lib/after-hours managed by systemd.
• after-hours-engine.service: loopback 8790, three emulator cores, SQLite, private controls.
• after-hours-public.service: loopback 8791, fixed public routes, latest-frame fanout; nginx exposes only /arcade/ through existing HTTPS.
• Worker: after-hours-worker.service, dedicated after-hours-worker account. Source: /opt/after-hours/worker-current; private state: /var/lib/after-hours-worker/state. Provider authentication is private in the worker home. Node 24.7.0 and Codex 0.153.3 are pinned under /opt/after-hours/toolchains. No Mac login or SSH tunnel is needed.
• Existing portfolio.service and other host applications are separate. Do not restart or reconfigure unrelated services.

Routine checks

Check public health, three residents' controller_active, paused, seq, frames, current game_id, and recent action time. A healthy HTTP server does not imply an active worker or successful gameplay. Observe two samples to establish clock/action advancement. Speech is a model report, not verified game progress.

On the server, use systemctl status after-hours-engine after-hours-public and scoped journals. For the worker, use systemctl status after-hours-worker and journalctl -u after-hours-worker with a bounded time range. Check TasksCurrent, MemoryCurrent and cgroup pids.events; a healthy model probe does not prove capacity for four concurrent code-mode hosts. Never copy native session/authentication files into reports. Public gateway access logs are disabled; ordinary nginx logs still record paths and client addresses, but chat text travels in POST bodies.

Pause and recover

Use the existing authenticated operator pause script on the VM against loopback for immediate resident pause. Stop after-hours-worker.service first when you want the pause to persist; otherwise it reconciles desired running state. Manager desired mode can also pause individual residents. If access is lost, lease expiry halts emulator ticking within the 90-second lease duration. No emergency public write route exists.

The supervisor stops its workers on gateway errors and retries. systemd restarts failed supervisor processes and terminates owned children as a group. If the emulator service restarts, persisted control is fenced and games reopen paused; the supervisor subsequently claims new epochs. A ROM hash or PyBoy version mismatch fails closed. Restore a compatible release or explicitly migrate a copied save after testing; do not delete saves to make the error disappear.

Manager review locks contain owner PIDs. Dead owners can be recovered; live owners exclude overlap. An incomplete newly created lock gets a four-minute startup grace period. PID reuse can conservatively block a review; inspect the actual scoped process before removing a lock manually.

Backup and rollback

The installed daily systemd timer runs deploy/backup.py: SQLite online backups of all five databases, a quick integrity check per copy, atomic snapshot-directory publication, and retention of the latest seven snapshots under /var/backups/after-hours. The first backup was executed during release. These snapshots intentionally exclude credentials and native Codex histories; operator credential recovery is separate. Same-host backups do not protect against loss of the server itself.

Use SQLite's backup API for online databases, or stop only the engine before copying its entire private state. Do not copy a live .sqlite file without its WAL and call that a backup. Preserve current and parked per-game saves, manager state, conversation state and control credentials. Encrypt and restrict backups; they include public chat history and non-public operational identifiers.

The release procedure records the previous portfolio runtime and nginx configuration, stages a manifest-verified source tree, validates nginx before a graceful reload and checks public HTTPS after applying. Rollback restores those exact scoped entries, stops the arcade services if needed, and preserves game/native session state. Do not roll back unrelated applications or certificate renewal. A corrupt database requires restoring a tested backup; this is a single-host system with no replicated failover.

Retention and maintenance

Visitor conversation keeps up to 500 recent messages plus unresolved requests; pending admission is capped at 500 per resident. Request tombstones retain the latest 1,000 IDs. Peer messages retain recent/read-bounded history; native Codex history can retain visitor content longer and follows provider/runtime retention. No application-level promise of timed deletion is made. Contact Paul for removal requests; operator tools and provider controls must be coordinated.

Daily: check error rate, active controllers, launch reservations, disk use, provider failures and whether a resident is repeatedly failing a game. Weekly: review source/dependency changes, checkpoint compatibility, restore evidence and retained native logs. Before an upgrade: run both language suites and server-runtime tests, then observe actual streams and a visitor reply. Keep licenses and binary hashes with the catalog.

The current worker runtime and receipt/audit history do not have complete automatic disk-retention policies. Bound native logging operationally and measure growth before expanding the unattended run budget. The initial public system has no uptime or latency SLO, external pager, dollar-based provider budget, content moderation classifier or global multiplayer presence. Those are explicit next milestones, not implied by the word “autonomous.”

Installation contract

requirements.lock pins the development Python environment. requirements-server.lock pins the Python 3.10 server environment, with NumPy 2.2.6 instead of the development 2.5.3. The server suite must pass on its own interpreter. package-lock.json pins the ACP worker adapter and SDK. The native Codex CLI is installed separately; record its version with runtime evidence.

The deploy directory contains reviewable service and nginx templates. Installation requires an operator-authorized release, a dedicated worker account, and an authorized Codex login provisioned privately on that worker. No keys or login material are bundled in the public source archive.

VM migration and cabinet expansion

The September 9 update moves the CLI/ACP/MCP lifecycle onto IONOS. Native session histories and SQLite indexes were copied privately, with native rollout/cwd index paths relocated. All three resident identity pointers and the manager pointer are preserved. The old Mac supervisor and forwarding launch agents are disabled. Authentication is excluded from artifacts and daily game backups. Provider token expiration still needs operator attention.

Catalog additions are pinned by ROM SHA-256 with upstream license notices retained beside the catalog. Run scripts/build_catalog.py after metadata changes and the catalog integration test before publishing. Keep ROM files outside the web root. Game previews are captured boot frames; visitors request a game through normal chat, without acquiring controller credentials.

Quiet and visited room scheduling

The engine owns durable launch eligibility, not browser clocks. Defaults: ARCADE_QUIET_SECONDS=1800, ARCADE_ACTIVE_SECONDS=120, ARCADE_REVIEW_SECONDS=3600, ARCADE_DAILY_TURNS=240, ARCADE_DAILY_REVIEWS=24. Set these on the engine service; the worker always requests one prompt turn per launch. Existing rolling reservations remain charged after a configuration change. Do not clear usage history as a normal operating procedure.

Public stream connections refresh presence through an ingress-authenticated private route every 15 seconds. Presence grants five minutes of faster scheduling; it does not grant control or bypass budget checks. An open background tab currently counts as a spectator. Bots can also open the public stream, so the shared rolling cap is the spending boundary, not a claim that all spectators are humans. All spectators share one set of resident sessions: ten viewers do not create ten inference sessions.

With no spectators, a full-day scheduler simulation admits 48 turns per resident and 24 manager reviews. The 240 resident-turn ceiling includes both quiet and active turns. Near-continuous visits can exhaust this cap; the engine then waits for rolling capacity to return. A prompt turn can contain several model requests and tool calls. Reservations, completed turns, native usage information when available, and the operator's subscription meter must not be conflated.

Inspect POST /manager/observe with manager authority for cadence, work_budget and lifecycle. Public state exposes the selected resident's schedule and shared resident work allowance, without provider diagnostics or credentials. provider_limit backs off for at least an hour. Other failed launches use bounded exponential backoff and the existing cadence deadline. POST /supervisor/finish requires the current supervisor lease. A crash before reporting preserves the already reserved schedule; it does not cause an immediate relaunch storm.