![](\"https://sklvc.atlassian.net/wiki/download/thumbnails/2064842814/image-20260607-205726.png?version=1&modificationDate=1780865848116&cacheVersion=1&api=v2&width=1536&height=1024\")
A site is run by many agents acting at once \u2014 several robots, the fleet agents that assign and monitor them, human operators, and System-3 integrations (a customer WMS, a SynaOS AGV fleet). They share work, space, and equipment, and nothing keeps them from unknowingly stepping on each other unless something records what is being done and what is believed to be true. Switchboard is that something: the one place those agents synchronize. This spec defines the state model that makes the synchronization honest \u2014 what a subject is, what a fact is and where its truth lives, how fresh it is, who may write it, and where it is authoritative versus merely cached.
Two framings have to be rejected up front, because both are quietly assumed in the current system. The first is that Switchboard is the workflow: a robot comes up already bound to a workflow and is expected to fire transitions even while unlocalized, mislocated, or holding stale beliefs. The second is that Switchboard is the world model \u2014 that its Petri net is authoritative over reality. It is neither. It coordinates agents and replicates the state they need to coordinate against; the truth lives at the edge, and the net is one valid view of the work, not the territory. The robot knows reality better than Switchboard does, and the model is explicitly allowed to be a schematic approximation that reality is permitted to diverge from.
Getting this layer right also gives every other system one legible place to plug in. A WMS, a SynaOS AGV master, an operator console, and a robot are all just agents that read and write the same state through the same surface \u2014 which is what keeps integrations at arm's length from the robot stack instead of threaded through it.
The prototype makes the cost of not having this layer concrete:
A robot sat localizing, unable to resolve tf map \u2192 base_link, while coordination still counted it a participant.
Per-robot facts collided because the key namespace had no notion of which robot (push_facts had to be hand-namespaced robot1/foo).
Coordinator-side map/tag edits got clobbered because robots re-sync map data and "there is no clear ownership over map/tags between robots and coordinator."
A single-workflow-at-a-time regression crept in when map-sharing was implemented, because workflow and map were treated as interchangeable \u2014 "I thought it was interchangeable" \u2014 when in fact a robot should self-register to a workflow, pull its map, and decide its own assignments.
The whole stack assumes a single always-on coordinator, which does not survive a customer site where the VPN flaps.
These are all symptoms of an absent state model: no subject identity, no ownership, no freshness, no authority placement, no clean separation of the logical (workflow) and physical (map) axes.
The anchor is the Schaeffler pilot (Herzogenaurach, end of 2026; Schweinfurt to follow), which forces every one of these: small fleets that must coordinate without colliding, intermittent and locked-down connectivity, a SynaOS / SAP customer, navigation built on recorded trajectories and visual localization, and operators who need to see where everything is on a map.
Switchboard is the coordination primitive for everything that acts on a site \u2014 robots, fleet agents, operators, and System-3 integrations (SAP/WMS, the SynaOS AGV fleet). It is deliberately dumb: "it doesn't really do anything but answer queries and update its internal state." It does not plan, schedule, or optimize, and the Petri net is not authoritative over reality \u2014 multiple assignments may be valid at once and the robot agent (Reverb) chooses among them. It does not hold any agent's private context or reasoning; it holds only the shared state agents coordinate against. Switchboard's job is to ensure two agents never unknowingly do conflicting work, to preserve what actually happened, and to be the one legible place every agent plugs into.
"Dumb" applies to the coordination plane, not the config plane. Switchboard answers coordination queries and serializes coordination state without intelligence. It does hold registry and authorization configuration (fleets, sites, maps, workflows, groups) \u2014 but that is static data pushed from a cloud-authoritative config plane, never logic on the robot hot path. The doctrine "the registry is data, not schema" (\u00a714) applies to identity as much as to coordination.
The reframing this spec makes explicit: Switchboard is authoritative for almost nothing. It owns the coordination it computes \u2014 markings and leases \u2014 strongly and transactionally, and replicates everything else from a source of truth that lives at the edge.
\u200c
Identity and coordination are separated by lifetime and blast radius.
Backbone is global and owns device identity: every physical chassis as a hardware identity, its attestation, SPIFFE SVID, and certificate lifecycle (it already federates auth through Cognito). A device keeps its Backbone identity wherever it is deployed; nothing else mints device identity.
Switchboard is per-deployment and owns the fleet registry plus coordination state \u2014 fleets, sites, robot site deployments, maps, workflow definitions, workflow instances, executors, and assignments. It holds device identity only as a reference outward to Backbone and authorizes against its own registry. Identity is global; authorization is local.
Four identity/authority lifetimes are kept distinct (consistent with the Outpost spec): device identity (what hardware this is \u2014 long-lived), fleet identity (which fleet it may join \u2014 rotated), entitlement (what it may do \u2014 policy), and lease (what it may do right now \u2014 short-lived). This spec governs the last three as they appear in coordination; device identity is Backbone's.
This section exists to kill the single-workflow regression. That bug happened because workflow and map were conflated. They are distinct axes, joined by three explicit actions, and robots are never pushed a workflow \u2014 they poll for their assignment and pull what they need.
Fleet is the logical unit \u2014 the coordination domain and isolation/RLS boundary; all markings, leases, and facts live inside one fleet. Robots are fleet resources.
Site is the physical unit \u2014 a building. Maps describe a site; workflows move robots within a site.
A robot site deployment records which site a physical robot is currently operating at. Fleet\u00d7site deployment is deferred until there is a real site-sharing requirement.
A map is a versioned spatial model of (part of) a site. A site may have more than one map \u2014 different configurations, or deliberately broken up for reliability/stability \u2014 but a workflow definition belongs to exactly one site map.
An assignment binds an executor to a running workflow instance (and therefore, transitively, to that instance's workflow definition and pinned map version).
Implementation note from the Rust MVP. Maps and workflows are intentionally not packaged as a single durable "bundle" yet. A running instance records the workflow definition version and map it started against, and validation checks that a definition's referenced places exist on its target map. Studio or Reverb may later introduce import/export bundles for convenience, but Switchboard's coordination model keeps maps and workflow definitions independently versioned.
The three actions:
Enroll \u2014 a robot joins a fleet via its Backbone device cert and receives a fleet cert. Durable. From this point its resource facts (pose, power, localization) replicate at fleet scope whether or not it is doing anything.
Deploy \u2014 a robot is deployed to a site, constraining which maps and workflows can be assigned to its executor.
Assign \u2014 an executor is bound to a running workflow instance (via executor_instance_assignments, \u00a7A.1). The robot polls to discover its assignment, then pulls the workflow's map and the workflow itself from Switchboard.
fleet \u2500\u2500 robot \u2500\u2500(site deployment)\u2500\u2500 site \u2500\u2500 site_map \u2500\u2500 workflow_definition\n \u2514\u2500\u2500 executor \u2500\u2500(assignment)\u2500\u2500\u2500\u2500\u2500\u2500 workflow_instance \u2500\u2500 pins definition version + map version\n\ndeployment = robot \u00d7 site assignment = executor \u00d7 workflow_instance\n
Switchboard never pushes a workflow onto a robot. A human, UI, or System-3 agent records an assignment in Switchboard; the robot discovers it by polling. (A future push channel is permitted only if latency/bandwidth ever demands it, as an explicit exception \u2014 not the default.) Two robots in different workcells therefore run two workflows on two maps in one site, coordinated by per-robot assignment rather than by forcing a single shared map \u2014 exactly the "develop in workcell A while the other works in workcell B" case the regression broke.
A fleet deploys to one site at a time (coordination requires a shared site). It does not require a single shared map \u2014 "they must share the same map while operating in the same space," but distinct spaces within a site may use distinct maps. This resolves the v0 open question "is one-fleet-per-site a hard rule?": one site per fleet deployment, not one map.
The map registry row holds the version, metadata, and named places; the heavy map assets (occupancy grid, 3D semantic mesh, recorded trajectories) live in object storage and are referenced by URI. Switchboard does not store map rasters.
State attaches to a subject. A subject's kind is a plain logistics noun:
resource \u2014 equipment with live state: robots (location, power, localization).
load \u2014 things moved: totes, items, goods.
place \u2014 fixed locatable things: slots, tables, chargers, conveyors, zones \u2014 and map landmarks. A tag is just a place a map owns.
work \u2014 orders, tasks, locks, reservations (the tokens that flow through the net).
scope \u2014 tenant, fleet, site, map, workflow.
A subject's identity is a UUIDv7 \u2014 a time-ordered RFC 9562 id, generated app-side in the Rust service and stored as the built-in 16-byte uuid type. The millisecond-timestamp prefix keeps inserts local in the B-tree and recently-created rows sorted together, where random v4 scatters writes and fragments the index. App-side generation removes any dependency on a specific Postgres version or extension being present at every (often locked-down) customer site, and the service already mints ids for claims and leases. The scheme is forward-compatible: a site on Postgres 18 may switch to native uuidv7() with no schema change, since it is the same type and layout.
The id is the identity; kind is a separate column; a subject also carries a mutable label (a human name like r12) and an external_ref (a customer/protocol id \u2014 AGV serial, SAP order number, tote barcode). Identity is never parsed out of a string, and a physical thing keeps one durable subject identity for its whole life regardless of how it is currently represented (see \u00a76 \u2014 a tote is the same subject whether sitting on a table or being carried).
Display and logs render a subject by composing kind + label (or a short id): robot:r12, load:0197f96c\u2026. Throughout this doc, robot:r12-style strings are that rendered form, not the stored key. Integrations map their external ids onto external_ref without colliding with internal identity, so relabeling a robot or re-barcoding a tote never breaks a reference. (Registry scopes keep stable human slugs instead of UUIDs, because they appear in SPIFFE paths and URLs.)
Three kinds of state attach to a subject:
Markings \u2014 a token's place in a net. Changed only by transitions; owned by Switchboard; strongly consistent; bounded (sink places reap, \u00a712).
Facts \u2014 observed/declared/derived state; many writers; an eventually-consistent replica; inform guards. At MVP, links are just facts too (a carried_by fact whose value is the carrier's id); a typed link table is a later optimization.
Leases \u2014 exclusive claims to act; strongly consistent; serialize dispatch (\u00a710).
The central principle: a fact is an eventually-consistent replica of state whose source of truth lives elsewhere. Switchboard does not originate facts; it holds a cached copy for coordination, and that copy is allowed to lag its source.
source of truth replicated into Switchboard as governed by\n\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500 \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\nrobot (self-telemetry) \u2500\u2500\u25b6 resource facts (pose, power, loc.) TTL (perishable)\nrobot (perception) \u2500\u2500\u25b6 observed facts (load locations) TTL (perishable)\nmap version \u2500\u2500\u25b6 place facts (tags, zones) epoch (versioned, static)\noperator \u2500\u2500\u25b6 declared facts (overrides) TTL or sticky\nSwitchboard \u2500\u2500\u25b6 markings, leases authoritative + strong\n
Static vs. dynamic ownership \u2014 the key division. The map / world-model owns static truth: tags, places, zones, semantic anchors. Robot perception owns dynamic truth: where movable objects actually are right now. A tote does not self-report and the map is not expected to track it; instead a robot is told (loosely, by an order or a prior) that a tote is at a table, goes there, and its perception answers present or not \u2014 and if present, picks it up and carries it. This deliberately does not centralize dynamic object location in a single world-model writer; each robot reports what it sees, duplicates are expected, and the model is not required to be perfect. A stale or missing load fact fails a guard closed (safe) and surfaces as an exception (\u00a78), never as a fleet-wide bottleneck.
Tags and poses are the same kind of thing, differing in volatility and source. Both are locatable facts replicated from an external source. A pose is perishable \u2192 TTL-governed, source must keep re-asserting (a replica that stops refreshing goes stale \u2014 the desired behavior). A tag is static \u2192 epoch-governed, invalidated only when the map version bumps. "Inherit semantics from the map" and "replicate pose from the robot" are the same operation; TTL/epoch fall out of replication rather than being bolted on. This is why tags are not first-class: a tag is a map-sourced place fact, nothing more.
Many writers, no collisions. Every fact has an authorized writer, and authority is granted per subject, not per fact kind. A fleet runs one Reverb per robot; each publishes pose, power, and localization.current only about its own robot (self-bound, \u00a7B.6), keyed by (subject, key) (\u00a75). Twelve Reverbs writing pose are twelve writes to twelve different subjects \u2014 not contention for one row. The map server is a different writer with different authority: it publishes the place facts the map owns (tags, zones). So a pose about a robot and a location about a tote both exist, written by different authorized writers, never in each other's way \u2014 and the tote's location comes from robot perception, not the map.
A tote is a load, and a load is locatable \u2014 but location is a fact (a pose); what the token encodes is the mode: relative-located or absolute-located.
Absolute-located (sitting at a place): the tote has a real, frozen absolute pose fact. It does not re-derive and so cannot go stale by drift; it changes only when the tote is next moved.
Relative-located (carried): the pick transition consumes the absolute-located token and produces a relative-located one whose pose is derived from the carrier's pose for as long as a carried_by link (a fact whose value is the carrier id, \u00a75) holds. The place transition consumes the relative-located token and produces an absolute-located token by collapsing the derived pose into a frozen absolute snapshot at the robot's current location.
The consequence for freshness: carrier-pose staleness can only mislocate a tote while it is being carried, never once it is set down \u2014 placement snapshots the pose. The staleness surface is bounded to the carry window, and it fails closed because a stale carried-token's derivation dies with the carrier's epoch (\u00a711). Through all of this the tote is one subject (stable UUIDv7); only its token's mode and place change \u2014 that stable identity is the continuity thread that lets us refer to "the same tote" as it moves between standalone and carrier-linked representations. A dropped tote is not silent staleness but an explicit carried \u2192 dropped transition that records last-known position and flags for an operator (\u00a78). (Conveyors \u2014 where a tote moves with no carrier holding it \u2014 are the genuinely tricky case and are out of scope here.)
The link is just another fact, so it rides the same replication, freshness, and authorization machinery as everything else: the executor that owns the pick transition is the one authorized to assert carried_by (\u00a7B.6).
Two mechanisms resolve where facts live and the idle-robot question:
Ingress replica (a real cached copy). A robot's pose/power/localization lives at fleet scope \u2014 the robot is a fleet resource, so its facts exist whether or not it is assigned to a workflow. That is the idle-robot answer: an idle robot's power is a fleet-scoped resource fact, no workflow required. Map places live at map scope, versioned. These are real rows replicated from an external SoT, hence TTL/epoch.
Workflow projection (a logical join, no copy). A running workflow instance does not own these facts and does not duplicate them. It projects the relevant subset \u2014 the resource facts of its assigned robots, the place facts of its map, the observed loads in scope \u2014 alongside the markings it owns, so guards evaluate against one consistent snapshot. "Assigned to a workflow" means projected-into, not stored-under.
fleet \u2500\u2500 robot (resource) \u2500\u2500 pose/power/loc facts \u2510 fleet-scoped replica\n \u251c\u2500\u25b6 workflow projects both + owns markings\nsite \u2500\u2500 map \u2500\u2500 place facts (tags, zones) \u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518 map-scoped replica\n \u2514\u2500 workflow (running instance)\n
The deployment (fleet \u00d7 site) and assignment (robot \u00d7 workflow) together make the join coherent: the fleet's robots, assigned to a workflow, projected onto that workflow's map.
Because facts live at their owning scope and workflows only project, the same replicated facts are reachable three ways \u2014 different join paths, one underlying store:
By fleet / robots (operations lens): every robot's resource facts \u2014 pose, power, localization, status \u2014 valid when idle. "Show me my fleet." No workflow needed.
By map (spatial / plotting lens): every locatable subject resolvable on map@v \u2014 the map's own places plus the resources and loads localized on it. "Plot the floor." Needs a map and fresh poses, not a workflow. This is the operator map view.
By workflow (coordination lens): the instance's markings plus the projected facts for its participating subjects. "Why isn't this transition firing." Needs a running instance.
Reality diverges from the schematic constantly. There is no single exception mechanism; there is a single principle: degrade to keep operating as reliably as possible, and flag the exception for a remote operator to resolve. Each case gets its own most-graceful local behavior, and all of them log a domain event and raise an operator-visible flag. They never hard-stop the fleet.
Anomaly | Graceful local behavior | Surfaced as |
|---|---|---|
Map update removes/moves a tag a running instance uses | Guards referencing it fail closed (wrong epoch/missing) | Operator flag; resolve by migration or re-tag |
Robot claims a non-conforming transition | Accepted and recorded (loud); token may move to | Conformance event; operator can resolve |
Robot drops a carried tote | Explicit | Operator flag |
Leased task abandoned (never claimed) | Lease TTL-expires; work returns to the pool | Conformance/timeout event |
Perception asserts something implausible | Fact fails plausibility/freshness; guard fails closed | Operator flag |
Token stranded by migration |
| Operator/agent resolves |
Quarantine and other reserved-place residents are bounded the same way as sink places (\u00a712): they are visible, alertable, and reaped on a policy so they don't become silent WIP leaks.
A workflow exists at two levels, and the data model must keep them distinct.
A definition is an immutable, versioned template \u2014 the canonical Petri net, authored against a particular map. The net is evaluated from a Starlark source (Appendix I): the author (human or agent) writes Starlark, the server evaluates it once into a normalized Petri net (the canonical IR) and hashes that, and the immutable artifact is the evaluated net, not the source text. New versions are appended; a version is never mutated. An instance is a running instantiation of one definition version on a map within a deployment. The instance holds the strongly-consistent markings and leases; the definition is just the template it currently conforms to. One definition may have many instances (testing, simulation, physical twins).
Starlark rather than hand-written YAML because definitions are now mostly machine-authored and increasingly parametric: a Python-subset language gives loops, conditionals, and reusable macros (one source emits N near-identical work cells), evaluates hermetically and deterministically (no clocks, no I/O \u2014 see Appendix I), and is far easier and more familiar for an authoring agent to emit, diff, and validate than soupy YAML. The runtime never sees Starlark; it sees the evaluated IR, so execution, conformance, and migration are unchanged from the Coordinator spec. Existing YAML workflows (the symphony2_*.yaml corpus) are handled by conversion tooling into Starlark/IR; YAML is not maintained as a perpetual first-class authoring input.
Map\u2194workflow references are soft. A workflow references its map's places by name/identity, but there is no hard foreign-key constraint between map and workflow \u2014 their lifecycles are decoupled (definitions immutable, maps versioned independently). References are validated on save (advisory): publishing a definition checks that every place it references resolves to a real place in the target map version, and rejects on a stale from version. If a later map update removes a place a running instance uses, that is handled by the exception principle (\u00a78), not by a constraint that would couple the two lifecycles. (The deeper authoring-agent issue \u2014 the agent omitting map tags so edits don't round-trip, HECO-3376 \u2014 is out of scope here and owned by the forthcoming Studio workflow-editing spec.)
Versioning a definition migrates its instances. Publishing a new version does not silently change anything running. Each live instance migrates explicitly: its markings move from the old version's places to the new version's places, a recorded action, not an automatic rewrite. The migration is a place mapping (old \u2192 new) produced by the same authoring agent that edits the definition, which validates and symbolically simulates the result before proposing it. Migration is advisory: inference proposes, a human approves, the version is published. Switchboard's role is narrow and mechanical \u2014 it applies the agent-supplied mapping transactionally to the strong markings and records it. Unmapped tokens quarantine, they don't vanish (__quarantine__).
In-flight leases during migration. Default is drain: stop admitting new transitions, let in-flight leased tasks finish (or expire), then rewrite markings in one transaction and bump definition_version. An operator may force migration, which reclaims outstanding leases (via the same transactional release \u2192 reacquire primitive used for intervention, \u00a710) and quarantines any token whose task was interrupted. Migration rewrites markings only \u2014 facts are untouched (subject/map-scoped, not instance-scoped) and re-project against the new places.
Two stores, two contracts. The coordination core \u2014 markings and leases \u2014 is strongly consistent, the authoritative transactional store. The fact replica \u2014 pose, power, places, everything from robots and maps \u2014 is relaxed, eventually consistent, bounded only by TTL and epoch.
The safety model splits cleanly into two concerns that are often both called "illegal," and separating them is the whole trick:
1. Exclusivity (leases) \u2014 strict, always. Leases serialize dispatch. The only hard safety property is that Switchboard must never offer/hand out the same exclusive claim to two agents. A lease serializes dispatch: one agent wins it, the other is not offered the conflicting work. This never relaxes. Note the precise claim \u2014 leases prevent Switchboard from dispatching conflicting work; they cannot and should not prevent a robot from doing something physical out-of-band and reporting it.
2. Conformance \u2014 advisory and loud. Switchboard is a schematic approximation; the robot knows reality. A robot may claim a transition that does not conform to the current net (wrong place, structural deviation). Switchboard accepts it, records it, moves the token (possibly to __illegal__/__quarantine__), and is loud about it \u2014 it does not silently reject reality. If the robot physically did the thing, refusing to log it only makes Switchboard lie. Two robots that somehow both grabbed the same tote is a real-world event that already happened; recording both loudly is more honest than pretending one didn't. Conformance becomes a metric computed over the event log (Coordinator-spec stance), not a gate.
Deviation is gated by capability. Who may fire a non-conforming transition is an authz decision (\u00a7B):
robot-exec may deviate on its own subject (it knows reality).
operators may intervene and force-resolve.
adapters (SAP/SynaOS) may not deviate \u2014 they only fire their permitted source transitions.
Intervention primitive: transactional release \u2192 reacquire. Handing a stuck robot's leased task to another agent (or a teleoperator), or reclaiming a lease during forced migration, is a single atomic reassignment \u2014 release the held lease and reacquire it under the new holder in one transaction \u2014 not an ad-hoc hack.
Marking intervention and reset. Operators may also edit an instance marking directly when recovering from a mismatch between reality and the net: move a token to another valid place, remove a token, reset an instance's marking from an explicit seed, or retire the instance and start a replacement. These are operator-only interventions, authorized by policy, validated against the compiled workflow when possible, and recorded as events. Normal restarts should prefer "start a new instance and retire the old one"; reset exists for the prototype-style recovery/debug path where preserving the same instance identity is useful.
Guard evaluation runs in two phases:
Eligibility (relaxed, fast). Compute which transitions could fire from the instance's projected facts (\u00a77), freshness applied (stale / wrong-epoch / wrong-map facts fail closed). A read against a relaxed view; may lag.
Commit (strong, transactional). A claim fires the claimed transition: re-read the marking, check the definition's place capacity, compare-and-set on marking.version (optimistic concurrency \u2014 a failed CAS is a retry, not a rejection of intent), acquire/clear/serialize the lease, write the event, update the marking \u2014 all in one transaction. Claims are idempotent by claim id, so a retried /work over flaky WiFi does not double-fire. A relaxed read can only cause a transition to be offered that then fails its strong dispatch check; it can never cause two agents to be dispatched the same exclusive work.
Per-instance guard cache. The coordinator already loads a marking and definition per instance to compute Petri updates; extend that working set with the instance's projected facts. Markings stay authoritative (read strongly at commit); projected facts are the relaxed, cached part serving eligibility only, so cache lag is never a correctness risk. The strong core shards by token/subject, so independent transitions don't contend.
Poses are reasoned about in the ROS2 map frame \u2014 globally consistent (it jumps on localization corrections but does not drift), unlike smooth-but-drifting odom. Frame identity is (frame, map_id) and it is a correctness constraint: two poses are only comparable within the same map version, and Schaeffler will re-record maps (update_map is planned, and SynaOS supplies its own LIF map), so distance/zone guards must require matching (frame, map_id) and fail closed otherwise.
Every fact carries observed_at (sensor time \u2014 what guards compare against), written_at, expires_at/TTL, epoch, and source. Stale facts remain stored but stop satisfying guards. Epoch invalidates cleanly across lifecycle changes: on reconnect a robot bumps its epoch and prior-epoch facts stop counting as fresh even inside their TTL \u2014 so a robot that blinked out can't have its old pose silently satisfy a guard. Epoch also bumps as a tote's token crosses pick/place transitions (\u00a76), which is what makes a stale carried derivation die at the carrier's epoch.
Clock skew. observed_at is the source's sensor time, and robot/server clocks drift; we have already seen tf extrapolation errors and use_sim_time leakage produce nonsensical timestamps. The server treats observed_at as advisory for ordering but enforces freshness against its own clock at write by clamping timestamps too far in the future, and requires observed_at to be monotonic per (subject, epoch). Guards fail closed on stale, wrong-epoch, wrong-map, or implausibly-timestamped facts.
External precondition \u2014 reverb localization cache. Switchboard's freshness guarantees assume that nothing else resurrects stale truth behind its back. Today reverb persists pose to on-disk caches (
~/.cache/reverb_localization_cache.json,/tmp/localization_last_pose.yaml) and republishes a setpose every ~2s, overriding manual init \u2014 which can revive a stale pose across a reboot, exactly the zombie this model is meant to kill. The reverb localization cache must be removed, or made epoch-aware (a cached pose carries its epoch and is invalidated by a bump), before the epoch/TTL guarantees here hold in practice. This is a reverb-side dependency, not solved by Switchboard alone; G.9 is its acceptance test.
A robot's durable relationship is to the fleet, not a workflow. It enrolls once, then polls for site and workflow assignment.
enrolled (fleet member, Backbone-rooted, holds fleet cert)\n \u2502 poll deployment\n \u25bc\nunassigned / idle \u2500\u2500 no workflow; resource facts still replicated at fleet scope\n \u2502 deployed to a site (gains active map(s))\n \u25bc\nlocalizing \u2500\u2500 pulls site map; visual + lidar localization\n \u2502 localization.current fresh AND score \u2265 threshold AND map_id matches\n \u25bc\nlocalized \u2500\u2500 eligible for work\n \u2502 assigned to a workflow (assignment join; robot polls and pulls map+workflow)\n \u25bc\nworking \u2500\u2500 runs the acquire-work loop\n
Three fixes fall out. A robot cannot execute transitions while delocalized \u2014 eligibility is gated on a localization.current fact (fresh, above threshold, for the active map), written by reverb as the self-bound robot-exec (the localization node computes confidence; reverb publishes the fact, same machinery as pose/power). The localizing-stuck case becomes an enrolled-but-idle robot, not a phantom participant. Enrollment is decoupled from tasking \u2014 a robot joins the fleet via its device cert; only then is it deployed and assigned. Assignment is poll-discovered, never pushed.
The work loop is robot-initiated and idempotent. Robots poll POST /work (outbound only \u2014 zero inbound to robots, which suits site NAT/firewalls), carrying facts, lease refreshes, and claims. Fact writes are latest-wins per (subject, key); lease refreshes are idempotent by lease_id; claims are idempotent by claim id. The loop is safe to retry through the WiFi saturation and AP flaps seen on site (pings >300 ms, "bumping" wifi). Switchboard stays dumb: it accepts registrations/assignments and answers "what may I do."
Bounded markings \u2014 sink places. Markings encode current state and must stay bounded. A place may be flagged a sink; tokens that land in a sink expire and are reaped from the marking table after a TTL, and transient subjects (work, load) are reaped with their tokens. Durable subjects (robot, place, scope) persist. History is not kept in the live marking \u2014 the append-only event log is the forever-history and ships to the OLAP/ClickHouse path (Appendix D) for audit and analytics. Schaeffler order history lives in the event log, not in the hot tables. (The prototype does this in-memory; the SQL translation is a periodic reap job / TTL on sink-resident tokens.)
The MVP is a single Rust service (Tokio, axum) over a single PostgreSQL (sqlx) for deployed stacks, while the standalone development binary can run against an in-memory store for rapid iteration. Local full-stack testing uses k3s/k3d so the runtime shape matches the EKS deployment path. HTTP/JSON is the main API \u2014 for the work loop too \u2014 because third parties (SAP, SynaOS adapters, operator UIs) integrate far more easily without being handed gRPC protos, and it caches/load-balances cleanly. JSON on the hot path is a deliberate simplicity bet, not a performance one (if JSON serialization is what brings a robot down, we have much bigger problems); a binary/gRPC framing for the work loop specifically is a possible later, internal-only optimization, never the partner-facing surface.
MVP scope: the single-fact-table schema (\u00a7A.1), no PostGIS and no per-type tables, pose math in the service; app-side UUIDv7; one Postgres primary, no sharding, fleet-as-RLS for isolation; the strong commit path (capacity + marking.version CAS + lease) in one transaction; sink-place reaping; the three actions (enroll/deploy/assign) and idempotent /work. Identity is static (\u00a7B): mTLS robots, a Cognito user pool with static scoped groups (HMND cross-site roles; customer fleet-scoped roles assigned in-app), service-account tokens for adapters \u2014 no customer-IdP federation yet. The same service is k8s-deployable for the cloud single-pane-of-glass and for a future site-authoritative deployment; the typed/PostGIS evolution (\u00a7A.2), read replicas, the per-instance guard cache, customer-IdP federation, and the full VDA5050 layer come only when measured or contractually required.
Subjects persist; a physical thing has one durable identity. Markings coordinate and are bounded; sink places reap. Facts replicate. Static truth is the map's; dynamic object location is the robot's perception. Coordination is strongly consistent; the fact replica is relaxed. Leases serialize dispatch; facts inform; the robot knows reality. Conformance is advisory and loud, not a gate; deviation is gated by capability. Definitions are versioned and immutable; instances migrate explicitly; map\u2194workflow references are soft. Exceptions degrade-and-flag, never hard-stop. Events preserve history; the OLAP log is the forever-record. TTL and epochs prevent zombie truth. Identity is global; authorization is local and, for now, static \u2014 federation is designed-for and deferred. Config authority is central; coordination authority is wherever the robots physically are; authority placement is the Outpost spec's call, and this model accommodates either. At the System-3 boundary, integration is zoned: we obey a partner's traffic authority only inside the regions it owns. Integrations bridge at the System-3 boundary, never into the robot. The coordination plane stays dumb and the registry is data, not schema.
S0 \u2014 single Rust/Tokio service + Postgres in k3s/EKS, plus a standalone in-memory binary for rapid iteration; registry tables; enroll/deploy/assign with the assignment join; single fact table (\u00a7A.1); app-side UUIDv7; definition (Starlark-evaluated, Appendix I) / instance / marking / lease; sink-place reaping; fleet-as-RLS; static Cognito groups; idempotent /work.
S1 \u2014 three query lenses; strong-commit / relaxed-eligibility guard path with serialize-dispatch / conformance-advisory semantics; (frame, map_id) identity; operator map view; the exception table (\u00a78) wired to operator flags.
S2 \u2014 robot lifecycle (enroll \u2192 deploy \u2192 poll \u2192 localize-gate \u2192 assign \u2192 work); localization.current written by reverb (reverb cache precondition resolved); epoch invalidation; carried-tote mode/snapshot + carried \u2192 dropped; map-version + workflow-place layering; instance migration with quarantine; transactional release \u2192 reacquire intervention.
S3 \u2014 SAP source adapter (Appendix E.1); SynaOS coexistence floor \u2192 mid (occupancy facts \u2192 virtual-fleet broadcast, Appendix E.2); typed snapshot evolution where measured (\u00a7A.2); per-instance materialized guard cache; site-authoritative placement with the Outpost spec; customer-IdP federation when contractually required.
-- Registry (config; mostly static; scopes use human slugs)\ncreate table tenant ( id text primary key, name text not null );\ncreate table fleet ( id text primary key, tenant_id text not null references tenant, name text not null );\ncreate table site ( id text primary key, tenant_id text not null references tenant, name text not null );\ncreate table map ( id text primary key, site_id text not null references site, version int not null,\n assets_uri text, recorded_by text, unique (site_id, version) ); -- many maps per site; rasters in object store\ncreate table workflow ( id text primary key, map_id text not null references map,\n version int not null,\n source_starlark text not null, -- authored source (Appendix I)\n definition jsonb not null, -- canonical net (IR) evaluated from source\n canonical_hash text not null,\n unique (id, version) ); -- immutable per version\n\ncreate table deployment ( id text primary key, fleet_id text not null references fleet,\n site_id text not null references site, active_map_id text references map,\n status text not null default 'active' ); -- one site per fleet deployment; multiple maps allowed\n\n-- Robots are fleet resources; device_ref points outward to Backbone\ncreate table robot ( id uuid primary key, fleet_id text not null references fleet,\n device_ref text not null, label text, role text not null );\n\n-- NEW: robot -> workflow assignment (the third action; poll-discovered, never pushed)\ncreate table assignment ( id uuid primary key, fleet_id text not null,\n robot_id uuid not null references robot(id),\n workflow_id text not null references workflow(id),\n instance_id uuid, -- the running instance, once instantiated\n status text not null default 'active',\n assigned_by text not null, assigned_at timestamptz default now(),\n unique (robot_id) where status = 'active' ); -- a robot runs one active assignment at a time\n\n-- A running instantiation of a definition version on a map (holds the strong markings)\ncreate table instance ( id uuid primary key, fleet_id text not null,\n workflow_id text not null references workflow(id), definition_version int not null,\n map_id text not null references map(id), deployment_id text references deployment(id),\n status text not null default 'active', created_at timestamptz default now() );\n\n-- Subjects: app-side uuidv7 identity, kind + label + external_ref; one durable identity per physical thing\ncreate table subject (\n id uuid primary key, -- uuidv7, generated app-side (plain uuid column)\n fleet_id text not null,\n kind text not null, -- resource | load | place | work | scope\n label text, -- human name ('r12'); mutable, not identity\n external_ref text, -- customer/protocol id (AGV serial, SAP order, barcode)\n status text not null default 'active', -- active | terminal (left the building / fulfilled)\n epoch int not null default 0 );\ncreate index on subject (fleet_id, kind) where status = 'active';\ncreate index on subject (fleet_id, external_ref); -- integration lookup\n\n-- Replicated facts: ONE table, JSONB bodies, RELAXED. SoT at robot self-telemetry / robot perception / map.\ncreate table fact (\n fleet_id text not null,\n subject_id uuid not null references subject(id),\n key text not null, -- 'pose.current', 'power.current', 'localization.current', 'carried_by', 'tag', 'occupancy', ...\n value jsonb not null,\n mode text, -- locatable facts: 'absolute' | 'relative' (relative = derived from carrier, \u00a76)\n source text not null, -- writing executor: 'robot:r12' | 'map:herzo@v17' | 'operator' | 'sap-adapter'\n observed_at timestamptz not null,\n written_at timestamptz not null default now(),\n expires_at timestamptz, -- null = non-perishable (map-sourced static, or collapsed absolute snapshot)\n epoch int not null default 0,\n map_id text, -- set on locatable facts; carries frame identity\n primary key (fleet_id, subject_id, key) );\ncreate index on fact (fleet_id, key);\ncreate index on fact (fleet_id, map_id) where map_id is not null;\n\n-- Switchboard-authoritative coordination state: STRONG. Bounded: sink-resident tokens are reaped (\u00a712).\ncreate table marking (\n fleet_id text not null, instance_id uuid not null references instance(id), subject_id uuid not null,\n net text not null, place text not null, -- place may be '__quarantine__' / '__illegal__' / a sink place\n token_type text, payload jsonb not null default '{}',\n version bigint not null, -- optimistic concurrency (CAS at commit)\n entered_at timestamptz default now(), -- for sink-place TTL reaping\n primary key (instance_id, subject_id, net) );\n\ncreate table lease (\n fleet_id text not null, instance_id uuid not null references instance(id), lease_id uuid primary key,\n subject_id uuid not null, task text not null, holder text not null,\n acquired_at timestamptz not null default now(), expires_at timestamptz not null );\n\n-- Places may be sinks; sink-resident tokens reap after ttl (def can also be derived from the IR)\ncreate table place_meta (\n map_id text not null, place text not null, is_sink boolean not null default false,\n sink_ttl interval, capacity int, primary key (map_id, place) );\n\n-- History (transitions + migrations; NOT high-frequency facts \u2014 see Appendix D). Ships to OLAP = forever-history.\ncreate table event (\n fleet_id text not null, event_id bigserial primary key, instance_id uuid not null,\n transition text, -- a workflow transition, '__migrate__', or a recorded deviation\n conformant boolean not null default true, -- false = recorded illegal/deviation (loud), \u00a710\n subject_id uuid, claim_id uuid, old_value jsonb, new_value jsonb,\n actor text, source text, observed_at timestamptz, written_at timestamptz default now(),\n trace_id text );\ncreate unique index on event (instance_id, claim_id) where claim_id is not null; -- claim idempotency\nThe three lenses over this:
create view fact_fresh as -- freshness applied once (relaxed read)\n select f.* from fact f join subject s on s.id = f.subject_id\n where (f.expires_at is null or f.expires_at > now()) and f.epoch = s.epoch and s.status = 'active';\n\nselect * from fact_fresh f join subject s on s.id=f.subject_id where f.fleet_id=:f and s.kind='resource'; -- fleet lens\nselect * from fact_fresh where fleet_id=:f and map_id=:m and key='pose.current'; -- map lens\n-- workflow lens: marking for the instance \u2295 projected facts for its subjects\n
Pose is JSONB (value->>'x'); spatial math is in the service. A migration is one transaction over marking (rewrite place per the agent's mapping, unmapped \u2192 __quarantine__, bump instance.definition_version) plus a __migrate__ event. A reap job deletes sink-resident tokens past sink_ttl and marks their transient subjects terminal. This schema carries the pilot.
Versioning note. Three counters could exist \u2014 map
version, factepoch, and the v0tag_set_version. This revision dropstag_set_version: tag/place currency is fully captured by the mapversion(tags are map-sourced place facts, invalidated on map-version bump) and per-factepoch(lifecycle/reconnect). A separate tag-set counter is redundant.
The read interface (fact / fact_fresh) is invariant to where a fact physically lives, so a fact graduates arbitrary \u2192 registered (schema + TTL) \u2192 promoted built-in without breaking consumers. Promotion moves a hot/spatial fact family into a typed snapshot table clustered by co-write group:
pose_current \u2014 kinematic snapshot, geometry(PointZ, 0) (SRID 0 = local Cartesian meters), GiST on the many static places (not the few hot robots), frame columns, mode for absolute/relative; attached-pose composition in the client.
device_current \u2014 telemetry snapshot (battery/power/health) as columns; a new scalar built-in is ADD COLUMN, not a new table.
link_carried_by \u2014 promote links from facts to a typed table with FK integrity (carrier_id references subject).
Each promoted table gets a UNION ALL arm in the fact/fact_fresh view, so the lens query count doesn't grow with built-ins. Do not promote until the generic table is a measured bottleneck.
Identity is established at the door (Appendix H) and reduced to a principal. Everything past that is one model with four enforcement tiers, with a Cedar policy engine for the fine-grained tier. For MVP the model is static \u2014 no customer-IdP federation, no claim mapping; the scoped-group/containment structure below is the seam that lets federation drop in later (~2027, EU posture) without reshaping anything.
Authn (ALB). Robots by mTLS, humans by OIDC (Cognito), adapters by a service-account token (Appendix H).
Isolation (Postgres RLS) \u2014 the hard fleet wall. The principal's fleet is a session GUC; every table carries fleet_id; even a policy bug cannot cross fleets.
select set_config('app.fleet', 'fleet:herzogenaurach', false);\nalter table fact enable row level security;\ncreate policy fleet_isolation on fact\n using (fleet_id = current_setting('app.fleet', true))\n with check (fleet_id = current_setting('app.fleet', true));\nCapability (Cedar) \u2014 the policy brain. Fine-grained permit/forbid over (principal, action, resource, context), evaluated per route.
Integrity (commit). Capacity / lease-serialize / marking.version CAS at strong commit (\u00a710).
Three principal types, each built at the door: Robot (mTLS SVID), User (OIDC token), Service (adapter tokens).
Groups are static and scoped, with a containment hierarchy:
HMND users get cross-site roles: admin, teleoperator, operator (members of HMND-tenant groups).
Customer users get fleet-scoped roles: fleet-admin, fleet-operator, assigned in-app by someone with the right.
Cedar's entity hierarchy gives transitivity on two axes:
Group membership (principal in Group): a robot's role bundles capabilities; user groups nest (fleet-operator@herzo in fleet-admin@herzo). Membership is directly assigned (group_membership), not derived from external claims (federation deferred).
Resource containment (resource in Scope): Subject in Fleet in Tenant; Instance in Workflow in Map in Site in Tenant; Fact in Subject; Lease in Instance. A policy at Fleet/Tenant transitively covers everything beneath.
Resources: Tenant, Fleet, Site, Map, Workflow, Deployment, Assignment, Instance, Subject, Fact, Lease.
Actions: Enroll, Deploy, Assign, PollDeployment, WriteFact, FireTransition, Deviate, AcquireLease, Claim, Intervene, InjectWork, PublishDefinition, Migrate, AddPlace, ReadFleet, ReadMap, ReadInstance, and Admin*. Deviate (fire a non-conforming transition, \u00a710) and Intervene (transactional release\u2192reacquire) are distinct, capability-gated actions.
// A robot acts only on its own subject, within its fleet \u2014 including deviation\npermit (principal, action in\n [Action::"WriteFact", Action::"FireTransition", Action::"Deviate", Action::"AcquireLease", Action::"Claim"], resource)\nwhen { resource in principal.fleet && context.subject == principal.subject };\n\n// Operators: fleet-scoped operator actions + intervention (release->reacquire)\npermit (principal in Group::"operators", action in [Action::"OperatorActions", Action::"Intervene"], resource)\nwhen { resource in principal.fleet };\n\n// Adapters may inject work / read status \u2014 and may NOT deviate (hard cap via forbid)\npermit (principal == Service::"sap-adapter",\n action in [Action::"InjectWork", Action::"ReadInstance"], resource);\nforbid (principal == Service::"sap-adapter", action, resource)\nunless { action in [Action::"InjectWork", Action::"ReadInstance"] };\nThe subject-binding condition (context.subject == principal.subject) keeps a robot from writing or deviating on another robot's subject \u2014 the same intent hand-namespacing (robot1/foo) approximated, now enforced.
Two correlated, append-only logs (a regulated customer like Schaeffler expects this):
Authz decision log. Every Cedar decision: principal, action, resource, context, allow/deny, determining policy ids, timestamp, trace_id. Who was permitted to do what.
Domain event log (event). actor/source/claim_id/conformant/trace_id on every transition (conformant or recorded-deviation) and migration. What actually changed, and by whom.
Correlate by trace_id. Both are high-volume on the hot path (every /work authorizes) and therefore ship to the OLAP path, not the OLTP store. RLS bounds reads to the fleet.
A workflow definition names an executor on every work_type and task \u2014 the actor that owns a token class and is bound to the transitions it may fire. Executors are not robot-only: a robot is robot-exec, a WMS is wms-exec, an operator console operator-exec, a button/PLC button-exec. The executor maps onto a Cedar principal; the subject-binding policy (B.4) holds a robot-exec to its own token. External executors carry no token \u2014 they drive a transition by writing a guarding fact or claiming completion. This is the System-3 boundary expressed as authorization rather than a special case.
Because facts are keyed by (subject, key), the writer is not part of the key \u2014 so fact-write authority must be declared, not inferred. Per fact key (or prefix), the definition states which executor(s) may assert it about which subject kinds:
robot-exec writes its own pose, power, localization.current (self-bound; the localization node computes, reverb publishes), and its own perceived load locations (dynamic object location is the robot's, \u00a76).
the map version writes static place facts (tags, zones).
an operator writes overrides.
The writing executor is recorded as the fact's source. Within the authorized set, last-write-wins by written_at; two different executors contending for the same (subject, key) is a definition smell to catch at validation, not a runtime race \u2014 facts inform, they never enforce.
Authentication and authorization stay split. Cognito is the identity provider \u2014 the Backbone-owned pool that holds human credentials and emits the OIDC token; Switchboard never stores a credential. It stores the authorization projection: its own scoped Cedar groups, the containment graph, direct group memberships, and a thin user record for audit.
create table app_group (\n id text primary key, -- slug: 'fleet-operator@herzo' or 'operator' (HMND cross-site)\n tenant_id text not null references tenant,\n scope_kind text not null, -- 'tenant' | 'fleet' | 'site'\n scope_id text not null,\n name text not null,\n unique (tenant_id, name) );\n\ncreate table group_parent ( -- containment edges (Cedar parents); no cross-tenant nesting\n child_id text not null references app_group(id),\n parent_id text not null references app_group(id),\n primary key (child_id, parent_id) );\n\ncreate table app_user ( -- one row per Cognito subject, for audit + direct grants\n id uuid primary key, -- uuidv7\n tenant_id text not null references tenant,\n idp_sub text not null unique, -- Cognito 'sub'\n email text, display_name text,\n is_internal boolean not null default false, -- HMND staff vs customer user\n status text not null default 'active' );\n\ncreate table group_membership ( -- DIRECT, in-app assignment (no claim mapping at MVP)\n user_id uuid not null references app_user(id),\n group_id text not null references app_group(id),\n granted_by text not null, granted_at timestamptz not null default now(),\n primary key (user_id, group_id) );\n
Deferred \u2014 customer-IdP federation (~2027). Brokered federation (customer SAML/OIDC \u2192 Cognito external provider, per-tenant), claim\u2192group mapping, and SCIM deprovision are designed-for but not built. The scoped-group + containment structure above is precisely what lets them drop in: a future
idp_provider+idp_group_mappingresolve external claims onto these same scoped groups at request time, structurally tenant-bound (an IdP can only map onto groups in its own tenant). EU security posture makes this likely next year; it is out of MVP scope. Until then, all membership is direct in-app assignment, and HMND staff authenticate against our own pool (break-glass independent of any customer SSO).
GET /v1/admin/users # list users in the caller's tenant (status, groups)\nPATCH /v1/admin/users/$id # enable / disable\nGET /v1/admin/groups\nPOST /v1/admin/groups/$g/members # direct grant a user into a group\nDELETE /v1/admin/groups/$g/members/$user # revoke\nPOST /v1/admin/robots/$r/role # robot capability group\n
Two admin tiers from one hierarchy: tenant admins (a customer's own, or our deployment engineers acting within that tenant) manage direct grants inside their tenant; HMND super-admins authenticate against our pool, administer the hmnd tenant, and are the only principals who may cross tenants. Every call writes both audit logs. The federation/IdP-broker endpoints arrive with the deferred work in B.7.
Refines the Coordinator spec's "tenant = RLS" to fleet = RLS (a tenant may own several fleets); the capability tier is Cedar. Site is not in the identity path, so redeployment doesn't reissue certs. SPIFFE segments use scope slugs and the robot label; the internal id is the app-side uuidv7.
HTTP/JSON (cacheable, easy third-party integration; main API including the hot path). Fleet always from the SVID, never the path.
# Registry (cloud-authoritative config)\nPOST /v1/fleets POST /v1/sites POST /v1/maps POST /v1/workflows\nPUT /v1/workflows/$id?from=$ver # publish a new definition version (Starlark source; evaluated to IR + validated vs map places, Appendix I)\nGET /v1/deployment # what the robot polls (site, active map, assigned instance)\n\n# The three actions\nPOST /v1/fleets/$f/enrollments # ENROLL: mTLS = Backbone device SVID -> fleet cert\nPOST /v1/robots/$r/deployments # DEPLOY: robot -> site\nPOST /v1/assignments # ASSIGN: executor/robot -> running workflow instance (poll-discovered; never pushed)\nGET /v1/robots/$r/assignment # what the robot polls to learn its workflow + map\n\n# Instances, migration, intervention\nPOST /v1/instances # instantiate a definition version on a map\nPOST /v1/instances/$id/migrate # apply agent-authored place mapping; quarantines unmapped\nPOST /v1/instances/$id/intervene # transactional release -> reacquire of a lease (operator)\n\n# Work loop (hot path) \u2014 facts + lease refreshes + claims; idempotent\nPOST /v1/instances/$id/work\n\n# Reads (the three lenses)\nGET /v1/robots # fleet lens (resource facts; idle ok)\nGET /v1/locatables?map=$m&zone=$z&fresh=true # map lens (plotting)\nGET /v1/instances/$id/state # workflow lens (markings \u2295 projected facts)\n\n# Places (a workflow may add an operator-owned place; map places come from the map version)\nPOST /v1/workflows/$w/places\n
POST /work body carries facts (routed to their home by registry, stamped with the subject's epoch and sensor observed_at; locatable facts carry mode), leases (refreshes, idempotent by lease_id), and claims (idempotent by claim_id; a claim may be a recorded deviation, \u00a710). Eligibility is computed from the relaxed projection; the claim commits strongly. Subjects are addressed by uuidv7, or resolved server-side from (kind, external_ref) for integrations.
Pilot (Herzogenaurach) | Design envelope | |
|---|---|---|
Robots per fleet | ~2\u20138 | 50\u2013200 |
Pose write | ~10 Hz/robot | ~10 Hz/robot |
Telemetry write | ~1 Hz/robot | ~1 Hz/robot |
Work-loop poll | ~1\u20132 Hz/robot | ~1\u20132 Hz/robot |
Perceived-load writes | sporadic, per robot at a station | per robot; bounded by stations in view |
Per-robot \u2248 12\u201314 current-state upserts/sec plus sporadic perceived-load writes (each robot writes only loads it perceives \u2014 there is no single world-model fan-out bottleneck, \u00a76). Pilot (8 robots) \u2248 ~110/sec \u2014 trivial for one Postgres. Design (1,000 robots) \u2248 ~13k/sec, contention-free (each robot upserts only its own rows), partitioned by fleet; a single primary per region handles it.
Decisions that hold this: relaxed-fact writes don't contend (PK = subject); the strong core shards by token so independent fires don't contend; HF facts and both audit logs never enter the OLTP path \u2014 pose/telemetry history and Cedar decisions go to the OLAP/ClickHouse path; markings are bounded by sink-place reaping so the hot table doesn't grow with completed work; global reads go to a read replica; the map snapshot materializes at ~1 Hz; the per-instance guard cache absorbs eligibility reads off the hot store. Rate-limit /work per principal, coalesce fact writes latest-wins within a window, keep lease TTLs short.
Latency (measured, office): robot\u2192local LAN \u2248 3 ms; robot\u2192local WiFi \u2248 25 ms; robot\u2192cloud London \u2248 33 ms; robot\u2192cloud US \u2248 88 ms. WiFi dominates; a UK-region cloud adds ~8 ms, so the work loop tolerates WAN \u2014 on-prem is an availability decision (Outpost spec / Appendix F), not a latency one. JSON serialization on this path is immaterial next to WiFi.
Integrations attach at the System-3 boundary as adapters \u2014 Switchboard is the seam, and robots never talk to SAP or SynaOS directly. Each adapter is a capability-bounded principal (Appendix B); adapters may not deviate (\u00a710). There are two, deliberately distinct.
SAP/WMS is where orders originate. The adapter is a System-3 principal authorized only to inject work and read status \u2014 the same shape as mock-wms in the martur_fompak stack. A SAP order becomes a work token via a source transition (claimed instantaneously, no lease), with the SAP order number on the subject's external_ref; fulfilment status is reported back from the marking and the OLAP event history. Robots fulfil the work without any knowledge of SAP.
Our robots share floor with Schaeffler's KUKA AGVs under SynaOS. The SoW commits us to integrate; SynaOS wants deep integration; the right engineering posture is a continuum, and authority is zoned, not global: SynaOS is the traffic authority only in the regions it owns (its transit aisles), and is not authoritative over our work cells. A place/zone therefore carries an authority-owner attribute, and a single mechanism spans the whole continuum \u2014 the only thing that changes between tiers is how much of the map SynaOS owns and how hard we work to align the two maps.
The continuum (pilot targets the floor + a slice of mid):
Floor \u2014 heuristic coexistence (pilot scope). Treat AGVs/AMRs as dynamic obstacles: announce presence, pull over, avoid. This is the same behavior we need for all dynamic obstacles (humans, manual trains), not just KUKA AGVs, and it is viable given the low AMR density at the pilot sites. The only Switchboard-side state this needs is node/zone occupancy facts \u2014 a robot broadcasts which zones it occupies so peers (ours and, where bridged, SynaOS) avoid deadlocks. This is the answer to the v0 open question "does AWx need AWy's state to break deadlocks": yes, via occupancy facts served at the edge. (This matches the 2026-06-05 pilot decision: heuristic coexistence, "no full VDA 5050 integration," node-occupancy broadcasting, SynaOS mock against the LIF format.)
Mid \u2014 "virtual fleet" state broadcast. A VDA5050 adapter presents our robots to SynaOS as a virtual fleet: SynaOS sees state updates synthesized from our replicated facts (pose, power, carried_by, marking progress), so it knows what our robots are doing \u2014 but the integration is intentionally incomplete: we broadcast discrete state, we do not yet coordinate over map locations.
Ceiling \u2014 zoned base/horizon coordination (future / Schweinfurt / MSA scale). Align the two very different maps (SynaOS LIF nodes/edges \u2194 our spatial + mindmap + 3D maps), represent SynaOS nodes as places (semantic anchors, perception-refined), and in SynaOS-owned zones request and obey base/horizon as lease tokens (an execution reservation), with edge traversal evaluated by conformance ("close enough"), not exact match. In our work cells, SynaOS holds no authority and base/horizon don't apply. This is the hard tier and is explicitly post-pilot.
The mapping onto this doc's model, where it applies:
VDA node (LIF) \u2192 a place with authority_owner = synaos in SynaOS-owned regions.
VDA order \u2192 a work token via a source transition.
VDA action (e.g. pick) \u2192 an internal workflow; the adapter reports a clean action result and hides internal transitions.
VDA base/horizon \u2192 lease tokens only inside SynaOS-owned zones.
VDA state/telemetry \u2192 synthesized from replicated facts + marking progress.
Two boundaries to hold regardless of tier: VDA navigation gets the robot roughly there at AMR tolerances \u2014 task-level cm precision is local manipulation refinement, never grasp from the map; and customer-specific shims (SynaOS) sit on top of any VDA5050 core, expected to vary per site. This is distinct from Safe Coexistence (HMND-19), the runtime AGV/AMR coexistence behaviour; E.2 is the protocol/authority by which the customer fleet and ours coordinate.
This model accommodates either authority placement; it does not own the placement decision, node topology, or HA tiers \u2014 those belong to the HMND Outpost & Site Authority spec, which is canonical for them. HA/node-count is an infrastructure axis orthogonal to this state model.
Only one thing: where the writer of operational truth lives. Two cases, and the state model behaves identically in coordination terms under both:
Cloud-authoritative. Robots reach the cloud coordinator over the (locked-down, proxied) site WAN. Acceptable where the WAN is reliable and ~33 ms (UK region) is fine. Phase 1 of the Schaeffler pilot leans here.
Site-authoritative. A local Switchboard runs at the site and is authoritative for that fleet's live coordination; the cloud observes via async, one-directional sync. Survives WAN loss. Phase 2 shifts here to hit availability KPIs.
Registry / config (definitions, maps, fleet membership, groups) \u2014 cloud-authoritative, pushed down; the site reads a cached copy.
Live coordination state (markings, leases, facts) \u2014 local-fast; site-authoritative when a site authority is present (markings may be local-fast even under cloud authority, per the Outpost Tier-1 contract). Streamed up to the cloud read-only.
One authority per data class means sync is one-directional per class with no merge conflicts \u2014 an event/log stream, not two-master reconciliation. Epoch and version travel with the records so the observer can order and dedupe.
For a site authority to enroll robots and mint fleet certs during a WAN outage, Backbone issues it an intermediate CA, granted only for airgap/in-factory/offline operation (the Outpost spec's "certificate pi\u00f1ata" warning applies \u2014 the default is not to hand every site a long-lived intermediate). The default is cloud-mints, edge-validates. Enrollment and the work loop keep functioning offline; revocation/rotation reconcile on reconnect.
Fleet/site cardinality \u2014 resolved (\u00a74): one site per fleet deployment, multiple maps allowed.
Fleet SVID minting \u2014 resolved by the Outpost spec: cloud mints by default, edge validates; delegated local intermediate only for offline modes.
Cross-robot deadlock info \u2014 resolved (\u00a7E.2 floor): node/zone occupancy facts, served at the edge.
Sync transport and lag budget \u2014 owned by the Outpost spec.
Map places: replica vs reference \u2014 replicate-on-version-load (uniform query, chosen in A.1).
Decision. Sync across the site\u2194cloud boundary happens at the application layer, over the same authenticated HTTP surface everything else uses \u2014 never DB-layer replication. Two rules: DB replication (read replicas, HA) is permitted only inside a trust-and-version domain; anything crossing the site\u2194cloud boundary \u2014 different trust, different versions, different authority \u2014 goes through the application protocol. Rationale: the sync contract is per data class with a direction (F.2), which WAL-level replication cannot express; the site authority becomes just another principal (SVID at the edge, Cedar-capped to its own fleet, RLS, both audit logs) so the sync channel inherits the entire four-tier security model instead of needing a parallel one for a database connection; a versioned JSON surface tolerates version skew across N customer sites where synchronized upgrades are impossible; outbound-only HTTPS from the site matches the same NAT/firewall posture as the robots; and authority transfer (Phase 2 cutover) becomes an epoch-scoped recorded event rather than a database failover. The work loop is the template: a site authority syncs to the cloud the way a robot syncs to Switchboard \u2014 the same idempotency, epoch, and latest-wins machinery, one level up.
Markings sync by replay. The event log is the transport. The cloud never stores authoritative markings for a site-authoritative fleet; it holds a read-model derived by folding events \u2014 G.6's oracle ("the event log replays to the same marking") promoted from an invariant to the mechanism, and kept honest as a property test (replay(log) == marking) run against every store. Four conditions make the fold sound:
Replay is a mechanical fold, never re-execution. Events record what happened, with a payload sufficient to apply blind \u2014 replay never re-evaluates guards, consults facts, or needs the workflow definition. All commit-time decisions are baked into the event at write time; migrations, resets, and interventions are events carrying their inputs (the place mapping, the seed).
Reaping is an event, not a clock. Sink-place reaping must appear in the log (a __reap__ event) so an observer neither re-implements TTL with its own clock nor fails to prune. No clocks in anything that must replay deterministically.
Gap detection needs a per-instance monotonic sequence. Time-ordered UUIDv7 event ids order the stream but cannot prove completeness. A per-instance seq makes the cursor protocol self-verifying: the observer sees 41 then 43 and requests backfill.
Bootstrap is snapshot + suffix. Periodic marking checkpoints (snapshot + the cursor it corresponds to) let a new or lagging observer join without genesis replay, and bound site-side log retention \u2014 the OLAP path remains the forever-record.
Facts sync as an LWW map. The fact replica is merged with merge(a, b) = max_by(epoch, observed_at, writer) \u2014 a join-semilattice (commutative, associative, idempotent), i.e. a state-based CRDT without importing one. The design already supplies what makes LWW safe: write authority is single-writer per (subject, key) (two writers contending is a definition smell, \u00a7B.6), observed_at is enforced monotonic per (subject, epoch), and epoch dominates across reconnects \u2014 so "last" never depends on synchronized wall clocks. TTL/epoch make tombstone GC unnecessary: an old entry is inert, not dangerous. Do not adopt a generic CRDT library (vector clocks, OR-sets); the domain-specific causality is smaller, already implemented, and easier to audit.
Leases and claims never sync as writable state. They are meaningful only inside the single strong authority. The cloud observes a site's leases read-only \u2014 effectively more TTL-governed facts. The moment a lease looks writable on both sides of a partition, the one-authority-per-class design has been silently abandoned.
Protocol summary. Up: the event stream (per-instance seq + cursor; the same stream feeds the webhook outbox and the OLAP export \u2014 sync is a third consumer of the J.3/J.4 outbox-and-cursor machinery, not a new subsystem) plus coalesced fact deltas merged LWW. Down: immutable config versions, pulled. Bootstrap/recovery: snapshot + cursor, then suffix replay. Partition behavior: the cloud twin goes TTL-stale and says so \u2014 a partitioned site degrades in the operator view exactly like a robot that went dark; on reconnect, replay the gap and merge \u2014 nothing that syncs is multi-master, so there is no reconciliation logic.
Work items implied (all small, all site-side): the per-instance event seq column; reap-as-event; marking checkpoint snapshots; the replay-equivalence property test. Transport choice and lag budget remain the Outpost spec's (F.4); this section fixes the mechanism.
Scenarios double as the use-case catalogue and the integration-test suite, drawn from real pilot/demo contexts: Symphony 2 two-robot tote handling (east/west, aw7/aw12), RoboStore/ring picking demos, the martur_fompak stack with mock-wms, the June Schaeffler navigation workshop, and recurring deployment/troubleshooting threads. Each is preconditions \u2192 steps \u2192 oracle and asserts a model property.
Pre: r12 working, holding a NavigateToStack lease, carrying load:tote_3. Steps: take r12 offline past TTLs; bring it back (epoch 42\u219243), re-localize, re-assign. Oracle: during downtime r12's pose leaves fact_fresh; the lease expires and work returns to the pool; r12's marking is unchanged; tote_3's relative pose resolves unknown. On return, epoch-42 facts are not fresh even within TTL; r12 is ineligible until localization.current(epoch=43) is fresh and above threshold (written by reverb); then it resumes. No phantom participation.
Pre: deployment on herzo@v17; r7, r12 localized. Steps: record herzo@v18; point active_map at v18; robots re-pull and re-localize. Oracle: v17 place facts invalidate by map epoch and v18 places replicate in; a pose with map_id=v17 is rejected for a v18 guard (frame identity, fail closed); each robot is ineligible until it re-asserts localization.current(map_id=v18); workflow places re-resolve against v18; no cross-version pose comparison succeeds. (No tag_set_version needed \u2014 map version + epoch suffice.)
Pre: r12 working A on M1; r7 working B on M2; both maps in site:herzo. Steps: operator records a new assignment r12\u2192B; r12 polls, de-registers from A, pulls M2+B, re-localizes, runs B. Oracle: r12's A-markings release (or quarantine if mid-task) and its A lease frees; r12's fleet-scoped resource facts survive the switch (not workflow-owned) \u2014 proving the replica/projection split; r12 is ineligible in B until localized on M2; r7 unaffected; the fleet lens shows both robots continuously. (This is the single-workflow regression, now supported.)
Pre: instance i1 on empty_stacks v3, tokens in P1,P2,P3; agent publishes v4 renaming P2\u2192P2', removing P3. Steps: PUT v4 (validated vs map places); agent supplies {P1\u2192P1, P2\u2192P2'}; POST /migrate (drain). Oracle: drains in-flight leases, then quiesces; tokens move P1\u2192P1, P2\u2192P2'; the P3 token \u2192 __quarantine__; definition_version=4; a __migrate__ event records the mapping; facts untouched; eligibility recomputes against v4. Forced migration uses release\u2192reacquire and quarantines the interrupted token.
Pre: two robots, two stacks, central table with N slots, hall locks. Steps: both concurrently reserve totes/slots and attempt the same hall. Oracle: no tote/slot reserved twice (lease serializes dispatch \u2014 even though pose facts are relaxed); the second hall claimant is not offered the lock while held; when one robot drops (G.1) its reservations release by TTL; total delivered never exceeds available totes/slots.
Pre: running instance, idle robots; a WMS principal authorized only for NewOrder. Steps: WMS injects NewOrder \u00d73 (instant, no lease), each with a SAP order number on external_ref. Oracle: three work:order tokens via source transitions; the WMS principal cannot fire any other transition and cannot deviate; orders fulfilled by eligible robots; status maps back by external_ref; the event log replays to the same marking; fulfilled tokens reach a sink and reap, history retained in OLAP.
Pre: the SynaOS adapter registered as a System-3 principal; LIF aligned to herzo@v17; r7 localized. Steps (pilot floor): r7 traverses a SynaOS-owned aisle while a KUKA AGV approaches; r7 broadcasts zone occupancy and pulls over. Oracle: the occupancy fact is visible to peers; r7 treats the AGV as a dynamic obstacle and yields; inside the SynaOS-owned zone SynaOS is the traffic authority; once r7 enters its work cell, SynaOS holds no authority; no VDA semantics leak into the robot stack; the adapter cannot deviate. (Ceiling-tier base/horizon\u2192lease is exercised only in a future test once map alignment exists.)
Pre: r7 localized and moving; r12 enrolled but stuck localizing. Steps: operator opens the map lens for herzo@v17. Oracle: the map returns r7 at a fresh world pose plus static places; r12 does not appear as a located pose \u2014 it shows enrolled/idle in the fleet lens; a load carried by r7 resolves through its carrier (relative mode); nothing plots a stale or wrong-map pose.
Pre: r12 working; pose TTL 2 s; reverb localization cache removed or epoch-aware (\u00a711 precondition). Steps: r12 blips for 1.5 s (inside TTL) and returns with an epoch bump. Oracle: even within its TTL window the prior pose is non-fresh because of the epoch bump; r12 re-asserts at the new epoch before any guard trusts its pose; no on-disk cache republishes the stale pose; no flicker. (This test fails against today's reverb until the cache precondition is met \u2014 that is the point.)
Pre: r12 working; a CompletePick claim in flight. Steps: r12 sends the claim; the response is lost; r12 retries the identical claim. Oracle: the transition fires exactly once (idempotent by claim_id); fact upserts are latest-wins; a refreshed lease is extended, not duplicated; the event log shows one CompletePick.
Pre: load:tote_3 absolute-located on table_a; r7 localized and approaching. Steps: r7 fires pick (consume absolute token, produce relative/carried, assert carried_by=r7), navigates, fires place on table_b. Then a second run where r7 drops the tote mid-transit. Oracle (place): while carried, the tote's pose derives from r7's pose and is in relative mode; on place the pose collapses to a frozen absolute snapshot at r7's location and stops re-deriving; the tote is one subject throughout. Oracle (drop): a carried \u2192 dropped transition records last-known position and raises an operator flag \u2014 not silent staleness.
Pre: r12 working; the net expects Pick before Place. Steps: r12 (knowing reality) claims Place from a state the net considers illegal. Oracle: Switchboard accepts and records the claim with conformant=false, moves the token (possibly to __illegal__), and raises a flag \u2014 it does not reject reality; conformance metrics over the event log reflect the deviation; a wms-adapter attempting the same Deviate is denied (capability-gated); exclusivity (any lease involved) was still serialized.
Single Rust service (Tokio, axum) over one Postgres (sqlx) in k3s/EKS, Cedar for route authorization, and an in-memory standalone mode for rapid local testing. HTTP/JSON throughout, hot path included.
Robots \u2014 mTLS. ALB terminates mTLS and forwards the SPIFFE id (fleet SVID); the service parses it into a Robot principal + fleet + label. Robots reach only enrollment, deployment/assignment poll, and the work loop; outbound-only (zero inbound \u2014 suits site NAT/firewalls).
Humans \u2014 OIDC (Cognito). ALB validates the session and forwards signed claims including static group memberships; the service builds a User principal with its groups and scope. (Customer-SSO federation deferred, B.7.)
Adapters / System-3 \u2014 service-account token. SAP and SynaOS adapters present a token and become a capability-bounded Service principal; they cannot deviate.
ALB (mTLS | OIDC | token) \u2500\u2500headers\u2500\u2500\u25b6 axum middleware\n 1. authenticate at ALB \u2192 identity headers\n 2. middleware builds Principal (Robot | User+static groups | Service)\n 3. set Postgres session app.fleet GUC (RLS isolation)\n 4. Cedar authorize(principal, action, resource, context) (capability; includes Deviate/Intervene)\n 5. handler runs; strong commit in one txn; RLS backstop (integrity)\n 6. emit Cedar decision + domain event, correlated by trace_id, to the OLAP path (audit)\n
A.1 schema only (no PostGIS, no typed snapshots); one Postgres primary (no sharding, no replica); fleet-as-RLS; cloud-authoritative footprint (Phase 1; site authority deferred to Outpost for Phase 2); app-side UUIDv7; static Cognito groups (no federation); sink-place reaping; the three actions + idempotent /work. The same binary is k8s-deployable for the cloud single-pane-of-glass and a future site authority.
Definitions are authored as Starlark and compiled to the canonical Petri-net IR the runtime executes. Starlark (a hermetic Python subset) gives loops, conditionals, and reusable macros without a Turing-complete escape hatch, evaluates deterministically, and is far easier and more familiar for an authoring agent than soupy YAML. The runtime never sees Starlark \u2014 only the evaluated IR.
Pipeline. source (Starlark) \u2192 parse \u2192 evaluate \u2192 normalize \u2192 hash \u2192 validate \u2192 store. The published artifact is the normalized net plus its canonical_hash (A.1 workflow.definition + canonical_hash); the source is retained (source_starlark) for editing/provenance but never executed. PUT /v1/workflows/$id?from=$ver carries the source; the server evaluates, validates that referenced places resolve in the target map version (soft reference, \u00a79), and rejects on a stale from.
Existing YAML. The symphony2_*.yaml corpus is migrated by conversion tooling to Starlark/IR. YAML is not maintained as a perpetual first-class authoring input; it is one possible serialization of the same net, useful for one-time conversion.
Evaluation constraints. Frozen globals; no clocks, randomness, or I/O; bounded execution-step and allocation budget; deterministic collection ordering. load() is disabled or restricted to a controlled module path for shared macros (open question below).
Builtins (DSL sketch).
place(name, capacity=, sink=, sink_ttl=) \u2014 a Petri place; capacity enforced at commit; sink/sink_ttl drive marking reaping (\u00a712).
transition(name, consume=[...], produce=[...], guard=, executor=) \u2014 bound to the executor that fires it (\u00a7B.6).
role(name, capabilities=[...], subject_binding=) \u2014 a capability bundle; capabilities may include Deviate for executors permitted to record non-conforming transitions (\u00a710).
executor(name, role=) \u2014 a named actor bound to a role.
guard helpers (e.g. near(subject, tag)) \u2014 evaluate at author-time into a guard AST, not free-text, so the validator can reason about them.
Example.
# symphony2.star \u2014 authored by the workflow agent, evaluated to the IR\nWORKCELLS = ["table_a", "table_b"]\n\nrobots_idle = place("RobotsIdle", capacity = len(WORKCELLS))\ncarrying = place("Carrying", capacity = 1)\ndone = place("Done", sink = True, sink_ttl = "10m") # tokens here reap\nrole("mobile", capabilities = ["*", "Deviate"], subject_binding = "self")\nrobot_exec = executor("robot-exec", role = "mobile")\n\ndef empty_stack(cell):\n src = place("tote_stack@%s" % cell, capacity = 1)\n dst = place("conveyor@%s" % cell, capacity = 1)\n transition("pick_%s" % cell, # absolute -> relative (carried)\n consume = [robots_idle, src], produce = [carrying],\n guard = near(subject = "robot", tag = "tote_stack@%s" % cell),\n executor = robot_exec)\n transition("place_%s" % cell, # relative -> absolute snapshot\n consume = [carrying], produce = [robots_idle, dst, done],\n executor = robot_exec)\n\nfor cell in WORKCELLS:\n empty_stack(cell)\nDeterminism. Identical source \u21d2 identical canonical_hash. This lets canonical traces act as regression tests and makes the (deferred) snapshot/patch authoring model reproducible.
Open questions.
Shared-macro distribution: allow load() of a vetted module path, or inline per definition?
Guards: author-time AST (preferred) vs runtime strings \u2014 settle before the validator is built.
Evaluation budget defaults and host interpreter (starlark-go / starlark-rust / a Python-native dialect).
Operator-facing SOP/visualization: parallel projection (Studio) or co-located docstrings? (Studio workflow editing \u2014 including the authoring-agent tag round-trip, HECO-3376 \u2014 is a separate forthcoming spec.)
The Rust implementation has added several concrete surfaces beyond the abstract state-model spec above. These are accepted MVP amendments unless later replaced by a narrower production design.
The local runtime has two supported shapes:
standalone binary is the fast local path: one Switchboard process using the in-memory store, with no container runtime or Postgres dependency.
k3s/k3d is the runtime-infra simulation path: ephemeral Postgres via memory-backed emptyDir, a scalable Switchboard Deployment, a ClusterIP service, and Traefik ingress. Kubernetes Ingress does not itself mutate request headers, so the local k3s stack uses Traefik Middleware resources to overwrite the same trusted identity headers that production edge auth provides.
The local trusted-header contract is: x-switchboard-principal-kind, x-switchboard-user-sub, x-switchboard-groups, x-switchboard-fleet, x-switchboard-service, and x-switchboard-robot-id. Direct insecure development access additionally requires x-switchboard-dev-token or Authorization: Bearer <dev-token> when no edge principal header is present.
The MVP embeds a map-server surface in the Switchboard binary instead of running a separate map service. It stores YAML/PGM map bundles in site_map* tables, maintains an active-map selector per fleet, extracts map tags into durable place subjects, publishes selected tags as keypoint.pose facts, and records map mutations in map_audit_log. This is broader than Appendix A's compact map metadata row: the compact workflow map catalog still exists, but map bundles and tag publication are owned by the embedded map-server module.
The implementation includes a durable webhook outbox over workflow events. Webhook subscriptions store projection, event-kind filters, retry bounds, retention windows, timeout, pending backlog limit, and optional signing secret. Deliveries are claimed with database locks, retried with bounded attempts, marked delivered or failed, and subscriptions can pause when backlog bounds are exceeded. Payloads are redacted before delivery; outbound hosts are checked against an allowlist so local/test deployments can opt into explicit receiver hosts.
The MVP records Cedar authorization decisions in a fleet-scoped authz_audit_log before protected operations proceed. The analytics export stub intentionally treats exactly three OLTP sources as exportable: workflow events, authz_audit_log, and map_audit_log. High-frequency fact updates are deliberately excluded from this business/audit export path.
The implementation adds practical catalog hardening that the simplified schema omitted: UUID text validation for externally visible IDs, soft-delete metadata, case-insensitive uniqueness for human names, bounded pagination/filter structs on list endpoints, fleet-scoped uniqueness, a commercial_model on fleets, and a region on sites.
The HTTP surface also includes concrete MVP routes beyond Appendix C's sketch: OpenAPI/Redoc (/openapi.json, /docs), map bundle CRUD and active-map selection, webhook CRUD/delivery inspection, instance event filtering, work-item injection/removal, marking move/reset/retire intervention routes, and the locatables map lens backed by both projected facts and embedded map-server places.
This amendment records the concrete model used by the Rust MVP. Where it disagrees with the earlier sketch above, this section is the source of truth until the spec is rewritten top-to-bottom.
One data model. Switchboard has one native schema and one runtime model. Prototype compatibility is an API translation layer over that model, not a second schema, compatibility database, or schema mode.
Fleets are the commercial boundary. The earlier tenant split is folded into fleets for now. Federation, customer IdP ownership, and cross-customer tenancy can be reintroduced when there is a real commercial or regulatory requirement.
Sites are stable operating locations. Sites remain fleet-scoped and keep a region field as coarse deployment/jurisdiction metadata. Whether this becomes richer jurisdiction policy is deferred.
Site maps are definitions plus versions. A site_map is the named, site-scoped logical map. A site_map_version is the pinned map artifact used by workflows and instances. Names are unique within a site, not globally. Inline bundles stay in the version row for the pilot; S3/LanceDB-backed map assets are a later storage seam.
Workflow definitions are source-only metadata plus versions. A workflow_definition is the named workflow attached to a fleet/site/site-map lineage and records the latest version. A workflow_definition_version stores the authored source, source format, version number, and canonical source hash. The compiled IR is recomputed on load and may be cached in RAM; it is not persisted in the catalog table.
Legacy YAML is first-class for the transition. New authoring is still expected to move toward Starlark, but old Coordinator YAML is accepted as a stored workflow source format and compiled into the same IR. This keeps the existing workflow corpus testable while Studio and the authoring agent move.
Runtime instances are separate from definitions. A workflow_instance pins the exact definition version it started from, may record migrated_from, and owns runtime markings, leases, events, and facts. Publishing a new definition version does not implicitly mutate live runtime state.
Robots are physical assets; executors claim work. Robots are enrolled in a fleet and deployed to at most one site. Executors are instance-scoped work claimants of kind robot or service; robot executors reference a deployed robot, while service executors do not need physical-site lifecycle. Assignment is to workflow instances, not definitions.
Markings carry stable entity identity directly. The separate generic subject/entity table is dropped. entity_id on a marking is the stable UUID that threads token lifecycles across create/destroy transitions.
Facts have two scopes. workflow_facts are instance-scoped facts used by the workflow itself. executor_facts are executor-scoped observations and heartbeats. Both use created_at, created_by, updated_at, and version naming; per-key write authority and stronger frame/epoch modeling are deferred until they are needed.
Place metadata belongs to the workflow IR. Runtime rows reference places by IDs/names resolved from the compiled workflow and map context. Extra metadata that describes a place belongs in the workflow/map source, not in a parallel mutable runtime table.
/v1 is the compatibility API. The old Coordinator/prototype-compatible paths are served at /v1, including old colon-style prototype subpaths where necessary. They translate to the native model and are marked deprecated.
/v2 is reserved for the native API. The native route shape should avoid colon actions and should be designed after the store-port proves the model against Studio, reverb, sockpuppet, and the workflow authoring agent.
Local tools do not require HTTP. switchboardctl is the local compiler, validator, diff, snapshot, simulator, and trace runner for Starlark and legacy YAML. Python callers use LocalSwitchboardTools, which shells out to the configured switchboardctl binary and avoids the service API entirely for validation and simulation workflows.
Deferred items are intentionally explicit: fact epochs and frame identity; tag-relative pose semantics; S3/LanceDB map assets; fleet-by-site sharing and multi-fleet sites; customer tenants and IdP federation; the /v2 resource design; richer workflow syntax such as subnets, entrypoint constraints, actionless transitions, and tag reservations; per-fact-key write authority; and PostGIS or gRPC promotion if measured bottlenecks justify them.
Summary: Managed K8S service for simulation fleets with optional display,
workflow execution, and developer access, replacing direct remote-workstation
and Helm workflows with a service-owned control plane.
Owner: Tony Kinsley
Date:
Approvers: Cody Griffin Dmitriy Shingarey
Informed:
Related:
Source spec: https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1501626408
Fleet Control Plane: https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1170341889
Core Control Plane: https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1151303693
Nexus Sandbox Capability:
https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1963851838
API and CLI: https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/2099871761
Runtime: https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/2099806230
Catalog: https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/2099904528
Workflow runs: https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/2099871787
HMND needs a scalable way for engineers to test robot applications without
depending on direct access to physical robots. The company will never have
enough hardware for every developer, every integration branch, and every
automated test lane to use real robots on demand. A managed virtual fleet lets
us make more tests automatic, repeatable, and deterministic while shortening
the developer loop from "find hardware and reproduce state" to "launch the
scenario and inspect the result."
The current simulation workflow is also constrained by developer machines.
Most engineers work on macOS laptops, while realistic simulation requires an
NVIDIA platform and a Linux graphics/runtime stack. Setting up a host that can
run Isaac Sim is itself a project: GPU drivers, CUDA/runtime compatibility,
Isaac Sim dependencies, remote display, and Wayland/X11 behavior all become
part of the developer experience. The simulator should be a managed service,
not a workstation setup exercise.
Single-machine simulation is insufficient for the systems we need to validate.
Fleet orchestration, robot coordination, Switchboard behavior, policy serving,
Reverb integration, and cloud control-plane interactions all need tests that
can operate fleets of virtual robots. Those fleets should be able to run
against real hosted cloud systems for demos, canaries, product review, and
feedback, while also supporting isolated test environments for development and
CI.
The service needs to support two primary usage modes:
User-operated simulation, where a person launches a complete virtual system
built from prebuilt components and interacts through the same product
surfaces used elsewhere: CLIs, web pages, teleop, logs, and observability.
Developer-targeted simulation, where any component in the virtual system can
be replaced with a development version, such as a local Switchboard server,
policy engine, Reverb instance, robot application, or scenario runner.
The current simulation stack already has useful K8S building blocks:
Kubernetes, Helm, ECR-hosted images, Karpenter GPU pools, and sim_cli
integration. Those pieces prove that GPU-backed interactive simulation can run
in shared infrastructure, but the user-facing abstraction is still a
developer-operated pod workflow.
On main, hmnd_training/infra/helm/simulation deploys an interactive hmnd-sim Kubernetes Deployment. sim_cli wraps this chart and provides
per-user Helm releases, image tag selection, Karpenter GPU profile overrides,
DCV or VNC display mode, workspace persistence, logs, shell access, scene
config injection, and pod readiness reporting.
The current model has several service-level gaps:
the user's local process owns authentication, Helm execution, and Kubernetes
state interpretation;
authentication depends on local awscli and kubectl setup;
each simulator instance is a separate deployment, which makes shared
components across instances difficult;
TTL, idle detection, and cleanup are not central service behavior;
cloud simulation cannot schedule the existing E2E orchestration, assemble
sim/fleet/robot/policy components, seed Switchboard workflows, drive
localization/WBC/Reverb initialization, track workflow progress, or return a
structured pass/fail result.
Detailed E2E workflow behavior is covered in
Simulation Workflow Runs.
Provide a service-owned API, CLI, and web UI for launching and operating
simulator fleets on k8s.
Authenticate developers through a Cognito pool with Google SSO.
Avoid requiring normal users to run Helm or kubectl for deployment,
lifecycle, and happy-path access.
Preserve current developer workflows from sim_cli, including scene setup,
GPU selection, display access, logs, shell/debug access, robot bringup, and
controller launch.
Support workflow execution for E2E and canary simulation runs.
Optimize the developer loop by keeping components small and schedulable
together, so iterating on one component does not require rebuilding and
uploading the full simulator image.
Keep fleet and instance state Kubernetes-native for v1.
Replacing every kubectl workflow. The service should still provide
generated kubectl logs, kubectl exec, and kubectl port-forward
commands for diagnostics and development.
Supporting multi-instance fleets in Phase 1. The API should be designed for
multi-instance fleets, but the first implementation should enforce SimFleet.size == 1.
Making display a required part of every run. Display is a capability that can
be enabled for visual debugging and user-operated simulation, not a fleet
type.
Introducing an external database for authoritative fleet state in v1.
The backend should be implemented as a single binary process with two logical
responsibilities: a user-facing Simulator Service API and a Kubernetes-facing
Simulator Operator. Splitting these names in the spec is about ownership and
interfaces inside the service, not about requiring two separately deployed
services.
The Simulator Service API is the product and workflow layer. Users interact
with it through a web UI and CLI backed by a well-defined API. It authenticates
developers, walks users through scene setup and configuration, exposes catalog
inputs that users can choose from, records fleets, instances, and runs, and
returns connection details for active simulations. Its job is to turn "I want
this simulation" into validated desired state.
The Simulator Operator is the scheduling and reconciliation layer. It consumes
validated desired state from the Simulator Service API and turns it into K8S
workloads. The operator schedules simulator fleets, places workloads on GPU
nodes, wires companion containers or services, watches runtime health, reports
status, and cleans up resources.
The operator will be a Kubernetes "operator" for simulation workloads.
"Operator" is intentionally in quotes: the exact implementation is TBD. We may
introduce Custom Resource Definitions such as SimFleet and SimInstance, or
we may implement the same reconciliation model inside the backend controller
without exposing custom Kubernetes resource types. In either case, the
user-facing service API should remain stable while the operator implementation
evolves inside the same backend binary.
The service should avoid storing authoritative fleet or instance state outside
Kubernetes for v1. If we introduce CRDs, SimFleet and SimInstance
resources should hold desired state and status. If we do not introduce CRDs,
the backend should use Kubernetes labels, annotations, and ConfigMaps to
maintain the same state. The expected scale is low enough that Kubernetes API
performance should not be a limiting factor any time soon.
---\nconfig:\n theme: base\n themeVariables:\n darkMode: true\n background: "#0f172a"\n mainBkg: "#111827"\n primaryColor: "#111827"\n primaryBorderColor: "#64748b"\n primaryTextColor: "#e5e7eb"\n lineColor: "#94a3b8"\n textColor: "#e5e7eb"\n clusterBkg: "#111827"\n clusterBorder: "#64748b"\n clusterTextColor: "#e5e7eb"\n edgeLabelBackground: "#111827"\n layout: elk\n flowchart:\n htmlLabels: false\n---\nflowchart TB\n user["User CLI / Web UI / Agent"]\n\n subgraph backend [Single simulator backend process]\n subgraph service [Simulator Service API]\n api["API server"]\n auth["Cognito + Google SSO"]\n setup["Scene and configuration setup"]\n catalog["Simulation catalog"]\n records["Fleet, instance, and run views"]\n access["Connection details and access tokens"]\n end\n\n subgraph operator [Simulator Operator]\n desired["Validated SimFleet / SimInstance desired state"]\n reconcile["Reconcile desired state"]\n schedule["Schedule simulator fleet workloads"]\n watch["Watch pod/job state and runtime health"]\n cleanup["TTL, idle, and artifact cleanup"]\n credentials["Short-lived access credentials"]\n end\n end\n\n subgraph kube_state [Kubernetes-native state]\n kstate["CRDs or labels/annotations + ConfigMaps"]\n end\n\n subgraph k8s [k8s GPU cluster]\n subgraph instance [Configurable SimInstance pod]\n sim["sim: Isaac Sim + GPU"]\n robot["robot: hmnd_robot bringup"]\n coordinator["coordinator: workflow API"]\n map_service["map-service: fleet map API"]\n robot_map["robot-map: robot map state"]\n warehouse_agent["warehouse-agent: task executor"]\n policy["policy: selected policy service"]\n runner["runner: readiness, workflow, result collection"]\n display["Optional display / debug sidecars"]\n exported["Artifacts/logs exported on completion"]\n end\n end\n\n user --> api\n api --> auth\n api --> setup\n api --> catalog\n api --> records\n api --> access\n access --> credentials\n api --> kstate\n kstate --> desired\n kstate --> records\n desired --> reconcile\n reconcile --> schedule\n reconcile --> watch\n reconcile --> cleanup\n schedule --> instance\n watch --> kstate\n instance --> records\n runner --> exported\n cleanup --> kstate\n\n sim --> robot\n robot --> coordinator\n coordinator --> map_service\n coordinator --> robot_map\n coordinator --> warehouse_agent\n robot --> policy\n runner --> sim\n runner --> robot\n runner --> coordinator\n runner --> policy\n\n classDef actorNode fill:#7c2d12,stroke:#fdba74,color:#fff7ed\n classDef serviceNode fill:#1e3a8a,stroke:#93c5fd,color:#eff6ff\n classDef operatorNode fill:#14532d,stroke:#86efac,color:#f0fdf4\n classDef stateNode fill:#713f12,stroke:#fde68a,color:#fef9c3\n classDef workloadNode fill:#581c87,stroke:#d8b4fe,color:#faf5ff\n\n class user actorNode\n class api,auth,setup,catalog,records,access serviceNode\n class desired,reconcile,schedule,watch,cleanup,credentials operatorNode\n class kstate stateNode\n class sim,robot,coordinator,map_service,robot_map,warehouse_agent,policy,runner,display,exported workloadNode\n\n style backend fill:#0f172a,stroke:#64748b,stroke-width:2px\n style service fill:#0f1f3d,stroke:#60a5fa,stroke-width:1px\n style operator fill:#0f2a1a,stroke:#4ade80,stroke-width:1px\n style kube_state fill:#261b09,stroke:#facc15,stroke-width:2px\n style k8s fill:#1f1235,stroke:#c084fc,stroke-width:2px\n style instance fill:#2e1065,stroke:#d8b4fe,stroke-width:1px\n
The service should expose two managed states: SimFleet and SimInstance.
These are product concepts first. They may become Kubernetes CRDs, or they may
be represented through native Kubernetes objects, labels, annotations, and
ConfigMaps.
A SimFleet is the user-visible unit for a virtual system. It describes the
desired scene, selected catalog inputs, fleet size, shared services, display
configuration, workflow execution, access capabilities, lifecycle policy, and
developer overrides.
The service should model those capabilities directly rather than introducing a
separate runtime_profile state field. The CLI and UI can still offer
templates such as "interactive debug", "E2E workflow", or "canary", but those
templates should expand into explicit SimFleet configuration before the
operator reconciles Kubernetes resources.
A SimFleet records:
id / name
owner
state: queued, starting, running, stopping, stopped, failed, expired
size: desired number of simulator instances
scene: scene/world configuration
catalog_inputs: selected simulator, robot, policy, fleet, and scenario
catalog entries
gpu_profile: for example g6_l4 or g7e_rtx6000
display: enabled flag, backend such as DCV or WebRTC, resolution, and
access policy
workflow: optional scenario, autostart flag, inputs, timeout, and
completion criteria
access: logs, exec, port-forward, Foxglove, Sockpuppet, Reverb/app, IDE
attach, and local development connectivity options
shared_services: Switchboard, policy, Reverb, map services, or external
endpoints used by the fleet
networking: local-development connectivity mode and exposed access points
lifecycle: TTL, idle timeout, hard timeout, cleanup policy
status: aggregate instance readiness, workflow progress, and conditions
failure_reason
Phase 1 restriction: MVP fleets support exactly one simulator. This lets
the operator schedule shared services in the same pod as the simulator while
the service contract is still maturing. The restriction should be lifted in a
later phase by integrating with the
Nexus/Sandbox
model for shared services used by all SimInstance pods in a fleet.
A SimInstance is one simulator pod in a SimFleet. Each pod should contain
the containers required to operate that simulated robot. At minimum, that means
an Isaac Sim container and an hmnd_robot container. Depending on the selected
fleet configuration, the pod may also include display/viewer, policy, runner,
artifact-export, or debugging sidecars.
A SimInstance records:
id / ordinal
fleet_id
pod_name
node
state: pod phase plus normalized service state
containers: Isaac Sim, hmnd_robot, and selected companion containers
image_refs: image repository, tag, digest, and artifact channel
robot: robot ID, ROS domain, bringup config, pc_type, and map config
access: display, Foxglove, Sockpuppet, Reverb/app, shell, logs, and IDE
attach endpoints or commands
runtime: tmux/debug handle, port-forward status, and local connect state
health: simulator readiness, robot bringup readiness, fleet registration,
workflow progress, and last Kubernetes event
artifacts: logs, reports, screenshots, recordings, and run metadata
failure_reason
The service should have one SimFleet runtime model. Interactive debugging,
E2E workflow execution, and canary/demo fleets should be represented as
different combinations of explicit configuration fields, not as separate
managed resource categories.
The v1 runtime should use a split SimInstance pod model with separate
containers for major components instead of treating the current monolithic
interactive pod as the long-term abstraction:
sim owns Isaac Sim and the GPU.
robot owns robot bringup.
fleet services run as separate containers or service dependencies.
policy is either an internal endpoint or a sidecar selected by the fleet.
display.enabled adds a display/viewer sidecar and browser access endpoint.
workflow.autostart adds or activates a runner that owns readiness,
scenario start, progress checks, result collection, and shutdown.
access controls which debug and developer connectivity surfaces are
exposed through the API and CLI.
Display should not be required for workflow execution. Automated E2E runs
should default to display.enabled=false so they do not pay the CPU, memory,
startup, and operational cost of the display sidecar unless a developer
explicitly requests visual debugging.
Runtime details are covered in
Simulation Runtime on k8s.
API and CLI compatibility details are covered in
Simulation Service API and CLI Compatibility.
Add the single backend binary with Simulator Service API and Simulator
Operator responsibilities.
Add Kubernetes-backed SimFleet and SimInstance state.
Enforce SimFleet.size == 1.
Dispatch the current Helm chart from the operator instead of from sim_cli.
Preserve current GPU profile behavior and display access.
Add TTL cleanup, list/get/terminate, and normalized failure reporting.
Update CLI to call the service for create/list/get/terminate/connect.
Define and implement a configurable split SimInstance pod shape on k8s.
Start with app_regression as the first supported scenario.
Use a real reachable policy service or an explicitly requested test policy
sidecar.
Default workflow runs to display.enabled=false, with explicit visual debug
opt-in.
Persist runner reports and per-container logs as fleet artifacts.
Add scenario registry and runtime image compatibility metadata.
Add UI/API support for scene setup and catalog selection.
Add scenario input validation and explicit artifact outputs.
Support promoted images and release channels instead of ad hoc image tags.
Add richer run history and search.
Add queueing and capacity-aware scheduling.
Add better spot/on-demand fallback policy if cost pressure requires it.
Revisit one-Pod workflow runtime versus separate Kubernetes services.
Evaluate whether WebRTC or another browser display path should augment or
replace DCV on k8s.
Version 1 is ready when:
A user can create a display-enabled fleet without running Helm or kubectl
for deployment.
A user can choose a scene, configuration, and catalog inputs through the
service API.
When display is enabled, the service returns a browser display URL and log
access for each instance.
When display is disabled, the operator does not schedule display-only
sidecars.
Each instance schedules on the selected GPU profile or fails with an
actionable admission error.
The user can terminate the fleet and the service cleans up all Kubernetes
resources.
TTL and idle cleanup work without manual intervention.
A user can launch an app_regression workflow through the service API.
The workflow run reports pass/fail, current step, final task counts, and
links to logs/artifacts.
Failures surface the real blocker, such as policy unreachable, sim not ready,
robot bringup failed, localization failed, or workflow timeout.
The CLI no longer requires normal users to manage kubeconfig, Helm releases,
or direct pod names for happy-path deployment and lifecycle usage.
Engineers can still use service-generated kubectl logs, kubectl exec,
and kubectl port-forward commands for diagnostics and development
connectivity.
Audience: Systems & Product
Purpose: Show how the FMS requirement set (the 129 FMS functional requirements + 30 new user stories) maps onto the fleet software architecture we are actually building, and where we are on each.
Status: Draft for review \u2014 Cody Griffin
Related: Switchboard State Model & Fleet Registry \u00b7 Switchboard Coordinator \u00b7 HMND Outpost & Site Authority \u00b7 Spec: Insight \u00b7 The Problem With Central Planning
Software doesn't march down the requirements list line-by-line, which makes it hard to tell whether what's been asked for is being built. This document is the bridge: every requirement has a home \u2014 an owning service, a one-line how, and a status \u2014 including the ones deliberately deferred or owned outside fleet software.
Three things explain why the build order differs from the requirement order:
Substrate first, product second. Many requirements (dashboards, lists, maps, reports) are products on top of a shared coordination/identity/state substrate. Build that once (Switchboard, Backbone, Outpost) and the surfaces come fast and reliable; build them first and you re-plumb the foundation repeatedly.
Same intent, different mechanism. Where a requirement assumes a central scheduler, we coordinate decentrally \u2014 robots self-select advertised work, strong consistency prevents conflicts \u2014 and optimize advisorily, off the critical path. That meets the throughput intent while keeping graceful degradation; full central scheduling stays an optional layer. (See "The Problem With Central Planning.")
Status, not just ownership. Each row carries a maturity, so you can see the pilot path versus scale-out.
Code | Service | Owns | Source of truth |
|---|---|---|---|
SB | Switchboard | Fleet/site/map/workflow registry, coordination state (markings, leases), the replicated fact model, the three actions (enroll/deploy/assign), query lenses, authz projection, System-3 adapters | Coordination state; replicates everything else |
CO | Coordinator | Workflow/Petri-net execution semantics (the engine under Switchboard) | Workflow execution model |
BB | Backbone | Global device identity, attestation, certificate lifecycle, global asset registry, OS-level device ops (SSH/reboot/OTA/rollback), capability distribution, IAM/Cognito pool | Device identity & global asset truth |
OP | Outpost | Authority placement (cloud vs site), HA tiers, provisioning state machine, network/power qualification, sync, local service stack, PKI placement | Authority placement & site infra |
RV | Reverb | On-robot System-2 agent: edge work-selection, perception, localization, manipulation, local safety/coexistence behavior | Reality at the edge |
ST | Studio | Cloud workflow authoring/editing/inference, simulation & what-if, authoring-agent + eval harness | Workflow definitions (authoring) |
MAP | Map / Semantic | Map ingest, versioning, 3D/semantic assets, spatial pipeline | Static spatial truth |
OPT | Optimization Agent (advisory) | Goal\u2192priority translation; reprioritizes/affinity-tags advertised work to bias selection. Off the critical path | Nothing authoritative (advisory) |
INS | Insight | Derived-truth plane (read-only): KPIs/SLAs, dashboards, alerting, incident detection, benchmarking, explainability, learning corpus. Four stores \u2014 Mimir/Tempo/Loki (system health) + ClickHouse (work events); spans fleet scope (from Switchboard) and device scope (from Backbone). See Spec: Insight. | Nothing authoritative (derived) |
FC | Fleet Console / Fleet API | The unified experience layer (single pane of glass): composes the substrate into facility/customer/global views; NL tasking + intent confirmation; role-scoped access | Nothing authoritative (composition) |
N/A | \u2014 | Not fleet software (hardware handling, manufacturing QA) | \u2014 |
Built \u2014 substrate exists in the current implementation or an approved spec.
Next \u2014 on the pilot path (Schaeffler EOY 2026).
Later \u2014 post-pilot / scale-out.
Deferred \u2014 designed-for but deliberately parked (with a reason).
Out \u2014 outside fleet-software scope.
The requirements include a Scheduling Engine, Workload Balancing, Multi-Robot Coordination "optimising material flow," and autonomous charging coordination. Here is how we satisfy the intent of all of them:
Conflict-free coordination is guaranteed by Switchboard today: leases serialize dispatch, so two robots are never handed the same exclusive work. This is strong and non-negotiable.
Work selection is decentralized: work is advertised, and robots (via Reverb) self-select based on their own state (charge, distance, health) and peer awareness. Better-suited robots respond more eagerly; the first valid claim wins. No central brain decides every move.
Optimization is real but advisory: the Optimization Agent translates facility goals ("1,000 units/day," "expedite this order") into priorities and affinities on the advertised work. It tunes the bias, not the mechanism. If it degrades, the fleet falls back to making steady progress rather than stalling.
This is a deliberate reliability trade chosen for humanoids in unstructured spaces. The throughput target is met; the fragility of central planning is avoided by default. Where a requirement says "scheduling engine," the default is goal-driven prioritization over a decentralized, self-balancing fleet \u2014 with full central scheduling available as an optional layer for deployments that deliberately choose optimality over availability.
Requirement text is quoted verbatim from the source export (the bold lead is a short label for scanning). Rows ordered by maturity: Built \u2192 Next \u2192 Later \u2192 Mixed \u2192 Deferred \u2192 Out.
ID | Requirement | Service(s) | How it's delivered | Status |
|---|---|---|---|---|
3239 | Facility fleet orchestration (6+ robots) \u2014 The Fleet Management System shall orchestrate the facility (local) fleet, optimising material flow, and maximising local fleet utilisation. (6+ robots per deployment) | SB \u00b7 RV \u00b7 OPT | Coordination substrate + edge selection + advisory prioritization | Built (coord) / Next (opt) |
3308 | Fleet & robot monitoring \u2014 The Fleet Management System will have monitoring and alerting capabilities for errors impacting robot performance. For example, network degradation, battery, task performance, etc. | SB \u00b7 INS | Facts/lenses provide data; Insight computes alerts | Built (data) / Next (alerting) |
3303 | Workflow simulation & what-if \u2014 The Fleet Management System should allow users to test scenarios before implementation. | ST | Symbolic simulator exists in Studio | Built |
3298 | Event replay \u2014 The Fleet Management System shall allow users to replay robot operations (map + actions + sensor context) for debugging and training. | SB \u00b7 INS | Append-only event log is the spine; replay UI in Insight | Built (log) / Later (UI) |
3292 | Integration with enterprise systems \u2014 The Fleet Management System shall integrate with customer systems (e.g. WMS, MES) for task ingestion and reporting. | SB \u00b7 OP | SAP/WMS adapter at System-3 boundary; Factory Integration Gateway | Built / Next |
3257 | Workflow execution \u2014 The Fleet Management System shall execute defined workflows with metrics and validation. | CO \u00b7 SB \u00b7 INS | Execution engine + markings/leases + conformance metric | Built |
3252 | Intervention \u2014 The Fleet Management System shall enable operators to intervene via teleoperation or control overrides in a safe way. | SB \u00b7 RV \u00b7 FC | Transactional release\u2192reacquire + marking intervention; teleop transport | Built (coord) / Next (teleop) |
3249 | Robot swap management \u2014 The Fleet Management System shall support swapping robots in and out of tasks dynamically. | SB | Assignment + release\u2192reacquire | Built |
3248 | Multi-robot coordination \u2014 The Fleet Management System shall coordinate multiple robots executing tasks simultaneously. | SB \u00b7 RV | Leases serialize dispatch + edge selection | Built |
3247 | Dynamic repurposing \u2014 The Fleet Management System shall enable reassignment of robots across tasks and use cases. | SB | Reassign via assignment join | Built |
3245 | Task assignment \u2014 The Fleet Management System shall allow operators to assign tasks to robots via the Fleet UI. | SB \u00b7 FC | Assignment / InjectWork action; Console UI | Built |
3243 | Robot onboarding \u2014 The Fleet Management System shall onboard robots into the facility fleet with readiness validation. | SB \u00b7 OP \u00b7 BB | Enroll + localization gate; provisioning state machine; device identity | Built / Next |
3242 | POI & zone definition \u2014 The Fleet Management System shall allow definition of points of interest, workstations, safety, maintenance and storage zones. | SB \u00b7 MAP \u00b7 FC | Zones/POIs as map-owned places; authoring UI | Built (model) / Next (UI) |
3241 | 2D facility map \u2014 The Fleet Management System shall provide an interactive 2D map of the facility for context showing real-time robot locations, health and allow drill-down to individual robots. | SB \u00b7 FC | Map lens (operator map view); Console render | Built (data) / Next (UI) |
321 | Robot list \u2014 The Fleet Management System shall offer a searchable and filterable robot list view for fleet overview. | SB \u00b7 FC | Fleet lens; Console list view | Built (data) / Next (UI) |
274 | Robot status \u2014 The Fleet Management System shall report the operational status such as Running, Idle, Need Intervention. | SB \u00b7 FC | Lifecycle state + resource fact; Console | Built |
273 | Robot ID / name \u2014 The Fleet Management System shall report a unique identifier or label for each robot. | SB \u00b7 BB | Subject id + label (SB); device identity (BB) | Built |
3307 | Workflow editor \u2014 The Fleet Management System shall provide the ability to edit workflows intuitively through the interface. | ST | Cloud authoring/editing (authoring agent prototype exists) | Next |
3300 | Local worker interaction \u2014 The Fleet Management System shall enable workers to interact with robots locally whilst updating the FMS. For example, a local worker might want to "send home" or "swap out" a robot and this is reflected in the UI. | SB \u00b7 RV \u00b7 FC | Operator-override facts + assignment change; reflected in Console | Next |
3299 | Natural language tasking \u2014 The Fleet Management System shall allow users to assign tasks using natural language via the FMS interface. | FC \u00b7 ST | NL front-end \u2192 assignment/work; authoring agent exists | Next |
3291 | Integration with 3rd-party FMS \u2014 The Fleet Management System shall have the API to integrate with third party fleet management applications for visibility, and control. For example; our robots can be visualised in 3rd party fleet management application like NAiSE. | SB \u00b7 OP | VDA5050 virtual-fleet state broadcast | Next (floor) / Later (full) |
3261 | VDA5050 protocol support \u2014 The Fleet Management System shall implement the VDA5050 communication standard for message exchange between the Fleet Management System and robots, including all mandatory message types and data structures. | SB \u00b7 OP | Zoned-authority continuum; pilot does floor tier | Next (floor) / Deferred (full set) |
3255 | Charging coordination (manual) \u2014 The Fleet Management System shall enable the ability to manually instigate a battery swap between a user selected robot and a user selected swap station. | SB \u00b7 FC | Manual swap as work injection + lease; triggered in Console | Next |
3253 | Safety controls \u2014 The Fleet Management System shall implement and visualise safety features including stop, slow zones, stop conditions and go home for individual robots and fleet. | RV \u00b7 SB \u00b7 FC | Zones as places + Console visualisation; enforcement is robot-side | Next (viz) / RV (behavior) |
3240 | 3D facility map \u2014 The Fleet Management System shall provide an interactive 3D map of the facility for context. | MAP \u00b7 SB \u00b7 FC | 3D/semantic assets + locatables; Console render | Next |
346 | Fleet dashboard \u2014 The Fleet Management System shall include relevant user dashboards. | INS \u00b7 FC | KPI data + Console dashboards | Next |
3713 | Remote robot control \u2014 The Fleet Management System support remote control of an individual robot including bring-up, move to work location, start/stop task, etc. | SB \u00b7 RV \u00b7 FC | Commands as polled operator intent over the work loop (preserves zero-inbound) | Next |
3498 | Environment change action \u2014 The Fleet Management System shall suggest actions for unforeseen environmental changes. For example, suggesting rerouting options to be confirmed. | RV \u00b7 FC | Robot reroutes locally; suggestion surfaced for confirmation | Later |
3493 | Fleet/facility goal setting \u2014 The Fleet Management System will have goal setting ability to set operational performance based goals. For example, the facility needs to achieve 1000 orders or units per day. | FC \u00b7 OPT | Goals set in Console; translated to advertised-work priorities | Later |
3306 | Environment change detection \u2014 The Fleet Management System shall detect and present to the user changes in the environment. | RV \u00b7 MAP | Perception + map-delta detection | Later |
3302 | Facility digital twin \u2014 The Fleet Management System maintain a digital twin of each facility for simulation and planning. | ST \u00b7 MAP \u00b7 SB | Twin from spatial assets + simulation instances | Later |
3301 | Intent confirmation \u2014 The Fleet Management System shall confirm and validate user intent (with confidence levels) before executing commands. | FC | NL intent rendered for confirmation before commit | Later |
3297 | Remote diagnostic \u2014 The Fleet Management System shall automatically identify and present root causes for task failures, robot failures or performance degradation (with dynamic upgrade and confirm). | INS | Root-cause over event/telemetry streams | Later |
3296 | Decision explainability \u2014 The Fleet Management System shall explain why a robot made a decision (e.g. path, failure, task choice) in a human-readable format. | INS \u00b7 RV | Event+authz logs correlated by trace_id; agent rationale | Later |
3295 | Customised report management \u2014 The Fleet Management System shall allow users to create, edit, and store reports that include images, annotations, timestamps, and metadata for traceability and audit purposes. | INS | Reporting product over the event store | Later |
3294 | Image capture & association \u2014 The Fleet Management System shall enable live and time-based image capture from robot sensors or external devices and associate images with robots, tasks, locations, or events within the system. | RV \u00b7 INS | Robot capture; associate to subject/task/event | Later |
3260 | Fleet operational reporting \u2014 The Fleet Management System provide reporting on task status, productivity and utilisation at robot and fleet level. | INS | Productivity/utilisation reporting over the event store | Later |
3258 | Workflow optimisation \u2014 The Fleet Management System shall provide feedback and suggestions to optimise workflows. | ST \u00b7 OPT | Inference proposes improvements; advisory | Later |
3256 | Charging coordination (auto) \u2014 The Fleet Management System shall coordinate charging across robots and swap stations to maximise uptime and operational flows. | OPT \u00b7 RV \u00b7 SB | Swap station as place + lease + power facts; advisory coordination | Later |
3246 | Scheduling engine \u2014 The Fleet Management System shall schedule and balance workloads across robots with constraints, prioritising operational value to the high level operation goal. | OPT \u00b7 RV | Advisory prioritization + decentralized selection (central scheduling = optional layer) | Later |
153 | Robot goal setting \u2014 The Fleet Management System will have goal setting ability to be able to set operational performance based goals. | FC \u00b7 OPT | Goals in Console \u2192 priorities | Later |
278 | System logs \u2014 The Fleet Management System shall collect and report system logs, messages, and historical status. | BB \u00b7 INS | Device logs collected by Backbone; surfaced in Insight | Later |
277 | Robot adv. states \u2014 The Fleet Management System shall report CPU usage, memory usage, current, data usage, and uptime. | BB \u00b7 INS | Device-scope telemetry (Backbone) surfaced via Insight | Later |
129 | Individual robot monitoring & control \u2014 The Fleet Management System shall support remote control and reporting of the individual robot. | SB \u00b7 BB \u00b7 FC | Operational state via SB; device telemetry via BB; Console | Mixed |
599 | Sub-fleet & group operations \u2014 The Fleet Management System shall enable grouping of robots and execution of group operations like updates, reboots, and dispatch commands. | BB \u00b7 SB | Split concept: OTA/OS cohorts \u2192 Backbone; tasking affinity \u2192 Switchboard | Deferred (needs spec) |
322 | Mass actions \u2014 The Fleet Management System shall support bulk actions on multiple robots, such as updates, reboots, and task assignments. | BB \u00b7 SB \u00b7 FC | OS bulk ops \u2192 Backbone; bulk tasking \u2192 affinity | Deferred / Next |
3305 | Infrastructure interaction \u2014 The Fleet Management System shall enable robots to interact with facility infrastructure (doors, elevators, conveyors). (REMOVE) | N/A | Marked REMOVE in source | Out |
ID | Requirement | Service(s) | How it's delivered | Status |
|---|---|---|---|---|
3285 | Customer-level policy configuration \u2014 The Fleet Management System shall allow configuration of policies, thresholds, and workflows at the customer level - applicable across a single customer network of sites. | SB \u00b7 BB \u00b7 FC | Tenant-scoped groups/policies in registry/authz; Console UI | Next (registry) / Later (UI) |
3238 | Customer fleet monitoring & growth \u2014 The Fleet Management System shall monitor and alert on performance, reliability, utilisation, ROI and provide features that grow and adapt the fleet to new facilities and operations based on customer fleet data (1000+ robot per customer). | INS \u00b7 FC | Customer-level analytics + dashboard | Later |
3304 | Cross-facility learning transfer \u2014 The Fleet Management System shall transfer learnings from one customer facility to others where applicable. | INS \u00b7 ST | Fleet-wide data pipeline feeds learning/authoring | Later |
3290 | Fleet growth forecasting \u2014 The Fleet Management System shall forecast required fleet size based on workload and performance trends, and proactively recommend fleet expansion. | INS | Forecasting over utilisation/throughput trends | Later |
3289 | Performance recommendations \u2014 The Fleet Management System shall provide recommendations to improve utilisation and performance. | INS | Recommendation product over the event store | Later |
3287 | SLA tracking \u2014 The Fleet Management System shall track and report performance against defined SLAs. | INS | SLA computation/reporting | Later |
3286 | Stakeholder reporting \u2014 The Fleet Management System shall generate reports for operational, and business stakeholders. For example: inspection reports for operations, financial reports for business stakeholders. | INS | Operational/financial report generation | Later |
3284 | Site-level alerting \u2014 The Fleet Management System shall notify users of failures, anomalies, and performance degradation at the site level i.e. this site isn't achieving its UPD target. | INS | Per-site KPI/target alerting | Later |
3282 | Aggregated task performance \u2014 The Fleet Management System shall measure aggregated task-level performance including cycle time, success rate, and throughput across all sites (operated by the customer). | INS | Cycle time / success / throughput across sites | Later |
3280 | Reliability monitoring (MTBF) \u2014 The Fleet Management System shall track reliability metrics including uptime, failure rates, and MTBF. | INS \u00b7 BB | Device-scope reliability metrics (joins fleet at availability) | Later |
3279 | ROI reporting \u2014 The Fleet Management System shall calculate and present ROI metrics including throughput, cost savings, and productivity gains. | INS | ROI/productivity metrics | Later |
3278 | Utilisation tracking \u2014 The Fleet Management System shall track robot utilisation including active time, idle time, and task allocation. | INS | Active/idle/allocation tracking | Later |
3277 | Customer fleet dashboard \u2014 The Fleet Management System shall provide a unified dashboard showing fleet performance, utilisation, and status across all customer facilities. | FC \u00b7 INS | Unified cross-facility dashboard | Later |
ID | Requirement | Service(s) | How it's delivered | Status |
|---|---|---|---|---|
3313 | Multi-tenant isolation \u2014 The Fleet Management System shall isolate customer data, fleets, and operations securely with customers having visibility and control (through UI) of what data is sharable. | SB \u00b7 OP \u00b7 FC | Fleet-as-RLS + tenant boundary; segregation; sharability-control UI | Built (isolation) / Later (UI) |
3271 | Identity & access control \u2014 The Fleet Management System shall enforce secure identity, authentication, and access control across all robots and operators. | SB \u00b7 BB \u00b7 OP | Cedar authz (SB) + Cognito/device identity (BB) + PKI placement (OP) | Built / Next |
3269 | Fleet lifecycle management \u2014 The Fleet Management System shall manage the full fleet lifecycle including onboarding, operation, maintenance, and decommissioning. | BB \u00b7 SB \u00b7 OP | Enroll/deploy/assign (SB) within full device lifecycle (BB) + provisioning (OP) | Built (enroll) / Later (full) |
3319 | Commercial model differentiation (CapEx vs. RAAS) \u2014 The Fleet Management System shall have the ability to differentiate between different commercial models (RAAS or CapEx), with different monitoring, alerting, and escalation paths. | SB \u00b7 INS \u00b7 FC |
| Next (model) / Later (paths) |
3273 | Data pipeline for learning \u2014 The Fleet Management System shall collect and structure fleet-wide data for training, simulation, and continuous improvement. | INS \u00b7 ST | Event log \u2192 analytical/archive tiers; structured for training | Next (log ships) / Later (full) |
3272 | Fleet segmentation \u2014 The Fleet Management System shall segment robots by customer, region, facility and commercial model while maintaining centralized control. | SB \u00b7 FC | tenant/fleet/site + region on site, commercial model on fleet | Next |
3262 | Global asset registry \u2014 The Fleet Management System shall maintain a global registry of all robots including identity, configuration, ownership, and lifecycle state. | BB | Backbone owns it; Switchboard references it (per-deployment fleet registry) | Next (BB) / Built (SB ref) |
3236 | E2E asset lifecycle & global observability \u2014 The Fleet Management System shall have end-to-end asset lifecycle management, global observability & health, upgrades & rollout ability for a large distributed fleet operating across multiple different customer facilities (185,000 robots in operation by 2030). | BB \u00b7 INS \u00b7 OP | Lifecycle/asset (BB) + observability (INS) + rollout substrate (OP) | Later |
3312 | Spare parts prediction & tracking \u2014 The Fleet Management System shall predict and recommend spare parts inventory based on usage patterns. | INS \u00b7 BB | Device-scope usage-pattern prediction | Later |
3311 | Remote diagnostics \u2014 The Fleet Management System shall allow remote diagnosis of robot issues without onsite intervention. | INS \u00b7 BB | Remote diagnosis without onsite intervention | Later |
3310 | Guided maintenance workflows \u2014 The Fleet Management System shall guide technicians through maintenance procedures step-by-step. | BB \u00b7 FC | Step-by-step maintenance procedures | Later |
3276 | Fleet expansion management \u2014 The Fleet Management System shall support rapid scaling from pilot to large-scale deployments without degradation in performance. | OP \u00b7 SB | Placement/scale tiers + Switchboard sharding path | Later (designed-for) |
3275 | Global map & knowledge sharing \u2014 The Fleet Management System shall enable sharing of maps, workflows, and learnings across facilities and customers. | MAP \u00b7 ST \u00b7 BB | Map/workflow/learning sharing across sites | Later |
3274 | Capability distribution \u2014 The Fleet Management System shall distribute new robot capabilities (skills/workflows) across the fleet (if permitted by customer). | BB \u00b7 ST | Capability rollout (customer-permitted) | Later |
3270 | Incident management \u2014 The Fleet Management System shall detect, log, and manage incidents across all robots and fleets. | INS \u00b7 SB | Exception events (SB) \u2192 detect/log/manage (INS) | Later |
3268 | Predictive maintenance \u2014 The Fleet Management System shall predict maintenance needs using fleet-wide data and trigger proactive servicing. | INS \u00b7 BB | Device-scope fleet-wide predictive servicing | Later |
3267 | Controlled software rollback \u2014 The Fleet Management System shall support controlled rollback of software, policies, and configurations across the fleet (sample to full fleet). | BB | Sample\u2192full rollback | Later |
3266 | Fleet-wide software deployment \u2014 The Fleet Management System shall deploy software updates, policies, and capabilities across the global fleet (sample to full fleet). | BB | Sample\u2192full OTA | Later |
3265 | Global performance benchmarking \u2014 The Fleet Management System shall aggregate telemetry data (performance, utilisation, and faults) across all robots for centralized analysis, and shall use this data to benchmark robot and fleet performance across sites, customers, and use cases. | INS | Cross-site/customer benchmarking | Later |
3263 | Fleet-wide health monitoring \u2014 The Fleet Management System shall continuously monitor health, diagnostics, and status of all robots across the global fleet. | INS \u00b7 BB | Device-scope health/diagnostics across fleet | Later |
Most user stories restate an FMS functional requirement at a higher level. The use-case text below is the verbatim user story; the bold lead is a short label. Mapped to FMS equivalent + service + status; rows ordered by maturity, as above. Physical-handling and QA stories are flagged as outside fleet software.
ID | Use case | FMS equiv. | Service(s) | Status |
|---|---|---|---|---|
3740 | Workflow Execution & Editing \u2014 As a Deployment Engineer/Fleet Manager, I want to execute defined workflows with real-time metrics and validation, and edit those workflows intuitively through the FMS interface, so that operations can be adapted quickly without engineering intervention. | 3257 \u00b7 3307 | CO \u00b7 SB \u00b7 ST | Built / Next |
3739 | Dynamic Repurposing & Robot Swap \u2014 As a Deployment Engineer/Fleet Manager, I want to dynamically reassign robots across tasks and use cases, and swap robots in and out of active tasks, so that I can respond to changing operational demands without interrupting workflows. | 3247 \u00b7 3249 | SB | Built |
3738 | Multi-Robot Coordination \u2014 As a Deployment Engineer/Fleet Manager, I want the FMS to coordinate multiple robots executing tasks simultaneously, optimising material flow and maximising fleet utilisation across deployments of 6 or more robots, so that throughput targets are met without conflicts. | 3248 | SB \u00b7 RV \u00b7 OPT | Built (coord) / Later (opt) |
3735 | Task Assignment \u2014 As a Deployment Engineer/Fleet Manager, I want to assign tasks to specific robots via the FMS interface so that I can direct the fleet to the right work at the right time. | 3245 | SB \u00b7 FC | Built |
3726 | 2D Facility Map \u2014 As a Deployment Engineer/Fleet Manager, I want an interactive 2D map of the facility so that I can see real-time robot locations and health, and drill down into individual robots directly from the map. | 3241 | SB \u00b7 FC | Built / Next |
3725 | Robot List View \u2014 As a Deployment Engineer/Fleet Manager, I want a searchable and filterable list of all robots in the fleet so that I can quickly locate, inspect, and act on individual robots. | 321 | SB \u00b7 FC | Built / Next |
3723 | Robot Onboarding Validation \u2014 As a Deployment Engineer/Fleet Manager, I want to onboard a robot into the facility fleet with automated readiness validation so that only robots in a verified operational state are admitted to active service. | 3243 | SB \u00b7 OP \u00b7 BB | Built / Next |
3722 | POI & Zone Definition \u2014 As a Deployment Engineer/Fleet Manager, I want to define points of interest, workstations, safety zones, maintenance zones, and storage zones on the facility map so that robots can navigate purposefully and operators can enforce spatial rules across the fleet. | 3242 | SB \u00b7 MAP \u00b7 FC | Built / Next |
3719 | Network Setup & Fleet Onboarding \u2014 As a Deployment Engineer/Fleet Manager, I want to connect the robot to the customer network and register it in the fleet management system so that it can be monitored, assigned tasks, and operated as part of the fleet. | \u2014 | OP \u00b7 SB \u00b7 BB | Built / Next |
3784 | Fleet Segmentation \u2014 As a Global Fleet Operations Manager, I want to segment robots by customer, region, facility, and commercial model while maintaining centralised control so that each customer's fleet is managed independently without sacrificing global visibility. | 3272 | SB \u00b7 FC | Next |
3741 | Safety Controls \u2014 As a Deployment Engineer/Fleet Manager, I want to define and visualise safety features including stop zones, slow zones, stop conditions, and go-home commands for individual robots and the fleet so that I can enforce safe operating boundaries at all times. (maybe this should just be visualisation) | 3253 | RV \u00b7 SB \u00b7 FC | Next (viz) / RV (behavior) |
3736 | Natural Language Tasking & Intent Confirmation \u2014 As a Deployment Engineer/Fleet Manager, I want to assign tasks to robots using natural language via the FMS interface, with the system confirming and validating my intent (including confidence levels) before executing, so that I can direct the fleet intuitively without risking unintended actions. | 3299 \u00b7 3301 | FC \u00b7 ST | Next |
3733 | Charging Coordination (Manual) \u2014 As a Deployment Engineer/Fleet Manager, I want to manually trigger a battery swap between a specific robot and a specific swap station so that I can intervene in charging schedules when operational priorities require it. | 3255 | SB \u00b7 FC | Next |
3727 | 3D Facility Map \u2014 As a Deployment Engineer/Fleet Manager, I want an interactive 3D view of the facility so that I can gain richer spatial context for robot positions, routes, and zone layouts. | 3240 | MAP \u00b7 SB \u00b7 FC | Next |
3724 | Fleet Dashboard \u2014 As a Deployment Engineer/Fleet Manager, I want a real-time fleet dashboard so that I can monitor the operational status, health, and performance of all robots at a glance and receive alerts for issues impacting robot performance. | 346 | INS \u00b7 FC | Next |
3721 | Local Mapping \u2014 As a Deployment Engineer/Fleet Manager, I want to instruct the robot to autonomously map its local environment so that it can navigate, manage tasks, and be orchestrated within the fleet. | \u2014 | MAP \u00b7 RV | Next |
3720 | Global Mapping \u2014 As a Deployment Engineer/Fleet Manager, I want to upload a facility layout into the FMS so that robots have a global reference map available before on-site deployment. | \u2014 | MAP \u00b7 SB | Next |
3737 | Scheduling Engine & Workload Balancing \u2014 As a Deployment Engineer/Fleet Manager, I want the FMS to automatically schedule and balance workloads across all robots respecting constraints and prioritising tasks by operational value so that the fleet consistently works towards facility-level goals. | 3246 | OPT \u00b7 RV | Later (decentralized) |
3734 | New Task Onboarding \u2014 As a Deployment Engineer/Fleet Manager, I want to onboard a new task into the robot's system within 2 hours so that the robot can learn and execute it consistently in its operational environment. | \u2014 | ST \u00b7 RV | Later (policy training partly Out) |
3732 | Charging & Battery Swap (Autonomous) \u2014 As a Deployment Engineer/Fleet Manager, I want the FMS to autonomously coordinate charging and battery swaps across all robots and swap stations so that operational uptime is maximised and battery health is maintained. | 3256 | OPT \u00b7 RV \u00b7 SB | Later |
3730 | Environment Change Detection & Response \u2014 As a Deployment Engineer/Fleet Manager, I want the FMS to automatically detect changes in the environment and surface suggested responses (e.g. rerouting options) for my confirmation so that I can keep the fleet operating safely when conditions change unexpectedly. | 3306 \u00b7 3498 | RV \u00b7 MAP \u00b7 FC | Later |
3729 | Facility Digital Twin \u2014 As a Deployment Engineer/Fleet Manager, I want the FMS to maintain a live digital twin of the facility so that I can use it for simulation, planning, and what-if analysis without impacting live operations. | 3302 | ST \u00b7 MAP \u00b7 SB | Later |
3728 | Individual Robot Monitoring & Control \u2014 As a Deployment Engineer/Fleet Manager, I want to view detailed status and telemetry for any individual robot including unique ID, operational state (Running / Idle / Needs Intervention), CPU usage, memory, current draw, data usage, uptime, and system logs and issue remote control commands (bring-up, move to work location, start/stop task) so that I can monitor and manage each robot precisely. | 129 / 3713 / 277 / 278 / 274 | SB \u00b7 BB \u00b7 FC | Mixed |
3731 | Sub-Fleet & Group Operations \u2014 As a Deployment Engineer/Fleet Manager, I want to group robots into sub-fleets and execute bulk operations such as software updates, reboots, and dispatch commands across all robots in a group simultaneously so that I can manage large fleets efficiently. | 599 | BB \u00b7 SB | Deferred |
3718 | Power-Down \u2014 As a Deployment Engineer/Fleet Manager, I want to follow a defined power-down sequence so that I can prevent data corruption, avoid system damage, and leave the robot in a safe, stable state. | \u2014 | N/A | Out (hardware procedure) |
3717 | Power-Up \u2014 As a Deployment Engineer/Fleet Manager, I want to follow a defined power-up sequence so that the robot starts safely and correctly without risking damage or malfunction. | \u2014 | N/A | Out (hardware procedure) |
3716 | Unpackaging \u2014 As a Deployment Engineer/Fleet Manager, I want to unpackage the robot carefully so that I can confirm it arrived safely and avoid causing damage during unpacking. | \u2014 | N/A | Out (physical handling) |
3715 | Packaging \u2014 As a Deployment Engineer/Fleet Manager, I want to package the robot for transport or storage so that it is protected from damage and can be safely handled during transit or warehousing. | \u2014 | N/A | Out (physical handling) |
3714 | End of Line Testing \u2014 As a QA Technician, I want to perform end-of-line testing on each robot before shipment so that I can certify it meets quality, performance, and safety standards. | \u2014 | N/A | Out (manufacturing QA) |
By owning service (primary or shared):
Switchboard (SB) \u2014 the coordination/identity/state substrate underpins ~30 requirements; it is the first dependency for assignment, isolation, the map/list lenses, intervention, and integration.
Backbone (BB) \u2014 owns global asset registry, device identity, OS-level ops, and OTA/rollout; primary or shared on ~18 requirements (mostly \u00a73 lifecycle/rollout).
Insight (INS) \u2014 the derived-truth plane (Grafana/Loki/Tempo/Mimir + ClickHouse) is primary on the large majority of \u00a72 and the analytics half of \u00a73 (~30 requirements). Spans fleet scope (work KPIs, from Switchboard) and device scope (health/MTBF, from Backbone) \u2014 the two meet at availability. Derived-only; see Spec: Insight.
Fleet Console (FC) \u2014 the experience layer is shared on most "view," "dashboard," "list," and "tasking" requirements; composition-only.
Outpost (OP) \u2014 owns the entire deployment/identity/network/provisioning cluster (onboarding, multi-tenant segregation, HA, offline).
Studio (ST), Reverb (RV), Map (MAP), Coordinator (CO), Optimization Agent (OPT) \u2014 own the authoring, edge-execution, spatial, workflow-execution, and prioritization concerns respectively.
By status:
Built / Next (pilot path): identity & isolation, the three actions + lifecycle, coordination & leases, assignment/repurposing/swap, intervention, map & list lenses, WMS integration, SynaOS coexistence floor, workflow execution + simulation, manual charging, remote-control-as-intent, zone/POI definition, network onboarding.
Later (post-pilot/scale): the analytics/BI surface (\u00a72 and analytics \u00a73), autonomous charging, digital twin, predictive maintenance, global rollout/rollback, cross-facility learning, full reporting.
Deferred (designed-for, parked): customer-IdP federation, the full VDA5050 mandatory message set, sub-fleet/group operations.
Out (not fleet software): packaging/unpackaging, power-up/down sequences, EOL testing, removed infrastructure-interaction requirement.
These are the places where the requirement and the architecture need a conscious decision rather than a clean mapping. Calling them out builds the trust this document is for.
Sub-fleet / group operations (599, 3731, 322) mixes two planes: device-management cohorts (OTA, reboot, SSH \u2014 Backbone) and tasking groups (coordinated dispatch \u2014 Switchboard, expressible as affinity on advertised work). It needs a small spec change to split cleanly; we are not adding a monolithic "sub-fleet" object.
Scheduling requirements (3737, 3246): the system is designed for decentralized coordination via Petri nets by default \u2014 and already works this way: robots self-select advertised work, with strong consistency preventing conflicts. That posture prioritizes availability and continuous progress. Central scheduling is then an optional layer on top: a deployment that wants more optimality can switch it on, explicitly accepting reduced availability/progress as the trade. So we satisfy the intent (throughput, balance, goals) decentrally, and offer central planning as an opt-in for sites willing to pay the fragility cost \u2014 rather than baking central planning in as the only mode. These requirements may be worth rewording so they don't presuppose it.
Remote control (3713) rides the existing outbound poll channel as recorded operator intent \u2014 fine for tasking commands. Safety stops (e-stop, go-home in 3253/3741) do not ride this path \u2014 and the issue isn't poll-vs-push latency, it's that safety cannot depend on the facility network at all. Relying on customer WiFi for a remote stop is unacceptable regardless of channel. Safety belongs to a local, hardware-backed interlock at the edge; any remote stop is best-effort supervision layered on top, never the primary mechanism.
Global asset registry (3262) lives in Backbone, not Switchboard. The unified registry experience the requirement implies is delivered by the Fleet Console composing Backbone (asset) + Switchboard (fleet) \u2014 built "on top, after the fact."
Full VDA5050 (3261) is a continuum: the pilot delivers heuristic coexistence + occupancy broadcast (floor tier); the full mandatory message set is post-pilot.
Metrics readiness (Insight). The SoW makes the KPI pipeline an acceptance gate (Annex D SAT-09) and ties Availability/OEE to service credits. Most KPIs are event-derived, not raw gauges, so the work between "we have Grafana" and "we pass SAT" is instrumentation: cause-classified downtime, the SoW error taxonomy, autonomy-vs-intervention tagging, station tagging, battery/swap events, network telemetry. See Spec: Insight.
This mapping is a living artifact. Service boundaries (notably Optimization Agent vs Coordinator, and how much of Device-Ops is Backbone vs a distinct plane) are still being refined; the requirement attributions will track those decisions.
" }, { "id": "1672577044", "title": "Spec: Insight - Fleet Observability", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1672577044", "last_modified": "2026-06-16T17:46:12.816Z", "created_at": "2026-02-25T10:37:37.970Z", "body_html": "Summary: Insight is the derived-truth plane for fleet software \u2014 the read-only analytics & observability layer that turns the streams every other system emits into what humans and contracts need to see and judge. It spans two scopes (fleet/work and device/hardware) over four stores (Mimir, Tempo, Loki, ClickHouse). It owns no source of truth and never feeds coordination.
Owner: Cody Griffin
Date: 2026-06-16
Status: Draft v1 \u2014 supersedes the former Core Events (ClickHouse) spec, which becomes Insight's analytical tier.
Related: Observability Stack (Grafana/Loki/Tempo/Mimir) \u00b7 Spec: Switchboard State Model & Fleet Registry \u00b7 Spec: HMND Outpost & Site Authority \u00b7 Fleet Software: Requirements \u2192 Services Mapping \u00b7 Schaeffler Deployment Pilot SoW (Annex D SATs, Annex E KPIs)
Naming note. "Insight" is a working name for the derived plane; it unifies the existing Observability Stack and Core Events pages under one identity. Alternatives: fold under "Core" (matches the existing "Core observability / Core Events" framing) or "Observability" (undersells the \u00a72/\u00a73 analytics + learning mission). Cheap to rename while it lives in one place.
We have the pieces of an observability stack (Grafana/Loki/Tempo/Mimir) and a speculative event store (ClickHouse), but no plane that says what they are for, where their boundaries sit, and how they produce the things the business and the pilot contract actually depend on \u2014 KPIs, SLAs, dashboards, incidents, audit, and (later) the learning corpus. This spec defines that plane. It subsumes the former Core Events (ClickHouse) spec, which becomes Insight's analytical tier (\u00a77, \u00a710).
Two pressures make this urgent rather than academic:
The Schaeffler SoW ties service credits and termination to Fleet Availability and OEE, and Annex D SAT-09 makes the KPI pipeline itself an acceptance gate \u2014 at commissioning we must demonstrate correct logging and calculation of KPIs and produce a 24-hour KPI report. The derivation layer has to be real and validated before go-live.
Most contractual KPIs are event-derived (success rate, OEE, UPH, intervention rate, error taxonomy), not raw gauges \u2014 so "we have Grafana" is not the same as "we can compute and defend the numbers."
Switchboard coordinates; Insight observes. Insight owns no source of truth: it consumes the Switchboard event log, robot/device telemetry, traces, and Cedar authz decisions, and computes dashboards, KPIs/SLAs, alerts, incidents, benchmarking, explainability, and reports. Everything it holds is derived and can always be rebuilt by replaying its sources.
The hard rule: Insight never closes the loop. A dashboard number must never become a dispatch input \u2014 that would re-introduce the central-planning fragility the architecture rejects. Goal\u2192work bias is the Optimization Agent's job, and even that reads facts, not dashboards. Insight writes reports and alerts; it never writes operational state.
Insight has two source scopes with different owners, identities, tenancy, and consumers. Conflating them is the main way this plane goes wrong.
Fleet scope (work). Sourced from the Switchboard event log. Answers "what is this fleet producing, and is it meeting the contract?" \u2014 UPH, OEE, task/mission success, intervention rate, task-readiness, the Annex E KPIs. Tenant/fleet-isolated (RLS): a customer sees only their own fleet. Bound to the deployment; relevant for the contract window and audit.
Device scope (hardware). Sourced from device/robot telemetry, rooted in Backbone (device identity is global and long-lived). Answers "what is this chassis doing, and how healthy is it?" \u2014 CPU/mem/thermal/current, uptime, MTBF, faults/diagnostics, component wear, predictive maintenance, spares. Primarily HMND-facing: HMND monitors its whole device fleet's reliability across all customers; a customer sees only the health of devices currently assigned to it. Follows the asset across deployments and customers.
Fleet scope | Device scope | |
|---|---|---|
Question | What is the fleet producing? | How healthy is the chassis? |
Source | Switchboard event log | Device telemetry (Backbone-rooted) |
Identity | fleet (rotates per deployment) | device (durable, global) |
Tenancy | tenant-isolated (RLS) | HMND-global; customer sees assigned devices only |
Consumers | customer dashboards, SoW KPIs (\u00a71/\u00a72) | HMND reliability, maintenance, asset mgmt (\u00a73) |
Where the two scopes meet \u2014 availability. A device fault (device scope) causes fleet downtime (fleet scope). The SoW's Fleet Availability definition excludes non-Supplier causes (network, materials, operator, charging), so to defend the availability number you must attribute each downtime interval to a cause \u2014 and that attribution is the join between the two scopes, made through the robot subject (device_ref \u2192 Backbone device id) at the point in time the fault occurred. Availability and MTBF are therefore the canonical place fleet and device meet, and the most contractually load-bearing place in the whole plane. This is the concrete reason we need both: you cannot defend the fleet KPI without the device signal.
The clean line the existing specs draw: metrics/logs/traces describe the system; events describe the work.
Mimir (metrics) \u2014 low-cardinality system health: availability/uptime, latency p50/p95, comm quality, battery levels, device-telemetry rollups. Real-time, cheap.
Tempo (traces) \u2014 per-task lifecycle: the live view, response-time (assign\u2192action) spans, and the drill-down ("show me the trace for this failed pick"). Generates span-metrics into Mimir. Fine at pilot scale; do not tail-sample task traces or success-rate denominators and UPH counts skew.
Loki (logs) \u2014 operational logs, audit/RBAC, incident text, safety-event records.
ClickHouse (events) \u2014 high-cardinality, ad-hoc SQL event store. The home for the Switchboard event log and the right engine for the work-outcome KPIs: UPH per station per bearing-range, success rate by error class, OEE quality by good/total over monthly windows. Exactly what Mimir is bad at (cardinality explosion) and what ClickHouse is for.
All four surface through one Grafana pane. Infra for Mimir/Loki/Tempo/Grafana is specced in Observability Stack; ClickHouse runs in the same footprint (same-host bootstrap, EBS hot / S3 cold), detailed in \u00a710.
The seam between Switchboard and Insight is the append-only event log. Switchboard events ship (via Vector) into the ClickHouse events table; Switchboard dimensions (instance_id, transition, conformant, subject_id, station, error_class, autonomy flag, cause-of-downtime) live in attrs and are promoted to columns when they get hot. That is the same arbitrary \u2192 registered \u2192 promoted evolution the Switchboard fact table uses (A.2) and the same model ClickHouse uses for attrs \u2192 column \u2014 one doctrine across the stack, so the pipeline scales without a schema rewrite.
trace_id is the universal join: one id correlates a ClickHouse event \u2194 a Tempo trace \u2194 Loki logs \u2194 the Cedar authz decision. That correlation is explainability and replay.
Work-outcome KPIs (success rate, OEE quality, UPH, intervention rate, error taxonomy) \u2014 ClickHouse SQL over the event log. High-cardinality, monthly windows.
System-health KPIs (Fleet Availability, uptime, latency, comm quality, battery) \u2014 Mimir time-series; availability needs cause-classified downtime events (a ClickHouse event with cause in attrs, plus a Mimir uptime series).
Response time (assign\u2192action) \u2014 Tempo span \u2192 Mimir.
The 24-hour SAT-09 report and the monthly contractual numbers read from materialized ClickHouse aggregates + Mimir series \u2014 not raw-trace scans. Traces are the per-task substrate and the dispute drill-down, not the monthly aggregation engine.
Instrumentation that must exist for the above (gaps today): cause-classified downtime, the SoW error taxonomy on pick/place outcomes, autonomy-vs-intervention tagging per task, station tagging on completions, battery/swap events, and a robot-side network exporter. Transport (Tempo/events) is not instrumentation \u2014 Reverb/Switchboard must emit these attributes regardless of sink.
Three horizons; the former ClickHouse spec covers only the middle:
Live (Grafana stack) \u2014 days\u2013weeks; real-time ops and alerting.
Analytical (ClickHouse) \u2014 1 day hot \u2192 S3 cold, 180-day delete as currently specced. Covers the full pilot and the May\u2013Nov tracking window, but is not the forever-record.
Archive / learning (lakehouse / Parquet on S3) \u2014 forever; audit, cross-facility learning, training corpus (\u00a73 reqs 3273/3304). Currently out of scope; needed when the forever/learning mission lands. Note the tension against Switchboard's "event log is forever-history" doctrine \u2014 the analytical TTL must be reconciled with that, or archival must catch what ClickHouse drops.
Switchboard \u2014 source and seam; Insight is read-only on it.
Backbone \u2014 owns device identity and the device-telemetry feed; Insight ingests it for the device scope and the global asset/health view.
Console / Fleet API \u2014 presentation vs computation. At pilot scale Grafana is the ops console; at scale the Console composes Insight's outputs into the customer/global product. Overlap resolves with growth.
Optimization Agent \u2014 may read Insight aggregates (utilisation, congestion) to set bias, but the loop stays advisory and off the critical path.
Outpost \u2014 already names cloud as "a mirror and analytics plane"; Insight is structurally the cloud-side sink of the upward sync, and must be stale-aware (Outpost's shadow-vs-live warning). Under site authority, telemetry/events are emitted locally and streamed up.
Pilot is single-tenant (one customer, one site) and the current single-node footprint is fine. Scale (\u00a72/\u00a73 of the requirements) forces the multi-tenant authz the Core Events spec deferred \u2014 and the fleet-vs-device split is exactly what forces it: fleet analytics must be tenant-partitioned (a customer sees only their fleet), while device health is HMND-global with customer-scoped views (a customer sees only its assigned devices). Per Outpost's data-segregation rule, cloud analytics may be multi-tenant internally while the control plane preserves tenant boundaries. Cross-customer benchmarking and the learning pipeline live here and are HMND-internal by construction.
ClickHouse adds the high-cardinality, SQL-queryable event store to the observability footprint (hot on EBS, cold on S3), exposed to Grafana. It complements Loki/Tempo/Mimir; it does not replace them.
Topology (initial). Same EC2 host as Grafana/Loki/Tempo/Mimir for bootstrap (share /var/lib/observability EBS); move to a dedicated instance only if resource contention appears. Same ALB/Nginx front-door pattern and S3 bucket.
Storage. MergeTree with a hot_cold policy \u2014 hot on local EBS, cold on S3 via a ClickHouse S3 disk. Inserts land hot; TTL moves merged parts to S3.
Ingestion. Robots and Core services emit events; Vector forwards over HTTP with async_insert=1, wait_for_async_insert=0, minimal client batching (0.5\u20132s or N events) to keep events visible fast while coalescing server-side.
Event contract. Required: ts, robot_id, robot_type, event_name, severity, attrs. Recommended: trace_id, span_id, workflow_id, task_id, schema_ver. Design principle: low-cardinality fields become columns; high-cardinality/unknown fields live in attrs. The contract carries both scopes \u2014 device events are robot_id-keyed; work/fleet events add workflow_id/task_id/instance_id/subject_id/station/error_class/cause (in attrs, promoted to columns when hot).
CREATE TABLE events\n(\n ts DateTime64(3, 'UTC'),\n ingest_ts DateTime64(3, 'UTC') DEFAULT now64(3),\n\n robot_id String,\n robot_type LowCardinality(String),\n\n event_name LowCardinality(String),\n severity LowCardinality(String) DEFAULT 'info',\n\n trace_id String DEFAULT '',\n span_id String DEFAULT '',\n\n schema_ver UInt16 DEFAULT 1,\n\n attrs Map(LowCardinality(String), String),\n\n event_id UUID DEFAULT generateUUIDv4()\n)\nENGINE = MergeTree\nPARTITION BY toDate(ts)\nORDER BY (robot_type, robot_id, event_name, ts)\nTTL\n ts + INTERVAL 1 DAY TO VOLUME 'cold',\n ts + INTERVAL 180 DAY DELETE\nSETTINGS\n storage_policy = 'hot_cold',\n index_granularity = 8192;\n
Compaction. Background merges combine small parts on EBS; run OPTIMIZE TABLE events FINAL; off-peak (daily is typically enough at low rate) to avoid small-object pathologies in S3.
Access & security. Not exposed directly to the internet; private networking + optional ALB/Nginx front door. Grafana datasource credential in Terraform/Secrets Manager; Vector ingestion via basic auth (mTLS later).
Single-node ClickHouse: instance failure loses hot cache and short-term unshipped parts. Acceptable for bootstrap/pilot; revisit for fleet scale.
Multi-tenant authz deferred in Core Events \u2014 but \u00a79 shows the fleet-vs-device split is what brings it back for scale; treat as the first scale-up work item.
Alerting/SLOs are future work in the Observability Stack, yet the SoW SLA has Sev-1 response \u226410 min \u2014 incident detection/paging must exist for the pilot regardless.
Scaling path: single instance + EBS hot + S3 cold + one events table \u2192 dedicated instance/ECS later, keeping the table definition, storage policy, S3 cold tier, and Grafana datasource. No schema or pipeline rewrite.
The Switchboard spec makes the coordinator deliberately dumb: it answers queries, serializes coordination state, and holds no intelligence. That is the right design, and it creates an obligation \u2014 the intelligence has to live somewhere. Studio is that somewhere: the System-3 plane where agents reason about a deployment and act on it, always through Switchboard's API as authorized principals, never beside it.
We are not starting from zero. The workflow authoring prototype has already demonstrated the core loop end-to-end: natural-language intent in, validated multi-robot workflow out, with a symbolic simulator closing the verification loop. Running that agent in an eval-driven improvement loop over a single weekend took a 7-task suite from 57% to 100% pass, with median prompt-to-successful-workflow time around 30 seconds \u2014 including scenarios that build complex two- and three-robot workflows from an empty map. Product has run Symphony demos end-to-end without engineering in the room, including recording a new workflow and agentically editing it. The prototype proved the concept, built the verification machinery, and \u2014 through its eval suite \u2014 told us precisely where the capability frontier is. This spec is about industrializing that: splitting the prototype's two jobs into two agents, raising the natural-language bar from "engineer-crafted prompt" to "operator sentence," and making evaluation the mechanism by which these agents earn autonomy.
Three observations drive the design:
The prototype does two jobs, and they want different containment. The same agent session edits workflow source and operates live state (pushing runtime facts, resetting markings, editing workflows mounted on robots). The team correctly recognized the risk and added a coordinator-side guard that blocks edits to robot-loaded workflows \u2014 the right instinct, implemented at the only seam available. This spec gives that instinct an architecture: two agents with disjoint capabilities, so the guard becomes structural rather than a check.
Operating is as agent-shaped a problem as editing. During a recent investor demo, a robot stuck on the vinyl floor was given "fail task" instead of a teleop intervention, and the failure handling left the coordinator in a stuck state; the demo was abandoned. Diagnosing why a transition won't fire and how to safely unstick coordination state is exactly the kind of structured, tool-mediated reasoning an agent does well \u2014 and today it lives in the heads of two or three engineers.
The prompts are the product gap. Today, successfully driving the authoring agent requires carefully engineered requests \u2014 naming candidate workflow IDs, exact map tags, state naming conventions, which tool to prefer. That defeats the purpose. The bar is natural language from a non-domain expert: "make the robot bring totes from that stack to the table, three of them" should work; so should "why has robot 2 been standing there for five minutes." Recent eval additions (vague prompts, empty-map-to-complex-workflow) point exactly at this bar; this spec makes it the headline requirement.
The anchor remains the Schaeffler pilot: operators expressing intent conversationally, remote uptime management over flaky links, and workflow evolution on a live deployment without resetting the fleet.
Studio is the collection of System-3 agents that act on a deployment, plus the substrate they share: the intent log, the authoring workspace, the capability catalog, the evaluation harness, and the chat/UI surfaces. Two agents exist at MVP; the architecture admits more (a conformance/analytics agent is the obvious third).
The Operator agent acts on running instances: diagnosing blocked transitions, injecting work, intervening on leases, resolving quarantined or stuck tokens, recording operator-directed deviations.
The Deployment agent acts on definitions: turning intent into validated workflow source edits, publishing versions, and authoring the migrations that move live instances between versions.
Implementation shape: Switchboard is Rust; Studio agents are Python services speaking to Switchboard exclusively through its HTTP/JSON API via a thin typed client. Studio holds what Switchboard deliberately doesn't \u2014 agent context, drafts, intent history, reasoning \u2014 and holds none of what Switchboard does: Studio is never a second coordinator, never caches authoritative state, and every effect on shared state lands through the API, is authorized by Cedar, hits RLS, and is recorded in both audit logs.
In the Sockpuppet intervention tiering, these agents are System-3 intervenors: latency-tolerant, working entirely on Switchboard's replicated state, above remote-embodiment teleop and below nothing except humans. Deployment-by-demonstration (Sockpuppet recording \u2192 coordinator upload) remains the origin of many workflows; Studio is how they evolve afterward. The on-robot reactive stack \u2014 VLA policies and the visual/perception oracle that, for example, scans a shelf and reports which totes are where \u2014 is System-2; Studio's agents invoke those System-2 capabilities as grounding (\u00a78.1) but never replace them.
The split is real on three axes, not an implementation preference:
Blast radius and time constant. The Operator agent acts on the strong core \u2014 markings and leases \u2014 with immediate physical-world consequences, bounded by machinery Switchboard already provides: leases serialize dispatch, conformance is advisory-and-loud, exceptions degrade-and-flag. The Deployment agent's mistakes manifest at publish/migration time and are bounded by validation, human approval, and quarantine.
Grading shape. The Deployment agent is graded like a coding agent: mostly deterministic checks (does the source evaluate, do the traces replay, do the invariants hold). The Operator agent is graded like an intervention agent: a simulated deployment with an injected fault, judged on the resulting state plus how it got there. Two eval suites exist whatever we do \u2014 itself evidence these are two agents.
Context discipline. A single context holding both "edit the definition" and "poke the live instance" tools will, under pressure, fix a definition bug by mutating a running marking, or "fix" a stuck robot by hot-editing the workflow. The separation enforces on agents the same discipline the State Model enforces on the system:
The only path from a definition change to a running instance is an explicit, validated migration. The only path from a runtime observation to a definition change is the intent log.
Concretely: one agent harness, one model family, two principals, two tool mounts, two contexts, two state models (\u00a76). A conversation is either an operating conversation or an editing conversation. When a fault needs a definition fix, the Operator agent files an intent (\u00a711); it does not open an editor.
This is a product requirement with architectural consequences, so it gets its own section.
The requirement. A non-domain expert must be able to deploy and operate workflows with ordinary language. They will not know map tag names, state naming conventions, work-item vocabulary, candidate-ID discipline, which policies exist, or which verification step to run. Every piece of knowledge the user currently has to put into the prompt must move into one of three places:
The source format \u2014 the new-form Starlark (\u00a75) is legible enough that structure the agent used to be told about in prose is now visible in the artifact it reads.
Tools and validators \u2014 invariants that used to be prompt rules ("never invent a tag name", "all robot states start with Robots", "validate before claiming validity") become hard checks: a reference to a nonexistent place is a validation rejection, not a behavioral guideline.
Retrieved context \u2014 map tags, payload schemas, seeds, the capability catalog, scene context, and recent history are assembled by the harness per turn (\u00a78), not recited by the user.
The behavioral contract for vague input. Given an underspecified intent, the agent does exactly one of three things:
Proceed with stated assumptions when the gap is inconsequential ("I'll use three totes since three are on the stack").
Ask one question when the gap is consequential and answerable ("two tables here \u2014 which one?"). One question, not a form.
Refuse with a reason when the request depends on something the deployment doesn't have \u2014 a place not on the map, an object property nothing senses, a capability no policy provides or a question the visual oracle can't answer (all checked against the map and the capability catalog, \u00a78.1). Refusal happens before any mutation. The prototype's refusal discipline (verify feasibility before copying or editing) already implements this and carries forward.
What the agent must never do: silently map unsupported phrases onto the nearest available concept, or invent facts (weights, colors, locations) or policies to make a request feasible. The prototype's prompt encodes this well; the target architecture enforces it with graders (Appendix A's refusal scenarios) rather than prose alone.
The measurement. Every Deployment-agent eval task carries intent variations across specificity tiers \u2014 from engineer-precise ("copy X to candidate Y, move three seeded tote tokens from stack1 to table1/left") down to operator-natural ("get the totes from that stack onto the table"). Tier-3 (vague) pass rate is the product metric; tier-1 pass rate is the floor. The recent vague-prompt eval additions are the first instance of this and become the pattern.
The new-form Starlark workflow (the robostore port is the reference example) is deliberately low-abstraction: work items own their states; tasks declare token flow per alias with three verbs (move, hold, drop); guards are plain named functions over bound token payloads; dynamic values (a shelf tag derived from a basket's payload) are ordinary functions passed as params; payload shapes are declared with token_schema and closed with allow_extra = False; referenced_map_places declares the map dependency. One screenful per concern, no codegen layers, no string-embedded expression language.
This settles the artifact question: the Deployment agent edits this source directly. Not an abstracted projection of it, not a parallel semantic model, not generated YAML. The source is the thing that matters, and it is now legible enough for both a model and a human reviewer to read a diff of it and understand the change. Patches are source diffs; the snapshot is (source, evaluated net, canonical_hash) plus seed; evaluation is hermetic and deterministic, so snapshot regeneration is evaluate(), never an agent call.
Two design consequences worth stating explicitly for implementers:
The prelude stays thin. work_item, token_schema, state, task, move/hold/drop, workflow \u2014 plus, sparingly, helpers for patterns that have proven genuinely recurrent. Resist growing a macro tower: every layer of abstraction between the agent and the net is a layer it can misunderstand. Where the YAML-era prototype needed a heavyweight mutation tool (counted transfers, completion gates), the question is first "does the new source format make this pattern short enough to just write?" before "should this be a prelude helper?" \u2014 and never "should this be an out-of-band mutation tool."
Schemas and guard-functions make the old failure modes checkable. The prototype's eval analysis identified the dominant authoring failures: invented guard helpers, payload fields nothing writes, object lifecycles that look right but don't carry tokens. Closed token_schemas plus guards-as-functions turn most of these into evaluation-time or validation-time errors: a guard referencing a misspelled payload field fails against the schema; a payload field no task or seed ever sets is a lint; the token-flow verbs make the lifecycle explicit in the artifact rather than implicit in input/output plumbing. This is the prototype's hardest-won lesson \u2014 put the domain invariants where the agent can't violate them \u2014 carried from its semantic tool layer into the format itself.
The prototype's tool surface was built for a YAML world and it earned its keep there; in the Starlark world its roles redistribute rather than disappear:
Grounding tools survive intact: list workflows, query structure, get_map_tags (the only legitimate source of place names), list seed robots/roles, read schemas, and read the capability catalog (\u00a78.1). These feed context assembly (\u00a78).
Verification tools survive intact and become the spine: evaluate, validate, symbolic-simulate, and (new) trace replay. The prototype's discipline \u2014 validate after structural edits, simulate after behavioral edits, treat a failed simulation as a failed edit \u2014 graduates from prompt rules into harness enforcement: a patch that hasn't passed validation and simulation cannot be proposed, whatever the model says.
The render/review surfaces survive intact: the diagram, diagram-diff, YAML/source-diff, simulation player, and map-tag views are the human approval UI. This was prescient work \u2014 the tier-4 approval gate (\u00a79.2) is "a human looks at exactly these."
The undo journal and workspace survive intact: turn-scoped undo with snapshot verification, and the disk workspace, generalize into the Studio draft workspace \u2014 per-session, branchable, keyed off the intent log.
Imperative semantic mutation tools shrink to mechanical operations a text edit does badly (copy/branch a draft, rename-with-references) \u2014 because the source is now the editing surface.
Drafts live in Studio, not Switchboard. Switchboard's workflow table remains append-only published versions; the registry stays data; the coordination plane stays dumb. The flow (formalized as a state machine in \u00a76):
intent \u2192 draft branch \u2192 edit source \u2192 evaluate \u2192 validate \u2192 simulate/replay\n \u2192 present (diff + checklist + replay results) \u2192 human approves\n \u2192 publish version (PUT /v1/workflows/$id?from=$ver)\n \u2192 [if live instances] author migration mapping \u2192 human approves \u2192 Switchboard applies\n
Publishing changes nothing running. The Deployment agent authors the migration mapping (old places \u2192 new places) for live instances; the mapping is itself a verified artifact \u2014 every live token either maps or is deliberately quarantined with a reason, checked symbolically and by replaying the drain path on a simulated instance seeded with the production marking. A mapping without total token coverage never reaches the human. Migrate is held by neither agent: the Deployment agent proposes, a human approves, Switchboard applies mechanically.
Runtime-backed editing of mounted workflows \u2014 the prototype mode the approval_required guard was built to police \u2014 is superseded by this flow. The guard's instinct was right; the architecture now delivers it by construction. Similarly, "edit seed then reset-from-seed" is recognized for what it is: applying a seed to a live instance rewrites markings, which is an operating action (a forced migration to the initial marking) and moves to the Operator agent's mount, gated accordingly.
The two agents have two distinct state models that share exactly one object: the intent log. This section gives the Deployment agent's edit lifecycle (\u00a76.1\u20136.4) and the Operator agent's intervention lifecycle (\u00a76.5), then states precisely what is shared and what is not (\u00a76.6). Naming the states matters because at any moment it must be unambiguous what an edit or an intervention is, what is true of it, who can move it forward, and what moving it costs.
The Deployment edit lifecycle has four sub-lifecycles \u2014 two Studio objects (the intent, the why; the patch, the what) that produce two Switchboard objects (the version, an immutable definition; the migration, how live instances adopt a version). The governing rule, restated from \u00a73 as a lifecycle property: publish and migrate are two distinct, separately-gated transitions. A new definition existing, and that definition running on the floor, are different events with different failure modes, and the lifecycle keeps them apart on purpose.
A patch is a proposed change to workflow source, motivated by one or more intents.
State | What it represents | Invariant while here | Leaves via |
|---|---|---|---|
| source edited on a branch | nothing is promised; the agent may still be iterating | submit to validation \u2192 |
| passed validation tiers 1\u20133 (\u00a79.2) | machine-checked: evaluates, traces replay, diff in scope | freeze for review \u2192 |
| frozen and presented for human approval (tier 4, \u00a79.2) | source locked; validation evidence + rendered diff/replay attached | human approves \u2192 |
| a human accepted it | transient; proceeds immediately to publish | publish API call \u2192 |
| committed: an immutable Switchboard version now exists | the patch is now history; the live artifact is the version; nothing running has changed | terminal (for the patch) |
| human declined, with reason | terminal | usually spawns a follow-up intent or a fresh draft |
| author/agent pulled it (stale/superseded) | terminal | \u2014 |
Cross-cutting invariants: a patch cannot be proposed unless validated; cannot be published unless approved; validation is not sticky \u2014 any edit after validated returns it to draft; publishing creates a version but changes nothing running.
Publishing a version with live instances does not move them. The migration is its own proposable, approvable artifact.
State | What it represents |
|---|---|
| agent authoring the old-places \u2192 new-places mapping for a target set of live instances |
| total token coverage (every live token maps, or is deliberately quarantined-with-reason) + drain-path replay on a sim instance seeded with the production marking |
| presented for human approval |
| approved; Switchboard applying mechanically |
| instances now run the new version; recorded as |
A version with no live instances skips this entirely \u2014 publish suffices.
The intent record (its natural-language text + evidence) is immutable; its status is derived from what references it, plus an appended decision marker. open (recorded, unaddressed) \u2192 addressed (a draft/patch cites it) \u2192 implemented (a published version cites it), or declined (closed with a reason, no version). Append-only purity is preserved: we never mutate the text, only append references and decisions. The intent is the one object both agents touch (\u00a76.6).
intent:open\n \u2192 patch:draft \u2192 patch:validated \u2192 patch:proposed \u2192 [HUMAN GATE] \u2192 patch:approved\n \u2192 version:published \u2190 nothing running changes\n \u2192 migration:mapping_draft \u2192 mapping_validated \u2192 mapping_proposed \u2192 [HUMAN GATE]\n \u2192 migrating \u2192 migrated \u2190 instances now on the new version\n
Two human gates (patch approval, migration approval); Migrate is held by neither agent (\u00a713). Operator-agent interventions are not on this chain at all \u2014 they act on the markings and leases of a single live instance and never produce a version (their own lifecycle is \u00a76.5). That is the entire point of the two-agent split (\u00a73): the chain above is the only way a definition changes, and it is disjoint from the way a running instance is nudged.
Mapped to the intuitive intended \u2192 validated \u2192 proposed \u2192 applied: it holds, with one deliberate refinement \u2014 "applied" is two gated steps, published then migrated \u2014 because otherwise "I shipped the new workflow" and "the floor is running it" collapse into one word, and those are different states with different things that go wrong.
The Operator agent's unit of work is a session against one live instance, opened by a trigger (\u00a710.3). It has its own states, none of which ever produce a version:
State | What it represents | Invariant while here |
|---|---|---|
| an operator question, an exception flag, or a scheduled sweep opened a session | read-only |
| investigation: query lenses, event log, eligibility-explain, and optionally a visual query (\u00a78.1) | read-only; no mutation |
| a root-cause hypothesis + a proposed action at the current autonomy tier | read-only |
| a mutating action awaiting its gate (confirm-per-action or human approval), when the tier requires one | nothing mutated yet |
| executing the mutating action (intervene / inject / resolve / reset / deviate) | the one mutating window |
| re-reading state to confirm the fix took (token unstuck, lease single-granted, instance progressing) | read-only |
| state confirmed good; session closed | terminal |
| could not resolve; handed to a human or teleoperator | terminal |
Invariants: read-only states never mutate; a flag-triggered session always reaches at least diagnosed with a proposed action \u2014 it never silently mutates and never silently does nothing; acting is always followed by verifying (no fire-and-forget); autonomy tier (\u00a710.2) decides only whether diagnosed \u2192 acting needs to pass through proposed. From diagnosed, verifying, resolved, or escalated the session may also file an intent (evidence attached) when the root cause is a definition flaw \u2014 this is the only crossing into the edit lifecycle (\u00a76.4), via the shared intent log.
Deployment agent | Operator agent | |
|---|---|---|
State model | edit lifecycle (\u00a76.1\u20136.4) | intervention lifecycle (\u00a76.5) |
Acts on | definitions (source \u2192 version \u2192 migration) | one live instance's markings/leases/facts |
Principal / tool mount / context |
|
|
Produces | published versions + migrations | transient instance changes; no version |
Shared | the intent log (the only object both touch, \u00a76.3/\u00a711); the identity & Cedar model (\u00a713); the capability catalog grounding (\u00a78.1); the eval harness & vocabulary (\u00a712) |
If a future agent is added (e.g. conformance/analytics), it gets its own lifecycle and principal under the same rule: it may append intents, and it shares the catalog and harness, but it does not borrow another agent's tool mount.
Does Studio need its own database, with its own tenancy and RLS \u2014 or can it just be more rows on the Switchboard schema, written through the API, since the coordinator is dumb anyway? The answer is mostly the latter in spirit, with one sharp line.
Principle. Switchboard owns what is authoritative and shared; Studio owns what is proposed and process-private. The boundary between them is exactly the authorization seam from \u00a713: publish and migrate. Everything before a commit is Studio's; the commit and everything authoritative it produces is Switchboard's. "Dumb coordinator" never meant "stores little" \u2014 Switchboard already persists facts, markings, leases, versions, and events. Dumb means holds no reasoning. Adding a provenance stamp to a version is not making it smart; it is making it store one more piece of dumb, authoritative data.
What Switchboard stores (already, plus one addition). Definitions and versions (immutable, append-only); instances, markings, facts, leases, events, migrations; and the capability catalog (\u00a78.1) as published, versioned reference data alongside the map. New: a small immutable provenance stamp on each version (the intent ids + patch canonical hash that produced it) and on each migration (mapping hash + approver). This is the only Studio-origin data that lives in Switchboard; it is written at the seam through the existing publish/migrate API; and it earns its place because "what is running, and why" must be answerable by Switchboard alone, even with Studio down. This is the natural extension of the fact already true \u2014 Switchboard versions workflows \u2014 so the committed link from a version back to its intent belongs there too.
What Studio stores (its own schema). The full intent log \u2014 branchy, with declined entries and rich evidence (Switchboard holds only the committed subset, by reference); drafts, patches, and their entire pre-publish lifecycle (\u00a76.1); snapshots (source, net, hash); migration mappings before approval; Operator session records (\u00a76.5); and everything evaluation \u2014 scenarios, runs, transcripts, token telemetry, conformance computations. This is high-churn, branchy, and bulky; it has nothing to do with coordination; and it must never sit on the robots' lease-acquisition hot path.
Record shapes worth pinning now: an intent is (id, author principal, timestamp, verbatim text, evidence refs [event/trace/instance/version ids], branch, status). A patch is (id, branch, intent refs, source diff, state per \u00a76.1, validation evidence, snapshot ref). Verbatim intent text is load-bearing twice over: it is the provenance answer to "why," and it is the dataset of what operators actually say, which seeds new eval tasks (\u00a712).
Tenancy \u2014 the part not to duplicate. Studio does not reinvent the identity model. It federates against the same IdP, scopes its rows by the same tenant/site/fleet keys, and authorizes its own actions with the same Cedar principals already defined (\u00a713). One source of identity truth; Studio is a relying party. And Studio never copies Switchboard objects \u2014 it reads them live through the API under Cedar + RLS \u2014 so there is no second copy of coordination state to protect, and therefore no parallel RLS to maintain over it. Studio's own rows (intents, approvals) get row-scoping by the same tenant keys, but lighter than Switchboard's four-tier enforcement, because what they protect is proposals, not the strong core.
Physical placement \u2014 pragmatic for a small team. At MVP, a separate studio schema in the same Postgres instance as Switchboard: one thing to run, back up, and observe. The discipline that keeps this honest: even sharing an instance, Studio touches Switchboard state only through the HTTP API, never by reaching into its tables. The shared instance buys operational simplicity, not a license to break the API seam \u2014 which means the provenance links are soft ids, not foreign keys, exactly the convention the State Model already uses for map\u2194workflow refs. Because the references are soft, Studio can later split to a fully separate database \u2014 for failure isolation, or because a transcript flood threatens the hot path \u2014 with no schema change. The architecture pre-approves that split; we just don't pay for it on day one.
Why not all of it in Switchboard? Because the reason the coordinator is dumb is to keep its authoritative core minimal and stable, on a deliberately slow and careful (Rust) release cadence. Published versions belong there because they are authoritative and shared. Half-finished drafts, branchy intent history, and gigabytes of eval transcripts do not \u2014 putting them there would couple Studio's fast Python iteration to Switchboard's release train and drop editing churn onto the robots' hot path. Drawing the line at commit gives us both halves of the intuition: versions and their provenance live in Switchboard; the editing process that produces them does not.
The prototype's instruction set grew to roughly ten thousand words of carefully engineered rules, examples, and tool-preference guidance \u2014 and it worked, which is exactly why it's worth studying. Reading it closely, almost every rule is one of: (a) domain structure the new source format now expresses, (b) an invariant the validator should enforce, (c) context the harness should retrieve, or (d) genuine behavioral policy. Only (d) belongs in the prompt. The target: a system prompt an order of magnitude smaller, with everything else supplied as structured per-turn context.
What the harness assembles per turn, in priority order:
The working artifact: current draft source + seed (or, for the Operator agent, the instance's marking/facts/leases via the lenses). The prototype's pattern of appending the full current workflow context to workflow-scoped tool results \u2014 keeping only the latest snapshot live in history \u2014 is the right mechanism and carries forward.
Grounding: map tags for the target map version, payload schemas, seed robots/roles, declared invariants, and the capability catalog (\u00a78.1) \u2014 the policies that exist, the prompts they support, and the questions the visual oracle can answer. The user never supplies these; the agent never invents them.
History: the recent intent-log entries and patches for this draft branch (so the agent understands not just the state but the trajectory \u2014 the v0 Studio insight), plus a compacted conversation.
Validation state: the freshest validate/simulate results, clearly marked as possibly stale after any edit.
Compaction policy (the prototype's is correct and keeps its essentials): never touch the current turn; truncate historical tool outputs, messages, and reasoning aggressively; drop oldest-first under a hard budget; strip orphaned tool-call/return pairs after every drop; retain exactly one full artifact snapshot. Implementers should treat the prototype's session-history / history-processor modules as the reference implementation.
Two rules for whoever implements this:
Knowledge moves down the stack, never up. When a behavior needs fixing, the order of preference is: make it impossible in the source format \u2192 reject it in the validator \u2192 catch it in a grader \u2192 only then add a prompt rule. A prompt rule is a confession that the lower layers have a gap.
The user's words are intent, not instructions. The harness owns naming, seeding, candidate management, and tool choreography. If an eval task can only pass when the user's prompt names a state, the task (or the agent) is wrong.
The single most effective way to stop the agent inventing "random shit" is to give it the closed set of things the deployment can actually do, and to make anything outside that set a hard error rather than a hopeful guess. That closed set is the capability catalog: the grounding source that enumerates what the robots and the perception stack can be asked to do, on this site, at this software version.
It has two parts:
Supported policies and their prompts. Every executor-backed action a task can invoke \u2014 navigate, policy, pick_policy, place_policy, and so on \u2014 with the prompt templates each trained policy actually supports and the slots they accept (e.g. a pick policy that takes a target description and a shelf level), plus preconditions (a placement policy may require the target below torso height; a pick may require a tag-relative approach offset). A task whose executor+task_type+prompt does not resolve to a catalog entry is a tier-1 validation rejection (\u00a79.2), exactly as a navigation tag that isn't on the map is. This is what makes "pick the {item} and juggle it" fail loudly instead of producing a plausible-looking workflow that no policy can run.
Visual-query (System-2) capabilities. The questions the perception oracle can answer about the scene, as a typed query schema \u2014 for CEVA, "which totes are present in the shelf and where (which column, top vs bottom)"; in general, occupancy, presence, identity-of-object, pose-relative-to-tag. The catalog states each query's inputs and the shape of its answer, so the Deployment agent can wire a visual query as a workflow oracle (feeding a guard or a navigation offset) only where one really exists, and the Operator agent can issue a visual query as a diagnostic \u2014 "is there actually a tote in front of the robot, or is the marking lying?" Crucially, the answer to a visual query is the perception system asserting reality, not the agent; this is what lets the Operator agent ground a diagnosis in what the robot sees without violating the no-impersonation rule (\u00a710.1). It is the sanctioned bridge from System-3 reasoning down to System-2 perception.
The catalog is published, versioned reference data, read via the API \u2014 authoritative-and-shared in the sense of \u00a77, pinned per site and software version so that grounding always matches what is actually deployed. Both agents read it; the validator enforces against it; refusal scenarios (\u00a712) test that out-of-catalog requests are declined rather than approximated. Exact ownership \u2014 whether it is generated from the trained-policy registry on the ML side, declared in the robot software, or assembled by Switchboard \u2014 and whether it participates in a definition's canonical hash the way the prelude does, are open questions (Appendix D).
One intent-log entry drives one agent turn-sequence: assemble context (\u00a78) \u2192 propose a source patch \u2192 evaluate \u2192 validate \u2192 simulate/replay \u2192 either iterate on failures (the prototype's repair discipline: a non-quiescent or wrong-outcome simulation is a failed edit to keep working, not a caveat to report) or present for approval. Multi-turn is the normal case, not the exception: incremental evolution of a live workflow over many intents is the product, and the intent log is the unit of work. Each turn moves the patch through the states of \u00a76.1.
Four tiers, ordered by cost; every patch passes them in order; a patch is not proposable until tiers 1\u20133 pass (the draft \u2192 validated gate of \u00a76.1), and tier 4 is the proposed \u2192 approved gate.
Deterministic-structural (every edit): source evaluates hermetically within budget; every referenced place resolves in the target map version; every task's policy and prompt resolve in the capability catalog (\u00a78.1); schemas close (no undeclared payload fields, no unread/unwritten fields); every task has an executor; every token class can reach a terminal state; capacity sanity; named invariants attached to the definition (e.g., "this net supports N concurrent robots") assert per version \u2014 so an edit that accidentally serializes a parallel workflow is caught before any human reads the diff.
Behavioral \u2014 trace replay: the simulator runs from the draft seed; canonical traces tagged must-pass replay identically; traces the intent deliberately changes are re-tagged, and the re-tagging is part of the reviewed diff. The validator is the runtime: the Python symbolic simulator is acceptable for the inner loop exactly insofar as it implements the engine's semantics; as the Rust engine matures, behavioral truth migrates to replaying through a simulated instance of the real engine (the State Model already permits sim instances of a definition). Divergence between validator and engine semantics is the class of bug that produced the Robostore basket-swap regressions \u2014 the canonical-trace corpus exists to catch it from both sides.
Model-based \u2014 scope check: a rubric-graded judgment that the diff implements the intent and nothing else, and that the regenerated SOP projection is consistent with the net.
Human approval, made cheap: source diff, green checklist, replay results, diagram diff. The reviewer confirms judgment, not correctness \u2014 correctness was already machine-checked.
Once a version is live, observed execution (the event log) is compared against the definition: a conformance score per instance, computed over events, with deviations attributable to a specific version and therefore to a specific patch and intent (the provenance chain, \u00a711). When conformance drops \u2014 robots improvising, operators repeatedly intervening at the same transition, the same token class quarantining \u2014 that is the system telling us the definition is wrong. The adaptive loop from the v0 Studio spec becomes concrete and stays propose-only: a conformance drop files an intent with the deviating traces attached as evidence; the Deployment agent proposes a patch; a human approves. No autonomous definition changes from conformance signals.
Almost entirely existing Switchboard actions, mounted as tools on the operator principal:
Diagnose \u2014 the three query lenses, the event log, and one new API requirement this spec levies on Switchboard: an eligibility-explain endpoint (GET /v1/instances/$id/eligibility?explain=true) returning per-transition why-not \u2014 failed guard (and which payload comparison failed), stale/wrong-epoch/wrong-map fact, capacity block, missing token, lease held by whom. The prototype's analyze/list_enabled simulator actions are the offline prototype of exactly this; the live endpoint is the same computation against the relaxed projection, with freshness annotations. Without it the agent reconstructs guard logic from raw facts \u2014 the dominant token cost observed in prototype sessions. The agent may also issue a visual query (the System-2 perception oracle, \u00a78.1) to ground a diagnosis in what the robot actually sees \u2014 "is there really a tote there?" \u2014 where the answer is the perception system asserting reality, not the agent.
Intervene \u2014 transactional release \u2192 reacquire (existing primitive, capability-gated). This is the correct fix for the vinyl-floor incident class: rather than "fail task" wedging coordination state, the lease is reassigned atomically and the work returns to the pool or a teleoperator.
Inject \u2014 work tokens via source transitions, same shape as the WMS adapter.
Resolve \u2014 quarantined or stuck tokens: map to a place or terminal-ize the subject; the "operator/agent resolves" cell of the exception table.
Apply seed / reset marking \u2014 gated like forced migration; this is where the prototype's reset-from-seed capability lives now.
Record deviations \u2014 conformant=false, actor=agent:<session>, loud \u2014 only as operator-directed execution. Deviate exists for actors that know reality better than the schematic; a robot qualifies, an LLM reading Switchboard's own replica does not. A human asserts the reality claim; the agent mechanizes the resolution.
Declare facts \u2014 operator-class declared facts only (TTL or sticky, source recorded). The Operator agent never writes a fact whose declared writer is robot-exec: it does not perceive reality and must not impersonate telemetry. Role changes route through operator-declared facts or assignment.
Reads are free. Intervention and injection start confirm-per-action. Resolution, marking resets, and deviations start human-approved. Each tier graduates to autonomous-with-notification only when its eval suite holds a consistency bar (\u00a712.3) over a defined window. Autonomy is deployment configuration, not prompt content \u2014 a site can pin the agent read-only. In lifecycle terms (\u00a76.5), the tier decides only whether diagnosed \u2192 acting must pass through the proposed gate.
Three entry points open a session (\u00a76.5): an operator asks ("why is r2 just standing there"); an exception flag fires (quarantine, dropped tote, lease timeout, conformance event) and the agent investigates and proposes; scheduled sweeps (stale leases, growing quarantine, eligibility deserts). Flag-triggered sessions always produce a diagnosis plus a proposed action at the current autonomy tier; they never silently mutate.
With the state models (\u00a76) and storage (\u00a77) defined, traceability is what they buy:
Provenance, both directions. The committed chain \u2014 operational incident \u2192 intent \u2192 patch \u2192 version \u2192 migration \u2192 events \u2014 is queryable end to end. "Why does this workflow do X" resolves to a person and a sentence (the intent text); "what did that change on the floor" resolves to event-log deltas and conformance movement. The provenance stamps in \u00a77 are what let the chain survive even with Studio offline.
The intent log is the only seam between the agents. The Operator agent appends intents (with trace evidence) and cannot author patches; the Deployment agent consumes intents and observes live instances only through published read lenses. No shared context, ever \u2014 an append-only log is the entire interface. This is how a stuck-robot session ends properly: resolve the token now (Operator), file the intent so it stops happening (the log), patch the definition later (Deployment).
Undo is traceable, not magical. Within a draft, the prototype's turn-scoped undo journal applies. Across published versions, undo is just another migration with its own mapping \u2014 recorded, approved, reversible in turn.
This section assumes no prior familiarity with agent-evaluation practice; the vocabulary it establishes is used by the appendices and should be used by the implementation.
An evaluation gives an agent a defined starting situation and a request, lets it run, then applies graders \u2014 checks against what it produced \u2014 to decide pass/fail. A single defined situation+request+graders is a task; one attempt at a task is a trial. Because model outputs vary, a task is run as multiple trials and we look at rates, not single results. The complete record of a trial \u2014 every message, tool call, and result \u2014 is the transcript; the final state of the environment is the outcome. A suite is a set of tasks measuring one capability area, and the harness is the infrastructure that sets up a clean environment per trial, runs trials, grades, and aggregates.
Two non-obvious rules that practice has taught everyone who does this:
Grade outcomes, not paths. Asserting the agent called tools in a particular order produces brittle suites that punish valid creativity. Assert what ended up true: the tokens are where they should be, the lease was reassigned exactly once, the workflow's simulated execution moves every tote through pick\u2192carry\u2192place. (Our existing path assertions on token trajectories are outcome checks \u2014 they grade the workflow's execution, not the agent's tool sequence \u2014 and are exactly right. A path assertion on the agent's tool calls is justified only when it encodes a safety property, e.g. "map tags were fetched, therefore not invented.")
Each trial starts clean. Shared state between trials (a reused database, a leftover workspace) produces correlated failures and phantom successes. For us: fresh compose stack, fresh Postgres, fresh workspace, every trial.
Three grader families, used together: deterministic (code checks: counts, paths, validity, quiescence \u2014 fast, objective, the backbone), model-based (an LLM judging against a written rubric \u2014 for qualities code can't check, like "did the diff stay in scope"; must be spot-calibrated against human judgment), and human (expensive; reserved for calibrating the model graders and for approval itself).
A capability suite asks "what can the agent do?" It is supposed to have a low pass rate: it encodes the frontier, and its score is the hill the team climbs. The realistic-suite baseline is a textbook example \u2014 it measured exactly where the agent stood and decomposed the gap into named failure categories, which is what made the weekend improvement loop possible. A regression suite asks "does it still do everything it used to?" and should sit near 100%; any drop is an alarm. Tasks graduate from capability to regression as they're conquered, and the regression suite then gates every change \u2014 agent prompts, model swaps, harness changes, and Switchboard engine changes, since the canonical traces exercise the engine and the agent simultaneously.
Two corollaries we adopt as policy:
A task at 0% across many trials is presumed a broken task before a broken agent \u2014 check the spec and graders first. Every task should have a known reference solution that passes its own graders, proving the task is solvable as written.
When a suite is recalibrated, the baseline resets: suite changes and agent changes are reported separately, never blended into one number. The calibration round established this discipline; it is policy.
For a per-trial success rate p over k trials: the chance of at least one success ("pass@k") rises with k; the chance that all k succeed ("pass^k") falls with k. Which matters depends on the product moment. For an engineer iterating in a workspace, pass@k is fine \u2014 retry is cheap. For an operator at a customer site, consistency is the metric: pass^k is the headline number for both agents, and it is the number autonomy tiers (\u00a710.2) ratchet on. Cost and latency (tokens per accepted patch, time to proposal) are tracked alongside as budgets, not afterthoughts \u2014 the baseline's worst single task burned ~870k tokens, which is a cost bug a budget grader would have flagged.
Deployment agent. Tasks are (baseline workflow source, map, seed, intent[, follow-up intents]) \u2192 outcome properties, with intents in specificity tiers (\u00a74). Corpora in order of acquisition cost: the existing mini/realistic/fresh suites, recalibrated (the fresh suite \u2014 empty map to multi-robot workflow \u2014 is the strongest capability signal and stays hard); YAML\u2192Starlark conversion of every existing workflow, where "convert and preserve behavior" is a task with trace replay as oracle, so conversion tooling and eval corpus are one body of work; named historical regressions (the basket-swap/duplicate-item coupling, with the traces that would have caught it); migration-mapping tasks (live marking + definition change \u2192 total mapping). Multi-turn tasks are first-class \u2014 the round-trip finding (one-shot prompts under-test incremental editing) generalizes: the suite must contain author-then-amend sequences because that is the product.
Operator agent. The State Model's G.1\u2013G.12 scenarios are the seed corpus, nearly verbatim: each is already preconditions \u2192 fault \u2192 oracle. The harness injects the fault into the compose stack (expired lease, epoch-stale fact, quarantined token, interrupted migration, a wedged "fail task"), and graders check outcome state (token unstuck, no double-granted lease, expected events with actor=agent:*), transcript constraints (no mutating call absent from the proposed plan; call budget), and a diagnosis-quality rubric. New tasks are minted continuously from production exception flags: every real incident that reaches a human becomes a task.
Refusal and restraint, both directions. Suites must be balanced: tasks where the agent should act and tasks where it must not (unsupported requests, infeasible placements, requests requiring a policy the catalog doesn't list or a visual query the oracle can't answer, requests requiring facts nothing senses). Refusal prompts must not lead ("sort totes by weight", not "refuse to sort totes"); refused means no mutating calls, reads allowed. One-sided suites produce one-sided agents \u2014 grade both the doing and the declining.
Read the transcripts. Scores are not trusted until someone reads transcripts from failing and passing trials and confirms the failures are fair and the passes are real. The eval reports written for the baseline and calibration rounds set the standard; one such report accompanies every suite revision.
Two principals, expressible almost entirely in existing State Model actions:
// Deployment agent: config plane only. Never touches live coordination.\npermit (principal == Service::"studio-deployment",\n action in [Action::"PublishDefinition", Action::"ReadMap", Action::"ReadInstance"], resource);\nforbid (principal == Service::"studio-deployment", action, resource)\nunless { action in [Action::"PublishDefinition", Action::"ReadMap", Action::"ReadInstance"] };\n\n// Operator agent: operator-shaped; no config plane.\npermit (principal == Service::"studio-operator",\n action in [Action::"Intervene", Action::"InjectWork", Action::"WriteFact",\n Action::"ReadFleet", Action::"ReadMap", Action::"ReadInstance"], resource)\nwhen { resource in principal.fleet };\nforbid (principal == Service::"studio-operator",\n action in [Action::"PublishDefinition", Action::"Migrate"], resource);\nWriteFact for the operator principal is constrained at the fact-writer-declaration layer to operator-class keys \u2014 never robot-exec-owned telemetry. Migrate is held by neither principal; the human-approval flow fires it. Deviate is granted to the operator principal only under an operator-directed context flag. Sessions are distinguished in the event log via actor = agent:<session-uuid> rather than per-session principals. Whether one Python service or two hosts both agents, the audit log distinguishes the principals \u2014 which is what matters when a customer asks who moved a token.
This authorization seam is also the data seam (\u00a77): the two actions the Deployment agent and the approval flow can perform \u2014 PublishDefinition and Migrate \u2014 are exactly the points where Studio-origin data crosses into Switchboard.
Placement: the Deployment agent is a development-loop service and can live in the cloud. The Operator agent must run with the deployment \u2014 it is useless if it cannot reach Switchboard during the WAN flap that caused the fault \u2014 making it part of the Outpost footprint at site-authoritative tiers.
Switchboard is dumb; Studio is smart; Studio acts only through Switchboard as an authorized principal. The agent edits the source that matters, not an abstraction of it; abstraction lives in a thin prelude and a strict validator, never between the agent and the net. Natural language is the bar: knowledge moves into the format, the tools, and the retrieved context \u2014 the prompt shrinks, the user's words stay intent. The agent grounds every prompt against the capability catalog and never invents a policy; the same catalog tells it what the System-2 visual oracle can be asked, and a request outside it is refused, not approximated. Two agents, two state models, two principals, two contexts \u2014 sharing exactly one object, the intent log. An edit is always in exactly one named state; definitions change by draft \u2192 validate \u2192 propose \u2192 publish \u2192 migrate, and publish and migrate are two gated transitions, never one; instances change by operator interventions that never produce a version; the two paths never share a context. The data seam is the authorization seam is the API seam \u2014 publish and migrate: Switchboard owns what is authoritative and shared, Studio owns what is proposed and process-private, and the committed provenance stamp is the only Studio-origin data in Switchboard. The intent log is the only seam between the agents and the provenance root for every change. The validator is the runtime. Conformance closes the loop, propose-only. Neither agent migrates \u2014 humans do. Autonomy is earned by consistency on evals, not asserted in config. The Operator agent never impersonates telemetry; when it needs to know reality, it asks perception. Grade outcomes, not paths; keep one suite climbing and one suite green; reset baselines when suites change; read the transcripts.
The existing scenario format is sound and carries forward with targeted extensions. A scenario is one YAML document:
scenario_id: <stable id>\ndescription: <one sentence: what capability this measures>\nagent: deployment | operator # NEW: which suite this belongs to\nsuite: capability | regression # NEW: lifecycle stage; graduation flips this\ntarget_workflow_id: <candidate id the harness manages> # deployment tasks\nbaseline_workflow: <path or version ref> # starting source; omit for fresh tasks\ncapability_catalog: <ref> # NEW: the policy/visual-query universe in scope (\u00a78.1)\nsimulation_seed: <path or inline tokens/facts> # ALWAYS seed physical state inline or by ref;\n # never expect the agent to invent seeds\nfault: <operator tasks only \u2014 injected condition, see below>\ncanonical_intent: <one sentence>\nintent_variations: # specificity tiers (\u00a74)\n tier_1: [...] # engineer-precise\n tier_2: [...] # informed user\n tier_3: [...] # operator-natural / vague\nfollow_up_intents: [...] # multi-turn tasks; applied after first success\nprompt_sampling: {sample_size, seed}\nsimulation: {max_steps}\nbudgets: {max_tool_calls, max_total_tokens} # hard on regression, tracked on capability\nreference_solution: <path> # proves the task solvable as written\nsuccess_criteria: [...] # assertion list, below\nAssertion vocabulary (existing types retained, semantics unchanged unless noted):
workflow_exists, workflow_valid \u2014 candidate present; tier-1 validation passes.
policy_supported {task} \u2014 NEW: the task's executor + task_type + prompt resolve to a capability-catalog entry (\u00a78.1); the structural complement of "tag is on the map."
visual_query_supported {query} \u2014 NEW: a workflow-wired or operator-issued visual query resolves to a catalog query type.
workflow_has_work_items / workflow_lacks_task_types \u2014 structural presence checks (renamed from work_types to match the new source vocabulary).
workflow_state_capacity {state, expected} \u2014 buffer/capacity properties.
invariant_holds {name} \u2014 a named invariant attached to the definition (parallelism, boundedness) asserts true post-edit.
task_param {task_name, param_key, expected_value} \u2014 exact param checks (e.g., a policy prompt).
token_count {state, expected} \u2014 final-marking counts after simulation.
sim_token_path_count {work_item, work_data?, expected, comparison, sequence} \u2014 the workhorse: asserts N tokens of a class traverse a given lifecycle (with consumed/produced/alias locations and repeat blocks). Assert paths for every relevant token class \u2014 object and robot and order \u2014 final counts alone miss role merging and order/flow decoupling.
quiescent \u2014 simulation terminates with no enabled transitions (no infinite loops, no premature termination).
trace_replay {trace_id, expectation: identical | retagged} \u2014 canonical-trace replay for regression tasks.
migration_total {instance_ref} \u2014 every live token maps or is deliberately quarantined with reason.
refused \u2014 no mutating tool calls; read-only calls permitted. Used for out-of-catalog and infeasible requests.
state_check {path, expected} \u2014 operator tasks: arbitrary checks against Switchboard state (lease tables, quarantine, event log entries incl. actor).
tool_used {tool_name} \u2014 permitted only when it encodes a safety property (e.g., get_map_tags before authoring navigation), never as path preference.
max_tool_calls, max_total_tokens \u2014 budget graders.
Operator-task fault vocabulary (initial set, drawn from G.1\u2013G.12 and incident history): lease_expired, lease_wedged_by_failed_task, fact_stale {key, epoch_behind}, token_quarantined {reason}, migration_interrupted, capacity_deadlock, robot_delocalized. Each fault is an idempotent setup script against the fresh compose stack.
Naming discipline. Terminal/state names in oracles follow the standard conventions (TotesOn_<place>, RobotsAt_done, CompletedOrder), and seeds + prelude establish those names so the agent inherits rather than invents them. Do not build semantic-equivalence graders; standardize the names. (Calibration finding; adopted as policy.)
Deterministic graders run in-process against the harness's Switchboard client and the evaluated net: structural checks (A's vocabulary), final-marking queries, event-log queries, replay comparison. They are the backbone; prefer them whenever the property is expressible in code.
Trace replay is the behavioral regression mechanism: a canonical trace is a recorded sequence of transition firings with bound tokens from a known-good run (real or simulated). Replay = instantiate the definition on a sim deployment, drive the trace through the real commit path, diff resulting markings and events. identical expects exact agreement; retagged records a reviewed new expectation when an intent deliberately changes behavior. Trace corpus sources: golden demo runs, eval reference solutions, and production event-log extracts (sanitized).
Model-based graders are used for exactly two judgments at MVP: scope ("does this diff implement the intent and nothing else" \u2014 rubric over intent text + source diff) and diagnosis quality (operator tasks \u2014 rubric over the transcript's stated diagnosis vs the injected fault). Rules: one rubric per judgment, graded by an isolated call (don't ask one judge for five dimensions); give the judge an explicit "Unknown" escape; spot-calibrate against human judgment on a sample each suite revision; record judge model+rubric version with every score.
Partial credit applies to multi-property tasks: an operator trial that correctly diagnoses but proposes the wrong remedy is meaningfully better than one that mutates blindly; score dimensions separately and report the vector, not only the conjunction. The pass/fail used for pass^k remains the conjunction.
Trial mechanics: fresh compose stack + Postgres + workspace per trial; fixed prompt-sampling seeds for reproducibility; N trials per task (default 3; raise for graduation decisions); transcripts persisted with per-request token usage (the prototype's usage plumbing carries forward); one written report per baseline or suite revision, in the established report format (summary table, per-scenario results, failure-assertion histogram, interpretation, trustworthiness caveats).
Prototype component | Disposition |
|---|---|
Symbolic simulator, validator, reachability analysis | Carry forward; spine of tiers 1\u20132; converges toward replay through the Rust engine |
Render surfaces (diagram, diffs, sim player, map tags) | Carry forward; they are the tier-4 approval UI |
Undo journal (turn-scoped, snapshot-verified) | Carry forward; draft-workspace primitive |
Disk workspace | Carry forward; becomes the branchable, intent-keyed Studio workspace |
Session compaction + tool-result context retention | Carry forward; reference implementation for \u00a78 |
Grounding tools (map tags, seed robots, structure queries) | Carry forward; feed context assembly; extended with the capability catalog (\u00a78.1) |
Visual/image-query oracle (e.g. CEVA shelf scan) | Promote to a catalogued System-2 capability (\u00a78.1): wired by the Deployment agent, queried by the Operator agent for perception-grounded diagnosis |
Semantic mutation tools | Honorably retired as the editing surface; their invariants live on in the prelude + validator; mechanical ops (copy/branch/rename) remain |
Raw YAML edit tools | Become source-text edit tools |
Runtime-backed editing + | Superseded by draft \u2192 publish \u2192 migrate (the guard's instinct, structural) |
| Move to the Operator agent's mount, gated as marking-rewrite operations |
Eval scenarios, harness, reports (incl. the weekend improvement loop) | Carry forward; recalibrated per Appendix A; the improvement loop is the development methodology |
Capability catalog ownership & versioning \u2014 is it generated from the trained-policy registry (ML side), declared in the robot software, or assembled by Switchboard? How is it pinned per site/software version, and does it participate in a definition's canonical hash the way the prelude does?
Visual-query schema \u2014 the formal set of questions the System-2 oracle answers (inputs + answer shape), and how a new query type is registered so the catalog stays truthful.
Intent schema details \u2014 evidence attachment types and retention; whether declined intents need a review queue.
Provenance stamp shape \u2014 exactly which fields land on the Switchboard version/migration rows (intent ids + patch hash at minimum) vs stay Studio-side by reference.
Studio store split trigger \u2014 concrete signal (transcript volume, hot-path contention) that moves Studio from the shared studio schema to a separate database.
Eligibility-explain consistency \u2014 relaxed projection with freshness annotations (position) vs strong read; and whether explain should include counterfactuals ("would fire if fact X were fresh").
Prelude governance \u2014 who may add helpers, and versioning: the prelude is pinned per definition and participates in the canonical hash.
Operator-agent inference at site-authoritative tiers \u2014 local model vs degraded explain-only mode during WAN loss (Outpost question).
Per-session principals vs actor=agent:session in the event log (position: the latter, until an auditor demands otherwise).
Trace sanitization for production-derived canonical traces at customer sites.
move/hold/drop exact semantics \u2014 confirm the consume/produce behavior of drop against the prelude before finalizing \u00a75 prose.
Summary: One logical control plane with movable operational authority \u2014 cloud or on-prem outpost \u2014 for site robotics deployments.
Owner: Cody Griffin
Date:
Approvers:
Informed:
Related:
Spec: HMND Fleet Control Plane
As we move robots out of demo areas and into real customer factories, a single architectural question keeps recurring: where does operational truth live? Some deployments are happy talking to HMND cloud over WAN. Others need low latency, local autonomy, offline operation, or on-prem provisioning, and cannot tolerate a factory that stops working when the internet hiccups.
The naive response is to fork the product: a "cloud" version and an "edge" version, plus a separate factory provisioning tool, plus a pile of bespoke install scripts. That path leads to two codebases, two mental models, YAML archaeology, and a field engineer who is the only person who knows the working incantation.
We want a single control plane where authority is placeable, not a family of divergent products. The same Switchboard, the same identity model, the same provisioning machinery \u2014 deployed with the authority either in the cloud or on a local outpost cluster, depending on what the customer is paying for.
A second recurring problem is that we do not own the customer's network. In most real deployments the customer provides Wi-Fi, switching, routing, firewall policy, DNS, DHCP, NTP, power, and egress. The outpost has to qualify and survive that infrastructure rather than pretend it owns it.
In Scope:
The site / outpost / authority abstraction and the product tiers (Tier 0, 1, 3, Airgap) derived from it.
Placement model for the Site Authority: cloud-hosted vs. local outpost cluster, with one authority per site.
Local edge software stack (k3s, Postgres/PostGIS/pgvector, NATS JetStream, ZFS blob store, OTel/Prometheus/Grafana).
State classification and sync semantics: runtime-local vs. durable-authored, and lost-write / split-brain behavior per tier.
Outpost as a fleet device (identity, enrollment, versioning, lifecycle) and as a provisioning appliance.
Identity & PKI model separating device identity, fleet identity, entitlement, and lease.
BYON (bring-your-own-network) as a first-class deployment mode, including network & power qualification.
Segregation model across tenant, network, trust, and data dimensions.
Reference hardware profile and rough BOM targets.
Out of Scope:
Detailed workflow / coordinator semantics \u2014 owned by Spec: HMND Fleet Control Plane.
Full PKI cert lifecycle implementation detail \u2014 see Spec: HMND Core Control Plane; this spec only covers placement and trust posture.
Local GPU inference on the outpost (slot is reserved in the hardware profile; not part of the default operational surface).
Multi-tenant outposts (default customer outpost is single-tenant).
Concrete airgap transfer tooling (acknowledged as a separate operational contract, priced and specced separately).
The system is built around one abstraction: a site has an operational control plane, and that control plane can be placed either in HMND cloud or on customer premises. The outpost is the hardware/software substrate that lets us place the control plane locally when the customer needs autonomy, low latency, provisioning, or offline operation.
The key idea is that outpost size and authority placement are separate concepts. An outpost may have zero, one, or three local nodes. Authority may live in the cloud or at the site. Those two axes combine into a small set of product tiers \u2014 not a product fork.
![\"image-20260605-130347.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/2061434886/image-20260605-130347.png?version=1&modificationDate=1780664629536&cacheVersion=1&api=v2&width=1536&height=1024\")
Invariant: Every site has exactly one operational authority. For Tier 0 and Tier 1 that authority is the cloud; for Tier 3 and Airgap it is the local outpost cluster. Everything else is cache, proxy, mirror, provisioning appliance, or analytics sink.
Site \u2014 the physical deployment boundary: one factory, lab, warehouse, customer location, or logically isolated operating area. A site contains robots, maps, workflows, semantic zones, operators, provisioning state, local integrations, and runtime state.
Outpost \u2014 HMND-managed edge infrastructure at the site. Absent, single-node, or three-node HA. It can run robot gateway services, local caches, artifact stores, provisioning flows, factory integration adapters, local UI, observability, and (in Tier 3) the Site Authority itself. The outpost is treated as a fleet device: device identity, enrollment state, software version, certificates, health, logs, cloud-managed lifecycle.
Authority \u2014 the writer of operational truth. Operational truth includes workflow markings, robot leases, task ownership, active map versions, active policy versions, fleet configuration, command authorization, and site-local runtime decisions. In Tier 0/1 the cloud is authoritative; in Tier 3 the local outpost is authoritative and cloud receives an async mirror.
Segregation \u2014 whether customer workloads, robot traffic, factory integrations, management traffic, and cloud egress share the same network and trust surface. The default is logically segregated even when the customer provides the physical network: separate identities, separate mTLS policies, separate entrypoints, VLAN separation where available.
The two axes (outpost size, authority placement) collapse into four product modes with explicit contracts:
Tier | Local nodes | Authority | Use case | WAN-outage behavior |
|---|---|---|---|---|
Tier 0 | 0 | Cloud | Demo / trial / lab | Fleet operation stops or degrades to robot-local safe behavior. Acceptable \u2014 not sold as production autonomy. |
Tier 1 | 1 | Cloud | Small deployments wanting local latency, buffering, aggregation, better operator UX | Local cache/UX degrades; durable config unaffected because it lives in cloud. Node replacement hydrates from cloud. |
Tier 3 | 3 (HA) | Site | Serious production factories needing local autonomy | Becomes a visibility / remote-management problem, not a factory-death problem. Local operation continues. |
Airgap | 3 (usually) | Site | Regulated / disconnected sites | No cloud sync, or delayed/manual/sanitized. Separate operational contract; priced for the field burden it transfers. |
This avoids a product fork. There is no "cloud Switchboard" and "edge Switchboard" \u2014 there is one Site Authority implementation with different placement. Tier 0 still uses device and fleet identity so a demo robot is never a snowflake that has to be reprovisioned through a bespoke pipe later.
The edge stack stays close enough to cloud deployment patterns that engineering does not maintain two religions. Cloud may run ECS and Kubernetes; the outpost runs a small Kubernetes-compatible environment (k3s + containerd). ECS is not reproduced on-prem \u2014 product services are containerized with a placement abstraction (cloud ECS / cloud k8s / site k3s).
OS: Ubuntu LTS (immutable/container-focused Linux later)\nRuntime: k3s, containerd, Helm/Flux/Argo-style deploy, local image cache\nStorage: ZFS local disks; content-addressed blob store on ZFS; MinIO only if S3 semantics needed\nDatabase: Postgres + PostGIS + pgvector; WAL archiving / snapshot / backup\nMessaging: NATS JetStream for local fanout; Postgres remains source of truth\nSecurity: mTLS, local trust bundles, TPM-backed node identity, sealed/SOPS secrets, outbound-only tunnel\nObserve: OpenTelemetry Collector, Prometheus, Grafana, structured logs, local health UI\n
Main local services: Switchboard, Robot Gateway, Map/Semantic API, Artifact Cache, Sync Agent, Provisioning Service, Factory Integration Gateway, local Operator UI. The same components run in a gateway/cache profile for Tier 1 and an authority profile for Tier 3.
Switchboard is Postgres-backed. Workflow markings live in Postgres; transition firing is transactional; robot leases use epochs/fencing tokens; command state is durable and idempotent.
The map system starts boring and expandable \u2014 Postgres + PostGIS + pgvector for metadata, semantic objects, geometry, spatial indexes, and embeddings; ZFS-backed blobs for the chunky cursed spatial things (voxel chunks, point clouds, map tiles, MCAP logs, meshes, TSDFs, occupancy grids). A clean split matters:
Observations: what robots/sensors saw\nCanonical semantic facts: what the site currently accepts as operationally true\nMap versions: what robots are allowed to operate against\nBlob chunks: heavy spatial data streamed as needed\n
Robots never download the whole factory. They request the active map version, route-corridor chunks, local workcell chunks, and semantic objects in relevant bounds; dense voxel blobs stream on demand and cache locally. Tier 1 commits map/semantic edits to cloud (and caches active chunks fast); Tier 3 commits map activation and canonical facts locally, then mirrors up.
Tier 1 is the subtle case: it wants local speed but cloud truth. The answer is to classify state.
Rule: Runtime state may be local-fast. Durable authored state must be cloud-committed.
Runtime state (local-first, async replicated): heartbeats, command acks, local workflow progress, transient Petri markings, telemetry, traces, local UI state, buffered observations.
Durable authored state (cloud-committed in Tier 0/1): workflow definitions, fleet configuration, robot enrollment, site config, map activation, policy activation, semantic-zone edits, factory integration config, user/RBAC changes, artifact version activation.
If a Tier 1 node accepts an authored edit, it treats it as a draft/pending intent until cloud acknowledges. Die before cloud commit \u2192 the write didn't happen. Die after cloud commit \u2192 the replacement node hydrates from cloud and converges. This prevents dropped writes and split-brain without sacrificing operator responsiveness. In Tier 3, the local Site Authority commits operational and authored state locally, emits an append-only event log, and replicates events / snapshots / logs / blobs upward; cloud is a mirror and analytics plane, not the writer of site truth.
Tier 3 cloud views must be explicitly stale-aware: a shadow copy fed by replicated events/snapshots is a different thing from a live proxied view through a healthy tunnel, and the two should never be blurred.
The same outpost handles factory provisioning. This is not an optional convenience \u2014 it is how the system becomes installable without every field deployment turning into performance art. A site bring-up is a state machine, not a bag of scripts, and each transition emits durable events for auditability, repeatability, supportability, and recovery after someone unplugs the wrong cable (because of course they did):
uninitialized -> claimed by tenant -> network qualified -> outpost enrolled\n-> robots discovered -> robots enrolled -> artifacts staged\n-> map imported/created -> frames calibrated -> factory integrations connected\n-> workflows staged -> acceptance tests passed -> production ready\n
The same provisioning model runs inside the HMND factory, where the outpost can act as a secure local provisioning authority \u2014 issuing local enrollment material even when cloud is unavailable, then batch-syncing manufacturing/provisioning records back. That factory case has a different (stronger) trust posture than customer fleet operation: offline signing requires stronger key protection and clearer audit semantics.
The model separates four things that must not be collapsed:
Concept | Answers the question | Lifetime |
|---|---|---|
Device identity | What physical HMND-managed thing is this? | Long-lived / immutable (provenance) |
Fleet identity | Which customer/site/fleet may this thing participate in? | Medium-lived, mutable, rotated |
Policy / entitlement | What is this thing allowed to do? | Per entitlement doc |
Lease | What is this thing allowed to do right now? | Short-lived, renewable |
Device identity roots in AWS PCA. During manufacturing/provisioning we mint a device cert for each robot, outpost node, gateway, or managed device. Outpost nodes are first-class devices (type e.g. outpost_node) \u2014 visible in the cloud control plane with serial, SKU, cert status, software version, site assignment, health, audit trail.
Fleet identity is mutable enrollment identity, minted under a per-fleet/per-site intermediate CA in the HMND hierarchy when a device is assigned to a customer/site/fleet. The outpost terminates mTLS using fleet trust: it validates that a presented cert chains to the correct fleet CA, isn't expired, isn't locally revoked, matches the right tenant/site/fleet, and corresponds to an enrolled robot. Same fleet-identity model whether robots connect to cloud authority (Tier 0/1) or local Site Authority (Tier 3).
Outpost identity has at least three layers: device identity (physical node), cluster-membership identity (allowed to join this outpost cluster), and service identity (local services present mTLS certs). A Tier 3 Site Authority needs local serving certs plus the fleet trust bundle to authenticate robots without a cloud round trip.
Local CA delegation: For ordinary connected deployments, cloud mints fleet certs and distributes trust bundles; the outpost validates, it does not mint long-lived fleet identity. Only factory provisioning / airgap / special offline modes get a delegated local intermediate CA or short-lived signing authority \u2014 TPM/HSM-backed, tightly audited, rotated, scoped. Do not casually give every customer outpost a powerful intermediate CA. That's how you create a certificate pi\u00f1ata.
Revocation is bounded by connectivity. Online sites sync revocation/denylist from cloud; offline periods rely on last-known revocation state plus cert expiry to bound risk. Lifetimes: device cert long-lived (bootstrap/provenance), fleet cert medium-lived (rotated), session/workload cert short-lived (runtime mTLS). If a robot is removed from a fleet while the site is offline, that revocation can't take effect locally until sync resumes \u2014 unless the local operator performs it. No amount of PKI incense changes physics.
Assume the customer provides Wi-Fi, switching, routing, firewall policy, DNS, DHCP/static addressing, NTP, power, rack space, and egress unless contracted otherwise. BYON is not an exception \u2014 it is a first-class deployment mode. The outpost must qualify and survive customer infrastructure, not pretend it owns the network.
Connectivity is outbound-only to HMND cloud for Tier 1/3 \u2014 no inbound firewall holes for normal operation; remote access runs over a mutually authenticated tunnel the outpost initiates. The architecture supports network segregation where available:
robot network: robot-to-outpost traffic\nfactory integration network: WMS/MES/VDA5050/OPC UA/MQTT/etc.\nmanagement network: BMC, admin UI, support access\negress network: outbound tunnel and cloud sync\n
Small deployments may collapse these onto one customer LAN; production should strongly prefer VLAN separation; regulated deployments may mandate it. The outpost should not depend on customer Wi-Fi being good \u2014 it should measure it, complain loudly, and refuse production readiness if the network is below minimum.
The customer is responsible for power and network meeting defined requirements. We provide a qualification checklist and an automated qualification tool. The network-qualification phase tests outbound cloud reachability, DNS, NTP sanity, TLS interception, proxy requirements, MTU, packet loss, latency, jitter, robot VLAN reachability, multicast assumptions, firewall rules, and required ports; power covers outlet/circuit/UPS/grounding/rack/thermal.
Tier 1: one of everything (one node, one path, one UPS \u2014 degrades if any dies, by contract)\nTier 3: three servers, two switches, dual-homed nodes, redundant power/UPS,\n separate management access \u2014 enough redundancy to avoid cosplay HA\n
WAN redundancy is optional for Tier 3 because the factory continues locally without WAN; dual WAN helps cloud visibility/support/analytics freshness but is not required for local autonomy.
Four dimensions, kept distinct:
Tenant/site \u2014 each site belongs to exactly one tenant; the default customer outpost is single-tenant and assigned to exactly one site (or a tightly controlled set).
Network \u2014 robot / factory-integration / management / egress traffic separable; not mandatory for every pilot, but cleanly supported.
Trust \u2014 device PKI, fleet PKI, service/workload identity, and entitlement policy are separate. Manufacturing provenance does not authorize joining ACME's fleet; a fleet cert does not authorize every action; a lease grants current operational authority.
Data \u2014 runtime telemetry, customer maps, semantic facts, robot logs, support sessions, training exports all carry explicit tenant/site ownership and retention policy. Cloud analytics can be multi-tenant internally, but the control plane preserves tenant boundaries.
The contract per tier should be explicit and surfaced in the UI so operators and support never have to infer which side owns truth (that's how you get ghost stories):
Tier 0 \u2014 cloud authority, no local autonomy guarantee.
Tier 1 \u2014 cloud authority with local acceleration; fast while gateway + WAN are healthy; runtime data may buffer locally; durable config requires cloud commit; if the node dies, cloud truth survives.
Tier 3 \u2014 local HA authority; continues during WAN outage; cloud receives a replicated mirror; remote support/analytics degrade when disconnected but local operation continues.
Airgap \u2014 local authority with restricted sync; operational responsibility and update mechanics are special and priced accordingly.
A site page in the UI should show: authority placement, outpost size, current authority health, cloud sync lag, local mode, certificate status, robot enrollment status, network qualification status, last successful backup, last successful artifact sync.
Device identity proves the hardware.\nFleet identity proves enrollment.\nPolicy proves entitlement.\nLease proves current authority.\n\nCloud authority gives low-footprint deployments.\nSite authority gives local autonomy.\n\nThe outpost is the movable substrate:\n zero nodes for demos,\n one node for gateway/cache/proxy/provisioning,\n three nodes for local HA authority.\n
The system is sold and built as one logical control plane with movable authority \u2014 not cloud product plus edge product plus factory tool plus bespoke provisioning scripts. That way lies madness, YAML archaeology, and a field engineer named Dave who knows the only working incantation.
The same server hardware is usable for Tier 1 and Tier 3 \u2014 Tier 1 ships one node, Tier 3 ships three. Same image, same provisioning path, same spare strategy, same thermal/power assumptions.
short-depth 1U/2U edge server\n24-32 CPU cores\n256 GB ECC RAM\n2x NVMe OS mirror\n4x SSD/NVMe data drives\n2x 10/25GbE\ndedicated BMC/IPMI/iDRAC-style management\nTPM 2.0\ndual PSU\nPCIe x16 GPU-capable slot (GPU optional; reserved for possible local inference)\nPSU + airflow headroom for a future GPU\n
Rough installed-BOM targets (excluding field labor, support margin, spare pool, and customer network remediation \u2014 where the little goblins hide):
Tier | Range | Target SKU |
|---|---|---|
Tier 1 | \u00a312k\u2013\u00a330k | ~\u00a320k for a serious SKU |
Tier 3 | \u00a340k\u2013\u00a390k | ~\u00a360k\u2013\u00a380k for production HA |
This builds on the May framework doc \u2014 the proposal that introduced maturity levels (L1\u2013L5) and the case for organizing engineering goals by capability rather than by team. What follows extends that into a complete planning system: where each layer lives, who owns it, what rituals run them, and how AI maintains the day-to-day state so we don't end up with another 100KB Google Doc.
Working draft. Tooling and schemas are in an appendix to keep the body focused on intent.
Our previous planning doc was trying to do many things at the same time \u2014 strategic roadmap, cross-team planning, research bets \u2014 but it failed as a tool to make engineering work legible to the rest of the org. Product and other stakeholders had a difficult seeing what engineering is working on and why. Additionally, there were a number of other artifacts like robot allocations and goals that existed which often could fall out of sync. This doc outlines a new capability planning process which aims to resolve these challenges without adding too much additional toil onto engineering leaders.
Instead of trying to flatten everything out, we\u2019ll consider 3 broad layers to planning:
Opportunity is what the market wants \u2014 customer asks, partner interest, expansion targets \u2014 written as outcomes, not as solutions. "Amazon wants box handling by July, mixed SKUs, 80% accuracy." Not "implement box handling." Opportunities are owned by Product and live in the PB board; engineering reads them but doesn't write them.
Capability is what the robot can do, at some maturity. Capabilities are generally never completed \u2014 they mature over time, potentially regressing in maturity if expectations shift. Capabilities should be granular enough to allow for meaningful planning (\u201cThe robot can robot\u201d would be a capability, but a pretty terrible one), but not so granular as to overwhelm planning efforts. Think of efforts like \u201cOne-Click Bringup\u201d, \u201cWorkflow Authoring\u201d or \u201cTote Handling at Kinematic Limits\u201d as broad capabilities we may consider here.
Delivery is the work. A capability isn't owned by any one team \u2014 it pulls work from controls, applications, simulation, navigation, and so on \u2014 and that work is broken into epics, mostly team-bounded. Each team owns its own epics under the capability; the collection of those epics across teams is what makes cross-team dependencies visible naturally. When a capability is stuck, the gating epic (in whichever team's board it lives) is the locator. Stories live under their team's epic in whatever board the team already uses (HEAP, HECP, HELM, HERE, HERM, HESA, HECO, DCP, etc.).
The Capability layer here is the glue which ties ongoing engineering deliverables (the \u201chow\u201d) to specific product opportunities (the \u201cwho\u201d, \u201cwhere\u201d and \u201cwhy\u201d).
Note that this relationship isn\u2019t strictly hierarchical. Delivery items may contribute to many capabilities, and capabilities may enable many product opportunities. The goal here isn\u2019t to perfect model all work and dependencies in this dogmatic way. We want to make the simplest approximation we can to keep engineering legible to our stakeholders while minimizing our own planning overhead.
\nMaturity is the shared language between engineering and every other function. It replaces the unstable, jargon-heavy way engineering currently describes progress. The five levels are TRL-style and named for what each one unlocks externally:
Level | Name | What it unlocks |
|---|---|---|
L1 | Developer-ready | Internal eng testing and integration. Works enough to develop against. Not repeatable under observation. Validates feasibility. |
L2 | Film-ready | Repeatable under controlled conditions. Failures recoverable and editable around. Unlocks team updates, recruiting content, PR video, early marketing. |
L3 | Demo-ready | Reliable enough to run live for VIPs without a developer on standby. Failure modes known and recoverable. Unlocks investor demos, partner exec meetings, Tier-1 customers. |
L4 | POC-ready | Works reliably outside Humanoid's lab. Operable by a Humanoid deployment engineer (no specialized R&D eng on standby). Unlocks CEVA, Amazon, Schaeffler POC programs. |
L5 | Pilot-ready | Hits customer-agreement reliability and performance targets. Ideally operable outside of Humanoid entirely \u2014 by the customer's own staff. Incident response and update mechanisms exist. Unlocks pilot agreements, recurring revenue, SLAs. |
Not all capabilities are expected to reach all levels of maturity. For example, we don\u2019t really expect to expose our CI infrastructure to customers, but we may peel off pieces as part of the capability factory. Similar for Simulation and other more engineering-centric capabilities.
\nThe names matter because they're what non-engineers read. When an investor asks "what can I see today?" \u2014 L3 answers. When a customer asks "will this work in my facility?" \u2014 L4 answers. Marketing wants L2. Production sales targets are L5 commitments.
Each capability carries its current maturity as its workflow status, with the level names as the status values. Transitions are auditable \u2014 when a capability advances to L3 the date and author are captured in history automatically, no separate maturity log needed. Transitions are allowed in both directions because capabilities can regress (a stable L3 demo can become L2 if reliability degrades) and honest regression tracking matters.
As mentioned above, Capabilities are implemented by various engineering teams tracking deliverables as epics, stories and tasks. In addition to deliverables, we also may want to track goals when it makes sense. Capabilities have a \u201cGoal\u201d field to allow tracking goals at the capability level, in addition to the various epics that teams may be executing on to make the capability more mature. In this framework, its possible that a goal could be achieved prematurely, without ever closing out epics. Its also possible that finishing all the planned work doesn\u2019t advance the goal as far as desired.
Goals define a measurable outcome related to some capability. Epics define the scope of work to make progress toward these goals.
\n![\"image-20260601-174333.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1968111692/image-20260601-174333.png?version=1&modificationDate=1780335831237&cacheVersion=1&api=v2&width=1338&height=423\")
Some efforts may be more amenable to goals while others may be more about landing big changes as epics. Depending on the level of maturity, different perspectives may be needed to effectively manage the effort. We can at least organize these around top-level capabilities to provide a consistent view to external stakeholders, while internally we can choose to emphasize one or the other as it makes sense. In this example above, notice that there are a few existing HECO tickets linked as implementation details, along with 2 goals defined in the right-hand side.
Note also the Amount field - this, along with the alpha-wheelbase-poc-symphony label, is used to compute robot allocations dynamically.
Every capability is expected to be a long-duration effort which doesn\u2019t really have a defined start or end date, but not all capabilities are \u201cactive\u201d at the same time. Each capability has a Target Start Date and Target End Date which can be used to create a simple timeline to help assist in planning activities. This is how we know which capabilities are the focus of any given month or quarter.
![\"image-20260601-174515.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1968111692/image-20260601-174515.png?version=1&modificationDate=1780335921313&cacheVersion=1&api=v2&width=1330&height=691\")
We can use the monthly filter (which will be updated each month) to see the current month\u2019s capabilities. In the monthly planning meeting for example, we can go through the in-scope capabilities (see the left-hand side) and ensure that we have the right epics linked and goals defined.
![\"image-20260601-174923.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1968111692/image-20260601-174923.png?version=1&modificationDate=1780336172267&cacheVersion=1&api=v2&width=1321&height=681\")
What\u2019s nice about this is that we can see a direct view of this in Confluence itself for a nice table:
| Type | Key | Summary | Assignee | Priority | Status | Updated |
|---|---|---|---|---|---|---|
![\"Capability\"](\"https://sklvc.atlassian.net/rest/api/2/universal_avatar/view/type/issuetype/avatar/10310?size=medium\") | HMND-10 | One-Click Bringup | ![\"Matt](\"https://avatar-management--avatars.us-west-2.prod.public.atl-paas.net/712020:7655017b-55c7-4902-aafd-678b431ce466/ccc59fd7-ece7-4b2d-ab53-f2ce4d68278d/48\") Matt Klingensmith | ![\"Medium\"](\"https://sklvc.atlassian.net/images/icons/priorities/medium_new.svg\"){kind=icon} | L3 Demo-ready | 16 Jul 2026, 13:15 |
| HMND-14 | OTA and Release Lifecycle Management | ![\"Daniel](\"https://avatar-management--avatars.us-west-2.prod.public.atl-paas.net/712020:1c39ff9c-40f1-4814-8ddd-22275692d623/bbeb01d5-1ca3-44b1-ac36-4de63b40d017/48\") Daniel Hoffman | | L4 POC-ready | 03 Jul 2026, 15:17 |
| HMND-15 | AI-enabled Workflow Editing | Matt Klingensmith | | L3 Demo-ready | 21 Jul 2026, 08:44 |
| HMND-16 | Map & Workflow Sharing and Synchronization | Matt Klingensmith | | L3 Demo-ready | 16 Jul 2026, 13:50 |
| HMND-21 | Operational Metrics and KPIs | ![\"Cody](\"https://avatar-management--avatars.us-west-2.prod.public.atl-paas.net/712020:15d56293-96ff-40d2-a06d-912f776e2042/a630846a-6205-4c5f-ad3e-4fa46817e742/48\") Cody Griffin | | L1 Developer-ready | 10 Jul 2026, 14:37 |
| HMND-25 | Site Mapping | | L3 Demo-ready | 25 Jun 2026, 08:42 | |
| HMND-27 | Fleet Visualization | Matt Klingensmith | | L1 Developer-ready | 17 Jul 2026, 15:26 |
| HMND-79 | Execution (System 2) Agent | Matt Klingensmith | | L1 Developer-ready | 24 Jul 2026, 15:48 |
| HMND-39 | Workflow Deployment | Matt Klingensmith | | L1 Developer-ready | 24 Jul 2026, 15:48 |
| HMND-38 | Robot Enrollment | | Awaiting implementation | 03 Jun 2026, 12:46 | |
| HMND-47 | Fleet Health & Diagnostics | Matt Klingensmith | | L1 Developer-ready | 17 Jul 2026, 15:26 |
| HMND-53 | Universal Cloud Client for all Devices | Daniel Hoffman | | Awaiting implementation | 26 Jun 2026, 21:51 |
| HMND-54 | Scaled Secure Manufacturing - Test Harness and SQL Database | Daniel Hoffman | | Awaiting implementation | 10 Jul 2026, 18:58 |
| HMND-55 | Device Platform Security (Secure Boot) | Daniel Hoffman | | Awaiting implementation | 25 Jun 2026, 15:40 |
| HMND-56 | Realistic Simulation | ![\"Dmitriy](\"https://secure.gravatar.com/avatar/f0a051628aecd1e437b2727d2045e491?d=https%3A%2F%2Favatar-management--avatars.us-west-2.prod.public.atl-paas.net%2Finitials%2FDS-5.png\") Dmitriy Shingarey | | L1 Developer-ready | 21 Jul 2026, 18:28 |
| HMND-57 | CI [Placeholder] | ![\"Ricardo](\"https://secure.gravatar.com/avatar/831d6988a914f8d065871e450bd83377?d=https%3A%2F%2Favatar-management--avatars.us-west-2.prod.public.atl-paas.net%2Finitials%2FRD-5.png\") Ricardo Delfin | | Awaiting implementation | 06 Jul 2026, 16:22 |
| HMND-58 | Capability Factory | ![\"Artem](\"https://avatar-management--avatars.us-west-2.prod.public.atl-paas.net/712020:0a41c0f4-871a-4fcb-a0b5-a76cb1fbadf5/4ae5b15b-e260-478e-8943-382abe4f5f04/48\") Artem Shutak | | Awaiting implementation | 26 Jun 2026, 15:20 |
| HMND-63 | Runtime Performance | Cody Griffin | | L1 Developer-ready | 26 Jun 2026, 14:12 |
| HMND-68 | Teleoperation throughput readiness for Pilot | ![\"Giulio](\"https://avatar-management--avatars.us-west-2.prod.public.atl-paas.net/712020:9cc8637e-abee-4674-93b2-db86a8818e0f/85373a38-4848-41d2-a115-3ccdb68d95f2/48\") Giulio Cerruti | | Awaiting implementation | 23 Jul 2026, 17:02 |
| HMND-69 | Single WBC interface for upper-body and base | Giulio Cerruti | | Awaiting implementation | 25 Jun 2026, 13:32 |
| HMND-70 | [Biped] Full-body Teleop Human-like Locomanip | Giulio Cerruti | | Awaiting implementation | 17 Jul 2026, 12:58 |
| HMND-71 | Whole-body manipulation (VLA) | ![\"Diogo](\"https://avatar-management--avatars.us-west-2.prod.public.atl-paas.net/712020:dc96becb-2860-44b9-a22c-54a8419f4360/aa29fbcd-d488-4a0d-a94e-fcc84af94621/48\") Diogo Almeida | | Awaiting implementation | 25 Jun 2026, 10:32 |
| HMND-72 | VLA post-training recipe | Diogo Almeida | | Awaiting implementation | 25 Jun 2026, 12:23 |
| HMND-73 | Reinforcement learning for manipulation | ![\"Oleg](\"https://secure.gravatar.com/avatar/9cff63edec3eecd6c64557405e974b91?d=https%3A%2F%2Favatar-management--avatars.us-west-2.prod.public.atl-paas.net%2Finitials%2FOS-0.png\") Oleg Sinavski | | Awaiting implementation | 21 Jul 2026, 18:28 |
| HMND-74 | Foundational model | Oleg Sinavski | | Awaiting implementation | 21 Jul 2026, 18:28 |
| HMND-75 | 3D navigation planning | ![\"Karim](\"https://secure.gravatar.com/avatar/eefb653672f5b0d16f598325cab8a0c9?d=https%3A%2F%2Favatar-management--avatars.us-west-2.prod.public.atl-paas.net%2Finitials%2FKS-0.png\") Karim Shaban | | Awaiting implementation | 10 Jul 2026, 14:57 |
| HMND-76 | Visual global localization | Karim Shaban | | Awaiting implementation | 25 Jun 2026, 16:47 |
| HMND-77 | Visual-Lidar SLAM | Karim Shaban | | Awaiting implementation | 25 Jun 2026, 17:16 |
| HMND-78 | Mindmap | Karim Shaban | | Awaiting implementation | 25 Jun 2026, 17:05 |
| HMND-80 | Agentic Platform | ![\"Mustafa](\"https://avatar-management--avatars.us-west-2.prod.public.atl-paas.net/712020:14696d05-fc0a-461c-9ba5-8565a0981437/1862e4ad-72fb-4bf7-9b0a-7cb76ceb69d1/48\") Mustafa Atakan | | Awaiting implementation | 06 Jul 2026, 13:54 |
| HMND-81 | Storage & Compute Infra | Mustafa Atakan | | Awaiting implementation | 06 Jul 2026, 14:21 |
| HMND-90 | Fleet-Scale Simulation | | Awaiting implementation | 16 Jul 2026, 13:15 |
Unfortunately Jira\u2019s goals are a different product and aren\u2019t yet visible via the API - though this may get addressed in a future release.
Each layer has its own forum, with AlphaClaw-prepared briefings replacing verbal status updates.
Quarterly capability planning brings product and engineering leads together to agree the 7\u201310 capabilities for the quarter, set their targets and trade-offs, and sign owner+sponsor commitments. This is where the contract gets signed.
Monthly capability review is engineering leads only. Each owner reports maturity delta vs target, surfaces cross-team blockers, and confirms or corrects the AlphaClaw-generated briefing from 24 hours ago. No story-level discussion \u2014 that's not what this forum is for.
Monthly product backlog grooming brings product back in with engineering leads to debate new opportunities, surface engineering discoveries, and adjust the capability portfolio. This replaces the current weekly grooming meeting \u2014 the cadence shifts because the artifact (capability portfolio) only changes at monthly rhythm, and weekly grooming without commitment teeth is theater.
Team planning runs at whatever cadence each team picks \u2014 weekly, biweekly, async, whatever fits. The capability is the cross-team commitment, but how each team gets to it is theirs.
Weekly opportunity refinement is product only \u2014 refining PB ideas, writing outcomes clearly, gathering customer and sales input. Engineering doesn't attend.
Work flows in both directions across the layers.
Forward. An opportunity pulls a capability into existence (or accelerates one that exists). The capability commits to a maturity by a date. That commitment opens delivery epics across the teams that need to contribute, which deliver stories. Each layer's commitment is the input to the layer below. In addition to the maturity goal, additional goals may be associated with a capability.
Reverse. Delivery work sometimes reveals a capability nobody asked for \u2014 "we accidentally got X working at a level that's actually demo-able." That surfaces upstream as a candidate opportunity. Half the real roadmap shifts in any technical org come from this reverse path; it isn't an edge case. Capabilities also propose new opportunities back to product when they unlock conversations that weren't possible before.
When both directions work, the system breathes \u2014 product pulls, engineering pushes back with reality, and the negotiation surface is real. When only one direction works, it devolves into either a product wishlist that engineering ignores or an engineering roadmap that product doesn't recognize.
Capability (long-lived parent, renamed Initiative, persists across quarters):
Status = current maturity (workflow): L1 Developer-ready / L2 Film-ready / L3 Demo-ready / L4 POC-ready / L5 Pilot-ready / Retired. Transitions allowed in both directions. No "Done" state.
Target Maturity (this quarter) \u2014 single-select L1\u2013L5
Health \u2014 On Track / At-Risk / Slipped
Engineering Owner \u2014 single user (builder)
Sponsor \u2014 single user (buyer: product, strategic, or research)
Capability Type \u2014 Customer-pulled / Strategic-foundational / Operations
Linked Opportunities \u2014 relationship to PB Ideas (multi-select)
Dependencies \u2014 linked Capabilities (issue links)
For bets: Hypothesis, Kill Criterion, Spend (eng-weeks)
The three things you read on any capability tile: where it is (Status), where it's going this quarter (Target), how that's going (Health).
Delivery Epic (typically team-bounded, multiple per Capability):
Parent: linked Capability
Owning team: implicit in the project the epic lives in
Start/End dates: quarter window (or shorter if scope fits in a sprint)
Standard Done workflow at scope completion
The capability advances when enough of its underlying epics close to satisfy the maturity transition criteria
Story:
Standard story fields plus Parent (the Epic)
Trace upward: Story \u2192 Epic \u2192 Capability \u2192 Opportunity
Lives in team boards (HEAP, HECP, HELM, HERE, HERM, HESA, HECO, DCP, etc.)
Auto-created from PRs by AlphaClaw
The AI stuff here will likely need to wait until after we run this process manually a full times to get things started.
Before the rituals: the operating model of the system itself. AI is the new front-end. Jira is the database; almost nobody opens it directly. AlphaClaw \u2014 the AI maintaining the system day-to-day \u2014 is what each role talks to.
Devs interact through Slack and git. Each capability has a Slack channel with a AlphaClaw-maintained pinned summary at the top \u2014 current maturity, target, recent signals, open decisions. Pull request descriptions and commit messages auto-link to capabilities by inference. Stories get auto-created from PRs without anyone filing a ticket. The PR is the unit of dev intent; everything else is bookkeeping the bot handles.
Leads interact through weekly DM digests and pre-meeting briefings. Monday morning, each owner gets a summary of what changed last week on their capabilities, with proposed updates and the evidence. Reply in natural language to confirm or correct. Twenty-four hours before the monthly review AlphaClaw generates the meeting briefing; the meeting itself is spent on decisions, not verbal status.
Product and business interact through conversational queries: "what's ready for Schaeffler phase 3?", "which capabilities are at risk for June?", "who owns box handling?" The answer comes from live state. Product stops trying to interpret engineering Jira because it doesn't need to.
AlphaClaw operates with confidence scores and provenance on every change. High-confidence updates auto-apply; low-confidence queue for human review. Maturity transitions always require human confirmation, regardless of confidence. The audit trail is reversible.
Trust is earned over time. First six-plus weeks AlphaClaw is read-only \u2014 proposes, never writes. Per-capability write access unlocks after proposal accuracy stays above ninety percent for several weeks. Maturity changes stay human-confirmed forever.
The strategic bet underneath: once the AI front-end is the only surface anyone touches, the backend is implementation detail. If Jira becomes painful we migrate; if the agent platform pivots we migrate. The durable thing is the taxonomy and the rituals, not the tooling.
Runtime stack:
Anthropic Managed Agents for runtime (~$200\u2013500/month operational)
MCPs: Atlassian (official Remote MCP), GitHub (official), Slack
Webhook routing from GitHub/Slack/Jira to the Managed Agent
System prompt + capability manifest = bulk of "code"
Operator UI = Slack (no dashboard to build)
Half FTE for one month to MVP; 0.1\u20130.2 FTE ongoing
Git integration:
PR opens \u2192 bot infers capability from branch name, PR title, files touched, description; auto-creates Story; comments with link
PR merges \u2192 bot updates story status, posts to capability channel; proposes maturity transition if the PR closed a maturity-blocking item
Periodic sweep (hourly) synthesizes signal across Slack/git/Jira
The one hard gate: PR cannot merge to main without a capability link. Soft everywhere else.
Cross-team dependency detection: PRs touching another team's component propose capability dependency links
Stale detection: no git activity for 3 weeks against an open epic gets flagged
Accidental discovery: signals that something works better than expected surface as candidate new capability or upstream opportunity
One of the more annoying aspects of our current autonomy solution is the \u201cpose \u2192 approach \u2192 prompt\u201d problem, where we need to carefully orchestrate the robot before we can hand over to the VLA to run policy. There are a few reasons we do this. First, we often need to adjust the pose of the robot (specifically the torso) to avoid collisions or to put the robot close to the object being manipulated. Second, we sometimes need to start the policy from an in-distribution pose to get any success (though this has improved greatly with recent tote-handling policies). But overall we still suffer some major issues here:
The sequence needs to be carefully preserved during workflow editing/deployment - which makes things a bit more brittle and fragile
The distinct \u201cphases\u201d, even when executed end-to-end without pauses, looks jerky and \u201crobotic\u201d since control itself is being handled by ~3ish different systems during this time
It mixes the what and how at the system-2 and system-1 boundaries. System-2 needs to know too much about how to position the robot or the last ~10-20cm of navigation to perform the task.
Future vision-based navigation may not be precise enough to reliably do this anymore, and some points of interest may not be fixed so precisely in the real world anyway.
Broadly, it should be possible for the policy to do all of this smoothly. System-2 just needs to get the robot \u201cclose enough\u201d, rather than trying to get centimeter precision within navigation and posing the robot blindly. System-1 should be able to pose appropriately conditioned on the prompt and the scene, close the last few centimeters using base navigation while executing the manipulation policy as one fluid motion. This will look better, perform better and ultimately be more robust to real-world dynamic environments and workflow edits.
One possibility is to actually use system-2 to record the sequence and just train a policy to do the things. Unfortunately this has some major downsides. For one, there will be almost no data diversity in the pose and approach phases since they are so deterministically executed. The second problem is that this preserves the stilted, disjoint motion - even if the sequence itself runs as a single longer-horizon behavior.
![](\"https://sklvc.atlassian.net/wiki/download/attachments/2002059272/Untitled%20Diagram-1779372547043.drawio.png?api=v2&version=1\")
WBC and Data Collection
In order to get this fluid, overlapping motion, we need to actually get this addressed during data collection. Base control has been a missing piece, and because policy can\u2019t do the \u201capproach\u201d, it forces us to break things up into these distinct pieces to be orchestrated.
The most recent idea here is to effectively extend the WBC with a 3dof planar \u201cvirtual joint\u201d with some fake impedance between the base and the floor. The \u201cstiffness\u201d of this would determine how far we allow the base to move - thus bounding the approach. Its almost like the robot is attached to some springy tether when we enable policy.
In this world, system-2 merely gets the robot close the desired point of interest. Perhaps a meter from the table. The WBC (and thus teleoperators or policy) can then smoothly extend the torso, move the base forward and position the arms over the tote for grasping before picking it up and returning back a location which is safely clear of the table.
This is superior to having the DCs doing manual base control, as it would be extremely difficult to get this natural overlapping motion with a wheelbase in this way.
The \u201cPrecision Problem\u201d
One of the areas highlighted above as a problem is that currently we rely on the base being positioned with centimeter level accuracy using lidar mapping and localization before we kick off the policy. In the real world, we cannot expect our various \u201cpoints of interest\u201d to be located so accurately anyway - it will be impossible for us to actually hit pilot-level performance on something like de-stacking totes from a pallet (dropped off by a forklift or a human) unless the base can make some \u201clast-mile\u201d adjustments.
![](\"https://sklvc.atlassian.net/wiki/download/attachments/2002059272/Untitled%20Diagram-1779384960197.drawio.png?api=v2&version=1\")
Risks
Moving the base adds a bit of risk since now the robot can move about a bit. Obviously a base lock (similar to the torso lock) can be implemented to avoid accidental base motion. Its not clear how well the policy will learn to approach and pose to avoid obstacles though. We have some evidence that the policy can learn to avoid obstacles, just by the nature of the task its performing in tight spaces like the shelves or trolley - so why wouldn\u2019t this extend to the base on approach?
First - the policy is memory-less. If we blindly move the base forward because of some policy error while we are already up against the table, the robot could push further into the table resulting in some pretty severe collisions. Second - the safety system is active on the base which could result in strange behaviors if the safety is tripped while policy is executing.
" }, { "id": "1653407830", "title": "Robostore Demo Setup Instructions - DEPRECATED", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1653407830", "last_modified": "2026-05-19T06:54:11.516Z", "created_at": "2026-02-19T19:39:42.022Z", "body_html": "![](\"https://sklvc.atlassian.net/wiki/download/attachments/1653407830/Untitled%20Diagram-1771530753132.drawio.png?api=v2&version=1\")
See Robostore + Ring picking demo running Guide for the latest demo guide
TL;DR State of the Demo (Alpha06 is the stationary robot, Alpha08 should be the moving robot. Place the moving robot at init, place the stationary robot at table_1.) /home/hmnd/robostore/hmnd, /mnt/ssd/robostore/hmnd, branch: releases/2026.03.02)
We can have a single robot pick and offer item 1 and item 3 in the above example.
Item 2 fails because the policy will grab item 1 (maybe due to how popcorn was never trained with the left arm?)
Item 4 fails because its on the shelf - which we cannot safely approach with the existing workflow (need more approach waypoints)
We cannot ask for items 1 and 3 at the same time - the robot will simply oscillate between Table 1 and Table 2 (simple workflow change needed).
![\"1000002973.jpg\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1653407830/1000002973.jpg?version=1&modificationDate=1771532697722&cacheVersion=1&api=v2&width=609&height=458\")
RTPC Bringup
For robots with F/T sensors (check status here):
cd hmnd_robot\npixi run reverb_robostore_rtpc_alpha_ft\n
For robots without F/T sensors:
cd hmnd_robot\npixi run reverb_robostore_rtpc_alpha\n
As always, sometimes this just doesn\u2019t work. No point in continuing if its not happy. Make sure to localize w/ Foxglove.
Initialization/Checkout
Navigate robot to "init" point (\u26a0\ufe0f Use it only for mobile robot):
pixi run ros2 action send_goal /navigate_to_tag hmnd_msgs/action/NavigateToTag "{tag_name: init, controller_id: omni}" --feedback\nSet arms to initial pose (\u26a0\ufe0f without obstacle avoidance):
pixi run ros2 action send_goal /whole_body_controller_interface/reset_robot_joint_config \\\nhmnd_msgs/action/ResetRobotJointConfig "{config_name: home, group_name: arms, skip_planning: true}" --feedback\nThis will put the robot in the middle of the space on the right hand (from the audience) side.
Orin Bringup
Option 1: Run coordinator etc. on a robot (alpha11 for example).
For a \u201cStationary\u201d robot, it will move to the table_1 tag (the left side of the table if you\u2019re sitting at it).
pixi run reverb_robostore_orin_alpha_local_stationary\n
I\u2019m not super thrilled about the robot moving when the stack is started - this is from the original mock setup and I\u2019ll probably tweak this to be a bit safer, especially as it takes a bit for the robot to actually boot and move.
The stationary robot will only attempt items 1 and 2.
For a \u201cMobile\u201d robot, it will move to the table_2 tag (the right side of the table)
pixi run reverb_robostore_orin_alpha_local\n
And it will reach all points.
For some reason (maybe Sam Pfeiffer knows?) the launch file doesn\u2019t fully clean up the docker-compose stack for the fleet services, leaving an orphaned coordinator. Its very important to clean this up when resetting the demo:
docker stop robostore-coordinator-1\ndocker ps # verify nothing is running\n
The coordinator (switchboard) contains all the workflow state, and so if its not reset properly weird shit can happen.
Option 2: Run Coordinator on a laptop
Connect a laptop to the same network as the robots.
run ifconfig on the laptop to find your IP address. It should be under something like wlp... under a field called inet call this \u201cLAPTOP_IP\u201d
On the orin of the robot make a change like so. Edit hmnd_fleet/switchboard/examples/robostore/config/robostore_demo_config.yaml so that the IP addresses in laptop correspond to the laptop\u2019s IP address. Afterwards, your git diff should look like this:
hmnd@orinalpha8:/mnt/ssd/robostore/hmnd/hmnd_robot$ git diff\ndiff --git a/hmnd_fleet/switchboard/examples/robostore/config/robostore_demo_config.yaml b/hmnd_fleet/switchboard/examples/robostore/config/robostore_demo_config.yaml\nindex 12aa8a81f..c72bcb245 100644\n--- a/hmnd_fleet/switchboard/examples/robostore/config/robostore_demo_config.yaml\n+++ b/hmnd_fleet/switchboard/examples/robostore/config/robostore_demo_config.yaml\n@@ -118,9 +118,9 @@ fleet:\n cims: "http://localhost:20003"\n laptop:\n # Remote peer fleet services (e.g., another robot running docker-compose)\n- coordinator: "http://10.44.68.111:20000"\n- robot_exec: "http://10.44.68.111:20001/robot"\n- map_exec: "http://10.44.68.111:20001/map"\n+ coordinator: "http://<LAPTOP_IP>:20000"\n+ robot_exec: "http://<LAPTOP_IP>:20001/robot"\n+ map_exec: "http://<LAPTOP_IP>:20001/map"\n
On the laptop run ./hmnd_fleet/switchboard/examples/robostore/scripts/launch_kiosk.sh coordinator
docker ps to confirm at least coordinator and map are running.
On the robot orin run:
For the stationary robot (alpha 06):
pixi run reverb_robostore_orin_alpha_laptop_stationary
For the moving robot (alpha 08):
pixi run reverb_robostore_orin_alpha_laptop
Ordering via test script
Once the system is up, and the robot has moved to the the correct table location depending on the stationary/mobile role, you can send orders
hmnd@orinalpha11:/mnt/ssd/codg/hmnd/hmnd_fleet/switchboard/examples/robostore$ ./robostore_order.sh 1 2\n
The arguments are the item types. You can pass 1 or 3. (as mentioned before, one at a time for now). The robot should get to the right basket (swapping with the shelf in the best of times) and then offer the item to the audience member.
WTF is Happening?
When you place an Order, its injected as an IncomingOrder to the workflow. The \u201cpos-agent\u201d takes care of breaking that IncomingOrder into PendingItems (one item per object being picked/offered), with some metadata like the specific item or left/right needed for policy execution. This is sorta the \u201csystem 3\u201d agent, and where Yuriy Strezhik 's kiosk app will eventually integrate.
The robots will take pending items and work to offer them to audience members according to the workflow. Once all the items of an order are completed, the pos-agent will receive the event notifying that its completed (which we can key off to update a GUI or some voice line).
I\u2019m not entirely satisfied with how the data is split over tokens and facts - but for now I was just trying to get something mostly working.
Item mappings are here: https://github.com/HumanoidTeam/hmnd/blob/codg/heco-2589-robostore-workflow/hmnd_fleet/switchboard/examples/robostore/config/robostore_demo_config.yaml#L13 ( don\u2019t mind all the leftover Boschisms)
Frontend Kiosk Setup
On a different machine (not on a robot), you will need to launch the kiosk app. The kiosk app consists of two components:
The Order Service handles receiving user orders and converting them into commands to the fleet service (the coordinator on the \u201cserver\u201d robot).
The Frontend creates a react UI that can run in firefox in kiosk mode, and be displayed on the touch screen.
Running with real Robots
Set up your environment. You will need to have node and docker installed locally.
Set up the ports and IP address of the \u201cserver\u201d robot. This will mean port forwarding from the server robot to your laptop.
For me, this is a command like so ssh -A -p 2022 -L 20000:localhost:20000 -L 20001:localhost:20001 -L 20002:localhost:20002 -L 20004:localhost:20004 -o ExitOnForwardFailure=yes -N hmnd@<your_robot_ip>
NOTE: if the \u201cserver\u201d is just a laptop, and that laptop is the same IP as the frontend \u2013 you DO NOT need to port forward!
First, try running the kiosk order service without voice.
export VOICE_ENABLED=0; bash ./hmnd_fleet/switchboard/examples/robostore/scripts/launch_kiosk.sh service
This should start running without errors.
In another terminal, try running the kiosk frontend.
bash ./hmnd_fleet/switchboard/examples/robostore/scripts/launch_kiosk.sh frontend
Launch Firefox in kiosk mode pointed at the frontend
firefox --private-window --kiosk "http://localhost:5173/"
This should pull up the kiosk UI. Since voice is disabled, you will have to \u201corder by touch.\u201d
![\"image-20260303-095844.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1653407830/image-20260303-095844.png?version=1&modificationDate=1772531926734&cacheVersion=1&api=v2&width=841&height=478\")
Now we can try to do the audio setup.
Start by plugging in the desired device(s).
Open up audio settings, and see if you see a new microphone device. You can verify this by unplugging and replugging the device.
Now you will need to set up the OPENAI_API_KEY. export OPENAI_API_KEY=<your_key> in a terminal. Ask Matt Klingensmith or Cody Griffin for a valid key in a private message.
This key can persist if you put it in ~/.bashrc
Now, we will try to run the voice agent on its own without the robostore component. This is just to verify that the audio and OpenAI connection is working.
export PA_ALSA_PLUGHW=1;PYTHONPATH=/home/mklingen/hmnd/hmnd_robot/apps pixi run python -m voice_agents.openai_voice_agent
This will pull up a list of audio devices. Select the device that you found in step (b) for the input device. Note the ID of the device (on the right side) not the index.
If this is successful you should be able to start talking to the agent.
Troubleshooting: linux is terrible at audio. Try unplugging and replugging the device several times. Make sure you don\u2019t have multiple apps trying to record audio simultaneously.
Noting the device indices that you found in step (g), rerun the order service with:
export PA_ALSA_PLUGHW=1; export ROBOSTORE_VOICE_INPUT_DEVICE_INDEX=6; export ROBOSTORE_VOICE_OUTPUT_DEVICE_INDEX=7; export VOICE_ENABLED=1; bash ./hmnd_fleet/switchboard/examples/robostore/scripts/launch_kiosk.sh service
Note that in my case, the input index was 6, and the output index was 7. Although they\u2019re called index, we\u2019re referring to the device ID that you found in step (g).
Verify that audio works. You should be able to just start talking into the microphone to order items.
Once Everything is Working: The Single Command
Once you\u2019ve verified everything is working, you can run the coordinator, order service, and kiosk frontend all with one command. Make sure that ROBOSTORE_VOICE_INPUT and output are correct first!:
export PA_ALSA_PLUGHW=1; export ROBOSTORE_VOICE_INPUT_DEVICE_INDEX=<input>; export ROBOSTORE_VOICE_OUTPUT_DEVICE_INDEX=<output>; export VOICE_ENABLED=1; bash ./hmnd_fleet/switchboard/examples/robostore/scripts/launch_kiosk.sh all\n
Running with Mock Mode
You will probably find it useful to debug the front end separately from the robots. After installing the dependencies (Docker), run a mock robot setup using:
KEEP_RUNNING=1 \\\nORDER_COUNT=0 \\\nWAIT_COMPLETED_ORDERS=999 \\\nPOS_MULTIMODAL_URL=http://localhost:20006 \\\nCOORDINATOR_URL="http://localhost:20000/workflows/robostore" \\\nMAP_EXEC_URL="http://localhost:20001/workflows/robostore/map" \\\nROBOT_EXEC_URL="http://localhost:20001/workflows/robostore/robot" \\\nWAREHOUSE_URL="http://localhost:20002/workflows/robostore" \\\nMOCK_WMS_URL="http://localhost:20004" \\\nbash hmnd_robot/apps/reverb/scripts/run_reverb_ros_mock_dual_robostore_multimodal.sh\n\n
This should boot up several docker instances that run simulated (mock) robots and services. Then all the other front end steps should work as expected.
Debug Monitoring
The debug monitor developed for Bosch can also be used in Robostore. To use it, simply navigate to the IP of the robot at port 5555
![\"image-20260305-150855.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1653407830/image-20260305-150855.png?version=1&modificationDate=1772723337344&cacheVersion=1&api=v2&width=1733&height=623\")
The Pause Acquisition button will prevent that particular robot from ever taking on orders. The Stop Robot button completely removes the robot from the workflow entirely and prevents it from moving (both policies and navigation). You can add the robot back to the workflow by pressed Start Robot.
Log in to either spark or thor.
docker logs hmnd-inference -f
You will now see the logs from the inference server, which should look something like this
[03/05/26 15:30:50] INFO hmnd_training.serve.server: === gRPC server.py:76\n Request [Client \n ipv4:10.44.52.106:54282] === \n E2E infer_ms: 220.06 \n policy_inference_ms: 211.83 + \n grpc_serialize_ms: 1.19 \n \n[03/05/26 15:30:50] INFO hmnd_training.serve.server: === gRPC server.py:76\n Request [Client \n ipv4:10.44.52.106:54282] === \n E2E infer_ms: 234.72 \n policy_inference_ms: 220.99 + \n grpc_serialize_ms: 1.18 \n \n[03/05/26 15:30:51] INFO hmnd_training.serve.server: === gRPC server.py:76\n Request [Client \n ipv4:10.44.52.106:54282] === \n E2E infer_ms: 235.79 \n policy_inference_ms: 222.13 + \n grpc_serialize_ms: 1.17 \n \n[03/05/26 15:30:51] INFO hmnd_training.serve.server: === gRPC server.py:76\n Request [Client \n ipv4:10.44.52.106:54282] === \n E2E infer_ms: 231.76 \n policy_inference_ms: 217.85 + \n grpc_serialize_ms: 1.16 \n \n[03/05/26 15:30:51] INFO hmnd_training.serve.server: === gRPC server.py:76\n Request [Client \n ipv4:10.44.52.106:54282] === \n E2E infer_ms: 226.94 \n policy_inference_ms: 218.03 + \n grpc_serialize_ms: 1.20\n
It doesn\u2019t seem like the robot can offer items from sku-2 (popcorn in this case), even with the specific prompt (Pick the popcorn from the left basket and give it to a person) - I\u2019m guessing this is due to the combinatorics and we could maybe shuffle the items around to make this work\u2026
Shelves are sketchy AF. Right now there is only a single shelf_approach way point, which is near the init position (but facing the shelves). Unfortunately this isn\u2019t enough - the robot will simply run the basket its carrying into the basket on the shelf, which can result in tipping the shelves over. These shelves are VERY easy to tip.
When grasping items from the side of the basket, the policy will grab the basket itself and try to hand it over, with predictable results. Occasionally the gripper can get jammed up on itself resulting in an \u201cempty offer\u201d. I noticed this most on baskets that are nearly empty.
robot IDs aren\u2019t being set properly (r1 is not right - should be 11)
You will often see the robots go into teleop mode the HRI system on the face will turn blue.
When this happens the only thing possible really is to go into sockpuppet and disable teleop, then kill the code that\u2019s running on the orin
Then move all the robots back to their initial states, and move all the baskets back.
Then reset the coordinator.
Then restart the front end.
This is a living document - I\u2019m sure there will be all sorts of snags onboarding here (especially with tools/MCPs/accounts) - let\u2019s try and pile all the context/comments/issues into this doc so we can get things cleared up.
\nClaude is Humanoid's AI assistant, powered by Anthropic. This guide walks you through logging in, the core tools available, and how to connect them to your everyday workflows.
Important: Use your thehumanoid.ai email
Humanoid's Claude access is provisioned through Google Workspace at thehumanoid.ai. Many team members are signed into their browsers with a personal or legacy skl.vc Google account \u2014 you must manually enter your @thehumanoid.ai email when signing in, rather than using the "Continue with Google" auto-fill. If you sign in with the wrong account, you won't have access to the team workspace, integrations, or shared memory.
Browse to https://claude.ai/login and use your @thehumanoid.ai login (or submit an IT ticket for an account)
Follow the link on the email issued to your account.
Click Sign in with Google
When prompted, manually type alias@thehumanoid.ai \u2014 do not use the pre-filled account if it shows a different email
Complete Google authentication
You should land in the Humanoid team workspace
If you see a personal or free-tier Claude workspace instead of the team one, you've signed in with the wrong account \u2014 sign out and repeat step 4.
Once you're in, here's what you need to know:
Chat: The main interface for asking questions, drafting content, reasoning through problems, and more. Think of it as a highly capable thinking partner.
Projects: Group related conversations and give Claude persistent context (e.g. a project brief, codebase context, or team norms). Great for ongoing workstreams.
Memory: Claude can remember things across conversations. You can review and edit your memories in Settings \u2192 Memory.
Artifacts: When Claude generates code, documents, or structured outputs, they appear as shareable Artifacts in a side panel \u2014 you can copy, download, or iterate on them.
Integrations / Connectors: Claude can connect to your tools (Slack, GitHub, Jira, Google Drive, etc.) and take actions on your behalf.
Claude Cowork is a desktop app for non-developers that lets you automate file management, document workflows, and everyday tasks \u2014 no coding required.
Download from claude.ai/download \u2192 select Claude Cowork
Install and sign in with your @thehumanoid.ai Google account (same caveat as above \u2014 type it manually)
From Cowork, you can also launch Claude in Chrome, Claude in Excel, and Claude in PowerPoint \u2014 browser and Office integrations for in-context AI assistance
Claude Code is a command-line tool and desktop app for agentic coding \u2014 it can read your repo, write code, run tests, and make commits.
Install via terminal:
npm install -g @anthropic-ai/claude-code\n
Or download the desktop app from claude.ai/download \u2192 Claude Code
Authenticate with your @thehumanoid.ai account when prompted
Open a project directory and run claude to start a session
Navigate to Settings \u2192 Integrations in Claude to connect the following. Each integration allows Claude to read context from and take actions in those tools.
![\"image-20260331-085417.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1816592386/image-20260331-085417.png?version=1&modificationDate=1774947258519&cacheVersion=1&api=v2&width=1282&height=710\")
Connect under Settings \u2192 Integrations \u2192 GitHub
Authorize with your GitHub account linked to the Humanoid org
Once connected, Claude Code can read repos, suggest PRs, and assist with code reviews directly
Connect under Settings \u2192 Integrations \u2192 Slack
Authorize for the thehumanoid Slack workspace
Claude can search message history, draft messages, and summarize threads
Connect under Settings \u2192 Integrations \u2192 Atlassian
Authorize with your @thehumanoid.ai Google SSO on Atlassian \u2014 this should match your sklvc.atlassian.net login
Claude can search pages, create Jira issues, update tickets, and draft documentation
Connect under Settings \u2192 Integrations \u2192 Google Drive / Gmail / Calendar
Use your @thehumanoid.ai account \u2014 again, type it manually rather than relying on auto-fill
Claude can search Drive, summarize docs, draft emails, and check your calendar
Connect applicants, etc
Be specific: The more context you give Claude, the better the output. Include the "why", not just the "what".
Use Projects for ongoing work: Set a project up with background context once, and every conversation in that project benefits from it automatically.
Iterate, don't redo: If Claude's first response isn't quite right, ask it to refine \u2014 it's faster than starting over.
Claude can use your tools: With integrations connected, you can ask Claude to "find the Jira ticket about X", "summarize last week's #engineering Slack messages", or "draft a doc based on this meeting note" and it will actually do it.
Check Artifacts: Any document, table, or code Claude generates can be saved as an Artifact \u2014 look for the side panel.
Don't paste sensitive data carelessly. While Claude conversations in the team workspace are covered by Anthropic's enterprise data handling agreements, treat anything truly confidential (customer PII, unreleased financials, private keys) with care. When in doubt, describe the situation rather than pasting raw data.
Claude can be wrong. Claude is a powerful reasoning tool but it hallucinates. Always verify important facts, code, or decisions \u2014 especially in production contexts.
Agentic actions are real. When Claude is connected to Slack, Jira, GitHub, etc., it can take real actions. Review before confirming any write operations (creating issues, sending messages, committing code).
Account confusion is real. If something looks off \u2014 missing integrations, wrong workspace, free-tier limits \u2014 the first thing to check is whether you're signed in with
@thehumanoid.aivs another account.
Internal questions \u2192 https://app.slack.com/client/T078M0VH3C6/C0AH0DKCGM7?cdn_fallback=2 or https://app.slack.com/client/T078M0VH3C6/C0813SXPKGF?cdn_fallback=2 on Slack
Anthropic documentation \u2192 docs.anthropic.com
Claude support \u2192 support.claude.ai
Drafted 2026-02-24. This is a \u201chere\u2019s what exists today\u201d map of the system (Alpha-era RTPC + Orin split). It is intentionally descriptive, not prescriptive.
The robot currently runs as a distributed ROS2 system split across two computers:
RTPC: real-time(ish) control stacks + nav/localization + hardware drivers + visualization bridge.
Orin: cameras/vision + policy inference + app runtime (teleop, workflows, fleet).
The guiding mental model for \u201cwhat talks to what\u201d today:
graph TB\n subgraph ORIN["Orin (application + perception + policy)"]\n SP[sockpuppet\nteleop + data collection]\n RV[reverb_node\nworkflow agent]\n BROKER[wbc_broker\ncommand arbitration + action proxy]\n PAS[hmnd_policy_execution\npolicy_action_server]\n VIS[hmnd_vision\nvision pipeline]\n end\n\n subgraph RTPC["RTPC (control + nav + hardware)"]\n NAV["hmnd_navigation\n(nav2 + action servers)"]\n LOC["hmnd_localization\n(map_server + neo_localization + slamtoolbox_service)"]\n WBC[whole_body_controller_interface\nWBC adapter + IK]\n CTRL["ros2_control_node + controllers\n(alpha_*_description)"]\n HW[EtherCAT + IO + sensors]\n FOX[foxglove_bridge]\n end\n\n %% Actions\n RV <-->|ROS action| NAV\n RV <-->|ROS action| BROKER\n\n %% Policy path\n RV -->|"/policy_execution (action)"| BROKER\n BROKER -->|"/policy_execution_internal (action)"| PAS\n PAS -->|"/wbc_broker/policy_commands (topic)"| BROKER\n\n %% Teleop path\n SP -->|"/wbc_broker/teleop_commands (topic)"| BROKER\n SP -->|"/teleoperation_server/* (topics)"| BROKER\n\n %% Command output\n BROKER -->|"/whole_body_controller_interface/robot_commands (topic)"| WBC\n\n %% Control + hardware\n WBC -->|controller interfaces| CTRL\n CTRL --> HW\n\n %% Perception\n VIS --> SP\n VIS --> PAS\n\n %% State/vis\n RTPC --> FOX\n
Notes:
The diagram shows the dominant paths; there are many additional topics (TF, joint states, diagnostics, etc.).
The Bosch demo launch files are good \u201cproxies\u201d for today\u2019s two partial-monolith bringups.
pixi run reverb_rtpc_alpha_wheelbase (controllers + WBC + nav stack + helper nodes)
Bosch proxy monolith: pixi run reverb_bosch_rtpc_alpha_ft
pixi run reverb_alpha_wheelbase (reverb + sockpuppet + vision + policy + broker when enabled)
Bosch proxy monolith: pixi run reverb_bosch_orin_alpha
hmnd_robot/apps/reverb \u2014 workflow agent + orchestration launch files (reverb.launch.py, reverb_mock.launch.py).
hmnd_robot/apps/sockpuppet \u2014 browser-based teleop + data collection; launches Quest teleop server and optionally vision/policy.
hmnd_robot/apps/wbc_broker \u2014 arbitration between teleop and policy WBC commands; proxies the policy action.
hmnd_robot/apps/alpha_bringup \u2014 RTPC bringup launch files (controllers + nav + helpers).
hmnd_robot/ros/control/whole_body_controller \u2014 whole_body_controller_interface (ROS-facing adapter around WBC/IK and controller forwarding).
hmnd_robot/ros/platforms/*_description \u2014 ros2_control + platform description + controller bringup.
hmnd_robot/ros/platforms/alpha_wheelbase_description/launch/controllers_bringup.launch.py \u2014 launches ros2_control_node, controller spawners, state publisher, joint-state merger, kinematics node, plus some real-hardware helpers.
hmnd_robot/ros/perception/hmnd_navigation \u2014 nav2 stack wrapper + action servers (navigate_to_pose, navigate_to_tag, etc.).
hmnd_robot/ros/perception/hmnd_localization \u2014 map server + localization + slam_toolbox lifecycle wrapper.
hmnd_robot/ros/perception/hmnd_vision \u2014 camera pipeline (hmnd_vision node) configured via YAML + intrinsics folder.
hmnd_robot/ros/robot_learning/hmnd_policy_execution \u2014 policy_action_server (default action name policy_execution).
This is not exhaustive; it\u2019s the set that tends to matter for reasoning about the system.
Provided by hmnd_navigation (via composed nodes + nav2):
/navigate_to_pose (hmnd_msgs/action/NavigateToPose)
/navigate_through_poses (hmnd_msgs/action/NavigateThroughPoses)
/navigate_to_tag (hmnd_msgs/action/NavigateToTag)
/navigate_path (hmnd_msgs/action/NavigatePath)
Notes:
These actions internally call nav2 actions like compute_path_to_pose, compute_path_through_poses, and follow_path.
/policy_execution (hmnd_msgs/action/ManipulationPolicy)
When the broker is enabled, the public action is typically provided by the broker:
Broker serves: /policy_execution
Broker forwards to: /policy_execution_internal
/wbc_broker/teleop_commands \u2014 teleop-origin WBC commands (source: sockpuppet)
/wbc_broker/policy_commands \u2014 policy-origin WBC commands (source: policy_action_server)
/teleoperation_server/teleop_active (Bool)
/teleoperation_server/teleop_commanding (Bool)
/whole_body_controller_interface/robot_commands \u2014 final WBC command stream consumed by RTPC WBC interface
/whole_body_controller_interface/wbc_solution_is_valid (Bool)
/whole_body_controller_interface/wbc_loaded_configuration (String)
/whole_body_controller_interface/diagnostics (DiagnosticArray)
/whole_body_controller_interface/pose_broadcaster/*/pose (PoseStamped)
/tf, /tf_static
/plan (nav2), and hmnd_navigation/nav_plan_publisher re-publishes as /plan_viz
lidar driver topics (as currently wired in RTPC bringup):
/lidar_1/scan_filtered, /lidar_2/scan_filtered
scan_merged, cloud_merged (from dual_laser_merger_node)
localization commonly publishes pose on /amcl_pose (see hmnd_localization config)
/merged/joint_states (due to robot_state_publisher remap)
Provided by whole_body_controller_interface:
/whole_body_controller_interface/reset_wbc_tasks
/whole_body_controller_interface/reset_wbc_targets
/whole_body_controller_interface/reset_joint_configuration
/whole_body_controller_interface/toggle_wbc_target_distance_check
/whole_body_controller_interface/get_last_grasp_primitive_command
Map server (nav2_map_server) provides map-serving services.
slamtoolbox_service.py owns starting/stopping the slam_toolbox mapping process and checks readiness via /slam_toolbox/serialize_map.
Operator uses sockpuppet UI (Orin) \u2192 Quest teleop server publishes WBC teleop commands.
wbc_broker arbitrates based on /teleoperation_server/teleop_active and /teleoperation_server/teleop_commanding.
wbc_broker publishes to /whole_body_controller_interface/robot_commands.
RTPC whole_body_controller_interface consumes commands, solves IK / checks safety-ish constraints, forwards to low-level controllers.
ros2_control_node + platform controllers drive hardware.
Client (often Reverb) sends action goal to /policy_execution.
wbc_broker proxies the action to /policy_execution_internal on policy_action_server.
Policy publishes WBC commands to /wbc_broker/policy_commands.
Broker forwards to /whole_body_controller_interface/robot_commands when teleop is not active.
Client (often Reverb) sends navigation action goal (e.g., /navigate_to_tag).
hmnd_navigation computes path \u2192 follows path (nav2).
Nav stack outputs base motion commands to the base controller path (exact topic wiring depends on platform; wheelbase vs biped differs).
RTPC bringup typically includes foxglove_bridge (topic whitelist to reduce bandwidth).
Many nodes publish diagnostics; WBC interface publishes DiagnosticArray and internal metrics.
TF/joint-state churn: many consumers reconstruct state outside the control process.
Multi-process nav stack composition is mixed: some components are composed, others intentionally not (performance degradation was observed for composing nav2 controller_server).
Cross-machine ROS graph can become \u201caccidental coupling\u201d (ad-hoc topics leak across RTPC\u2194Orin).
If we want, the next iteration of this doc can include:
a Phase 0 \u201ccontract v0\u201d section: the exact RTPC\u2194Orin interface list + rate/size budgets
a per-monolith \u201cinternal vs external topic\u201d rubric (what is allowed to cross machines)
a single canonical \u201cbringup matrix\u201d (Alpha wheelbase vs biped; real vs mock vs isaacsim)
There is some confusion perhaps around System 2 (agent-on-robot) and System 3 (agent-for-fleet) terminology. I think some of this stems from the public terminology found in https://thehumanoid.ai/ai/, or that our product team uses internally. In reality during implementation, we will typically need to be more precise.
Consider a service like Claude.ai. Clearly there is a model. And there is a product where users can chat with that model. But under the hood, there are going to be enormous amounts of bits and pieces to provide this product. KV caches, routing/batching, databases, authz, connectors/integrations, memory. Probably literally hundreds more.
Similarly, we should consider System-3 to be the broad term for how users interface with our fleet as a whole. This is where we talk about work, assign tasks, reason about overall efficiency. Its also where we\u2019ll edit workflows, re-assign robots and run analysis to improve performance over time. Similar to claude.ai there are many different pieces involved.
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1931476998/Untitled%20Diagram-1777560109616.drawio.png?api=v2&version=1\")
Switchboard
Switchboard specifically functions as a strongly consistent database where many services/agents can safely coordinate on some shared understanding of the world. Because we need to ensure only one robot at a time attempts to pick from some given tote stack or navigate through some specific passage, we need strong consistency guarantees so that everyone can agree on the both the current state of the world, and the recent history. The current implementation of Switchboard is primitive. It only recently gained the ability to persist its data to disk. But its role/function within the broader context of \u201csystem-3\u201d is basically as a shared database that other parts of system-3 can use to coordinate effectively.
Like most databases, Switchboard is dumb. It doesn\u2019t \u201cdo\u201d anything. It responds to queries, and doesn\u2019t generally take action on its own.
System-3 Agents and Services
In the same way that an authorization service like IAM may be used by hundreds of other services to gate access to resources in the cloud, Switchboard can provide coordination services to many other services within System-3, with some of these services being more \u201cagentic\u201d than others. Generally if we want the fleet as a whole to do anything, this functionality would be implemented within some System-3 agent or service which would in turn make queries to Switchboard to drive the workflow forward.
For example, Workflow Authoring is currently implemented as a System-3 agent. What makes it a System-3 agent? It has no body and it operates at a more abstract fleet-level. It can modify workflows and facts within Switchboard. This is analogous to having a conversation with Claude and stashing artifacts in a database.
A non-agentic System-3 service would be something like the SAP integration service (the non-A2A version), or a potential VDA5050 gateway to interop with some other fleet mgmt solution. These services bridge something on the outside world to the fleet as a whole.
We also have a non-agentic \u201cmap\u201d service where we can store maps/tags to be shared with other clients. Robots obviously need this data, but its not a big leap to imagine that some other system-3 service or agent may benefit from accessing this map data. The map service is probably more like switchboard (more foundational/data-basey), but its still fleet-wide and not specific to any robot. The map service is likely to become extremely sophisticated as we transition to large 3D maps - these maps may be so large as to be streamed in pieces to robots, with a bunch of additional indexing and spatial queries being supported. This starts to look increasingly like a spatial database, which complements switchboard which functions more as a coordination database.
Technically, robots talk to Switchboard in a way that\u2019s no different than any of the other System-3 agents that may exist. What makes System-2 special is that it controls a robot and can have direct action in the real world via the available tools.
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1931476998/Untitled%20Diagram-1777560744026.drawio.png?api=v2&version=1\")
System-2 is where the robot has agency. Switchboard is just another tool which can be used to query information about the current workflow, available workflows, the current assignments, etc. Switchboard is a relatively sparse, abstract and inaccurate view of the world. The robot has MUCH more context about what\u2019s happening, and so System-2 should ultimately be responsible for what the robot does. This of course can exist on a sliding scale. At one extreme, the system is fixed to ONLY execute assignments from Switchboard as they arrive. This has largely been how we\u2019ve operated through the robostore demo. It\u2019s predictable and simple. At the other extreme, the robot can choose to consider the assignments from switchboard as a suggestion. Maybe it decides to do something else because its closer. Or improvise in some other way. It\u2019s flexible and adaptable, but could be problematic. The value of the System-3 / System-2 split is that we can control this tradeoff directly. For example, a reasonable middle ground is that we follow assignments in the happy path, but if something goes wrong, the robot can \u201cskip\u201d the rest of the workflow and return home to await another task and resume the current one later.
Generally, we should minimize the coupling between the different parts of our system. Most interactions should be able to take place via Switchboard. This is simple and easy to scale. Having robots directly communicating with specific System-3 agents greatly increases the coupling in the system. Remember that we will need to apply software updates to all these pieces - and its unlikely that we can update things simultaneously, and its unlikely that we can tolerate downtime. This means rolling updates. Switchboard\u2019s API is designed to be simple enough that compatibility isn\u2019t a big challenge, and it isolates the individual components of the system to a degree that makes rolling updates relatively simple.
If individual robots start to integrate with OTHER System-3 services or agents directly, this can quickly become a very tight knot that is impossible to untie without significant pain. Similarly, robots generally shouldn\u2019t communicate peer-to-peer. 1st, there are generally n2 different pairs where this communication could happen, which is far more than the n channels needed to communicate with Switchboard as a hub/spoke model. 2nd, these \u201csidebar\u201d conversations would be invisible to the rest of the fleet which could cause some challenges with coordination.
Agent to Agent
One of the massive benefits of A2A comms is that it sorta bypasses the nitty gritty of API compatibility trivially, but it replaces it with a lot of non-determinism and unpredictability. We should be very mindful about how and where we use this.
Generally, the happy path of the system should go via Switchboard. Robots should be leasing/firing transitions, updating tokens and facts along the way. Other robots and system-3 agents see these updates reflected immediately and can take their own actions. Everything moves forward, its easy to audit and trace.
But what if something goes wrong? In these cases, it may make sense to open an A2A escalation path to a dedicated System-3 agent. For example, perhaps the robot is stuck in some complicated traffic jam. It can escalate as much context as possible via an unstructured A2A protocol and this agent can manipulate switchboard in such a way or give the Robot\u2019s System-2 agent additional info necessary to resolve the issue. These exceptional cases are the best usage of A2A protocols.
Switchboard is a dumb database, and doesn\u2019t \u201cdrive\u201d anything. Because its dumb, we can put it in the middle of the system to help with coordination challenges at fleet scale. At the fleet level, we may have a number of \u201csystem 3\u201d services which interface with switchboard, dispatching orders, adjusting priorities, resolving issues, authoring changes. At the robot level, Reverb is the system-2 agent which has autonomy and can use its tools to perceive the world, retrieve assignments from switchboard and take action via manipulation or navigation.
" }, { "id": "1929904147", "title": "hmnd-cloud -> hmnd", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1929904147", "last_modified": "2026-04-30T10:43:22.229Z", "created_at": "2026-04-30T10:37:58.285Z", "body_html": "This should be a boring lift/shift. Keep the hmnd-cloud repo intact under hmnd/hmnd_cloud, preserve history with git subtree, then wire the old pipelines into the root hmnd_ci workflow behind normal detect-changes gates.
The important rule: do not reorganise Terraform during the migration. Run Terraform from the same module roots, just with the new monorepo prefix:
hmnd_cloud/infrastructure
hmnd_cloud/ml-infrastructure
hmnd_cloud/deployables
hmnd_cloud/hmnd_deployment
hmnd_cloud/onprem-infrastructure
Terraform state tracks resource addresses, module names, provider config, backend config, and variables. It does not care that the Git checkout path changed, as long as the module internals and backend keys stay the same.
Create the monorepo branch.
cd hmnd\ngit checkout -b migrate/hmnd-cloud\n
Import hmnd-cloud with history.
git subtree add \\\n --prefix=hmnd_cloud \\\n git@github.com:HumanoidTeam/hmnd-cloud.git \\\n main\n
This keeps the old commits reachable in hmnd history while putting the old repo under one directory. Avoid --squash.
Do the minimum mechanical path fixes.
Move or copy the old GitHub workflows into root .github/workflows with new names, then prefix repo-local paths with hmnd_cloud/. Examples:
infrastructure -> hmnd_cloud/infrastructure
ml-infrastructure -> hmnd_cloud/ml-infrastructure
deployables/... -> hmnd_cloud/deployables/...
hmnd_deployment/... -> hmnd_cloud/hmnd_deployment/...
deployables/common.pkrvars.hcl -> hmnd_cloud/deployables/common.pkrvars.hcl
Add hmnd_cloud to hmnd_ci change detection.
Add one detector output, for example hmnd_cloud_changed, watching:
hmnd_cloud:\n - hmnd_cloud/**\n
Then roll it into plan exactly like the other areas. The old standalone on.pull_request.paths filters are no longer the primary guard; hmnd_ci.detect-changes should decide whether the migrated cloud jobs run.
Add reusable cloud workflows under hmnd_ci.
Keep the old jobs mostly intact, but make them workflow_call workflows and call them from hmnd_ci only when hmnd_cloud_changed, workflow files changed, or the run is forced.
Suggested split:
test_hmnd_cloud_infrastructure.yaml: fmt, lint, security, validate, plan for hmnd_cloud/infrastructure.
test_hmnd_cloud_ml_infrastructure.yaml: matrix, fmt, lint, security, validate, plan for hmnd_cloud/ml-infrastructure.
build_hmnd_cloud_amis.yaml: wraps the existing Packer jobs, still manually runnable if wanted.
deploy_hmnd_cloud_robots.yaml: keep manual only unless there is a strong reason to run from CI.
Open a PR with only the subtree import and CI/path wiring.
Do not rename Terraform resources, modules, backend keys, buckets, tags, or project names as part of this PR. Those can be follow-up cleanups after the plan proves stable.
Expected local proof after the subtree import and path-only CI changes:
cd hmnd/hmnd_cloud/infrastructure\nterraform init -input=false -no-color\nterraform plan -input=false -no-color -lock=false\n
Expected result: no repository-path-driven changes. A clean run should end with:
No changes. Your infrastructure matches the configuration.\n
For ML infrastructure, run the same plan per backend:
cd hmnd/hmnd_cloud/ml-infrastructure\nterraform init -input=false -no-color -reconfigure \\\n -backend-config=backends/backend-dev-eu-west-2.hcl\nterraform plan -input=false -no-color -lock=false \\\n -var='environment=dev' \\\n -var='region=eu-west-2'\n
Repeat for:
backend-dev-eu-west-2.hcl
backend-dev-us-west-1.hcl
backend-prod-eu-west-2.hcl
CI proof should be a deliberately harmless PR change, for example a small README edit under hmnd_cloud/README.md. That should trigger hmnd_ci.detect-changes, run only the hmnd-cloud workflows, and show the same clean Terraform plans in CI.
I scanned the current hmnd-cloud checkout on 2026-04-30. These are the likely hiccups.
Hard-coded workflow paths need prefixing. The old workflows assume they live at repo root:
.github/workflows/deploy-infrastructure.yml: working-directory: infrastructure, scan-ref: infrastructure, cache keys, artifact paths.
.github/workflows/deploy-ml-infrastructure.yml: working-directory: ml-infrastructure, backend discovery with find ml-infrastructure/backends, cache keys, artifact paths.
.github/workflows/shared-ami-build.yml: deployables/common.pkrvars.hcl and Packer directories.
.github/workflows/build-base-gpu-ami.yml, build-github-runner-ami.yml, build-simulation-ami.yml: packer_dir values under deployables/....
.github/workflows/deploy-to-robots.yml: checkout path and script paths explicitly use hmnd-cloud.
Repo-name references exist and should be reviewed, but most should not block the first lift/shift:
ml-infrastructure/config.yaml allows ARC runners for both HumanoidTeam/hmnd-cloud and HumanoidTeam/hmnd. After migration, keep hmnd and remove hmnd-cloud only after old repo CI is retired.
infrastructure/domains/shared-between-teams/github-runners/main.tf defaults GITHUB_RUNNERS_REPO to HumanoidTeam/hmnd-cloud. This is probably overridden by TF_VAR_GITHUB_RUNNERS_REPO; confirm in repo variables before changing it.
infrastructure/variables.tf, deployables/common.pkrvars.hcl, infrastructure/domains/simulation/coder/templates/simulation-workspace/main.tf, and infrastructure/domains/shared-between-teams/skypilot/skypilot-config.yaml contain hmnd-cloud as project/tag/bucket naming. Do not rename these during the migration unless the plan explicitly proves it is harmless.
infrastructure/domains/shared-between-teams/skypilot/skypilot-config.yaml uses s3://hmnd-cloud-skypilot-jobs2. Treat that as infrastructure naming, not a repo path.
Path-sensitive Terraform appears okay if the subtree root is stable:
Terraform module sources are relative inside the repo, for example source = "./domains", source = "./modules/...", and source = "../../../modules/...". These continue to work when running from hmnd_cloud/infrastructure or hmnd_cloud/ml-infrastructure.
One path crosses from infrastructure to sibling ssh-keys: infrastructure/domains/reasoning/eu-west-2/main.tf uses ${path.root}/../ssh-keys. This still resolves correctly if ssh-keys remains at hmnd_cloud/ssh-keys.
Cross-region training modules reference ../../../../../infrastructure/modules/...; this remains valid as long as the internal hmnd_cloud/infrastructure tree is not rearranged.
Commit/hash usage does not look like a blocker:
CI artifact names use ${{ github.sha }}. In the monorepo this becomes the hmnd commit SHA, which is fine for artifacts and PR comments.
hmnd_deployment/pre_alpha/deployment-scripts/deploy_to_robots.py has a default commit='main', but I did not find a hard dependency on old hmnd-cloud commit SHAs.
Existing local plan evidence is not clean yet, but the failures are credential/access related, not repo path related. The checked-in infrastructure/plan.log shows a previous plan ending with Plan: 173 to add, 26 to change, 2 to destroy, followed by missing Coder token and Kubernetes Unauthorized errors. The migrated CI proof should therefore be treated as the source of truth for the clean plan.
hmnd_cloud exists in hmnd with hmnd-cloud history preserved.
hmnd_ci detects hmnd_cloud/** changes.
The migrated Terraform and Packer workflows are called through hmnd_ci, not standalone root path filters.
A README-only PR under hmnd_cloud/ triggers the cloud workflows and no unrelated monorepo jobs.
CI Terraform plans for infrastructure and all active ml-infrastructure backends are clean, or any non-clean output is explained as pre-existing drift/access rather than migration fallout.
A fleet of loosely coordinated, autonomous humanoids will beat a rigid, centrally planned scheduler - not by being \u201csmarter\u201d, but by staying in motion and constantly making progress.
Centrally planning the paths and tasks of a fleet of humanoids will result in a very brittle and fragile system. There is no graceful failure. With autonomous agents, individual agents may fail and degrade performance, but the fleet as a whole can continue making progress. We are already betting these robots on local autonomy. We are already assuming that AI will work. We should double-down on this and also assume that we can make our robots sufficiently intelligent such that global coordination (rather than planning) is viable.
This is aligned with how technology as a whole has been evolving for decades at this point. Expensive vertically scaled databases have been horizontally scaled across cheap commodity devices. Routers and application servers too. More recently, even satellites have been hit with this trend as billion dollar geostationary satellites are replaced with fleets of cheap low-earth-orbit satellites (Starlink being the best-known with more coming). The first reason for this is that you can get unit costs down (this is aligned with us scaling a large fleet). The second reason is that failures become more gradual rather than abrupt - this is the point that\u2019s relevant here.
There\u2019s a seductive idea underneath central planning: if you can just see everything, you can assign perfectly. Your agents can plan around other agents, predict where they will be in the future, minimize congestion and generally execute tasks at maximum throughput. It\u2019s a lie. Not because optimization is hard or impossible (heuristics are good enough that being NP hard isn\u2019t really a meaningful problem these days), but because the thing you\u2019re optimizing against doesn\u2019t really exist. The world you model centrally is always:
slightly out of date (latency is non-zero)
slightly incomplete (you aren\u2019t seeing everything)
slightly wrong (its an abstraction of an abstraction, not reality)
This is fundamental. For a large fleet of humanoid robots, the data exists at each robot. Each robot is moving, sensing and reacting to its local environment. It makes much more sense to push decision making to where the data lives, rather than trying to pull the data from multiple robots into a single \u201cbrain\u201d.
Central planning works (kind of) for AMRs because the world could be simplified:
flat floors and static routes
limited actions & predictable outcomes
You can reduce everything to nodes and edges and call it a day. Humanoids won\u2019t live in a graph, because the range of actions they can take in the world are far greater than an AMR. AMRs can accept a payload and take it somewhere. Humanoids can actually manipulate payloads end-to-end, with a number of complex failure and recovery behaviors along the way. The actions that an AMR can take are pretty predictable. Maybe it takes longer to load/unload - but there aren\u2019t many other possibilities. A humanoid may have a lot of variance around the tasks it may need to perform. Who knows how many items are in the bin, or how many times it may need to retry grasps\u2026
The important information isn\u2019t just hard to centralize - it\u2019s not even representable in a clean way without drastically over-simplifying things. So the planner makes a \u201cgood\u201d decision based on bad abstractions, and the robot fixes it locally anyway.
The real goal isn\u2019t picking the \u201cbest\u201d task for any robot at any time. It\u2019s just keeping the system moving. A centrally planned system looks efficient only when nothing goes wrong. But the moment anything deviates (and it always will) it stalls:
robots wait for reassignment
plans are repeatedly invalidated
no guarantee that replanning results in continuous changes - lots of thrashing and inefficiency
Throughput and efficiency collapses not because robots are dumb, but because they\u2019ve given up local autonomy. A decentralized system ensures robots can continue to make progress, even if other robots have unexpected issues.
a robot hits friction \u2192 it retries, escalates or abandons the task
a task becomes weird \u2192 someone else picks it up later
state is slightly off \u2192 it gets corrected in-line
You trade perfect coordination for continuous progress, because reliability is traditionally the bigger barrier to throughput than \u2018optimality\u2019 with robotics (especially something like humanoids).
A core mistake is treating the center as the source of truth. It isn\u2019t and it can\u2019t be.
Truth lives at the edge:
in the camera frame
in the gripper
in the blocked aisle
in the thing that just changed
You can stream summaries of that upstream, but you can\u2019t centralize the experience of it (at least not without somehow fixing wireless networks: Wireless Networking Sucks )
So every centralized decision is made on a compressed, approximated, delayed abstraction of reality. But local decision is made on the thing itself.
Central planners love to claim they prevent deadlocks. You can plan optimal route for many agents through the same map, ensuring robots never collide, congest or otherwise impede one another. Great, right?
What they actually do is synchronize failure. When the central system hesitates, everything hesitates. When it\u2019s wrong, everything is wrong together. One robot\u2019s pose being off (perhaps delayed due to faulty wireless) can result in the whole plan unwinding.
In a decentralized system, conflicts still happen - but they stay small:
two robots contend \u2192 one backs off
a path blocks \u2192 local re-route
a resource is busy \u2192 retry later
This friction may reduce the efficiency of one robot, but the rest of the robots can continue to make progress.
Let\u2019s consider an example where we have items that need to be moved across a warehouse. In a traditional centrally-planned scheme, the central planner knows the battery level and the locations of all the robots and can identify the best robot for the job. In a decentralized scheme, the coordinator doesn\u2019t \u201cassign\u201d work to individual robots - it advertises assignments on a billboard and robots are free to take these assignments.
As a general rule, robots that are \u201cbetter\u201d for the assignment (closer to the source, higher charge-levels, better health) will respond more eagerly. Robots that are \u201cworse\u201d will back-off and respond after some amount of time. On average, this will result in the best robots being assigned the work - but unlike a central coordinator, it is much more robust to the chaos of real-world operations.
![\"image-20260429-152958.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1793425450/image-20260429-152958.png?version=1&modificationDate=1777476600416&cacheVersion=1&api=v2&width=790&height=526\")
Let\u2019s re-consider the centrally-planned scheme. The central task scheduler has the robot locations and their paths, so naturally it can find the optimal assignment to maximize throughput. But what if the scheduler chooses a robot that\u2019s just failed or maybe was manually re-assigned concurrently? The system will experience some sort of the thrashing as that plan times out (the robot fails to acknowledge the assignment perhaps), another choice is made, and the plan needs to be adjusted again.
In the decentralized approach, the assumption is that eventually SOME robot will respond. The coordinator can advertise priorities and operators can tune the eagerness depending on what factors they may care about (power, distance, priority). The key thing is that a robot will be assigned, and other robots don\u2019t need to worry about it. The coordinators strong consistency semantics will ensure that only the first robot to respond will actually successfully take the task.
Of course, the system SHOULD allow manual assignment. There are all sorts of reasons that someone may want to assign a specific task to a specific robot - this can be handled by roles/affinities in the advertised task (meaning only the matching robot would respond - regardless of any other facts).
In addition to this assignment problem, there are a number of other coordination challenges that can be solved similarly. Some simple schemes can be used to break symmetry to allow multiple robots to navigate around one another or establish right-of-way.
We are making smart robots. We should avoid the temptation to remove autonomy from robots. This is trading local intelligence for networking and coordination overhead - which is probably a bad trade given what I know about wireless networks. This doesn\u2019t mean that robots don\u2019t have some level of awareness of their peers though. Its relatively cheap to broadcast to each robot any peers within a 10m radius for example and see some historical navigation data. The key thing is that this data would be used by the robot itself to do planning, with the robot having the ability to modify these plans on the fly as it learns new information.
Maybe in the future the pendulum will swing the other way. We will train a mega-VLA that takes 100s of cameras and outputs 1000s of task-space trajectories to control 100s of robots. But for now let\u2019s build robots, not networks.
" }, { "id": "1619853323", "title": "Spec: Switchboard Coordinator", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1619853323", "last_modified": "2026-04-24T10:33:50.748Z", "created_at": "2026-02-08T13:04:14.704Z", "body_html": "Summary: System-3 Coordination
Owner: Cody Griffin
Date:
Approvers:
Informed:
Related: Spec: HMND Fleet Control Plane
Enabling robots, agents, operators and 3rd party systems like ERP, WMS or POS to coordinate requires some shared understanding of the world. This model can be used to ensure only one robot attempts to fulfill a given order at a time, or that robots use shared resources like chargers efficiently. A consistent, transactional database ensures that all agents (human or otherwise) agree on the state of the world while allowing each one to act independently.
Imperative models like Behavior Trees or State Machines allow for individual robots to be controlled precisely - even scripted to follow exact instructions. At the fleet level though, these representations begin to fall apart due to the partial observability of the world and the concurrency of many robots and other simultaneous processes.
Switchboard is a database that stores the state of the fleet\u2019s workflow. It forms the backbone of System 3 - where agents can coordinate about who is doing what at any point in time. Switchboard is not a workflow optimizer, scheduler or planner. It simply tracks workflow execution. A perfectly conformant agent could simply execute assignments one at a time to progress perfectly through any defined workflow, but we should also expect reality to differ considerably and for events to deviate entirely from any current workflow definition.
Authoritative - changes should be serialized, recorded and audited.
Consistent - because agents have side-effects, we need to prevent double-execution
Flexible - \u201cillegal\u201d transitions should be logged regardless so that we can make workflows more complete over time
Simple - limited scope to workflow state
Scalable - should scale to hundreds or thousands of agents coordinating simultaneously across dozens of tenancies
Inference or optimizing - leave this to operators or external agents
Scheduling - done externally
The Switchboard API is used by authorized agents to fire transitions, which the are recorded to the event log and used to update the marking of the current workflow state. This workflow is encoded as a Petri net, with each row being a particular token at some place. By breaking a Petri net into tokens as rows in a SQL database, we can transactionally update the workflow state, while enabling different agents to update different tokens concurrently, ensuring our workflow execution can scale to 10s or even 100s of thousands of tokens.
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1619853323/Untitled%20Diagram-1770558123165.drawio.png?api=v2&version=1\")
The robots' system 2 agents can use Switchboard as a tool to enroll robots or determine which tasks should be executed to progress the workflow. System 3 agents like WMS integrations (think SAP\u2019s Joule or similar) can fire transitions which create work items representing orders. Other system 3 agents may be responsible for updating priorities (scheduling) or handling escalations. Human operators can also use switchboard to re-task robots or recover from failure.
We have a few core components to manage within Switchboard:
Tenants - top-level accounting used to isolate specific customers. Ideally this is implemented with Postgres RLS as an extra layer of protection. The assumption here is that many customers will coexist within the same Switchboard instance, but we should also expect that some customers may want their own infrastructure entirely (a good reason to use something standard like Postgres).
Principles - authorization bounds within a tenancy (ensuring multiple robots or operators within a tenancy don\u2019t interfere)
Definitions - immutable descriptions of workflows: states, valid tasks, work items. New versions are appended.
Instance - logs and materialized state for a given instantiation of a particular workflow definition. Note that any given workflow instance could include illegal events which may not adhere strictly to the current definition.
Events - log of how the workflow has evolved over time. Should be replay-able, though order isn\u2019t strictly serialized (concurrent events at independent places should be tolerated)
Items - used to encode the materialized state of the workflow as tokens
Leases - exclusive claims to execute tasks which may have real-world side-effects.
Facts - mutable KV storage which allows agents to coordinate about the state of the world. Facts have a TTL - meaning they need to be re-asserted to remain valid.
Switchboard clients need to be authenticated principals. Robots may use mTLS, operators some sort of OIDC or we may start with simple pre-shared tokens. Each principal needs to be authorized to allow some specific transitions within the workflow, including various \u201cbuilt-in\u201d transitions to support internal mechanisms like re-prioritization.
Our workflow definitions can include which roles are permitted and which capabilities should be granted. Clients do not directly manipulate events or token states: they must interact through the workflow\u2019s transitions. In some cases, clients may be permitted to make arbitrary \u201cillegal\u201d transitions which don\u2019t fit the workflow description.
For example, some system 3 agent may be permitted to fire a \u201csource transition\u201d which introduces new order tokens into the workflow. This agent isn\u2019t permitted to do do anything else. A system 2 robot agent may be permitted to enroll itself, and to fire any transition involving that specific robot token (but no others). Each token is associated with some principal (who created it), and this create_by can be used to ensure that different agents don\u2019t interfere.
Agents can fire transitions which inject new tokens marked with their created_by into the workflow. Some agents may be able to move those tokens through the workflow (like a robot performing different actions). These agents needs to acquire leases to actually fire those transitions with any coordination. Robots for example should call Switchboard periodically to update the status of previous claims and to see which assignments are available.
Leases are used to track which tasks are supposedly being executed to ensure that multiple robots do not attempt to do conflicting work. Each workflow state also has a capacity to prevent multiple robots from colliding. So the workflow knows that if robot A is in state S1, which has a capacity 1, it cannot fire any more transitions into that state and thus will not advertise those transitions to other robots. Even after robot A starts transitioning out of the state, we don\u2019t complete until robot A updates about the status of the lease (a claim). There is a lot of intricacy around \u201creversing\u201d transitions here if we want to make robust workflows.
When agents make claims that transitions have been executed, Switchboard will log the event, clear the lease and update the materialized state of the workflow instance. In cases where the logged event is illegal, the marking may involve transitioning tokens to special \u201cquarantine\u201d states.
Sometimes there may be multiple transitions that are valid for any agent. How do they determine which ones to prioritize? A simple priority value on the tokens can help order things, and some agents may have permissions to call built-in \u201creprioritize\u201d transitions on specific states. For example, a system 3 optimizing agent could decide to expedite a pending order (allowing some PendingOrder.Reprioritize transition) without allowing in-flight orders to be touched. Another agent or human operator may have permission to cancel or undo in-flight orders.
Some tasks can be canceled (reversed) if we know the inverse. For example, navigating from A to B is easy to reverse (navigate to A). Picking up a tote can be reversed by placing the tote back. Some tasks can\u2019t reasonably be canceled though (like dumping the contents of a tote into a hopper or placing a tote on a conveyor).
At any time, we can run events through a workflow definition and keep track of how many follow legal transitions vs illegal transitions. We can use the ratio here as a simple conformance metric. If workflow conformance falls too low, we can trigger some external agent to look at the event log to propose a new workflow definition.
New workflow definitions are published as new versions. Instances migrate explicitly via recorded actions. When structure changes invalidate existing state, unmapped tokens move to quarantine rather than disappearing. Workflow regeneration is advisory: inference proposes, humans approve, and new definitions are published.
Human operators (or some system 3 agent) can move quarantined tokens to the correct states after the migration.
Switchboard is a single authority that records what agents do, prevents them from stepping on each other, and preserves execution reality so workflows can evolve without forgetting what actually happened.
M0. In-memory POC (what we have for MF/Bosch/Robotstore)
M1. SQL database + stateless API
M2. Simple authz model and multi-tenancy
M3. Conformance
For the Switchboard API, we\u2019ll do a simple REST interface. The big benefit of something like REST is that we can easily cache, load balance, etc - but also its just much simpler for 3rd parties to integrate (in comparison to gRPC or something like that).
Managing Definitions
One of the simpler interactions with switchboard should be to POST new workflow definitions, and to PUT new versions of workflows. Some care needed at the API to make sure versions increment safely. For example, PUT should include the old version so that if something has changed in the meantime we get a conflict.
The workflow definition is a YAML document (for now) describing the various types of workflow items, states, transitions and even principals that may attempt to access the workflow. We should assume that the API layer will validate these definitions before writing them to the database.
So a simple CRUD, with some additional complexity around the versioning to ensure things are consistent.
POST /v1/tenants/$tenant/workflows\nPUT /v1/tenants/$tenant/workflows/$id?from=$old_version\nGET /v1/tenants/$tenant/workflows/$id?version=$version\nDELETE /v1/tenants/$tenant/workflows/$id?version=$version\n
The WorkflowDefinition resource:
tenant_id: str\nworkflow_id: uuidv7\nversion: int\nname: str\ndescription: str\ndefinition: str # a blob\n\ncreated_at: datetime\ndeleted_at: datetime # tombstone\ncreated_by: str # how much authz do we do here?\nmetadata: JSON blob\n
Managing Instances
In addition to workflow definitions, we also have specific INSTANCES of those workflows. For any workflow definition, we may have a number of independent instances (think testing, simulation, or physical twins). While versioning of the workflow definitions is relatively simple, versioning a running instance is much more challenging its a living thing.
To simplify the API, we won\u2019t nest the workflow instances under the workflows above.
POST /v1/tenants/$tenant/instances\nGET /v1/tenants/$tenant/instances/$id?...\nDELETE /v1/tenants/$tenant/instances/$id\n
We can create a new instance of a workflow (given some definition/version) easily enough. We can also retrieve the workflow state. This effectively returns the current marking (all the tokens) of the workflow. This can be filtered down a bit. The workflow can also be deleted.
Unlike definitions, workflow instances are sharded over many rows, which enables many robots to update concurrently without stressing our database out (even a relatively small fleet could cause issues with a SQL database if there are lots of conflicting rows). This isn\u2019t what\u2019s exposed via the API though. To update a workflow, we must fire a transition - and to accommodate real-world side effects we must do this using leases and claims:
POST /v1/tenants/$tenant/instances/$id/work\n
This is a more complicated API, and not strictly very RESTful. The reasoning is we want to keep things effortless transactional for the clients. So when we update Switchboard, we\u2019ll provide facts, leases or and/or claims:
Facts - mutable facts about the world. These are things that evolve faster than our workflow, and can be used to guard transitions. For example, we may not want to allow a robot to attempt to pickup a tote unless its nearby (pose) or we may want a robot to navigate to a battery swap if its battery is too low.
Leases - exclusive rights to perform a transition with side-effects. This is how we avoid sending multiple robots down the same corridor or to pick up the same tote or trigger the same async business process via some baroque ERP system.
Claims - status of any tasks performed.
facts:\n pose: {x: 0, y: 12}\n battery: 0.3\nleases: \n - task: PickTote\n id: <leasing id>\n result: success\n ... more context about our lease\nclaims: \n - task: PickTote\n id: <leasing id>\n result: success\n ... more context about completed task\nSo a robot for example may update its pose or battery state periodically, and when executing long-running tasks (like policy or navigation) it may need to update the lease to keep it alive so that Switchboard doesn\u2019t hand it out to someone else.
When Switchboard sees something like the above, it will log the event for the incoming facts, claims or lease extension and then fire any claimed transitions (in this case, firing the PickTote transition). Switchboard can then evaluate what transitions could fire next (eligible) and hand out leases.
leases:\n - task: NavigateToTable\n id: <leasing id>\n ... additional context\n... other leases\n
The robot\u2019s system 2 agent in this case can evaluate the options (or use some sort of priority scheme) to decide which lease to hold. Leases have a time-to-live (TTL) that\u2019s relatively short, and so expire quickly. In this case, let\u2019s assume that the robot is going to navigate to the table, which takes longer than the TTL. The robot should simply call the work endpoint repeatedly (with new facts) refreshing this lease. Its possible that we could estimate how long it should take and extend the TTL more precisely but this isn\u2019t strictly necessary.
At some point the task is completed, in which case we also make a claim that will any transitions within Switchboard and move to the next task. Note that its possible that a robot could claim success for ANY tasks - not necessarily those leased (think exceptional cases or human intervention).
Other Work
In addition to robot\u2019s leasing/completing work and updating facts, we should expect other services or agents in system 3 to participate here. A WMS for example may need to add orders to the queue for fulfillment. In this case, the WMS can immediately claim a task completed (\u201cNewOrder\u201d for example), without worrying about leasing semantics since everything is instantaneous.
More advanced workflows may require some long-running asynchronous work at System 3 though - such as handling interventions. In this case, some agent capable of handling interventions (or a human operator) will poll the work and see new leases show up which can then be acted on, much the same way as robots operate.
Any incoming requests to Switchboard need to be authenticated (who is accessing) and authorized (what can they do). We\u2019ll consider two possible schemes to start: mTLS (for robots/service-to-service) and JWT (for humans/external systems). mTLS will be terminated at some load balancer and the various claims within the certificate will be lifted into the header, and so we will look here at a more abstract authorization model where we assume that incoming requests have some sort of subject_id (who is the principal making the request) along with what capabilities they have.
Principals and Tokens
Some principals actually are represented by tokens in the Petri net, like a robot for example. Other principals (like human operators) have no such representation internally. Both of these scenarios are relevant for authorization.
We define roles in our workflow definition - so we know for example what a robot is or what a WMS is and which transitions can be fired.
roles:\n- name: robot\n capabilities: [*, EnrollRobot]\n subject_binding: self\n- name: wms\n capabilities: [NewOrder]\n subject_binding: none\n\n... rest of definition\n
In this simple example, we define a robot role, which can fire any robot-involved transition (meaning only tokens bound to that subject), along with a wms role which is only permitted to fire the NewOrder transition (though it has no binding).
To support this, our token schema needs to track some sort of (optional) principal/subject_id. In the above example, a new robot should be able to fire the EnrollRobot on startup to \u201cinject\u201d a token containing the robot\u2019s subject_id into some idle state, after which the robot is authorized to transition freely as long as it involves that token.
Workflow Definitions
For completeness, we have another layer of authorization outside of the workflow at the tenancy level. Operators or system 3 agents may have access here to modify workflow definitions over time.
Our core SQL schema involves three tables: definitions, events and markings.
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1619853323/Untitled%20Diagram-1770639976528.drawio.png?api=v2&version=1\")
Definitions
Workflow definitions are written as immutable blobs (YAML-formatted right now), with versioning so that workflows can evolve over time. Creating a new workflow inserts a new row with a new workflow ID at version 0. Updating a workflow keeps the same ID and increments the version.
Its not expected that the schema needs to see \u201cinside\u201d the definition itself (like JSONB) since we won\u2019t be strictly validating that events or markings adhere to the definition.
Events
As clients fire transitions via the Switchboard API, events are appended to the table - referencing the current version of the workflow. This ensures that over time we can replay the events (and any workflow definition updates) to test for conformance or to simulate the workflow.
Events will include information about the source and destination states, the action taken and any tokens involved.
Marking
As the events are written, we can update the current state of the workflow in the marking table. This is almost a materialized view of a given workflow instance. It should be possible to recreate the marking table at any time by replaying the events.
Markings ought to be indexed so that we can quickly locate a token. The expectation is that our API will load the marking and the workflow definition into RAM when needed and compute any Petri updates there - rather than trying to do this work directly in SQL.
Leases
New table? Or is this just in the marking? We could \u201cacquire\u201d the lease in the marking table for some input token? Probably cleaner to have a small table tracking outstanding leases - assume explicit for now.
Facts
Just KV storage in SQL. Easy.
Here we can show:
using built-in transitions to re-prioritize work on the petri net
canceling work
reversing tasks (if its reversible)
This page replaces the earlier Teleport-based proposal.
We did not proceed with Teleport because the required OIDC-based human login flow is gated behind Teleport Enterprise. The current MVP architecture keeps human identity in Cognito/OIDC through Backbone and uses a simple bastion transport proxy instead.
This document describes the current SSH access design for HMND.
The goals are:
Device identity is anchored in Backbone mTLS.
Human identity is anchored in Cognito/OIDC through Backbone.
Bastion acts as a transport and policy enforcement point, not as a human IdP.
Human operators use normal ssh ergonomics.
The client-side helper is portable across macOS and Linux.
The robot side is handled by the existing hmnd-backboned package.
This is the current MVP architecture. It intentionally prefers a simple, reliable path over a fully general SSH access platform.
There are three planes:
Control plane: Backbone
Data plane: bastion proxy + device tunnel
Endpoint auth: robot sshd
Backbone is responsible for authenticating and authorizing both sides:
devices authenticate to Backbone with mTLS
humans authenticate to Backbone with Cognito/OIDC
Backbone does not terminate the final SSH session.
Instead, the bastion exposes a byte-stream proxy over WSS on 443, and the client uses that path as an OpenSSH ProxyCommand. This makes the human side feel like native SSH.
The robot is enrolled into Backbone.
hmnd-backboned authenticates to Backbone using its device certificate.
hmnd-backboned fetches ssh.bootstrap.json from:
GET /v1/devices/{device_id}/ssh/bootstrap
The bootstrap payload contains the information needed for the device tunnel:
device tunnel WSS URL
device tunnel session endpoint
bastion proxy TLS cert for pinning
robot login name
hmnd-bastion-tunneld calls:
POST /v1/devices/{device_id}/ssh/tunnel-session
Backbone issues a short-lived device tunnel token bound to that device.
hmnd-bastion-tunneld uses that token to keep a WSS device tunnel up to the bastion.
When a human attaches, the tunnel daemon connects locally to 127.0.0.1:22 and bridges raw SSH bytes.
hmnd-backboned reports SSH readiness and host key metadata in its session heartbeat.
The human runs either:
ssh aw8-orin with ProxyCommand configured, or
the helper directly: hmnd-ssh aw8-orin
hmnd-ssh checks for a cached Cognito ID token.
If needed, hmnd-ssh opens the Cognito Hosted UI in the browser and captures the callback locally.
hmnd-ssh calls Backbone:
POST /v1/ssh/session/{device_id}
Backbone verifies:
valid human identity
operator/admin authorization
device exists
device heartbeat is fresh
device SSH metadata is internally consistent
device tunnel is ready
Backbone returns:
a short-lived bastion proxy session token
the bastion WSS URL
the host alias for the target device
a known_hosts line for the robot host key
the bastion proxy TLS certificate for pinning
hmnd-ssh writes the known_hosts line into the local cache.
hmnd-ssh opens a WSS connection to the bastion and bridges stdin/stdout.
Local OpenSSH talks end-to-end to the robot sshd over that stream.
The robot handles box-local SSH auth.
For MVP, the robot may still use its existing password or public key auth policy.
Device identity is real and anchored in Backbone mTLS.
The robot must present a valid Backbone-issued device certificate to call device-scoped endpoints.
This is the mechanism that allows the device to retrieve SSH bootstrap data and short-lived device tunnel sessions.
Human identity is real and anchored in Cognito/OIDC.
The bastion does not talk to Cognito directly.
Backbone is the policy engine for humans. It converts a valid Cognito-authenticated human request into a short-lived, device-scoped session token for the bastion proxy.
The bastion trusts only Backbone-issued short-lived session tokens.
It does not independently decide who a human or device is. It only validates that Backbone already authorized the session.
The robot remains the final SSH server.
That means:
host key verification is against the robot host key, not the bastion host key
SSH auth methods offered by the robot are the ones the client sees
if the robot only offers publickey, the client will see publickey
if the robot allows password auth, the user will get a password prompt from the robot
This is why the design behaves like native SSH rather than like a remote shell product.
The bastion is not the SSH server the human authenticates to.
Instead, the bastion proxies raw SSH bytes between:
the human client on one side
the robot's reverse-tunneled socket on the other side
That gives three concrete benefits:
Native OpenSSH ergonomics
ssh, scp, rsync, sftp can use the same transport model later
Human auth stays centralized in Backbone
The bastion remains simple and policy-limited
Backbone owns:
device bootstrap endpoint
device tunnel session endpoint
human session endpoint
human authorization
SSH readiness evaluation
robot host key publication
Relevant runtime settings include:
BACKBONE_SSH_BASTION_HOST
BACKBONE_SSH_ROBOT_LOGIN
BACKBONE_SSH_PROXY_PUBLIC_URL
BACKBONE_SSH_PROXY_TLS_CERT_PEM
BACKBONE_SSH_PROXY_SHARED_SECRET
BACKBONE_SSH_PROXY_AUDIENCE
BACKBONE_SSH_DEVICE_TUNNEL_AUDIENCE
BACKBONE_SSH_PROXY_ISSUER
BACKBONE_SSH_PROXY_SESSION_TTL_SECONDS
BACKBONE_SSH_DEVICE_TUNNEL_SESSION_TTL_SECONDS
BACKBONE_SSH_HOST_ALIAS_PREFIX
hmnd-backboned is the robot-facing Backbone daemon.
It now owns the robot-side SSH flow at the package/install level.
Responsibilities:
load enrollment artifacts
authenticate to Backbone using device mTLS
heartbeat session data to Backbone
fetch ssh.bootstrap.json
inspect local SSH service state
publish host key metadata
ensure the reverse tunnel helper is installed/enabled via packaging
read tunnel status emitted by hmnd-bastion-tunneld
The installed package also enables/restarts hmnd-bastion-tunneld.service.
hmnd-bastion-tunneld keeps the device tunnel alive.
Responsibilities:
read ssh.bootstrap.json
prepare tunnel runtime assets
fetch short-lived device tunnel sessions from Backbone using device mTLS
establish the WSS device tunnel to the bastion
bridge bastion traffic to local robot sshd
This helper is still a separate service/process, but it is part of the robot-side SSH behavior managed by the hmnd-backboned package path.
The bastion stack provisions:
an EC2 host
an Elastic IP
Route53 DNS
a WSS proxy daemon on 443
a pinned TLS certificate for the proxy
a shared secret used to validate Backbone-issued session tokens
The bastion proxy daemon:
accepts WSS connections
validates Backbone-issued bearer tokens for both humans and devices
maintains the live registry of connected device tunnels
attaches a human proxy session to the authorized device_id
relays bytes between the human websocket and the device tunnel
hmnd-ssh is the portable human-side helper.
It is a standalone uv script with an executable shebang:
#!/usr/bin/env -S uv run --script\n
That means either of these are valid:
/abs/path/to/hmnd_services/backbone/scripts/hmnd-ssh aw8-orin\n
or
uv run --script hmnd_services/backbone/scripts/hmnd-ssh aw8-orin\n
Current modes:
hmnd-ssh <ssh-alias>
hmnd-ssh login
hmnd-ssh logout
hmnd-ssh proxy %h %p
Responsibilities:
browser OIDC login if needed
local token caching
request a short-lived session from Backbone
write robot host key entries into local known_hosts cache
open the WSS proxy session
act as ProxyCommand transport for OpenSSH
Run the one-time setup:
cd hmnd_robot\npixi run setup_ssh\n
or from the repo root:
pixi run -e ssh setup_ssh\n
That:
installs hmnd-ssh to ~/.local/bin/hmnd-ssh
creates or updates a managed block in ~/.ssh/config
makes reruns idempotent by replacing only the HMND-managed block
The installed SSH stanza looks like:
Host aw* ab* sp* ot*\n User hmnd\n ProxyCommand ~/.local/bin/hmnd-ssh proxy %h %p\n UserKnownHostsFile ~/.cache/hmnd/ssh/known_hosts\n StrictHostKeyChecking yes\n ServerAliveInterval 240\n
Then connect with:
ssh aw8-orin\n
If the SSH config is not installed yet:
/ABS/PATH/TO/hmnd_services/backbone/scripts/hmnd-ssh aw8-orin\n
For local testing where the target login user is not hmnd:
/ABS/PATH/TO/hmnd_services/backbone/scripts/hmnd-ssh aw8-orin --ssh-user <local-user>\n
These are intentional for now:
the robot may still use legacy password auth or manually-managed public keys
the bastion proxy TLS cert is pinned and self-signed, not ACM-integrated product polish
ssh:// launch links are best-effort and depend on local terminal / OS URL-handler support
terminal fidelity depends on client/server TERM support
audit is strongest at Backbone/bastion level; robot-local shell identity may still be a shared local account depending on the box configuration
The host key the human client verifies is the robot host key.
This is critical.
Because the bastion is only transporting bytes, the client should not pin the bastion SSH host key for the final shell session.
Instead:
hmnd-backboned reads the robot host public key
Backbone publishes that key through the SSH session API
hmnd-ssh writes that line into ~/.cache/hmnd/ssh/known_hosts
local OpenSSH verifies the robot host key normally
This is why debug logs show the target alias host key coming from the cache populated by hmnd-ssh.
A proof VM is sufficient if:
it is enrolled into Backbone
hmnd-backboned is running
hmnd-bastion-tunneld is running
the VM can maintain the device tunnel
the VM runs sshd
If the proof VM does not use the real robot login name, override the SSH user locally.
Build and install the updated hmnd-backboned package on the VM.
Restart hmnd-backboned.service and hmnd-bastion-tunneld.service.
Confirm ssh.bootstrap.json exists.
Publish and deploy the updated Backbone container.
Apply Terraform for the bastion/backbone changes.
Add the ~/.ssh/config stanza locally.
Run:
ssh aw8-orin\n
Expected result:
browser OIDC login if no cached token exists
successful host key verification
shell prompt from the robot VM
If SSH reaches the robot, debug logs will show:
the robot OpenSSH banner, not the bastion
the robot OpenSSH banner, not the bastion
the robot host key matching the cached alias
the robot's offered auth methods
If the auth methods say only publickey, that is a robot-side SSH policy issue, not a bastion issue.
This design intentionally avoids pushing human OIDC auth into the bastion and avoids requiring Teleport or paid SSO features.
It gives:
strong device identity via mTLS
strong human identity via Cognito/OIDC
native SSH ergonomics
minimal operator-side dependencies
simple bastion responsibilities
It also keeps the architecture evolvable.
Later improvements could include:
automatic SSH config installation
better terminal capability handling
stronger robot-local per-user SSH identity
session recording or richer audit
more productized packaging for operator machines
The current SSH system works like this:
Backbone authenticates devices with mTLS
robots fetch device tunnel bootstrap data from Backbone
robots fetch short-lived device tunnel session tokens from Backbone
robots maintain WSS device tunnels to the bastion
humans authenticate to Backbone via Cognito/OIDC
Backbone mints short-lived human proxy session tokens
hmnd-ssh uses that session to act as an OpenSSH ProxyCommand
local OpenSSH talks end-to-end to the robot sshd
the robot remains the final SSH server
There is no shared device bastion key in this design.
no bastion SSH listener required for robot tunnels
A device-side SSH user cert would be a big improvement over the current shared key, but it still leaves an awkward server-side authorization problem:
the device would still be using OpenSSH remote forwarding
the client chooses the remote unix socket path
we would still need a strong way to guarantee that a device can only bind its own socket path and not another device's path
That can be solved, but it pushes us into extra bastion-side complexity such as per-device users, per-device filesystem isolation, or awkward sshd policy tricks.
Since nothing is deployed yet, the cleaner move is to remove that entire class of problem instead of hardening around it.
A custom device tunnel over WSS lets the bastion bind:
device identity
active tunnel connection
target device socket/stream registration
all in one place, with no shared private key and no remote-forward path ambiguity.
The current implementation matches the design above:
Backbone exposes:
GET /v1/devices/{device_id}/ssh/bootstrap
POST /v1/devices/{device_id}/ssh/tunnel-session
POST /v1/ssh/session/{device_id}
hmnd-backboned fetches the bootstrap payload and publishes host key plus tunnel readiness metadata.
hmnd-bastion-tunneld fetches a short-lived device tunnel token using device mTLS, then keeps a WSS device tunnel alive with reconnect/backoff.
The bastion runs one TLS/WSS daemon on 443 that accepts:
human proxy sessions at /v1/proxy
device tunnels at /v1/device-tunnel
The bastion tracks the active device tunnel by device_id and attaches the human byte stream to that live tunnel.
No shared device bastion private key is distributed to robots.
The most important validation points are:
Device bootstrap no longer contains private key material.
Device tunnel sessions are short-lived and device-scoped.
Human proxy sessions are short-lived and human/device-scoped.
Bastion 443 is sufficient for both human and device traffic.
Native ssh aw8-orin still works through ProxyCommand.
it can proxy bytes to local 127.0.0.1:22
device websocket registration by device_id
human websocket rejected if device tunnel absent
channel bridging between human proxy and device tunnel
duplicate/replayed device session tokens rejected
enroll proof VM
start hmnd-backboned
confirm device tunnel is registered on bastion
run ssh aw8-orin
complete OIDC if needed
verify host key
land on the proof VM shell
Because the current shared device tunnel key is not in production use, do this as a single cut:
remove the shared device key path entirely
replace device-side OpenSSH reverse tunneling with the new device tunnel websocket path
keep the human-side hmnd-ssh UX stable
Do not spend time building migration shims for a path we do not intend to keep.
The current shared device bastion key is only acceptable as a temporary proof-of-concept shortcut.
The next implementation should not harden around that key. It should remove it.
The selected next design is:
human side remains hmnd-ssh + OIDC + short-lived Backbone proxy token
device side moves from shared-key OpenSSH reverse tunnel to a Backbone-authenticated WSS device tunnel
bastion becomes a single transport service on 443
no shared device private key remains in the system
Summary: Add OTA release publishing, channel-based desired state, per-device channel assignment, and live deployment status to Backbone.
Owner: Cody Griffin
Date: 04/09/2026
Status: Draft
Backbone currently solves device enrollment, device identity, certificate renewal, and basic presence via the devices and sessions tables. It does not yet solve OTA. We can see that a device exists and when it was last seen, but we cannot publish release artifacts, assign desired versions, resolve what a device should be running, or observe an update in flight.
The first OTA design needs to stay honest to the current Backbone model rather than pretending we already have a richer fleet registry than we do. In this codebase, devices are the deployable targets. robot_id is primarily a grouping key over those devices rather than a separately managed object. For Alpha this means a single robot group contains two OTA targets today: one RTPC device and one ORIN device. That is awkward, but it is workable.
The main mistake to avoid is turning OTA into a giant updates join table with one row per device per release. That model creates write amplification, stale rows, awkward rollbacks, and the wrong ownership boundary between desired state and observed state. The control plane should store intent and let devices report observation. It should not precompute every possible deployment.
Backbone OTA should therefore add three missing control-plane layers on top of the existing identity service:
A release catalog and artifact storage path.
A desired-state model based on channels and assignments.
A live status and execution-history model that makes in-flight OTA behavior visible.
Phase 1 is deliberately narrow. It should be good enough for the current fleet and the current Alpha topology without locking us into a bad long-term model.
Publishing release artifacts via the Backbone API, either from bearer-token clients or from the OIDC/Cognito-protected dashboard.
Storing release bytes in S3 and release metadata in Postgres.
Defining immutable releases per (robot_type, device_type) compatibility pair.
Defining channels and assigning one desired release per (robot_type, device_type) within each channel.
Assigning individual devices to channels.
Resolving desired state for a device over the existing mTLS devices entrypoint.
Extending presence/status so operators can see current version, current OTA state, download progress, throughput, and estimated time remaining.
Recording deployment attempts and outcomes as execution history.
Updating the dashboard so devices are still grouped by robot_id, but each device row shows its current and desired OTA state.
Rollout percentages, canary waves, or staged rollouts.
Robot-level memberships as a first-class data model.
Release bundles or trains that coordinate multiple targets atomically.
Peer-to-peer artifact distribution, torrent-style fanout, or CDN edge distribution.
Delta updates or binary patching.
Device-side install policy beyond a simple reconcile/apply/report loop.
Replacing the existing enrollment or certificate-renewal flows.
Backbone OTA extends the existing registry with a simple desired-state loop.
Operators publish immutable releases, map those releases into channels, and assign devices to channels. Devices ask Backbone what they should be running, download the artifact, verify it locally, apply it, and report progress back through the existing devices plane. Backbone stores the desired state and the observed state separately, which keeps the model tractable.
Backbone already has the right foundational split for Phase 1:
devices is the durable identity inventory.
sessions is the current observed-state heartbeat.
the devices entrypoint is already protected by mTLS.
the public and api entrypoints are already protected for human operators.
That means OTA does not need a second identity system. Devices remain the OTA targets.
Compatibility in Phase 1 should be expressed directly by the existing registry dimensions:
robot_type
device_type
That is enough for the current fleet and is cleaner than introducing a synthetic compatibility key such as alpha_wheelbase:ORIN. A release is compatible with a concrete (robot_type, device_type) pair, and channels point at releases for those same pairs.
Release compatibility pair
The pair (robot_type, device_type) defines which devices a release is intended for. In Phase 1, that pair is the compatibility class used for release resolution.
Release
An immutable artifact plus metadata for one (robot_type, device_type) pair. A release is never \u201cfor a device.\u201d It is for a compatibility class. Once finalized, the object bytes, digest, size, and metadata are immutable. If an image is bad, it is revoked or replaced by a new release, not mutated.
Channel
A named desired-state stream such as prod, staging, canary, or field-test. A channel answers the question: for each (robot_type, device_type) pair, what release should members converge to?
Device channel membership
A Phase 1 assignment from one device to one effective channel. This is intentionally device-level for now because the current fleet is small and the current registry treats devices as the primary managed objects.
Desired-state resolution
A pure lookup from device identity to desired release:
device -> (robot_type, device_type) -> assigned channel -> channel mapping -> desired release
This is the heart of the design. There is no materialized device x release update table.
Deployment attempt
An operational record for a concrete OTA attempt. This is where state transitions such as downloading, awaiting_reboot, verifying, succeeded, and failed belong.
Observed OTA status
The latest live heartbeat from the device, carried in sessions.data. This is the place for current version, progress counters, throughput, ETA, and the currently active attempt ID.
The design should make these three planes explicit.
Desired state is channel membership and channel target mappings. It changes when an operator publishes a release, repoints a channel, or changes an assignment.
Observed state is what the device reports right now: current version, current release ID if known, OTA state, progress, last error, and whether it is currently waiting for a reboot or verification.
Execution history is the durable record of what happened when the device attempted an update. This is the source of truth for audit and troubleshooting. It should not be overloaded with every heartbeat tick.
This split matters because progress updates are frequent, while deployment history should capture meaningful lifecycle boundaries. A device may update its session heartbeat every few seconds during a download. That should not force a permanent row for every minor progress change.
Release publishing
An admin creates a draft release in Backbone, including metadata such as robot_type, device_type, version, expected digest, expected size, and optional notes. Backbone returns a presigned direct-upload path to S3. The operator dashboard uses this from the OIDC-protected browser flow; API clients use the same contract from the token-authenticated API host.
Backbone should not stream multi-gigabyte images through FastAPI. Release bytes should go directly from the browser or CLI to S3. Because Alpha images may be large, the upload path should support multipart uploads rather than assuming a small single PUT.
Once the upload is complete, the operator finalizes the release. Finalization verifies the object exists and that the expected size and checksum match. Only then does the release become eligible for channel mapping.
Channel management
Admins create channels and assign one release per (robot_type, device_type) pair. For Alpha that means a usable channel generally needs two mappings:
one release for (alpha_wheelbase, RTPC)
one release for (alpha_wheelbase, ORIN)
A release does not become active merely by existing. It becomes desired for devices only when a channel points at it.
Assignment
Phase 1 assignments are per device. That is a conscious compromise for the current fleet size. The dashboard should still present a robot-centric workflow, which means it should offer a convenience action to apply the same channel assignment to every device sharing a robot_id. Under the hood, that convenience action simply writes multiple device memberships.
The dashboard should also flag robot groups whose devices are on different channels. This reduces the chance of accidentally splitting a two-target Alpha robot across incompatible desired states.
Each device should stay dumb.
It authenticates with the existing Backbone mTLS identity.
It calls a desired-state endpoint for itself.
It compares desired release versus current release.
If no update is needed, it continues heartbeating current status.
If an update is needed, it downloads the artifact, verifies digest and signature locally, applies it, reboots if needed, verifies success, and reports the outcome.
Backbone should not tell the device how to install the image beyond the release metadata and basic rollout instructions. Device-side policy should remain simple: ask, validate, apply, report.
Operators need to see more than \u201conline\u201d or \u201coffline.\u201d They should be able to tell whether a device is idle, actively downloading, waiting to reboot, or stuck.
Phase 1 should therefore define a typed OTA status contract inside sessions.data. At minimum it should support:
current version
current release ID if known
current OTA state
active deployment attempt ID
desired release ID being applied
artifact bytes total
artifact bytes downloaded
download bytes per second
estimated seconds remaining
whether a reboot is required or pending
last error code and detail
timestamp of the most recent OTA status update
The recommended OTA lifecycle states are:
idle
update_available
downloading
downloaded
installing
awaiting_reboot
rebooting
verifying
succeeded
failed
rolled_back
blocked
superseded
The dashboard should derive freshness from the existing session heartbeat. If a device has not reported recently, OTA status should be shown as stale rather than assumed current. An offline device that last reported downloading should be rendered as something like \u201coffline; last known OTA state downloading.\u201d
This design is intentionally modest.
It does not introduce robot bundles, staged rollout waves, or a full train system yet. It also does not collapse desired state, observed state, and history into one table. That is the right trade for the current service and the current fleet size.
It also leaves room for the obvious future upgrades:
replace device_channel_memberships with robot_channel_memberships
add release bundles once cross-target compatibility becomes harder
add rollout waves and canary percentages
move from direct S3 downloads to CloudFront or another distribution layer
None of those changes require us to revisit the core concepts of releases, channels, assignments, and observed status.
Phase 1 should adopt the following policies explicitly.
Releases are immutable once finalized.
A revoked release stays in the catalog but cannot be newly assigned.
Devices have one effective channel at a time.
Channels should be few; the dashboard should not encourage a channel zoo.
The service does not materialize device x release rows.
A channel mapping may be incomplete. In that case devices without a matching (robot_type, device_type) mapping simply have no desired release.
Rollback is implemented by repointing the channel to an older release.
The dashboard groups by robot_id, but desired state is still resolved per device.
The current Backbone uniqueness assumption of one (robot_id, device_type) pair remains valid in Phase 1.
Phase 1 should reuse the existing devices and sessions tables and add a small set of OTA-specific tables.
devices
Backbone should continue using devices as the durable identity table. No synthetic OTA compatibility column is required in Phase 1. Release compatibility is derived directly from the existing registry fields:
robot_type
device_type
The current uniqueness assumption remains:
one device per (robot_id, device_type)
That is acceptable for Alpha because the two targets are already distinct device types.
sessions
Backbone should continue using sessions as the current-status heartbeat table. The shape of data should become an explicit contract for OTA reporting rather than an unstructured blob.
Table | Purpose | Notes |
|---|---|---|
| Immutable release catalog | One row per artifact version for one |
| Named desired-state streams | Human-facing names like |
| Desired release pointer per channel and compatibility pair | One row per |
| Effective channel assignment per device | One row per device in Phase 1 |
| Execution history for OTA attempts | One row per attempt, updated on major state transitions |
releases
id uuid primary key
robot_type text not null
device_type text not null
version text not null
status text not null (draft, uploading, available, revoked)
artifact_bucket text not null
artifact_key text not null
artifact_size_bytes bigint
artifact_sha256 text
artifact_signature text null
artifact_mime_type text null
upload_session_id text null
metadata jsonb not null default '{}'::jsonb
release_notes text null
created_by_iss text not null
created_by_sub text not null
created_at timestamptz not null
updated_at timestamptz not null
unique (robot_type, device_type, version)
channels
id uuid primary key
name text not null unique
description text null
status text not null (active, paused)
created_by_iss text not null
created_by_sub text not null
created_at timestamptz not null
updated_at timestamptz not null
channel_target_releases
channel_id uuid not null references channels(id)
robot_type text not null
device_type text not null
release_id uuid not null references releases(id)
updated_by_iss text not null
updated_by_sub text not null
updated_at timestamptz not null
primary key (channel_id, robot_type, device_type)
device_channel_memberships
device_id uuid primary key references devices(device_id)
channel_id uuid not null references channels(id)
assigned_by_iss text not null
assigned_by_sub text not null
assigned_at timestamptz not null
This table is intentionally separate from devices even though Phase 1 only supports one channel per device. Identity data and rollout policy should not be collapsed into one row unless there is a strong reason.
deployment_attempts
id uuid primary key
device_id uuid not null references devices(device_id)
channel_id uuid null references channels(id)
release_id uuid not null references releases(id)
from_release_id uuid null references releases(id)
trigger_reason text not null (channel_change, manual, repair, rollback)
state text not null
current_version text null
target_version text null
bytes_total bigint null
bytes_downloaded bigint null
download_bps bigint null
estimated_seconds_remaining integer null
requires_reboot boolean not null default false
last_error_code text null
last_error_detail text null
started_at timestamptz not null
updated_at timestamptz not null
finished_at timestamptz null
This table should capture coarse transitions and final outcomes. It should not be written on every heartbeat tick unless we later decide we need that history.
sessions.dataThe existing sessions table should carry the current observed OTA state. A representative payload is:
{\n "software": {\n "current_version": "2026.04.1",\n "current_release_id": "6a706e58-8f12-4c43-9f30-2ebc63f7a2aa"\n },\n "ota": {\n "attempt_id": "67cc7b27-7e5d-4ff7-b7dc-cce258ba1d55",\n "desired_release_id": "e738bd7b-010f-4b47-b6cb-2a33ad0b4b2b",\n "state": "downloading",\n "bytes_total": 4294967296,\n "bytes_downloaded": 2147483648,\n "download_bps": 15728640,\n "estimated_seconds_remaining": 137,\n "requires_reboot": true,\n "last_error_code": null,\n "last_error_detail": null,\n "updated_at": "2026-04-09T10:15:12Z"\n }\n}\nThis keeps current-state rendering fast and simple for the dashboard while leaving durable attempt history in deployment_attempts.
Backbone should continue using the existing three entrypoints:
public: browser-facing/OIDC path and any anonymous public endpoints
api: bearer-token human API
devices: mTLS device API
The existing auth split should remain:
operators and admins have read access
only admins have write access
devices can act only as themselves over the devices entrypoint
These endpoints should be available on the public and api entrypoints. The browser dashboard will typically use public; automation and CLI tooling will typically use api.
Endpoint | Entrypoint(s) | Auth | Purpose |
|---|---|---|---|
|
| operator/admin | Render grouped robot/device status including OTA fields |
|
| admin | Create a draft release row |
|
| admin | Create S3 multipart upload session and presigned part URLs |
|
| admin | Complete multipart upload |
|
| admin | Validate artifact and mark release available |
|
| operator/admin | List releases |
|
| operator/admin | Get release details |
|
| admin | Create channel |
|
| operator/admin | List channels |
|
| operator/admin | Get channel details including compatibility-pair mappings |
|
| admin | Point a channel mapping at a release |
|
| admin | Assign or replace the effective channel for a device |
|
| admin | Remove assignment so the device has no desired release |
|
| operator/admin | Raw current status feed, now including OTA status in |
The dashboard may offer a robot-level \u201cassign all devices in this robot group to channel X\u201d action, but that should be implemented as multiple per-device writes in Phase 1 rather than a separate first-class robot membership API.
These endpoints should only be exposed on the devices entrypoint and require the existing mTLS device identity.
Endpoint | Entrypoint(s) | Auth | Purpose |
|---|---|---|---|
|
| device mTLS | Resolve desired release for the authenticated device |
|
| device mTLS | Mint a fresh presigned download URL for retries or resume |
|
| device mTLS | Start a deployment attempt |
|
| device mTLS | Transition state or publish coarse attempt progress |
|
| device mTLS | Heartbeat current version and live OTA status |
GET /v1/ota/desired should return enough information for a device to decide whether it needs to act.
Representative shape:
{\n "device_id": "...",\n "robot_id": "123",\n "channel": {\n "id": "...",\n "name": "staging"\n },\n "desired_release": {\n "id": "...",\n "robot_type": "alpha_wheelbase",\n "device_type": "ORIN",\n "version": "2026.04.1",\n "artifact_size_bytes": 4294967296,\n "artifact_sha256": "...",\n "artifact_signature": "...",\n "download_url": "https://...",\n "download_url_expires_at": "2026-04-09T10:30:00Z"\n }\n}\nIf a device has no channel assignment or if its channel has no release for its (robot_type, device_type) pair, desired_release should be null rather than forcing an error.
POST /v1/sessions remains the primary current-state heartbeat. OTA-capable devices should send:
current software version
current release ID if known
live OTA state and progress if an update is in flight
last error fields if the most recent attempt failed
During large downloads, devices should heartbeat frequently enough that the dashboard can render meaningful progress and freshness. A heartbeat every 10 to 30 seconds during active OTA is sufficient for Phase 1.
hmnd_infra/modules/backbone_stack already provisions ECS, Aurora, ALBs, Cognito, PCA, Secrets Manager, and a small S3 bucket for the ALB trust store. OTA requires real artifact storage. That should be added to the existing module rather than created ad hoc elsewhere.
Resource | Purpose | Notes |
|---|---|---|
Private S3 bucket for OTA artifacts | Store uploaded release bytes | Separate from the existing trust-store bucket |
S3 bucket versioning | Preserve immutable artifacts and support recovery | Enabled by default |
S3 public-access block | Keep artifacts private | Required |
Server-side encryption | Encrypt artifact bytes at rest | SSE-KMS preferred |
Lifecycle rules | Abort incomplete multipart uploads and optionally clean up abandoned drafts | Required |
Bucket CORS | Allow direct browser uploads from the OIDC dashboard origin | Include dev localhost origin as needed |
IAM policy additions on Backbone ECS task role | Allow multipart upload orchestration, object metadata reads, and presigned GET/PUT generation | Minimal least-privilege scope to the OTA bucket |
Configuration outputs/secret values | Bucket name, region, upload TTLs, download TTLs, allowed origins | Passed to Backbone service via config secret or env |
A simple immutable object layout is sufficient:
releases/<release_id>/artifact
releases/<release_id>/manifest.json
The database remains the source of truth for metadata. S3 stores bytes and optional manifest copies.
Backbone needs enough S3 permission to orchestrate direct uploads and signed downloads, but nothing broader.
Representative permissions:
s3:PutObject
s3:GetObject
s3:AbortMultipartUpload
s3:ListBucketMultipartUploads
s3:ListMultipartUploadParts
s3:GetObjectAttributes
If the bucket uses a customer-managed KMS key, the task role also needs the corresponding kms:Encrypt, kms:Decrypt, and kms:GenerateDataKey grants scoped to that key.
No new ALB or hostname is required. The existing public, api, and devices entrypoints are sufficient.
No new ECS service is required. OTA lives inside the existing Backbone service.
No CloudFront distribution is required yet. Direct S3 downloads are acceptable for the current fleet size.
CloudFront may become desirable later if artifact egress grows materially or if we want a different download control layer, but it should not be a Phase 1 dependency.
The existing Backbone dashboard already groups device rows by robot_id. OTA should extend that view rather than replace it.
Each device row should show:
device type
channel
desired version
current version
OTA state
progress percentage or bytes downloaded
download speed
estimated time remaining
last seen freshness
certificate status
Each robot group should additionally show a warning if its devices are assigned to different channels. That is the main operator footgun in the Phase 1 per-device assignment model.
If Phase 1 works as expected, the next obvious upgrades are:
robot_channel_memberships instead of device_channel_memberships
release bundles or trains so multi-target robots converge to a tested set
rollout waves and percentages
artifact distribution through CloudFront or another cache layer
typed device_status tables if sessions.data becomes too loose for reporting needs
The important point is that none of those require revisiting the core split between release catalog, desired state, observed state, and execution history.
Backbone OTA should treat the existing devices rows as the OTA targets, treat robot_id as the grouping key, use robot_type + device_type as the compatibility key, store artifacts in S3, store desired state in channels and device memberships, and store live progress separately from deployment history.
That gives us a practical Phase 1 system for the current fleet:
operators can upload and publish real images
channels express desired versions without a giant update table
devices can reconcile over the existing mTLS plane
the dashboard can show whether a device is downloading, waiting to reboot, verifying, or stuck
It is small enough to build now, but it does not paint us into a corner when we later move from per-device assignments to robot-level policy and multi-target release bundles.
" }, { "id": "1839235107", "title": "OSS Audit: hmnd_wholebody", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1839235107", "last_modified": "2026-04-08T10:22:33.997Z", "created_at": "2026-04-08T09:51:33.335Z", "body_html": "Risk Level: LOW (revised from HIGH)
Key Concerns: PyQt5 (GPL-3.0) is declared but only used in a dev-only visualization script \u2014 not in any distributed package
hmnd_wholebody has a relatively small dependency footprint (21 packages). Initial analysis flagged this module as HIGH risk due to PyQt5 (GPL-3.0), but actual code tracing reveals that PyQt5 is only imported in a single developer visualization script and is not part of any installed package or production pipeline.
The only import of PyQt5/PyQtGraph in the entire module is:
hmnd_wholebody/scripts/sys-id/presentation/simple_visualiser.py:
import pyqtgraph as pg\nfrom PyQt5 import QtCore, QtWidgets\n
This is a developer-only visualization tool used for system identification presentation/debugging. Key facts:
Not in any setup.py \u2014 this script is not part of any installable Python package
Not in any pyproject.toml entry point \u2014 not registered as a console script or GUI script
Located in scripts/ \u2014 a directory for ad-hoc developer tools, not production code
Not imported by any other module \u2014 no other file in hmnd_wholebody imports this script
Not in any CI/CD pipeline \u2014 no test or build step references this script
Verdict: PyQt5 is a dev-time convenience dependency with zero distribution risk. The GPL-3.0 obligation only triggers upon distribution of a derivative work, which does not happen here. Replacing with PySide6 (LGPL-3.0) is still recommended for cleanliness but is low priority.
Source | Name | Version | License | Actual Usage | Notes |
|---|---|---|---|---|---|
pixi (pypi) | pyqt5 | =5.15.11,<6 | GPL-3.0 | Dev-only script ( | Not in any setup.py or distributed package. No distribution risk. |
pixi (conda) | libstdcxx-ng | GPL-3.0 (w/ Runtime Exception) | Runtime library | Exception permits proprietary linking \u2014 safe. |
Name | Version | License | Category |
|---|---|---|---|
seaborn | 0.13.2 | BSD | Visualization |
pandas | 2.2.3 | BSD | Data |
tqdm | 4.67.1 | MIT | Progress |
h5py | 3.12.1 | BSD | HDF5 |
wandb | 0.19.11 | MIT | Experiment Tracking |
plotly | =6.5.2,<7 | MIT | Visualization |
cmaes | =0.12.0,<0.13 | MIT | Optimization |
pyqtgraph | =0.14.0,<0.15 | MIT | Plotting (note: MIT-licensed itself, but depends on PyQt5 which is GPL \u2014 however, only used in dev script) |
Name | Version | License | Category |
|---|---|---|---|
python | =3.11,<3.12 | PSF | Runtime |
pip | =24.0 | MIT | Package Manager |
setuptools | =68,<81 | MIT | Build |
wheel | MIT | Build | |
numpy | 1.26.4 | BSD | Numerical |
prettytable | BSD | Display | |
packaging | <24 | Apache-2.0/BSD | Packaging |
msgpack-python | =1.0.0,<2.0.0 | Apache-2.0 | Serialization |
platformdirs | MIT | Paths | |
hdf5 | BSD | Data | |
cmake | =3.22.0,<3.27 | BSD | Build |
xorg-libsm | =1.2.6,<2 | MIT | Display |
xorg-libxt | =1.3.1,<2 | MIT | Display |
libglu | =9.0.3,<10 | SGI-B-2.0 | Graphics |
Low priority: Replace PyQt5 with PySide6 in pixi.toml for cleanliness. PySide6 provides identical Qt Python bindings under LGPL-3.0 instead of GPL-3.0. This is a near-drop-in replacement. However, since this dependency is only used in a dev-only script that is never distributed, this is not urgent and carries no compliance risk as-is.
Risk Level: LOW (revised from HIGH)
Key Concerns: ffmpeg/x264/x265 present but used safely (PyAV is training-only, CLI is subprocess). All other RED items are build tools with GCC Runtime Exception.
hmnd_training has 571 total packages resolved in pixi.lock, plus bundled third-party code (lerobot, libero, robocasa, simplerenv, xdof_sdk). The training environment pulls in a large transitive closure of system libraries, many of which carry LGPL licenses. Initial analysis flagged this module as HIGH risk due to the GPL video codec chain, but actual code tracing reveals the risk is low because training code is not distributed externally.
1. PyAV in lerobot (linked, but training-only):
hmnd_training/third_party/lerobot/src/lerobot/datasets/video_utils.py contains:
import av\n
PyAV is a Python wrapper that uses C bindings to libavcodec, libavformat, etc. This is a library linkage (not subprocess), which technically triggers GPL copyleft. However, this code runs only in training environments and is never distributed to customers or third parties. The GPL obligation only arises upon distribution.
2. FFmpeg CLI in data conversion (safe \u2014 subprocess):
hmnd_training/tools/data/agibot/scripts/agibot_hdf5_to_lerobot_v3.py uses:
subprocess.run(["ffmpeg", ...])\n
This invokes ffmpeg as an external CLI process. Subprocess invocation does not create a derivative work under GPL. This usage is safe regardless of ffmpeg's license.
3. pixi.toml declaration:
ffmpeg = ">=5.1" is declared with the comment # Required by rerun-sdk for video decoding. The rerun-sdk (MIT licensed) uses ffmpeg for visualization during training. Again, training-only.
readline (GPL-3.0) appears in pixi.lock as a transitive dependency of IPython. However, readline is never directly imported in any hmnd_training code. It is only active when a developer uses the IPython REPL interactively. This is not a distribution concern.
All build tools (gcc 14.3.0, gxx 14.3.0, make 4.4.1, binutils 2.45.1, etc.) are GPL-3.0 but are build-time only. The GCC Runtime Library Exception explicitly permits linking the resulting compiled code into proprietary software. The runtime libraries (libgcc, libstdc++, libgfortran, libgomp) all carry this exception.
Source | Name | Version | License | Actual Usage | Notes |
|---|---|---|---|---|---|
pixi.lock | ffmpeg | 7.1.1 | GPL-2.0+ / LGPL-2.1+ | PyAV (training-only) + subprocess (safe) | Revised: Not distributed. Low risk. |
pixi.lock | libx264 | 3.0.2 | GPL-2.0+ | Transitive via ffmpeg | Same \u2014 training-only. |
pixi.lock | libx265 / x265 | 3.5 | GPL-2.0+ | Transitive via ffmpeg | Same \u2014 training-only. |
pixi.lock | readline | 8.3 | GPL-3.0+ | Never imported \u2014 transitive via IPython | Not a distribution risk. |
pixi.lock | dbus | 1.16.2 | GPL-2.0+ | System library | Inter-process communication. |
pixi.lock | gdbm | 1.18 | GPL-3.0+ | System library | Key-value database. |
pixi.lock | elfutils | 0.193 | GPL-3.0+ | Dev tool | ELF utilities. |
pixi.lock | hicolor-icon-theme | 0.17 | GPL-2.0+ | Desktop theme | Icon theme data. |
pixi.lock | lz4-c | 1.10.0 | GPL-2.0+ / BSD-2 | Compression | Dual licensed \u2014 BSD option available. |
pixi.lock | lzo | 2.10 | GPL-2.0+ | Compression | Lossless compression. |
pixi.lock | zstd | 1.5.7 | GPL-2.0+ | Compression | Dual-licensed BSD in some distributions. |
pixi.lock | xz-gpl-tools | 5.8.2 | GPL-2.0+ | CLI tool | XZ compression tools. |
pixi.lock | shellcheck | 0.10.0 | GPL-3.0+ | Dev tool only | Shell linter. |
pixi.lock | gcc / gxx (14.3.0) | various | GPL-3.0+ | Build-time only | GCC Runtime Exception applies. |
pixi.lock | binutils / ld_impl | 2.45.1 | GPL-3.0+ | Build-time only | Build tools. |
pixi.lock | make | 4.4.1 | GPL-3.0+ | Build-time only | Build system. |
pixi.lock | kernel-headers | 5.14.0 | GPL-2.0+ | Build-time only | Linux headers. |
pixi.lock | sysroot_linux_64 | 2.34 | GPL-2.0+ | Build environment | System root. |
pixi.lock | libgcc / libgcc-ng | 15.2.0 | GPL-3.0+ w/ Runtime Exception | Runtime library | Exception permits proprietary linking \u2014 safe. |
pixi.lock | libstdcxx / libstdcxx-ng | 15.2.0 | GPL-3.0+ w/ Runtime Exception | Runtime library | Exception permits proprietary linking \u2014 safe. |
pixi.lock | libgfortran / libgfortran5 / libgfortran-ng | 15.2.0 | GPL-3.0+ w/ Runtime Exception | Runtime library | Used by numpy/scipy. Exception applies. |
pixi.lock | libgomp | 15.2.0 | GPL-3.0+ w/ Runtime Exception | Runtime library | OpenMP runtime. Exception applies. |
pixi.lock | libsanitizer | 15.2.0 | GPL-3.0+ w/ Runtime Exception | Dev tool | Address/Thread sanitizers. |
pixi.lock | libgettext / libgettextpo | 0.25.1 | GPL-3.0+ | System library | GNU gettext. |
pixi.lock | libidn2 | 2.3.8 | GPL-2.0+ | System library | Internationalized domain names. |
pixi.lock | libunwind | 1.8.3 | GPL-2.0+ | System library | Stack unwinding. |
pixi.lock | liburing | 2.12 | GPL-2.0+ | System library | io_uring interface. |
pixi.lock | nettle | 3.10.1 | LGPL-3.0+ or GPL-2.0+ | Crypto library | Check which license variant applies. |
pixi.lock | freetype / libfreetype | 2.14.1 | GPL-2.0+ or FTL | Font library | FreeType License (FTL) is permissive option. |
Source | Name | Version | License | Notes |
|---|---|---|---|---|
pixi.lock | qt6-main | 6.9.2 | LGPL-3.0 or LGPL-2.0+ | Qt GUI framework. Not linked into distributed binaries. |
pixi.lock | gtk3 | 3.24.43 | LGPL-2.0+ | GTK graphical toolkit \u2014 transitive dependency. |
pixi.lock | pango | 1.56.4 | LGPL-2.0+ | Text layout and rendering. |
pixi.lock | librsvg | 2.58.4 | LGPL-2.1+ | SVG rendering. |
pixi.lock | gdk-pixbuf | 2.44.5 | LGPL-2.0+ | Image handling. |
pixi.lock | libglib | 2.86.4 | LGPL-2.0+ or MPL-1.1 | Core GLib library. |
pixi.lock | glib-tools | 2.86.4 | LGPL-2.0+ or MPL-1.1 | Core library tools. |
pixi.lock | graphite2 | 1.3.14 | LGPL-2.0+ or MPL-1.1 | Font shaping. |
pixi.lock | gmp | 6.3.0 | LGPL-3.0+ | Multiple precision arithmetic. |
pixi.lock | gts | 0.7.6 | LGPL-2.0+ | Geometric algorithms. |
pixi.lock | gnutls | 3.8.11 | LGPL-2.1+ (or GPL-3.0+) | TLS/SSL \u2014 verify LGPL variant is used. |
pixi.lock | lame | 3.100 | LGPL-2.0+ | MP3 encoding. |
pixi.lock | libacl | 2.3.2 | LGPL-2.1+ | POSIX ACL library. |
pixi.lock | libblasfeo | 0.1.4.2 | LGPL-2.1+ | BLAS/LAPACK library. |
pixi.lock | libfatrop | 0.0.4 | LGPL-2.1+ | Fast trajectory optimization. |
pixi.lock | libiconv | 1.18 | LGPL-2.1+ | Character encoding. |
pixi.lock | libmicrohttpd | 1.0.2 | LGPL-2.1+ | HTTP server library. |
pixi.lock | libnsl | 2.0.1 | LGPL-2.1+ | Network services. |
pixi.lock | libntlm | 1.8 | LGPL-2.1+ | NTLM authentication. |
pixi.lock | libsndfile | 1.2.2 | LGPL-2.1+ | Sound file library. |
pixi.lock | libsystemd0 | 259.1 | LGPL-2.1+ | Systemd library. |
pixi.lock | libtasn1 | 4.21.0 | LGPL-2.1+ | ASN.1 library. |
pixi.lock | libudev1 | 259.1 | LGPL-2.1+ | udev device library. |
pixi.lock | libunistring | 0.9.10 | LGPL-3.0+ | Unicode string library. |
pixi.lock | libusb | 1.0.29 | LGPL-2.1+ | USB device library. |
pixi.lock | libxcrypt | 4.4.36 | LGPL-2.1+ | Password hashing. |
pixi.lock | mpg123 | 1.32.9 | LGPL-2.1+ | MPEG audio decoding. |
pixi.lock | pulseaudio-client | 17.0 | LGPL-2.1+ | PulseAudio client. |
pixi.lock | pygame | 2.5.2 | LGPL-2.1+ | Game development. |
pixi.lock | pynput | 1.7.7 | LGPL-3.0 | Keyboard/mouse control. |
pixi.lock | num2words | 0.5.14 | LGPL-2.1+ | Number-to-words conversion. |
pixi.lock | humanfriendly | 10.0 | LGPL-2.1+ | Human-readable formatting. |
pixi.lock | keyutils | 1.6.3 | LGPL-2.1+ | Kernel key utilities. |
pixi.lock | fribidi | 1.0.16 | LGPL-2.1+ | Bidirectional text. |
pixi.lock | empy | 3.3.4 | LGPL-2.0+ | Template language. |
pixi.lock | eiquadprog | 1.3.1 | LGPL-3.0 | Quadratic programming solver. |
pixi.lock | eigen | 3.4.0 | MPL-2.0 | Linear algebra (file-level copyleft only). |
pixi.lock | jsonlines | 4.0.0 | MPL-2.0 | JSON Lines format. |
pixi.lock | tqdm | 4.66.0 | MPL-2.0 / MIT | Progress bar (dual licensed). |
pixi.lock | pytest-rerunfailures | 16.1 | MPL-2.0 | Test plugin. |
Source | Name | Version | Notes |
|---|---|---|---|
pixi.lock | icu | 78.2 | International Components for Unicode \u2014 likely Unicode License (permissive). |
BUNDLED | robocasa | 0.2.0 | Kitchen simulation benchmark. No license in setup.py. |
BUNDLED | xdof_sdk | custom | X-DOF hand SDK. No LICENSE file. |
Name | Location | License | Risk |
|---|---|---|---|
lerobot |
| Apache-2.0 | GREEN |
libero |
| MIT | GREEN |
robocasa |
| UNKNOWN \u2014 no license in setup.py | GRAY |
simplerenv |
| MIT | GREEN |
xdof_sdk |
| UNKNOWN \u2014 no LICENSE file | GRAY |
Name | Version | License | Category |
|---|---|---|---|
python | 3.11.14 | PSF | Runtime |
numpy | 1.26.4 | BSD-3 | Numerical |
scipy | 1.16.3 | BSD-3 | Scientific Computing |
torch | 2.2.1 | BSD-3 | Deep Learning |
torchvision | 0.21.0 | BSD-3 | Computer Vision |
transformers | 4.21.1 | Apache-2.0 | NLP |
diffusers | 0.35.2 | Apache-2.0 | Diffusion Models |
accelerate | 1.12.0 | Apache-2.0 | Distributed Training |
datasets | 4.0.0 | Apache-2.0 | Data |
huggingface-hub | 0.34.2 | Apache-2.0 | ML Hub |
safetensors | 0.4.3 | Apache-2.0 | Serialization |
peft | 0.18.0 | Apache-2.0 | Fine-tuning |
opencv | 4.11.0 | Apache-2.0 | Computer Vision |
pinocchio | 3.7.0 | BSD-2 | Robot Dynamics |
mujoco | 3.3.2 | Apache-2.0 | Physics |
gymnasium | 1.1.1 | MIT | RL Environments |
pandas | 2.2.2 | BSD-3 | Data |
h5py | 3.15.1 | BSD-3 | HDF5 |
matplotlib | 3.10.3 | PSF | Plotting |
wandb | 0.21.1 | MIT | Experiment Tracking |
rerun-sdk | 0.28.2 | MIT | Visualization |
flash-attn | 2.5.9 | BSD-3 | ML Optimization |
lerobot | 0.4.3 | Apache-2.0 | Robotics |
robomimic | 0.2.0 | MIT | Robot Learning |
robosuite | 1.4.0 | MIT | Robot Simulation |
hydra-core | 1.2.0 | Apache-2.0 | Config |
pytest | 8.4.2 | MIT | Testing |
pre-commit | 4.5.1 | MIT | Dev Tool |
ipython | 9.10.0 | BSD-3 | Development |
openssl | 3.6.1 | Apache-2.0 | Crypto |
protobuf | 6.31.1 | BSD-3 | Serialization |
grpcio | 1.60.0 | Apache-2.0 | RPC |
pillow | 12.1.1 | HPND | Image Processing |
numba | 0.56.4 | BSD-3 | JIT |
sqlite | 3.51.2 | Public Domain | Database |
zlib | 1.3.1 | Zlib | Compression |
libxml2 | 2.13.9 | MIT | XML |
libcurl | 8.18.0 | MIT | HTTP |
harfbuzz | 12.2.0 | MIT/ISC | Font |
fontconfig | 2.17.1 | MIT | Font Config |
ncurses | 6.5 | MIT | Terminal |
libarchive | 3.8.1 | BSD-2 | Archive |
opengl | 1.7.0 | MIT | Graphics |
vulkan-loader | 1.4.341.0 | Apache-2.0 | Graphics |
glfw | 3.4 | Zlib | Window |
cmake | 4.2.3 | BSD-3 | Build Tool |
ninja | 1.13.2 | Apache-2.0 | Build Tool |
s5cmd | 2.3.0 | MIT | S3 Tool |
Risk Level: HIGH (confirmed by code analysis)
Key Concerns: EtherCAT (GPL-2.0) ships in OS image, FFmpeg direct C linkage in quest_stream_viewer, spidev (GPL-2.0) on robot hardware
hmnd_robot is the largest module with 950+ total dependencies across conda (pixi), PyPI, Rust (Cargo), npm, and bundled third-party source code. It contains the most significant licensing risks in the entire repository due to the bundled EtherCAT driver (GPL-2.0) and the inclusion of GPL video codecs.
The following traces each problematic dependency to its actual usage in the codebase, determining real vs theoretical risk.
Full linkage chain:
Source: core/third_party/ethercat/etherlab-ethercat/ \u2014 bundled GPL-2.0 source code with COPYING file confirming GPL-2.0
Build: core/third_party/ethercat/CMakeLists.txt compiles this into libethercat.so (shared library)
ROS integration: ros/third_party/icube-ethercat-ros2/ethercat_interface/CMakeLists.txt contains find_library(ETHERCAT_LIB ethercat) and target_link_libraries(${PROJECT_NAME} ${ETHERCAT_LIB}) \u2014 links the GPL library
Driver: ros/hardware/alpha_universal_driver/CMakeLists.txt links to ethercat_interface, and src/ethercat_bus_driver.cpp includes #include "ethercat_interface/ec_master.hpp"
Packaging: targets/congatech_rtpc/BUILD packages hmnd-ethercat and hmnd-ethercat-modules as Debian package dependencies
Distribution: Installed in Congatech RTPC and Orin AGX OS images (squashfs and repo-installer targets)
Verdict: This is a hard GPL-2.0 linkage chain from source code through shared library, through ROS node, into the shipped robot OS image. This is the single highest-priority licensing issue.
Direct C library linkage in core/hmndlib_teleoperation/meta_quest_openxr/examples/quest_stream_viewer.cpp:
extern "C" {\n#include <libavcodec/avcodec.h>\n#include <libavutil/avutil.h>\n#include <libswscale/swscale.h>\n}\nAPI calls: avcodec_send_packet(), avcodec_receive_frame(), avcodec_find_decoder(AV_CODEC_ID_H264)
CMake linkage: pkg_check_modules(FFMPEG QUIET libavcodec libavutil libswscale) \u2192 target_link_libraries(${QUEST_VIEWER} PRIVATE ${FFMPEG_LIBRARIES})
Verdict: This binary directly links ffmpeg C libraries. Since the conda-forge ffmpeg is built with GPL codecs (x264/x265), this binary becomes a GPL derivative. Must either refactor to subprocess invocation, build an LGPL-only ffmpeg, or replace with a permissive codec.
Usage in ros/hardware/alpha_hri/hri_control/.../led_strip_spi.py:
from spidev import SpiDev\n
Used for SPI bus communication to WS2812 LED strips on the robot. Packaged as ROS2 node led_strip_node. This runs on the robot hardware at runtime.
Verdict: GPL-2.0 Python module imported at runtime on robot hardware. Whether this constitutes "distribution" under GPL depends on how the robot is delivered to customers. If the OS image is distributed, this is a GPL compliance issue.
Usage in:
ros/hardware/alpha_hri/hri_control/.../head_display.py \u2014 production head display rendering
ros/hardware/alpha_hri/hri_control/.../icon_loader.py \u2014 pygame.image.load(), pygame.surfarray.array3d()
Verdict: LGPL (weak copyleft). Used in production on the robot, but dynamic linking is standard for Python packages. Safe as long as pygame itself is not modified. No action required.
Declared in pixi.toml but never imported anywhere in the codebase. Zero matches for import pynput or from pynput across the entire repository.
Verdict: Dead dependency. Should be removed from pixi.toml for cleanliness.
Source | Name | Version | License | Category | Actual Usage | Notes |
|---|---|---|---|---|---|---|
BUNDLED | EtherLab EtherCAT | custom | GPL-2.0 + LGPL-2.1 | Hardware | LINKED \u2192 shipped in OS image | See linkage chain above. PRIMARY RISK. |
PIXI-CONDA | ffmpeg | =7.1.0,<8 | GPL-2.0 | Video | LINKED in quest_stream_viewer.cpp | Direct C library linkage. See analysis above. |
PIXI-CONDA | x264 | GPL-2.0 | Video | Transitive via ffmpeg | Linked into ffmpeg build. Risk depends on ffmpeg usage. | |
PIXI-PYPI | spidev | =3.7,<4 | GPL-2.0 | Hardware | IMPORTED at runtime | SPI LED strip control on robot hardware. |
PIXI-CONDA | gcc | =11,<12 | GPL-3.0 | Build Tool | Build-time only | Compiler \u2014 GCC Runtime Exception applies. |
PIXI-CONDA | gxx | =11,<12 | GPL-3.0 | Build Tool | Build-time only | Compiler \u2014 not linked into output. |
PIXI-CONDA | make | =4.4.1,<5 | GPL-3.0 | Build Tool | Build-time only | Build tool \u2014 not linked. |
PIXI-CONDA | binutils | =11,<12 | GPL-3.0 | Build Tool | Build-time only | Assembler/linker tools \u2014 not linked. |
PIXI-CONDA | libstdcxx | =15.1.0,<16 | GPL-3.0 (w/ Runtime Exception) | Core Library | Runtime library | Exception permits proprietary linking \u2014 safe. |
PIXI-CONDA | libcap | =2.71,<3 | GPL-2.0 | Core Library | System library | Linux capabilities library. |
PIXI-CONDA | kernel-headers | =5.14.0,<6 | GPL-2.0 | Linux | Build-time only | Development headers. |
PIXI-CONDA | git | GPL-2.0 | VCS | Tool only | Version control \u2014 not linked. | |
PIXI-CONDA | shellcheck | =0.10.0,<0.11 | GPL-3.0 | Code Quality | Dev tool only | Shell linter \u2014 not linked. |
Source | Name | Version | License | Category | Actual Usage | Notes |
|---|---|---|---|---|---|---|
PIXI-CONDA | zeromq | =4.3.5,<5 | LGPL-3.0 | Message Queue | Dynamic linking | Safe if dynamically linked. |
PIXI-CONDA | nlopt | =2.10.0,<3 | LGPL-2.1 | Optimization | Dynamic linking | Verify linking method. |
PIXI-CONDA | libusb | =1.0.29,<2 | LGPL-2.1 | Hardware | Dynamic linking | Typically dynamic. |
PIXI-CONDA | libudev | =257.4,<258 | LGPL-2.1 | Hardware | System library | Device enumeration. |
PIXI-CONDA | libsystemd | =257,<258 | LGPL-2.1 | Core Library | System library | systemd library. |
PIXI-CONDA | mesalib | MIT/LGPL | Graphics | Dynamic linking | OpenGL implementation. | |
PIXI-CONDA | eigen | =3.4.0,<4 | MPL-2.0 | Math | Header-only | File-level copyleft only. |
PIXI-PYPI | ruckig | =0.15.3,<0.16 | LGPL-3.0 | Control | Dynamic linking | Trajectory generation. Also bundled. |
PIXI-PYPI | pynput | LGPL-3.0 | Input | Never imported | Dead dependency \u2014 remove from pixi.toml. | |
PIXI-PYPI | pygame | =2.6.1,<3 | LGPL-2.1 | Graphics | Production use | Head display + icon loading on robot. Safe (LGPL, dynamic). |
Source | Name | Location | Notes |
|---|---|---|---|
BUNDLED | nrobotics |
| ARM/X86 binary distributions. No LICENSE file. |
BUNDLED | oculus_reader |
| Meta Quest tracking reader. Apache-2.0 per README but verify. |
BUNDLED | BehaviorTree.CPP |
| MIT \u2014 verified via LICENSE file. |
BUNDLED | BehaviorTree.ROS2 |
| Apache-2.0 \u2014 verified. |
BUNDLED | manus_ros_drivers |
| No LICENSE file found. |
BUNDLED | nr_tofcam_ros_dist |
| No LICENSE file found. |
BUNDLED | neo_* (common2, gz_worlds, msgs2, relayboard_v3, srvs2) |
| Neobotix platform libs. neo_gz_worlds is Apache-2.0; others need verification. |
BUNDLED | rox, rox_argo_kinematics |
| rox_argo_kinematics is BSD-3 (Neobotix). rox core has no LICENSE. |
BUNDLED | ros2_robotiq_gripper |
| BSD-3-Clause (PickNik Robotics) \u2014 verified. |
BUNDLED | ros2_xhand |
| No LICENSE file found. |
BUNDLED | serial |
| No LICENSE file found. |
BUNDLED | icube-ethercat-ros2 |
| Apache-2.0 \u2014 verified. |
BUNDLED | placo |
| MIT (Rhoban Team) \u2014 verified. |
BUNDLED | time_optimal_trajectory_generation_py |
| Contains pybind11 (BSD-3). Root package license unclear. |
Name | Version | License | Category |
|---|---|---|---|
cmake | =3.30.5,<4 | BSD-3 | Build Tool |
clangxx | =19.1.7,<20 | Apache-2.0 | Build Tool |
ninja | =1.12.1,<2 | Apache-2.0 | Build Tool |
lld | =19.1.2,<20 | Apache-2.0 | Linker |
python | =3.11.11,<3.12 | PSF | Runtime |
pybind11 | =2.13.6,<3 | BSD-3 | Binding |
libboost / libboost-devel | =1.86.0,<2 | BSL-1.0 | Core Library |
spdlog | =1.14.0,<1.15 | MIT | Logging |
yaml-cpp | =0.8.0,<0.9 | MIT | Config |
aws-sdk-cpp | =1.11,<2 | Apache-2.0 | AWS |
prometheus-cpp | =1.3.0,<2 | Apache-2.0 | Monitoring |
pinocchio | =3.4.0,<3.5.0 | BSD-2 | Robot Dynamics |
ompl | =1.6.0,<2 | BSD-3 | Motion Planning |
opencv | =4.10.0,<5 | BSD-3 | Computer Vision |
ceres-solver | =2.2.0,<3 | BSD-3 | Optimization |
glog | =0.7.1,<0.8 | BSD-3 | Logging |
gtest / gmock | =1.15.2,<2 | BSD-3 | Testing |
flatbuffers | =24.3.0,<25 | Apache-2.0 | Serialization |
libopentelemetry-cpp | =1.18.0,<2 | Apache-2.0 | Tracing |
ros-humble-desktop | =0.10.0,<0.11 | Apache-2.0 | ROS 2 |
ros-humble-navigation2 | Apache-2.0 | ROS 2 Navigation | |
ros-humble-slam-toolbox | BSD-3 | SLAM | |
ros-humble-foxglove-bridge | =0.8.2,<0.9 | Apache-2.0 | Visualization |
h5py | =3.13.0,<4 | BSD-3 | Data |
postgresql | PostgreSQL License | Database | |
rust | =1.90.0,<1.91 | MIT/Apache-2.0 | Language |
nodejs | =24.10.0,<24.11 | MIT | JS Runtime |
glfw | Zlib | Graphics |
Name | Version | License | Category |
|---|---|---|---|
grpcio / grpcio-tools | =1.78.0 | Apache-2.0 | RPC |
protobuf | =6.33.5 | BSD-3 | Serialization |
bleak | =0.22 | MIT | Bluetooth |
dbus-next | =0.2.3 | Apache-2.0 | IPC |
pyserial | =3.5 | BSD-3 | Serial |
websockets | =15.0.1 | BSD-3 | Networking |
fastapi | =0.68.0 | MIT | Web Framework |
uvicorn | =0.15.0 | BSD-3 | Web Server |
pydantic | =1.8.2 | MIT | Validation |
hydra-core | =1.3.2 | Apache-2.0 | Config |
mujoco | =3.3.2 | Apache-2.0 | Simulation |
open3d | <=0.19.0 | MIT | 3D Processing |
imageio | =2.37.0 | BSD-2 | Image I/O |
requests | =2.32.4 | Apache-2.0 | HTTP |
openai | =1.52.0 | MIT | AI |
dvc | *[s3] | Apache-2.0 | Data Versioning |
jupyterlab | =4.4.5 | BSD-3 | Notebook |
rich | MIT | Terminal | |
tqdm | MIT/MPL | Progress | |
trimesh | MIT | Geometry | |
flask | =3.1.0 | BSD-3 | Web Framework |
opentelemetry-api/sdk/exporter | =1.20.0 | Apache-2.0 | Tracing |
Name | Version | License | Category |
|---|---|---|---|
anyhow | 1.0.99 | MIT/Apache-2.0 | Error Handling |
axum | 0.8.4 | MIT | Web Framework |
chrono | 0.4.42 | MIT/Apache-2.0 | Time |
clap | 4.5.47 | MIT/Apache-2.0 | CLI |
git2 | 0.20.2 | MIT/Apache-2.0 | VCS |
jsonschema | 0.33.0 | MIT/Apache-2.0 | Validation |
nix | 0.30.1 | MIT | System |
serde / serde_json | 1.0.225 | MIT/Apache-2.0 | Serialization |
tempfile | 3.22.0 | MIT/Apache-2.0 | Filesystem |
thiserror | 2.0.16 | MIT/Apache-2.0 | Error Handling |
tokio / tokio-util | 1.47.1 | MIT | Async Runtime |
tracing / tracing-subscriber | 0.1.41 | MIT | Tracing |
utoipa / utoipa-axum / utoipa-swagger-ui | 5.4+ | MIT | API Docs |
uefi | 0.34 | MIT | Bootloader |
log | 0.4 | MIT/Apache-2.0 | Logging |
libfuzzer-sys | 0.4 | MIT | Testing |
Name | Version | License | Category |
|---|---|---|---|
react / react-dom | ^19.0.0 | MIT | Framework |
next | ^15.0.3 | MIT | Framework |
@radix-ui/* | various | MIT | UI Components |
@tanstack/react-query | ^5.59.14 | MIT | Data Fetching |
zustand | ^4.5.5 | MIT | State Management |
zod | ^3.23.8 | MIT | Validation |
tailwindcss | ^3.4.14 | MIT | CSS |
lucide-react | ^0.462.0 | ISC | Icons |
typescript | ^5.6.3 | Apache-2.0 | Language |
vitest | ^2.1.4 | MIT | Testing |
eslint | ^9.12.0 | MIT | Linting |
Date: April 8, 2026
Scope: All direct, transitive, and bundled third-party dependencies across Rust (Cargo), Python (pixi/uv/pip), Node.js (npm), and Bazel ecosystems
Method: Automated scan of all pixi.toml, pixi.lock, uv.lock, pyproject.toml, requirements.txt, package.json, Cargo.toml, Cargo.lock, MODULE.bazel, and third_party/ directories
The following analysis traces each RED dependency to its actual usage in code \u2014 determining whether it is compiled/linked into shipped binaries, used only at dev/build time, or declared but never actually imported.
Component | License | Shipped in Binary? | Actual Usage | Real Risk |
|---|---|---|---|---|
EtherLab EtherCAT | GPL-2.0 | YES \u2014 robot OS image | Compiled to | CRITICAL \u2014 hard GPL linkage chain into shipped robot binaries. Must resolve before commercial distribution. |
FFmpeg (quest_stream_viewer.cpp) | GPL-2.0 | YES \u2014 compiled binary | Direct C library linkage: | HIGH \u2014 direct library linkage in |
FFmpeg/PyAV (lerobot) | GPL-2.0 | No \u2014 training only |
| LOW \u2014 GPL trigger exists but training code is not distributed externally. |
FFmpeg CLI (flywheel/training scripts) | GPL-2.0 | No \u2014 subprocess only |
| NONE \u2014 subprocess invocation does not trigger GPL copyleft. |
spidev | GPL-2.0 | YES \u2014 robot hardware |
| MEDIUM \u2014 GPL Python module imported at runtime on robot hardware. Runs on the robot only (not distributed as a general product). |
pygame | LGPL-2.1 | YES \u2014 robot hardware |
| LOW \u2014 LGPL (weak copyleft). Dynamic linking is standard. Safe if unmodified and dynamically linked. |
PyQt5 | GPL-3.0 | No \u2014 dev script only | Only imported in | NONE for distribution \u2014 dev-only script. Could still be replaced with PySide6 for cleanliness, but not a distribution risk. |
pynput | LGPL-3.0 | No \u2014 dead dependency | Declared in | NONE \u2014 dead dependency. Can be safely removed from pixi.toml. |
readline | GPL-3.0 | No \u2014 transitive only | Pulled in by IPython via pixi.lock. Never directly imported in any HMND code. Only active when a developer uses the IPython REPL interactively. | NONE \u2014 transitive dev dependency, never imported or distributed. |
x264/x265 | GPL-2.0 | Transitive via ffmpeg | Linked into the conda-forge ffmpeg build. Risk depends on how ffmpeg is consumed (see rows above). | Varies \u2014 HIGH where ffmpeg is library-linked, NONE where ffmpeg is subprocess-only. |
Based on actual code usage tracing, the effective risk levels differ from the initial declaration-based scan:
Module | Initial Risk | Revised Risk | Reason |
|---|---|---|---|
hmnd_robot | HIGH | HIGH (confirmed) | EtherCAT GPL linkage chain is real and ships in OS image. quest_stream_viewer.cpp links ffmpeg directly. |
hmnd_training | HIGH | LOW | ffmpeg usage is either PyAV (training-only, not distributed) or subprocess (safe). readline is transitive/unused. All RED items are build tools with GCC Runtime Exception. |
hmnd_wholebody | HIGH | LOW | PyQt5 is only used in a dev-only script, not in any distributed package. |
hmnd_locomotion | MEDIUM | LOW | LGPL deps only (paramiko, psycopg2). Dynamic linking is standard. |
Component | License | Module(s) | Risk | Suggested Action |
|---|---|---|---|---|
EtherLab EtherCAT | GPL-2.0 + LGPL-2.1 |
| Bundled and compiled into robot binaries. GPL-2.0 requires entire linked work to be GPL-compatible if distributed. | Isolate into a separate process communicating via IPC, or obtain a commercial license, or evaluate SOEM (dual GPL/commercial). |
x264 / libx264 | GPL-2.0 |
| H.264 video codec. If linked into any binary, that binary becomes a GPL derivative. | Replace with libvpx (BSD) for VP8/VP9, use hardware encoding APIs, or ensure ffmpeg is invoked as a subprocess only. |
x265 / libx265 | GPL-2.0 |
| H.265 video codec. Same viral risk as x264. | Same as above \u2014 replace or isolate. |
FFmpeg | GPL-2.0 (when built with GPL codecs) |
| conda-forge builds include GPL codecs (x264, x265). Core libs are LGPL, but the full package is GPL. | quest_stream_viewer.cpp directly links ffmpeg C libraries \u2014 must refactor to subprocess or build LGPL-only ffmpeg. CLI subprocess usage in flywheel/training is safe. |
PyQt5 | GPL-3.0 |
|
| Low priority. Replace with PySide6 (LGPL-3.0) for cleanliness, but this is not a distribution risk. |
spidev | GPL-2.0 |
| SPI device interface for hardware. Links into Python runtime on robot. | Evaluate if this runs only on the robot (not distributed externally) or replace with a permissive SPI library. |
Component | License | Module(s) | Notes |
|---|---|---|---|
zeromq / libzmq | LGPL-3.0 |
| Safe if dynamically linked. Static linking requires providing object files for relinking. |
nlopt | LGPL-2.1 |
| Nonlinear optimization. Same dynamic vs static linking concern. |
ruckig | LGPL-3.0 |
| Trajectory generation. Verify linking method. |
libusb | LGPL-2.1 |
| USB device library. Typically dynamically linked \u2014 safe. |
libudev / libsystemd | LGPL-2.1 |
| System libraries. Dynamic linking is standard. |
pynput | LGPL-3.0 |
|
|
pygame | LGPL-2.1 |
| Used in production head display ( |
paramiko | LGPL-2.1 |
| SSH library. Must disclose source if modified. |
psycopg2-binary | LGPL-2.1 |
| PostgreSQL driver. Must disclose source if modified. |
Eigen | MPL-2.0 | Transitive (via pinocchio, ceres, etc.) | File-level copyleft only \u2014 must share modifications to Eigen source files, not your code. |
glib / gtk3 / pango / librsvg | LGPL-2.0+ |
| GUI/text rendering stack. Typically dynamically linked. |
GnuTLS | LGPL-2.1 (or GPL-3.0) |
| TLS library. Verify LGPL variant is used, not GPL. |
readline | GPL-3.0 |
| REVISED: Never directly imported in any HMND code. Transitive dependency of IPython \u2014 only active in interactive REPL sessions. Not a distribution risk. |
Qt6 | LGPL-3.0 |
| Verify not linked into distributed binaries. |
These bundled third-party components have no LICENSE or COPYING file in the repository:
Component | Location | What It Is |
|---|---|---|
nrobotics |
| ARM/X86 binary distributions |
xdof_sdk |
| X-DOF hand SDK |
manus_ros_drivers |
| Manus hand tracking drivers |
nr_tofcam_ros_dist |
| Time-of-flight camera ROS driver |
rox |
| Rox platform core libraries |
ros2_xhand |
| X-hand gripper ROS2 package |
serial |
| Serial communication library |
robocasa |
| Kitchen sim benchmark \u2014 no license in setup.py |
Component | Module(s) | Notes |
|---|---|---|
NVIDIA Isaac Sim (4.5\u20135.1) |
| Requires NVIDIA EULA. Cannot redistribute without permission. |
NVIDIA CUDA (12.1\u201313.0) |
| Redistributable under CUDA EULA terms. |
egl_probe |
| NVIDIA proprietary component. |
databricks-sql-connector |
| Databricks proprietary. |
google-genai |
| Google proprietary SDK. |
Module | GREEN | YELLOW | RED | BLUE | GRAY | Overall Risk | Key Concern |
|---|---|---|---|---|---|---|---|
hmnd_robot | 700+ | 10 | 12 | 0 | 8 | HIGH | EtherCAT GPL linkage ships in OS image. quest_stream_viewer.cpp links ffmpeg. spidev GPL on robot hardware. |
hmnd_training | 83 | 39 | 51 | 0 | 2 | LOW (revised) | ffmpeg used via PyAV (training-only) or subprocess (safe). readline transitive/unused. All other REDs are build tools with GCC Runtime Exception. |
hmnd_wholebody | 19 | 0 | 2 | 0 | 0 | LOW (revised) | PyQt5 only in dev-only script ( |
hmnd_locomotion | 73 | 2 | 4 | 2 | 0 | LOW | LGPL deps only (paramiko, psycopg2). Dynamic linking standard. isaacsim (proprietary). |
hmnd_sim | 32 | 0 | 3 | 2 | 1 | LOW | Build tools only (gcc/gxx/make). isaacsim (proprietary). |
hmnd_flywheel | 64 | 0 | 0 | 2 | 12 | LOW | No copyleft. ffmpeg via subprocess only. Some unknown licenses on npm devDeps. |
hmnd_fleet | 0 | 0 | 0 | 1 | 0 | LOW | Minimal external deps. |
hmnd_services | 11 | 0 | 0 | 0 | 0 | LOW | All permissive. |
hmnd_update | 9 | 0 | 0 | 0 | 0 | LOW | All permissive. |
hmnd_design | 38 | 1 | 0 | 0 | 0 | LOW | pygame (LGPL) in beta-hri-display only. |
Root / Bazel | 21 | 0 | 0 | 0 | 0 | LOW | All permissive (Apache-2.0, MIT, BSD, BSL-1.0). |
Priority 1 \u2014 Address before any commercial distribution:
EtherCAT (GPL-2.0): This is the #1 issue. The GPL linkage chain is confirmed: bundled source \u2192 libethercat.so \u2192 ethercat_interface \u2192 alpha_universal_driver \u2192 Debian packages \u2192 robot OS image. Isolate into a separate process behind IPC, obtain a commercial license from EtherLab/Beckhoff, or evaluate SOEM as an alternative.
FFmpeg in quest_stream_viewer.cpp: Direct C library linkage (libavcodec, libavutil, libswscale) makes this binary a GPL derivative. Refactor to use ffmpeg as a subprocess, build an LGPL-only ffmpeg (without x264/x265), or replace with libvpx (BSD) for VP8/VP9.
Priority 2 \u2014 Evaluate before external release:
spidev (GPL-2.0) on robot hardware: Used in LED strip control via SPI. Evaluate whether the robot OS image constitutes "distribution" under GPL. If so, replace with a permissive SPI library or provide GPL source compliance.
Obtain license files for the 8 bundled components with no license (nrobotics, xdof_sdk, manus_ros_drivers, nr_tofcam_ros_dist, rox, ros2_xhand, serial, robocasa).
Verify LGPL dynamic linking for zeromq, nlopt, ruckig, libusb, libudev. Ensure no static linking occurs in production builds.
Verify NVIDIA redistribution terms for Isaac Sim, CUDA, and egl_probe.
Priority 3 \u2014 Housekeeping:
Remove dead dependency pynput from hmnd_robot/pixi.toml \u2014 declared but never imported.
Replace PyQt5 with PySide6 in hmnd_wholebody/pixi.toml for cleanliness, even though it's dev-only.
Document the GCC Runtime Library Exception applicability for all C++ binaries. libgcc, libstdc++, libgomp are GPL-3.0 but the Runtime Exception permits linking into proprietary code.
Audit build-only vs runtime dependencies. Many RED items (gcc, gxx, make, shellcheck, pylint, git, kernel-headers) are build/dev tools that never ship \u2014 these are non-issues for distribution.
Detailed per-dependency tables for each module are available as child pages below.
" }, { "id": "1839693858", "title": "OSS Audit: hmnd_design, hmnd_playground, Root/Bazel", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1839693858", "last_modified": "2026-04-08T09:52:20.172Z", "created_at": "2026-04-08T09:52:20.172Z", "body_html": "Risk Level: LOW
Key Concerns: pygame (LGPL-2.1) in hmnd_design beta-hri-display; matplotlib (LGPL components) in playground projects
All 36 dependencies are MIT or Apache-2.0. No copyleft issues.
Name | Version | License | Category |
|---|---|---|---|
@emotion/react | ^11.14.0 | MIT | Styling |
@emotion/styled | ^11.14.1 | MIT | Styling |
@mui/icons-material | ^7.3.1 | MIT | UI |
@mui/material | ^7.3.2 | MIT | UI |
@mui/x-charts | ^8.11.0 | MIT | Charts |
@mui/x-data-grid | ^8.11.0 | MIT | Data Grid |
@mui/x-date-pickers | ^8.11.0 | MIT | Date Picker |
@mui/x-tree-view | ^8.11.0 | MIT | Tree View |
@react-spring/web | ^10.0.1 | MIT | Animation |
clsx | ^2.1.1 | MIT | Utilities |
dayjs | ^1.11.15 | MIT | Date |
react / react-dom | ^18.2.0 | MIT | Framework |
react-router-dom | ^7.8.2 | MIT | Routing |
@storybook/* | ^9.1.7 | MIT | Dev Tool |
@typescript-eslint/* | ^6.14.0 | BSD-2 | Linting |
eslint / eslint-plugin-* | various | MIT | Linting |
playwright | ^1.55.0 | Apache-2.0 | Testing |
typescript | ^5.2.2 | Apache-2.0 | Language |
vite | ^5.0.8 | MIT | Build |
vitest | ^3.2.4 | MIT | Testing |
Name | Version | License | Risk |
|---|---|---|---|
pygame |
| LGPL-2.1 | YELLOW \u2014 graphics library with weak copyleft |
moderngl |
| MIT | GREEN |
numpy |
| BSD | GREEN |
Pillow |
| HPND | GREEN |
noise |
| MIT | GREEN |
Playground projects are experimental / developer sandboxes. Not shipped in production. Key dependencies by project:
Name | Version | License | Risk |
|---|---|---|---|
opencv |
| Apache-2.0 | GREEN |
pinocchio |
| BSD-2 | GREEN |
ceres-solver |
| BSD-3 | GREEN |
matplotlib |
| PSF/BSD (some LGPL components) | YELLOW-LOW |
scipy |
| BSD | GREEN |
ros-humble-desktop | Apache-2.0 | GREEN |
Name | Version | License | Risk |
|---|---|---|---|
torch |
| MIT | GREEN |
torchvision |
| BSD | GREEN |
opencv-python | Apache-2.0 | GREEN | |
matplotlib | PSF/BSD | YELLOW-LOW | |
huggingface-hub | Apache-2.0 | GREEN |
Name | Version | License | Risk |
|---|---|---|---|
matplotlib |
| PSF/BSD | YELLOW-LOW |
pinocchio |
| BSD-2 | GREEN |
rerun-sdk |
| MIT | GREEN |
mcap-ros2-support |
| MIT | GREEN |
ros-humble-desktop | Apache-2.0 | GREEN |
Name | Version | License | Risk |
|---|---|---|---|
colmap | BSD-3 | GREEN | |
opencv | Apache-2.0 | GREEN | |
numpy | BSD | GREEN |
Name | Version | License | Risk |
|---|---|---|---|
ros-humble-desktop | Apache-2.0 | GREEN | |
opencv | Apache-2.0 | GREEN | |
All others | MIT/BSD/Apache | GREEN |
Name | Version | License | Risk |
|---|---|---|---|
ray[default] | Apache-2.0 | GREEN | |
polars | MIT | GREEN | |
All others | MIT/BSD/Apache | GREEN |
Name | Version | License | Risk |
|---|---|---|---|
gcc |
| GPL-3.0 | RED (build tool \u2014 not linked) |
gxx |
| GPL-3.0 | RED (build tool \u2014 not linked) |
binutils | GPL-3.0 | RED (build tool \u2014 not linked) | |
make |
| GPL-3.0 | RED (build tool \u2014 not linked) |
git | GPL-2.0 | RED (tool \u2014 not linked) | |
curl | MIT | GREEN | |
python |
| PSF | GREEN |
openjdk |
| GPL-2.0 + Classpath Exception | Safe (Classpath Exception) |
bazelisk |
| Apache-2.0 | GREEN |
pydpkg | MIT | GREEN |
Name | Version | License | Risk |
|---|---|---|---|
bazel-diff | 12.0.0 | Apache-2.0 | GREEN |
platforms | 1.0.0 | Apache-2.0 | GREEN |
rules_foreign_cc | 0.15.0 | Apache-2.0 | GREEN |
rules_pkg | 1.1.0 | Apache-2.0 | GREEN |
rules_shell | 0.6.1 | Apache-2.0 | GREEN |
bazel_skylib | 1.9.0 | Apache-2.0 | GREEN |
rules_cc | 0.2.16 | Apache-2.0 | GREEN |
rules_rust | 0.67.0 | Apache-2.0 | GREEN |
googletest | 1.17.0 | BSD-3 | GREEN |
flatbuffers | 25.12.19 | Apache-2.0 | GREEN |
grpc | 1.72.0 | Apache-2.0 | GREEN |
protobuf | 30.0 | BSD-3 | GREEN |
rules_proto | 7.1.0 | Apache-2.0 | GREEN |
rules_python | 1.6.3 | Apache-2.0 | GREEN |
aws_sdk | 1.11.321 | Apache-2.0 | GREEN |
boost.asio | 1.89.0 | BSL-1.0 | GREEN |
boost.beast | 1.89.0 | BSL-1.0 | GREEN |
boost.system | 1.89.0 | BSL-1.0 | GREEN |
spdlog | 1.17.0 | MIT | GREEN |
tl-expected | 1.1.0 | CC0-1.0 | GREEN |
All test/integration dependencies are permissively licensed (MIT, BSD, Apache-2.0). No copyleft concerns.
" }, { "id": "1840807976", "title": "OSS Audit: hmnd_fleet, hmnd_services, hmnd_update", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1840807976", "last_modified": "2026-04-08T09:51:46.738Z", "created_at": "2026-04-08T09:51:46.738Z", "body_html": "Risk Level: LOW
Key Concerns: None \u2014 all dependencies are permissively licensed.
Minimal external dependencies. The pixi.toml installs only a local editable package (hmnd-fleet). No third-party copyleft issues.
Source | Name | Version | License | Risk |
|---|---|---|---|---|
pixi (pypi) | hmnd-fleet | local editable | Proprietary (internal) | BLUE |
Backend API service using FastAPI, SQLAlchemy, and AWS. All 11 dependencies are permissively licensed (MIT, Apache-2.0, BSD). Project itself is marked as Proprietary.
Source | Name | Version | License | Risk |
|---|---|---|---|---|
pyproject.toml | alembic |
| MIT | GREEN |
pyproject.toml | boto3 |
| Apache-2.0 | GREEN |
pyproject.toml | fastapi |
| MIT | GREEN |
pyproject.toml | httpx |
| MIT | GREEN |
pyproject.toml | PyJWT[crypto] |
| MIT | GREEN |
pyproject.toml | psycopg[binary] |
| BSD (LGPL-free \u2014 note: psycopg3 is BSD, unlike psycopg2 which is LGPL) | GREEN |
pyproject.toml | pydantic |
| MIT | GREEN |
pyproject.toml | sqlalchemy |
| MIT | GREEN |
pyproject.toml | uuid6 |
| MIT | GREEN |
pyproject.toml | uvicorn |
| MIT | GREEN |
pyproject.toml (dev) | pytest | MIT | GREEN |
All resolved packages in uv.lock are MIT, Apache-2.0, or BSD licensed. No copyleft detected.
OTA update service using FastAPI and S3. All 9 dependencies are permissively licensed.
Source | Name | Version | License | Risk |
|---|---|---|---|---|
pyproject.toml | fastapi | MIT | GREEN | |
pyproject.toml | pydantic | MIT | GREEN | |
pyproject.toml | requests | Apache-2.0 | GREEN | |
pyproject.toml | uvicorn[standard] | MIT | GREEN | |
pyproject.toml | boto3 | Apache-2.0 | GREEN | |
pyproject.toml | sniffio |
| Apache-2.0 | GREEN |
pyproject.toml (dev) | pytest | MIT | GREEN | |
pyproject.toml (dev) | moto[s3] | Apache-2.0 | GREEN | |
pyproject.toml (dev) | pytest-asyncio | MIT | GREEN |
All resolved packages are permissively licensed. No copyleft detected.
" }, { "id": "1839071263", "title": "OSS Audit: hmnd_locomotion", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1839071263", "last_modified": "2026-04-08T09:51:17.061Z", "created_at": "2026-04-08T09:51:17.061Z", "body_html": "Risk Level: MEDIUM
Key Concerns: paramiko/psycopg2 (LGPL-2.1), build tools (GPL), NVIDIA Isaac Sim (proprietary)
hmnd_locomotion contains the locomotion training stack built on Isaac Sim / IsaacLab. It has 81 direct dependencies. The only RED items are build tools (gcc, gxx, make) and pylint (dev tool) \u2014 none of which ship in distributed binaries. The LGPL concerns (paramiko, psycopg2-binary) are weak copyleft and safe if used unmodified.
Source | Name | Version | License | Notes |
|---|---|---|---|---|
pixi (conda) | gcc | GPL-3.0 | Build tool \u2014 not linked. | |
pixi (conda) | gxx | GPL-3.0 | Build tool \u2014 not linked. | |
pixi (conda) | make | GPL-3.0 | Build tool \u2014 not linked. | |
pixi (pypi) | pylint |
| GPL-2.0 | Development linter \u2014 not shipped. |
Source | Name | Version | License | Notes |
|---|---|---|---|---|
pixi (pypi) | paramiko |
| LGPL-2.1 | SSH client library. Must disclose source if modified. |
pixi (pypi) | psycopg2-binary |
| LGPL-2.1 | PostgreSQL driver. Must disclose source if modified. |
Source | Name | Version | License | Notes |
|---|---|---|---|---|
pixi (pypi) | isaacsim | 4.5.0.0 | NVIDIA Proprietary | Physics simulator. Requires EULA. |
pixi (pypi) | egl_probe | 1.0.2 | NVIDIA Proprietary | EGL probe utility. |
Name | Version | License | Category |
|---|---|---|---|
torch |
| MIT | Deep Learning |
torchvision |
| BSD | Computer Vision |
triton |
| MIT | GPU Compilation |
gymnasium |
| MIT | RL Environments |
stable-baselines3 | 2.7.0 | MIT | RL |
diffusers |
| Apache-2.0 | Diffusion Models |
huggingface-hub |
| Apache-2.0 | ML Hub |
tensorboard |
| Apache-2.0 | Logging |
tensorboardX |
| MIT | Logging |
wandb |
| MIT | Experiment Tracking |
onnx |
| Apache-2.0 | Model Format |
flash-attn | (implicit) | BSD-3 | ML Optimization |
Name | Version | License | Category |
|---|---|---|---|
pinocchio |
| BSD-2 | Robot Dynamics |
mujoco |
| Apache-2.0 | Physics |
scipy |
| BSD | Scientific |
numpy | <2 | BSD | Numerical |
opencv-python |
| Apache-2.0 | Vision |
usd-core | Apache-2.0 | Scene Description | |
robomimic | (git) | MIT | Robot Learning |
xacrodoc |
| BSD | URDF |
Name | Version | License | Category |
|---|---|---|---|
fastapi |
| MIT | Web |
flask |
| BSD | Web |
pydantic |
| MIT | Validation |
hydra-core |
| Apache-2.0 | Config |
omegaconf |
| BSD | Config |
click |
| BSD | CLI |
rich |
| MIT | Terminal |
tqdm |
| MIT | Progress |
pyyaml |
| MIT | Config |
pytest / pytest-cov | various | MIT | Testing |
pre-commit |
| MIT | Dev |
black |
| MIT | Formatting |
isort |
| MIT | Imports |
mypy |
| MIT | Type Checking |
ipython |
| BSD | Development |
Name | Version | License | Category |
|---|---|---|---|
h5py |
| BSD | HDF5 |
pandas |
| BSD | Data |
boto3 / botocore |
| Apache-2.0 | AWS |
fsspec |
| BSD | Filesystem |
dask |
| BSD | Parallel Compute |
celery |
| BSD | Task Queue |
redis |
| BSD | Cache |
SQLAlchemy |
| MIT | ORM |
Jinja2 |
| BSD | Templating |
Name | Version | License | Category |
|---|---|---|---|
python | 3.10 | PSF | Runtime |
pip |
| MIT | Package Manager |
cmake |
| BSD | Build |
xorg-libsm |
| MIT | Display |
xorg-libxt |
| MIT | Display |
libglu |
| SGI-B-2.0 | Graphics |
Name | License |
|---|---|
isaaclab | BSD-3-Clause (NVIDIA) |
isaaclab-assets | BSD-3-Clause |
isaaclab-locomotion | Internal |
isaaclab-mimic | BSD-3-Clause |
isaaclab-rl | BSD-3-Clause |
isaaclab-tasks | BSD-3-Clause |
Risk Level: LOW
Key Concerns: No copyleft. Some proprietary SDKs (Databricks, Google GenAI). Several npm dev dependencies with unverified licenses.
hmnd_flywheel contains the data pipeline, dataset management, Databricks integrations, humanoid console, and prefect workflow orchestration. 78 unique dependencies were identified across 7 pixi.toml files, 10 pyproject.toml files, and npm package.json files. Zero GPL or LGPL dependencies were found. The module is clean for commercial distribution.
Source | Name | Version | License | Notes |
|---|---|---|---|---|
PyPI | databricks-sql-connector |
| Proprietary (Databricks) | Databricks SQL access. Check redistribution terms. |
PyPI | google-genai |
| Proprietary (Google) | Google AI SDK. Check terms of service. |
Source | Name | Version | Notes |
|---|---|---|---|
npm | @types/react | ^18.2.73 | DefinitelyTyped \u2014 likely MIT |
npm | @types/react-dom | ^18.2.23 | DefinitelyTyped \u2014 likely MIT |
npm | @vitejs/plugin-react | ^4.2.1 | Likely MIT |
npm | bufferutil | ^4.0.9 | Likely MIT |
npm | utf-8-validate | ^6.0.5 | Likely MIT |
npm | vite-plugin-electron | ^0.28.6 | Needs verification |
npm | vite-plugin-electron-renderer | ^0.14.5 | Needs verification |
npm | ws | ^8.18.3 | Likely MIT |
PyPI | hc-client | Internal package | |
PyPI | humanoid-console-models | Internal package | |
PyPI | hatchling.build | Likely MIT (part of hatchling) | |
PyPI | src/hmnd_console_api | Internal package |
Name | Version | License | Category |
|---|---|---|---|
boto3 |
| Apache-2.0 | AWS |
click |
| BSD-3 | CLI |
duckdb |
| MIT | Database |
fastapi | (implicit) | MIT | Web Framework |
ffmpeg-python |
| MIT | Video (Python wrapper \u2014 NOT the GPL binary) |
filelock |
| Unlicense | Filesystem |
fsspec |
| BSD-3 | Filesystem |
hatchling | MIT | Build Tool | |
httpx |
| BSD-3 | HTTP |
imageio-ffmpeg |
| MIT | Video |
ipykernel |
| BSD-3 | Notebook |
jupyter |
| BSD-3 | Notebook |
kubernetes |
| Apache-2.0 | Orchestration |
kubernetes-asyncio |
| Apache-2.0 | Orchestration |
lerobot |
| Apache-2.0 | Robotics |
marimo |
| MIT | Notebook |
mcap-ros2-support | ==0.5.5 | MIT | Data Format |
mcp |
| MIT | Protocol |
moto[s3server] |
| Apache-2.0 | AWS Testing |
nicegui |
| MIT | Web UI |
numpy |
| BSD-3 | Numerical |
openai |
| MIT | AI |
opencv-python | ==4.11.0.86 | Apache-2.0 | Computer Vision |
pandas |
| BSD-3 | Data |
pillow |
| HPND | Image |
plotly |
| MIT | Visualization |
polars[all] |
| MIT | Data |
prefect | ==3.6.5 | Apache-2.0 | Workflow |
prefect-databricks |
| Apache-2.0 | Workflow |
prefect-docker |
| Apache-2.0 | Workflow |
prefect-github |
| Apache-2.0 | Workflow |
prefect-kubernetes |
| Apache-2.0 | Workflow |
prefect-slack |
| Apache-2.0 | Workflow |
pyarrow |
| Apache-2.0 | Data |
pydantic |
| MIT | Validation |
pyspark |
| Apache-2.0 | Big Data |
pytest |
| MIT | Testing |
python-dateutil |
| BSD-3 | Time |
python-jose[crypto] |
| MIT | JWT |
pyyaml |
| MIT | Config |
ray[default] |
| Apache-2.0 | Distributed |
requests |
| Apache-2.0 | HTTP |
ruamel.yaml |
| MIT | Config |
sqlglot |
| MIT | SQL |
tenacity |
| Apache-2.0 | Retry |
tqdm |
| MIT | Progress |
vegafusion |
| MIT | Visualization |
vl-convert-python |
| BSD-3 | Visualization |
watchfiles |
| MIT | Filesystem |
websockets |
| BSD-3 | Networking |
Name | Version | License | Category |
|---|---|---|---|
@emotion/react | ^11.11.4 | MIT | Styling |
@emotion/styled | ^11.11.0 | MIT | Styling |
@mui/icons-material | ^5.15.15 | MIT | UI |
@mui/material | ^5.15.15 | MIT | UI |
@tanstack/react-query | ^5.28.9 | MIT | Data |
react / react-dom | ^18.2.0 | MIT | Framework |
zod | ^3.22.4 | MIT | Validation |
electron | ^29.1.6 | MIT | Desktop |
electron-builder | ^24.13.3 | MIT | Build |
typescript | ^5.4.3 | Apache-2.0 | Language |
vite | ^5.2.7 | MIT | Build |
Risk Level: LOW
Key Concerns: Build tools (gcc/gxx/make \u2014 GPL, not linked), NVIDIA Isaac Sim (proprietary)
hmnd_sim is relatively clean from a licensing perspective. The primary dependencies are ROS 2 (Apache-2.0), DVC (Apache-2.0), Ray (Apache-2.0), and the NVIDIA Isaac Sim + IsaacLab stack. The only GPL items are build tools that don't ship in binaries. The main concern is the proprietary NVIDIA Isaac Sim licensing.
Name | Location | License | Risk |
|---|---|---|---|
IsaacLab |
| BSD-3-Clause | GREEN |
IsaacLab ai_agent |
| BSD-3-Clause | GREEN |
Source | Name | Version | License | Notes |
|---|---|---|---|---|
pixi.toml | gcc | GPL-3.0 | Build tool \u2014 not linked. | |
pixi.toml | gxx | GPL-3.0 | Build tool \u2014 not linked. | |
pixi.toml | make | GPL-3.0 | Build tool \u2014 not linked. |
Source | Name | Version | License | Notes |
|---|---|---|---|---|
pixi.toml (system-req) | isaacsim | 5.0+ / 5.1.0.0 | NVIDIA Proprietary | Requires NVIDIA EULA. Cannot redistribute without permission. |
pixi.toml | pixi-pycharm | Proprietary | IDE integration. |
Source | Name | Version | License | Category |
|---|---|---|---|---|
pixi.toml | python |
| PSF | Runtime |
pixi.toml | numpy | BSD-3 | Numerical | |
pixi.toml | pandas | BSD-3 | Data | |
pixi.toml | scipy | BSD-3 | Scientific | |
pixi.toml | cmake | BSD-3 | Build Tool | |
pixi.toml | dvc[s3] | Apache-2.0 | Data Versioning | |
pixi.toml | ray-default | Apache-2.0 | Distributed Compute | |
pixi.toml | ros-humble-desktop | Apache-2.0 | ROS 2 | |
pixi.toml | ros-humble-foxglove-bridge | Apache-2.0 | Visualization | |
pixi.toml | ros-humble-ros-gz | Apache-2.0 | Gazebo Bridge | |
pyproject.toml | torch |
| BSD-3 | Deep Learning |
pyproject.toml | torchvision | BSD-3 | Computer Vision | |
pyproject.toml | gymnasium | MIT | RL Environments | |
pyproject.toml | opencv-python | Apache-2.0 | Computer Vision | |
pyproject.toml | matplotlib | PSF/BSD | Plotting | |
pyproject.toml | tensorboard | Apache-2.0 | Logging | |
pyproject.toml | wandb | MIT | Experiment Tracking | |
pyproject.toml | hydra-core | Apache-2.0 | Config | |
pyproject.toml | protobuf | BSD-3 | Serialization | |
pyproject.toml | pillow | HPND | Image Processing | |
pyproject.toml | mujoco | Apache-2.0 | Physics Simulation | |
pyproject.toml | pinocchio | BSD-2 | Robot Dynamics | |
requirements.txt | python-dotenv | BSD-3 | Config | |
requirements.txt | requests | Apache-2.0 | HTTP |
Name | Location |
|---|---|
isaaclab |
|
isaaclab-assets |
|
isaaclab-mimic |
|
isaaclab-rl |
|
isaaclab-tasks |
|
Name | Location | Notes |
|---|---|---|
urdf_pipeline |
| Local editable module \u2014 needs internal license header check. |
Claude | April 2, 2026
By end of 2026, operate a handful of robots in sustained, real operations at a customer site (Schaeffler), meeting the KPIs and metrics defined in the MSA.
That means: ring picking at 440 UPH, navigation at 320 UPH across 2 stations, and a secondary use case (end-of-line package handling) \u2014 all running continuously, not as a rehearsed demo.
We don't know all the things that will break in a sustained pilot. Nobody does. What we can do is close the product loop as fast as possible \u2014 get a robot autonomously executing an operator-authored workflow in a Schaeffler-like setup \u2014 and then run it until it fails. Fix what fails. Repeat.
This plan is organized around that principle. Phase 1 closes the loop. Phase 2 stress-tests it. Phase 3 hardens based on what we learn. Phase 4 is the pilot itself. The epics in Phases 2\u20133 are deliberately less specified because the whole point is to let real failure data drive the work.
April \u2013 May 2026
Goal: An operator can deploy a robot to a new space, teach it a task via teleop, and have it execute that task autonomously \u2014 without engineering involvement.
Epic | What it means | Status |
|---|---|---|
Symphony V2 \u2014 Agentic Workflow Authoring | LLM agent can create/edit workflows from operator interaction. Integrated with runtime executor. | In progress \u2014 PRs in flight |
Single-Shot Deployment MVP | Mapping pipeline (lidar \u2192 occupancy grid), deployment mode recording, workflow inference, package upload to S3/Coordinator. The full teleop-to-autonomous pipeline. | Demonstrated on robot |
Release Stabilization | Immutable, full-platform release images that fully capture all software running on the robot \u2014 OS, controls, policies, maps, agent runtime. Deployment team can run the latest release without chasing version mismatches between components. | Ongoing pain point |
Exit criteria: One robot, one site (internal), operator-authored workflow executing autonomously end-to-end. Shitty is fine. Working is mandatory.
Known fires running in parallel (not part of this plan, but consuming people):
Hannover Messe Expo: Apr 20\u201324
Siemens Amberg PoC: ~May \u2014 cancelled. This time is now available. Use it.
June \u2013 July 2026
Goal: Run the product loop continuously on a Schaeffler-representative setup internally. Find out where it falls over.
Epic | What it means | Notes |
|---|---|---|
Schaeffler Use Case Standup | Ring picking workflow authored and running on internal setup. Measure actual UPH. Identify gaps to 440 target. | Requires use-case-specific policy work + gripper readiness |
Multi-Robot Coordination (Minimum) | 2+ robots operating simultaneously without conflicts. Idle states, task handoff, charging hot-swap. | Fleet manager already needs idle state (Hannover blocker) \u2014 build on that |
KPI Instrumentation | Can we actually measure UPH, uptime, navigation success rate, cycle time? If we can't measure it, we can't hit it. | This is embarrassingly absent right now |
Operator Iteration Workflow | Operator reviews execution, identifies failures, edits workflow, redeploys \u2014 without engineering. The "iterate" part of the loop. | Conformance checking, Sockpuppet feedback, workflow versioning |
Exit criteria: Robots running Schaeffler ring-picking workflow daily, with KPI dashboards showing where we stand vs. MSA targets. A clear list of "this is why we'd fail the pilot today."
August \u2013 October 2026
Goal: Close the gap between "shitty pilot" and "meets MSA KPIs." This phase is intentionally underspecified \u2014 it gets filled by whatever Phase 2 reveals.
Epic | What it means | Notes |
|---|---|---|
[TBD] Top failures from v0 | Whatever the top 3\u20135 failure modes are from Phase 2. Could be controls, could be mapping, could be fleet coordination, could be something we haven't thought of. | This is the whole point of the approach |
Secondary Use Case Development | End-of-line package handling \u2014 the use case that hasn't been attempted yet. This is the highest-risk Schaeffler KPI. | Needs to start no later than August |
Release & Deployment Pipeline | OTA updates that work reliably. Map updates without rebuilding release images. Consistent policy versioning. The boring stuff that kills you in sustained ops. | OTA already in progress. Map-baking issue known. |
Fleet Ops Minimum Viable | Enough monitoring, alerting, and remote diagnostics to keep robots running without an engineer on-site 24/7. Not a full NOC \u2014 just enough. | On-robot logging service feeds into this |
Exit criteria: KPIs trending toward MSA targets. Secondary use case demonstrated. Team confident enough to put robots on a truck to Schaeffler.
November \u2013 December 2026
Goal: Robots operating at Schaeffler, meeting MSA commitments.
Epic | What it means | Notes |
|---|---|---|
Site Deployment | Physical deployment, network bringup, site acceptance test, operator training. | Deployment readiness work feeds directly here |
Sustained Operations | Daily operation against MSA KPIs. Issue triage, remote support, on-site presence as needed. | |
KPI Validation & Reporting | Formal measurement against SoW KPIs. Schaeffler sign-off. |
The 300+ PB backlog items. Most of those are noise until the product loop is closed and we have real operational data.
Flow requirements as written. The requirements doc is a feature wishlist, not a buildable spec. We will derive real requirements from pilot failure data.
Full fleet management / NOC / global platform. That's a 2027 problem. We need to prove we can run 3 robots at one site first.
Home / consumer product stream. Separate track, separate plan.
VDA5050, enterprise integrations, multi-tenant isolation. Important eventually. Not blocking the pilot.
Risk | Impact | Mitigation |
|---|---|---|
Gripper hardware not ready for ring picking | Blocks Phase 2 entirely | HECO-2526 (3-finger gripper v1.3.2) \u2014 track weekly, escalate early |
Symphony V2 + deployment MVP don't integrate cleanly by May | Phase 1 slips, everything shifts | Single integration owner needed. Treat PB-404 as the integration ticket. |
Release quality stays broken | Every phase is harder if deployment can't use latest software | Release stabilization is Phase 1 scope, not a nice-to-have |
Secondary use case (package handling) is harder than expected | MSA KPI miss | Start early in Phase 3, treat as highest-priority unknown |
Hannover + Siemens PoC consume key people through May | Phase 1 work slows | Accept it. Plan for reduced velocity April\u2013May. |
Beta hardware delays affect robot availability | Can't run enough robots for multi-robot testing | Monitor Beta-0 build schedule. Worst case, run on Alpha fleet. |
This is a direction, not a Gantt chart. The phases are real and the exit criteria matter. The specific epics in Phases 2\u20133 will evolve as we learn. That's not a bug \u2014 that's the entire strategy.
If the product loop works in Phase 1, we will have the fastest possible feedback mechanism to figure out what actually matters for the pilot. If it doesn't work, nothing else on any backlog matters anyway.
" }, { "id": "1821671441", "title": "Robot Observability", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1821671441", "last_modified": "2026-04-01T14:59:51.610Z", "created_at": "2026-04-01T14:55:44.488Z", "body_html": "This page is the practical guide to the HMND robot observability stack.
It is meant to answer four questions:
What do we collect, and why?
How does data get from a robot into Grafana?
How do I instrument a node in hmnd_robot?
How do I make that data show up in dashboards?
The goal is not exhaustive telemetry or full-fidelity replay of every internal signal. The goal is constant observability of enough data to diagnose real robot issues quickly and reliably.
In practice that means:
We continuously ship low-cost metrics and logs.
We preserve the metadata needed to segment by robot, platform, and target.
We use metrics for trends, rates, health, and distributions.
We use logs for sparse, structured events and explanations.
We do not try to stream arbitrary high-rate control internals as observability data.
If we need full-fidelity debugging or offline analysis, that is a rosbag / MCAP problem, not a Prometheus problem.
At a high level:
robot exporters + host/system logs -> Vector on robot -> Mimir/Loki/Tempo -> Grafana
The deployed stack today is:
Grafana for visualization and exploration.
Mimir for Prometheus-compatible time-series metrics.
Loki for logs.
Tempo for traces.
Vector on each robot as the local collection, enrichment, buffering, and forwarding agent.
Related background page:
Observability Stack (Grafana/Loki/Tempo/Mimir)
A useful way to think about observability on robots is:
Metrics answer: "is this healthy, how often is it happening, how big is it, what is the distribution?"
Logs answer: "what happened, where, and why?"
Rosbags / MCAP answer: "what exactly happened sample-by-sample?"
Those are different tools for different jobs.
A robot has many signals that update at high frequency. Some of those are control-loop internals, raw sensor streams, and dense state vectors. Continuously shipping all of that as centralized observability data would be expensive, noisy, and hard to query.
For example, trying to push impedance internals at 1000 Hz into a metrics backend is the wrong abstraction:
It creates unnecessary write volume.
It explodes cardinality if labels are added carelessly.
It makes dashboards harder to interpret.
It still does not preserve the full context needed for offline replay.
Instead, we prefer compact health signals such as:
loop period histograms
compute time histograms
overrun counters
queue drop counters
publish-rate gauges
delay / latency histograms
error counters by category
That gives us continuous fleet-scale visibility at a cost that actually scales.
Grafana is the operator-facing UI.
Use it for:
dashboards
ad hoc exploration in Explore
fleet comparisons
robot-specific drill-downs
PromQL / LogQL / Tempo queries
Grafana does not store the data itself. It queries the backends.
Mimir stores our Prometheus-style metrics. Exporters expose /metrics, Vector scrapes those endpoints locally, and Vector remote-writes the samples to Mimir.
This is the right fit for:
counters
gauges
histograms
derived rates
percentiles computed from histogram buckets
alertable health signals
Loki stores logs. On robot, Vector tails journald and internal logs, enriches them with robot metadata, and pushes them to Loki.
This is the right fit for:
structured error events
startup / shutdown records
retry / failure reasons
human-readable context around incidents
sparse lifecycle breadcrumbs
Tempo is available for distributed traces and OTLP spans. It is useful for request-like flows, especially for services such as sockpuppet or reverb.
This page focuses mainly on metrics and logs, since that is the core robot observability path today.
A counter only goes up.
Use a counter for things like:
messages received total
compute overruns total
queue drops total
errors total
episodes saved total
Typical query pattern:
rate(hmnd_wbc_command_missed_cycles_total[5m])\n
That tells you the recent rate of failures, which is usually more useful than the raw cumulative count.
A gauge is the current value of something.
Use a gauge for things like:
publish rate in Hz
current queue depth
current battery percentage
current overrun ratio in a recent window
latest observed delay
Typical query pattern:
hmnd_wbc_compute_overrun_ratio{hmnd_robot_id="$robot"}\nA histogram records a distribution into fixed buckets.
Use a histogram for things like:
loop period
compute time
frame poll delay
inference latency
end-to-end latency
Histograms are the most important metric type for timing-sensitive robot systems because they let us answer questions like:
what is the median?
what is the p95 or p99?
is the tail getting worse?
are we jittering or drifting?
which bucket mass is moving over time?
Typical query patterns:
histogram_quantile(0.95, sum by (le) (rate(hmnd_wbc_command_publish_period_seconds_bucket[5m])))\n
histogram_quantile(0.99, sum by (le) (rate(hmnd_vision_topic_publish_period_seconds_bucket[5m])))\n
hmnd_metrics also exposes a summary helper, but in practice histograms are generally the better default for shared dashboards because bucketed histograms aggregate across instances cleanly.
Use this rule of thumb:
If you want to know how many times something happened, use a counter.
If you want the current state, use a gauge.
If you care about the distribution of timings or sizes, use a histogram.
On robots, timing histograms are usually more valuable than one-off gauges because they preserve tail behavior.
For example:
A gauge saying "current loop time = 21 ms" is weak.
A histogram saying "p50 is 20 ms, p99 is 34 ms, tail got worse after 14:32" is useful.
Use metrics for continuous, cheap, queryable health signals.
Good examples:
WBC loop period
camera publish rate
dropped frames
policy inference latency
controller overrun ratio
battery percentage
Use logs for explanations and sparse events.
Good examples:
hardware device failed to initialize
configuration fallback was used
service reconnected after timeout
robot switched operating mode
a fault was raised or cleared
Use bags when you need full replay or dense signal inspection.
Good examples:
actuator command debugging at controller rate
raw sensor stream inspection
synchronization bugs requiring exact timestamps
offline postmortems needing complete context
The important distinction is that metrics/logs are always-on operational telemetry, while bags are episodic high-fidelity recordings.
Prometheus-style systems scale well when each metric has a bounded number of label combinations.
They scale badly when we attach unbounded labels such as:
raw IDs generated per request
timestamps as labels
paths or filenames with huge variety
exception strings
user-generated values
topic names that are dynamically synthesized without limit
Labels that are stable and bounded are good.
Examples we rely on heavily:
hmnd_robot_id
hmnd_robot_type
hmnd_robot_target
controller
topic
device
error_type
Avoid labels whose value count grows without a clear upper bound.
Examples:
UUIDs
free-form error messages
file paths
object names from untrusted input
per-frame IDs
If a label value could create thousands or millions of new time series over time, it probably does not belong in metrics.
Put that information in logs instead.
hmnd-shippingVector is the agent that runs on the robot and makes the whole system operationally sane.
It is responsible for:
scraping exporters
reading journald
reading Vector internal metrics/logs
accepting OTLP traces
enriching records with robot metadata
buffering on disk when the network or backend is unavailable
pushing data to Mimir, Loki, and Tempo
Without Vector, every node would need to know:
backend endpoints
auth details
retry policy
batching policy
buffer policy
per-robot metadata labels
That would be brittle and duplicated.
With Vector:
exporters stay simple
application code only exposes local metrics or writes local logs
enrichment and delivery policy are centralized
temporary outages are absorbed by the agent buffer
hmnd-shipping layoutThe deployed Vector config is packaged through:
platforms/common/hmnd-shipping
platforms/common/hmnd-shipping-robot
Important files:
platforms/common/hmnd-shipping/etc/hmnd/vector/common.yaml
platforms/common/hmnd-shipping-robot/etc/hmnd/vector/robot.yaml
platforms/common/hmnd-shipping/etc/systemd/system/vector.service.d/10-hmnd.conf
The common config defines the general data plane:
journald log ingestion
host metrics
internal Vector metrics/logs
OTLP trace ingestion
metadata enrichment transforms
sanitize transforms for problematic labels
sinks to Loki, Mimir, and Tempo
The key enrichment transform adds the robot identity labels from environment:
.tags.hmnd_robot_type = get_env_var("ROBOT") ?? "unknown"\n.tags.hmnd_robot_id = get_env_var("ROBOT_ID") ?? "unknown"\n.tags.hmnd_robot_target = get_env_var("HMND_ROBOT_TARGET") ?? "unknown"\nThat is what makes per-robot filtering in Grafana work consistently across exporters.
The robot-specific config adds the Prometheus scrape endpoints for robot exporters.
Current scrape list includes ports such as:
9201 hmnd_vision
9202 whole_body_controller
9203 policy execution
9204 general HMND ROS exporter
9205 sockpuppet
9206 alpha data collection
9207 reverb
9208 hmnd_vio
9209 controller overrun metrics
9210 teleoperation server
9212 FT gravity compensation diagnostics
That means adding a new exporter is usually a two-part change:
expose a new local /metrics endpoint in the node
add that endpoint to hmnd-shipping-robot so Vector scrapes it
Vector is configured to batch conservatively and buffer to disk.
That matters because robots can have intermittent connectivity, and we do not want every transient network failure to become dropped telemetry or backpressure inside application nodes.
The intent is:
preserve robot resources first
keep collection simple in the exporter
tolerate temporary backend or network problems
avoid aggressive reconnect storms
hmnd_metricshmnd_metrics is our shared library for exposing Prometheus metrics from robot code.
It exists in:
ros/utils/hmnd_metrics
It supports both:
Python exporters via hmnd_metrics
C++ exporters via hmnd::metrics
import hmnd_metrics\n\nhmnd_metrics.initialize_exporter("0.0.0.0:9207")\n\nrequests_total = hmnd_metrics.counter(\n "hmnd_mynode_requests_total",\n "Total requests processed",\n labels=["endpoint"],\n)\n\nlatency_seconds = hmnd_metrics.histogram(\n "hmnd_mynode_latency_seconds",\n "Request latency",\n labels=["endpoint"],\n buckets=hmnd_metrics.HistogramBuckets.STANDARD_LATENCY.value,\n)\n\nrequests_total.labels(endpoint="/foo").inc()\nlatency_seconds.labels(endpoint="/foo").observe(0.025)\nImportant behaviors:
initialize once per process
metrics are cached by name, type, labels, and buckets
use fixed, bounded label sets
use one exporter endpoint per process
#include <hmnd_metrics/metrics.hpp>\n\nhmnd::metrics::InitializeExporter("0.0.0.0:9209");\n\nauto& histogram = hmnd::metrics::HistogramFamily(\n "hmnd_mynode_latency_seconds", "Latency");\n\nhistogram.Add({{"topic", "/foo"}}, hmnd::metrics::StandardLatencyBuckets())\n .Observe(0.025);\nFollow these conventions:
prefix metrics with hmnd_<subsystem>_
use _total for counters
use _seconds for durations
use snake_case
prefer stable subsystem prefixes such as hmnd_wbc_*, hmnd_vision_*, hmnd_policy_*
Current convention is:
91xx for host daemons
92xx for robot nodes
The WBC node exposes loop health metrics on port 9202.
Relevant metrics include:
hmnd_wbc_controller_period_seconds
hmnd_wbc_compute_time_seconds
hmnd_wbc_compute_overrun_total
hmnd_wbc_compute_overrun_ratio
hmnd_wbc_commands_received_total
Example from whole_body_controller_interface.py:
self._metrics_controller_period = hmnd_metrics.histogram(\n "hmnd_wbc_controller_period_seconds",\n "Observed execution period for the WBC control loop.",\n)\nself._metrics_compute_time = hmnd_metrics.histogram(\n "hmnd_wbc_compute_time_seconds",\n "WBC loop compute time in seconds (before sleep).",\n)\nself._metrics_compute_overrun_total = hmnd_metrics.counter(\n "hmnd_wbc_compute_overrun_total",\n "Number of WBC control cycles that exceeded the period budget.",\n)\nself._metrics_compute_overrun_ratio = hmnd_metrics.gauge(\n "hmnd_wbc_compute_overrun_ratio",\n "Ratio of overrun cycles to total cycles in the last metrics window.",\n)\n
This is a good pattern because it captures:
the loop period distribution
the compute-time distribution
the cumulative failure count
the recent health ratio
That gives much more operational value than trying to continuously ship dense control internals.
hmnd_vision instrumentationhmnd_vision uses the C++ metrics helpers and exposes on port 9201.
Relevant metrics include:
hmnd_vision_topic_published_total
hmnd_vision_topic_publish_rate_hz
hmnd_vision_topic_publish_period_seconds
hmnd_vision_poll_delay_seconds
Example from hmnd_vision.cpp:
static MetricsFamilies families{\n .publish_counter = &hmnd::metrics::CounterFamily(\n "hmnd_vision_topic_published_total",\n "Number of compressed image messages published per topic."),\n .publish_rate = &hmnd::metrics::GaugeFamily(\n "hmnd_vision_topic_publish_rate_hz",\n "Observed publish rate per topic."),\n .publish_period = &hmnd::metrics::HistogramFamily(\n "hmnd_vision_topic_publish_period_seconds",\n "Observed publish period per topic."),\n .poll_delay = &hmnd::metrics::HistogramFamily(\n "hmnd_vision_poll_delay_seconds",\n "Duration spent waiting for each frame.")\n};\nIt then binds metrics per topic/device label pair and records observations for each stream.
This is another good pattern:
stable labels
bounded series count
direct operator value
enough information to see rate collapse, jitter, and frame acquisition delay
For local visualization, use the shared local observability stack in:
hmnd_infra/observability/local
Key files:
hmnd_infra/observability/local/vector/vector.yaml
hmnd_infra/observability/local/grafana/dashboard.json
The local setup mirrors the deployed structure:
Vector scrapes local endpoints
Vector remote-writes to local Mimir
Grafana queries local Mimir / Loki
Useful local docs:
ros/utils/hmnd_metrics/docs/LOCAL_METRICS_TESTING.md
ros/utils/hmnd_metrics/docs/CONTROLLER_OVERRUN_MONITORING.md
Start from the operator question, not from the code structure.
Ask:
what failure or degradation do we want to detect?
what metric will move when that happens?
do we care about current value, count, or distribution?
what labels are truly needed to segment the issue?
A small, high-value set beats a large noisy set.
In Python:
hmnd_metrics.initialize_exporter("0.0.0.0:92XX")\nIn C++:
hmnd::metrics::InitializeExporter("0.0.0.0:92XX");\nUse a unique 92xx port.
Prefer:
counters for totals
gauges for current state
histograms for timings
Be strict about labels.
Update:
local dev: hmnd_infra/observability/local/vector/vector.yaml
deployed robots: platforms/common/hmnd-shipping-robot/etc/hmnd/vector/robot.yaml
Add a new endpoint line such as:
- "http://127.0.0.1:9213/metrics" # my_new_node\n
Check locally on the process host:
curl -s http://127.0.0.1:9213/metrics | head\n
If the metric is not present here, Grafana will never see it.
Check in Grafana Explore or via the Prometheus API that the series exists.
Simple queries:
hmnd_my_node_requests_total\n
rate(hmnd_my_node_requests_total[5m])\n
Once the metric is visible in Mimir, create Grafana panels with PromQL.
A few patterns are especially useful for robots.
rate(hmnd_wbc_command_missed_cycles_total[5m])\n
histogram_quantile(0.95, sum by (le) (rate(hmnd_wbc_compute_time_seconds_bucket[5m])))\n
sum by (le) (rate(hmnd_wbc_command_publish_period_seconds_bucket[$__rate_interval]))\n
Use the metadata labels Vector adds:
hmnd_wbc_compute_overrun_ratio{\n hmnd_robot_id="$robot",\n hmnd_robot_type="$platform",\n hmnd_robot_target="$target"\n}\nThis is why the Vector enrichment layer matters so much. It gives consistent fleet-level filtering without requiring every exporter to know robot identity.
The local dashboard already contains practical examples for both vision and WBC.
Examples include:
vision poll delay heatmaps from hmnd_vision_poll_delay_seconds_bucket
WBC period heatmaps from hmnd_wbc_command_publish_period_seconds_bucket
WBC percentile panels with histogram_quantile(...)
WBC rate panels from histogram _count
Those queries are a good starting point when building new dashboards.
instrument at process boundaries and health boundaries
record distributions for timing-sensitive paths
keep labels bounded and semantically useful
expose one stable exporter per process
let Vector handle shipping, buffering, and metadata
use logs for sparse explanations and metrics for ongoing health
publishing dense high-rate control state as centralized metrics
adding unbounded labels
emitting a separate metric name per entity when labels would do
relying on raw cumulative counters without rate queries
using gauges where a histogram is needed to understand jitter or tail latency
Useful implementation references:
hmnd_robot/ros/utils/hmnd_metrics/README.md
hmnd_robot/ros/utils/hmnd_metrics/hmnd_metrics/__init__.py
hmnd_robot/ros/control/whole_body_controller/whole_body_controller/whole_body_controller_interface.py
hmnd_robot/ros/perception/hmnd_vision/src/hmnd_vision.cpp
hmnd_robot/platforms/common/hmnd-shipping/etc/hmnd/vector/common.yaml
hmnd_robot/platforms/common/hmnd-shipping-robot/etc/hmnd/vector/robot.yaml
hmnd_infra/observability/local/vector/vector.yaml
hmnd_infra/observability/local/grafana/dashboard.json
The intended workflow is:
instrument a node with hmnd_metrics
expose a local /metrics endpoint on a stable 92xx port
add that endpoint to the Vector scrape config in hmnd-shipping
let Vector enrich with robot metadata and ship to Mimir/Loki
build Grafana panels on top of the resulting series
If we keep following that pattern, we get observability that is:
always on
cheap enough to run fleet-wide
rich enough to debug most operational problems
simple enough that adding new instrumentation stays routine
Owner: Cody Griffin + Oleg Sinavski
Date: April 1, 2026
Status: Draft \u2014 for review
Enable engineers to hand off a task to a cloud coding agent via Slack or GitHub, have it run autonomously 24/7 in our own infrastructure (EKS DevPod with GPU + /mnt/shared), and steer it from Slack or GitHub review comments without needing to clone the repo or touch a terminal.
\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 ENTRY POINTS \u2502\n\u2502 Slack (@claude-agent do X) \u2502 GitHub PR comment / CI fail \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2534\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502 \u2502 webhook\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 CONTROLLER (Alphaclaw on EKS \u2014 extend existing) \u2502\n\u2502 \u2022 Parses task + context from Slack thread or GitHub event \u2502\n\u2502 \u2022 Manages pod lifecycle (helm install \u2192 monitor \u2192 teardown) \u2502\n\u2502 \u2022 Routes CI/review feedback back into running agent sessions \u2502\n\u2502 \u2022 Persists session state (SQLite + markdown on /mnt/shared) \u2502\n\u2502 \u2022 Exposes simple REST API for CloudCLI web UI \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u252c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n \u2502 helm upgrade --install \u2502 status / logs\n\u250c\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510 \u250c\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u25bc\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2510\n\u2502 AGENT POD (EKS DevPod) \u2502 \u2502 CloudCLI (web UI) \u2502\n\u2502 \u2022 Headless DevPod variant \u2502 \u2502 \u2022 List / attach sessions \u2502\n\u2502 \u2022 claude CLI (Agent SDK) \u2502 \u2502 \u2022 View turn-by-turn log \u2502\n\u2502 \u2022 /workspace (Longhorn PVC) \u2502 \u2502 \u2022 Inject steering message \u2502\n\u2502 \u2022 /mnt/shared (FSx Lustre) \u2502 \u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2518\n\u2502 \u2192 datasets, checkpoints \u2502\n\u2502 \u2192 agent memory (markdown) \u2502\n\u2502 \u2022 ANTHROPIC_API_KEY from k8s \u2502\n\u2502 secret \u2502\n\u2502 \u2022 Optional GPU (MIG slice) \u2502\n\u2514\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500\u2500-\u2518\n
Session identity: one Slack thread = one agent session = one pod. Replies in the thread steer the agent. The thread timestamp (thread_ts) is the primary key throughout.
Component | Status | Location |
|---|---|---|
DevPod Helm chart | \u2705 Exists |
|
| \u2705 Exists | PVC |
Longhorn PVC workspace | \u2705 Exists | Per-release, auto-remounted |
Alphaclaw controller | \u2705 Running on EC2, needs migration | PR #2996, |
Slack bot + session management | \u2705 In Alphaclaw |
|
Multi-backend support (Claude/Codex) | \u2705 In Alphaclaw | PR #2996 |
ANTHROPIC_API_KEY (enterprise) | \u2705 Just activated | Karen/IT \u2014 needs k8s secret |
What needs to be built: headless agent pod variant, pod lifecycle management in the controller, GitHub webhook handler, CloudCLI deployment.
Owner: Mustafa / Karen | Est: ~1 day
1. Create anthropic-api-key k8s secret on EKS
kubectl create secret generic anthropic-api-key \\\n --from-literal=ANTHROPIC_API_KEY=<key> \\\n --namespace=agent-system\n
2. Create github-token k8s secret
kubectl create secret generic github-token \\\n --from-literal=GITHUB_TOKEN=<pat-with-repo-scope> \\\n --namespace=agent-system\n
3. Create agent-system namespace (separate from stray-dog where Mustafa's monitoring agent lives).
Owner: Mustafa (k8s) + Atindra (image) | Est: 2\u20133 days
The goal is a pod that starts, runs claude headlessly, and exits cleanly. It is a new Helm values profile on top of the existing DevPod chart \u2014 not a new chart.
1a. New Docker image: hmnd-agent
FROM hmnd-interactive-base:latest\n\n# Install Claude Code CLI\nRUN npm install -g @anthropic-ai/claude-code\n\n# Agent entrypoint \u2014 waits for task file, runs claude, exits\nCOPY agent-entrypoint.sh /usr/local/bin/agent-entrypoint\nRUN chmod +x /usr/local/bin/agent-entrypoint\n\nENTRYPOINT ["/usr/local/bin/agent-entrypoint"]\n
agent-entrypoint.sh:
#!/bin/bash\nset -e\n\nTASK_FILE=/workspace/task.json\nMEMORY_DIR=/mnt/shared/agents/${SESSION_ID}/memory\nTRANSCRIPT=/mnt/shared/agents/${SESSION_ID}/transcript.jsonl\n\nmkdir -p "$MEMORY_DIR"\n\ncd /workspace/hmnd\n\nclaude \\\n --dangerously-skip-permissions \\\n --output-format stream-json \\\n --max-turns 100 \\\n "$(jq -r .prompt $TASK_FILE)" \\\n | tee -a "$TRANSCRIPT"\nNote on repo freshness: The Docker image contains a clone at
mainat build time. For Phase 1 this is acceptable \u2014 the agent's first tool call cangit pullto get latest. Incremental build caching comes in the Appendix.
1b. Helm values override: values-agent.yaml
hmnd-interactive-base:\n workload:\n purpose: agent\n image:\n repository: <ecr>/hmnd-agent\n tag: latest\n gui:\n enabled: false\n ssh:\n enabled: false\n resources:\n requests:\n cpu: "2"\n memory: "8Gi"\n limits:\n cpu: "4"\n memory: "16Gi"\n gpu:\n count: 0\n env:\n - name: ANTHROPIC_API_KEY\n valueFrom:\n secretKeyRef:\n name: anthropic-api-key\n key: ANTHROPIC_API_KEY\n - name: GITHUB_TOKEN\n valueFrom:\n secretKeyRef:\n name: github-token\n key: GITHUB_TOKEN\n storage:\n shared:\n mountPath: /mnt/shared\n pvcName: fsx-lustre-shared-pvc\n lifecycle:\n idleTimeoutSeconds: 7200\n
Deploy a single agent pod:
helm upgrade --install agent-<session_id> ./eks \\\n -f values-agent.yaml \\\n --set hmnd-interactive-base.user.name=agent-<session_id> \\\n --set hmnd-interactive-base.env[0].value=$SESSION_ID \\\n --namespace agent-system \\\n --wait\n
1c. Write task.json into the pod
kubectl exec -n agent-system \\\n $(kubectl get pod -l app.kubernetes.io/instance=agent-<session_id> \\\n -n agent-system -o jsonpath='{.items[0].metadata.name}') \\\n -- bash -c "echo '$TASK_JSON' > /workspace/task.json"\nOwner: Atindra (Alphaclaw) + Mustafa (k8s) | Est: 3\u20135 days
2a. Migrate Alphaclaw to EKS
apiVersion: v1\nkind: ServiceAccount\nmetadata:\n name: alphaclaw\n namespace: agent-system\n---\napiVersion: rbac.authorization.k8s.io/v1\nkind: Role\nmetadata:\n name: alphaclaw-role\n namespace: agent-system\nrules:\n - apiGroups: [""]\n resources: ["pods"]\n verbs: ["get", "list", "watch", "create", "delete"]\n - apiGroups: ["apps"]\n resources: ["deployments"]\n verbs: ["get", "list"]\n
2b. New Slack commands
Command | Behaviour |
|---|---|
| Spawns new agent pod, returns thread with session link |
| Lists running agent pods + their Slack thread links |
| Terminates the agent pod for this thread |
| Injects message into the running claude session |
Reply in thread | Same as steer \u2014 any reply is forwarded to the agent |
2c. Session lifecycle (pseudocode)
async def handle_slack_message(event):\n thread_ts = event.get("thread_ts") or event["ts"]\n session_id = make_session_id(thread_ts)\n\n if is_new_task(event):\n task = parse_task(event["text"])\n pod_name = await spawn_agent_pod(session_id, task, gpu=needs_gpu(task))\n db.create_session(session_id, thread_ts, event["channel"], pod_name)\n await slack.post(\n channel=event["channel"], thread_ts=thread_ts,\n text=f"\ud83e\udd16 Agent `{session_id}` starting on EKS...\n"\n f"View live: https://cloudcli.lan.thehumanoid.ai/sessions/{session_id}"\n )\n asyncio.create_task(stream_agent_output(session_id, thread_ts, event["channel"]))\n elif is_steering_message(event):\n await inject_into_agent(session_id, event["text"])\n\n\nasync def stream_agent_output(session_id, thread_ts, channel):\n transcript_path = f"/mnt/shared/agents/{session_id}/transcript.jsonl"\n last_posted = 0\n async for turn in tail_jsonl(transcript_path):\n if turn["type"] == "assistant" and turn["turn_index"] - last_posted >= 5:\n summary = summarise_turns(turns_since(last_posted))\n await slack.post(channel=channel, thread_ts=thread_ts, text=summary)\n last_posted = turn["turn_index"]\n if turn["type"] == "result":\n await post_completion(session_id, channel, thread_ts, turn)\n await teardown_pod(session_id)\n2d. Session state on /mnt/shared
/mnt/shared/agents/\n <session_id>/\n task.json \u2190 initial task written by controller\n transcript.jsonl \u2190 streaming claude output\n memory/\n context.md\n progress.md\n pr_url.txt \u2190 written by agent when PR is opened\n
Owner: Atindra | Est: 2\u20133 days
3a. GitHub webhook receiver
Register at https://alphaclaw.lan.thehumanoid.ai/webhooks/github for events: Check suites, Pull request review comments, Pull request reviews.
async def on_ci_complete(payload):\n pr_number = payload["check_suite"]["pull_requests"][0]["number"]\n session_id = db.lookup_session_by_pr(pr_number)\n if payload["check_suite"]["conclusion"] == "failure":\n failed_logs = await fetch_failed_ci_logs(pr_number)\n await inject_into_agent(session_id,\n f"CI failed on PR #{pr_number}:\n{failed_logs}
Please fix and push.")\n\nasync def on_review_comment(payload):\n pr_number = payload["pull_request"]["number"]\n session_id = db.lookup_session_by_pr(pr_number)\n comment = payload["comment"]["body"]\n author = payload["comment"]["user"]["login"]\n path = payload["comment"]["path"]\n line = payload["comment"]["line"]\n await inject_into_agent(session_id,\n f"Review comment from @{author} on `{path}` line {line}:\n> {comment}\n\nPlease address this.")\n3b. AGENTS.md skill for coding agents
# .agents/skills/agent-substrate/SKILL.md\n\nWhen working as a cloud coding agent:\n1. Always create a feature branch: git checkout -b agent/<session_id>/<short-description>\n2. Push and open a PR early (draft OK): gh pr create --draft --title "..." --body "..."\n3. Write the PR URL to /mnt/shared/agents/$SESSION_ID/pr_url.txt\n4. After each significant change, run: pixi run build && pixi run test\n5. Address CI failures and review comments before marking PR ready for review\n6. When done, post summary to Slack and mark PR ready: gh pr ready\n
Owner: Anyone | Est: half a day
CloudCLI is open source (GPL-3, github.com/siteboon/claudecodeui) and deploys in one command.
# Expose via existing internal ingress \u2192 https://cloudcli.lan.thehumanoid.ai\nkubectl apply -f cloudcli-deployment.yaml -n agent-system\n
At the end of Phase 4, you get:
Hand off work from Slack: @claude-agent implement the feature described in JIRA-1234 \u2192 agent pod spins up on EKS, implements, opens a PR
Steer via Slack thread: reply in the same thread to redirect, provide context, or stop the agent
CI/review loop: when CI fails or a reviewer comments, the agent fixes automatically
Access to your infra: /mnt/shared (FSx Lustre \u2014 datasets, checkpoints), optional GPU via --set gpu.count=1
Web UI: live session view at cloudcli.lan.thehumanoid.ai
Parallel agents: no limit \u2014 each Slack thread is an independent pod
Not yet (see Appendix): build caching, Codex as alternative harness, IsaacLab access from agent pods.
Phase | Work | Owner(s) | Days |
|---|---|---|---|
0 | Secrets, namespace, service account | Mustafa + Karen | 1 |
1a |
| Atindra | 1\u20132 |
1b/c | Helm values override + pod deploy/task | Mustafa | 1\u20132 |
2a | Migrate Alphaclaw to EKS | Atindra + Mustafa | 1\u20132 |
2b/c | Slack commands + session lifecycle | Atindra | 2\u20133 |
2d |
| Atindra | 0.5 |
3a | GitHub webhook receiver | Atindra | 1\u20132 |
3b | AGENTS.md additions | AI Captains | 0.5 |
4 | CloudCLI deployment | Anyone | 0.5 |
Total | ~2 weeks |
Phases 0\u20132 can run in parallel (Mustafa on infra, Atindra on controller).
Problem: Every agent pod runs pixi run build from a fresh clone. For the hmnd monorepo this is slow \u2014 C++ compilation, pixi env setup \u2014 and wastes GPU runner capacity.
Solution: Bazel Remote Cache on S3
The Q2 plan already calls for a Bazel migration (Ricardo + Tobias). The agent substrate is a strong forcing function to prioritise this.
# In .bazelrc\nbuild --remote_cache=https://cache.lan.thehumanoid.ai\nbuild --remote_cache_compression=zstd\nbuild --jobs=AUTO\n
Backend options (ordered by effort):
S3 bucket + nginx proxy \u2014 simplest, ~1 day to set up
BuildBuddy OSS \u2014 open source, dashboard showing cache hits/misses, Helm chart available
Bazel Remote Execution on EKS \u2014 longer term; agents submit build actions to an RBE cluster
For non-Bazel builds during transition: sccache with S3 backend works today with zero code changes.
Expected impact: ~5\u201310\u00d7 speedup on repeated builds. Agents on different pods share the same cache.
Problem: The Docker image contains a clone at main at build time. Agents always start slightly behind main and have to git pull, which on the hmnd monorepo (LFS, submodules) is slow and fragile.
Solution: Pre-bake a daily snapshot image in CI, triggered at 3am daily and on pushes to main. The controller always uses :latest \u2014 agents start with a repo at most 24 hours behind main.
Parameterize the harness in the Helm values:
# values-agent-codex.yaml\nenv:\n - name: AGENT_HARNESS\n value: "codex"\n - name: OPENAI_API_KEY\n valueFrom:\n secretKeyRef:\n name: openai-api-key\n key: OPENAI_API_KEY\n
# agent-entrypoint.sh\ncase "$AGENT_HARNESS" in\n claude) claude --dangerously-skip-permissions --output-format stream-json ... ;;\n codex) codex --approval-mode full-auto ... ;;\nesac\n
The session state layout is harness-agnostic. Pre-requisite: OpenAI contract.
Wire up the Sim-as-a-Service spec as an MCP server. Rather than running sim inside the agent pod, the agent submits a sim job via MCP and polls for results:
@mcp.tool()\nasync def submit_sim_job(branch: str, config: str) -> dict:\n response = await httpx.post("http://sim-service.lan.thehumanoid.ai/jobs",\n json={"branch": branch, "config": config})\n return {"job_id": response.json()["job_id"]}\n\n@mcp.tool()\nasync def get_sim_results(job_id: str) -> dict:\n response = await httpx.get(f"http://sim-service.lan.thehumanoid.ai/jobs/{job_id}")\n return response.json()\nThe agent never needs a GPU in its own pod for sim-validation tasks.
Same pattern as sim: expose Prefect training jobs and academic benchmark runs (Libero, IsaacLab evals) as MCP tools. Oleg's Agentification project has already validated this pattern. Standardise MCP tool signatures across sim, training, and evals so agents use a consistent interface regardless of backend.
Agent pods should sit between interactive DevPods and batch training in the Kubernetes Workload Priorities spec:
priorityClassName: agent-workload # new class, between devpod and training\n
This ensures agent pods don't starve human DevPod users, and can be preempted by training jobs if GPU contention spikes.
" }, { "id": "1818624031", "title": "Cloud Agents: Options Analysis", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1818624031", "last_modified": "2026-04-01T10:21:48.440Z", "created_at": "2026-04-01T10:21:48.440Z", "body_html": "Prepared for: Cody Griffin | Date: April 1, 2026
Based on conversations across #agentic-ai-practices, #team_eng-software, the Agentification project doc, and Q2 planning notes, the team is at an inflection point. Several parallel threads are converging:
What's working well:
Cursor Cloud Agents are in active use (Atindra set up snapshot environments, Andy is using Graphite/Cursor agents for PR generation and review)
Alphaclaw (Atindra's agent framework, spun off from nanoclaw) is running on a data-flywheel EC2 instance, supporting Codex, Claude Code, and OpenCode backends with Slack integration
Oleg's Agentification project has MVPs for overnight agents doing multi-hour training cycles, dataset analysis, and eval runs
DevPods on EKS are operational with PVC-backed persistent volumes (Longhorn), unison-based sync, and Cursor remote development
AGENTS.md and Skills are standardized across Cursor/Claude/Codex with cross-agent compatibility
AI Captains program is established with monthly syncs
Claude Enterprise account just activated; Anthropic contract signed; OpenAI coming next
Key pain points identified in Slack:
No self-hosted cloud runner pool yet \u2014 "single workers are buggy" (Oleg)
Cursor Cloud Agents don't use our Anthropic API keys
GPU runner contention on EKS (9 \u2192 30 MIG partitions, still hitting I/O limits)
Build caching is missing \u2014 Boris noted "a good straightforward fix will be a distributed build cache"
Submodules and LFS cause constant friction with agents
No persistent data access (no mnt/shared, HF rate limits, confused agents)
Review remains the bottleneck (200+ open PRs in hmnd)
Requirements:
Cursor Cloud Agent-like experience (background tasks, Slack integration, steering/monitoring)
Hosted on your infra (EKS cluster with GPUs and NFS)
Build caching
Review/steering capabilities
Documentation/speccing and planning
BYOK: at minimum Anthropic + OpenAI model support
What it is: Cursor's official self-hosted offering. Workers run on your infra (outbound HTTPS to Cursor's cloud for orchestration). Cursor provides a Helm chart and Kubernetes operator for fleet management.
Requirement | Coverage |
|---|---|
Cloud Agent experience | Native \u2014 same UX as Cursor Cloud Agents |
Self-hosted on your infra | Yes \u2014 workers run in your k8s, code never leaves your network |
Build caching | Not built-in |
Review/steering | Slack integration, attach to running agent from Cursor desktop, PR creation |
Documentation/planning | Skills, AGENTS.md, rules all supported |
BYOK API keys | Partial \u2014 Cursor controls model routing |
Tradeoffs:
Pro | Con |
|---|---|
Fastest path \u2014 team is already on Cursor | Cursor controls inference layer \u2014 limited BYOK flexibility |
Helm chart + k8s operator handles scaling | Vendor lock-in to Cursor's orchestration plane |
Slack integration works out of the box | Oleg found it "clumsy and buggy" |
Skills/rules/AGENTS.md all carry over | Cost: Cursor Pro/Business per seat + infra |
Graphite integration for stacked PRs | Doesn't solve the build cache problem natively |
Best for: Getting to a working state fast with minimal new tooling.
What it is: Anthropic's Agent SDK lets you build and host your own agent infrastructure. You control everything \u2014 the orchestration, sandboxing, model selection, and tooling.
Requirement | Coverage |
|---|---|
Cloud Agent experience | You build the UX \u2014 SDK handles agent lifecycle, tool execution, sessions |
Self-hosted on your infra | Full control \u2014 runs in containers on your k8s |
Build caching | You integrate it |
Review/steering | Build custom review flows, PR bots, Slack integration via MCP |
Documentation/planning | Full control over prompt engineering, skills, context management |
BYOK API keys | Native Anthropic support; OpenAI via their compatible API endpoint |
Tradeoffs:
Pro | Con |
|---|---|
Full BYOK \u2014 use any model via API keys you control | Significant build effort to reach Cursor-like DX |
No vendor lock-in on orchestration | You own maintenance and upgrades |
Alphaclaw is already a working prototype | Atindra is the primary maintainer \u2014 bus factor concern |
Works with your existing NFS/PVC setup | Need to build monitoring/observability |
Best for: Maximum flexibility and control. Ideal if you want agents deeply integrated with your specific workflows.
What it is: A fully open-source self-hosted remote coding agent platform built on k3s + Kata Containers + Cloud Hypervisor microVMs + Tailscale.
Requirement | Coverage |
|---|---|
Cloud Agent experience | iOS app + web interface, session pause/resume |
Self-hosted on your infra | 100% self-hosted, open source |
Build caching | Not built-in |
Review/steering | Basic \u2014 agents run in sandboxed microVMs |
BYOK API keys | Yes \u2014 direct Anthropic SDK integration + Ollama for local models |
Tradeoffs:
Pro | Con |
|---|---|
Fully open source \u2014 no vendor dependency | Young project \u2014 less battle-tested |
MicroVM isolation is best-in-class security | k3s-native; adapting to EKS may be nontrivial |
Snapshot/restore is powerful for agent experimentation | No Slack integration built-in |
Session pause/resume saves compute | Requires nested virtualization support |
Best for: Teams that want maximum isolation and open-source control, and are willing to invest in customization.
What it is: Use Cursor Self-Hosted Cloud Agents as the primary developer-facing coding agent, while running Claude Agent SDK-based agents (via Alphaclaw) for specialized workflows that need deeper integration (training pipelines, overnight eval loops, robot mock runs, documentation generation).
This is essentially what's already emerging organically: Cursor Cloud Agents for day-to-day coding + Alphaclaw/Oleg's agents for overnight training/eval loops + the #agent-requests Slack channel for on-demand tasks.
What you'd need to do:
Deploy Cursor self-hosted (Phase 1 \u2014 fast)
Productionize Alphaclaw on EKS with NFS (Phase 2 \u2014 medium effort)
Add shared build cache layer (Phase 2)
Build unified monitoring dashboard (Phase 3)
Standardize the agent-request interface across both systems
Tradeoffs:
Pro | Con |
|---|---|
Meets devs where they are (Cursor) while unlocking specialized agents | Two systems to maintain |
BYOK for specialized agents (training, eval, planning) | Model routing still Cursor-controlled for coding tasks |
Incremental \u2014 ship Cursor self-hosted fast, iterate on SDK agents | Potential UX fragmentation |
Leverages existing work (Alphaclaw, DevPods, skills) | Need clear ownership boundaries |
Start with Option 4 (Hybrid), phased:
Phase 1 (This week): Enable Cursor Self-Hosted Cloud Agents on your EKS cluster. Atindra has already requested dashboard access, Mustafa volunteered for k8s setup.
Phase 2 (Next 2\u20134 weeks): Productionize the Claude Agent SDK path. Merge and harden Alphaclaw on EKS with NFS mounts for shared state. This covers overnight agents, training loops, and full BYOK needs.
Phase 3 (Q2): Add the shared infrastructure layer \u2014 distributed build cache, unified monitoring, and a Slack-based steering interface that works across both systems.
Dimension | Cursor Self-Hosted | Claude Agent SDK | Netclode | Hybrid (Recommended) |
|---|---|---|---|---|
Time to production | Days | Weeks | Weeks | Days (P1) + Weeks (P2) |
BYOK (Anthropic) | Limited | Full | Full | Full (via SDK layer) |
BYOK (OpenAI) | Limited | Via API | Via Ollama/API | Full (via SDK layer) |
Build caching | BYO | BYO | BYO | BYO (shared) |
Review/steering | Good | Build your own | Basic | Best of both |
IDE integration | Native (Cursor) | None | App-based | Cursor for coding |
Vendor lock-in | Medium | Low | None | Low-Medium |
Fits existing workflow | High | Medium | Low | High |
Internal discussions: #agentic-ai-practices, #team_eng-software, #team_data-and-compute-platform
Key Slack threads: Self-hosted cloud agents (Mar 25\u201331), AI Captains sync (Mar 2), Agent framework selection (Mar 16), Claude Enterprise activation (Mar 31)
Google Drive docs: Agentification project (Oleg Sinavski), 2026 Q2 planning notes, Agentic Development Workflow Proposal (Andy Park)
" }, { "id": "1803386953", "title": "Potential Deployment MVP for Symphony", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1803386953", "last_modified": "2026-03-26T19:53:43.004Z", "created_at": "2026-03-26T18:38:53.924Z", "body_html": "This is a rough attempt to get some level of actionable scope for the Symphony demo, which I believe the core premise is this iterative specification of a workflow. There is a lot of ancillary scope in this document, various objects, capabilities, behaviors - but the real guts is being able to build up complicated workflows incrementally by natural language. First, some realities:
The map/demo area is chaotic - this is because we will be doing maintenance, people around, data collection, and everyone will be fiddling with the \u201cset\u201d up until the last moment
The policy isn\u2019t as good as we want it to be - there will be limitations about posing, locations, etc and we will be unlikely to get sufficient reliability on the behaviors covered in the scope doc on day 1
The UI will not be very friendly and the experience won\u2019t be very smooth
So given this - what do we do?
Single Shot Deployment
I think single shot deployment can address the first point above by making it trivial to deal with changes to the environment. But this isn\u2019t enough. This operation can give us a set of interesting tags and some default \u201chappy path\u201d behaviors to get started immediately, but it falls short in some serious ways:
Making a robust workflow in a single shot is much harder for the operator than for the robot (though the solution is fairly unreliable even for the robot if the instruction is good)
The inferred map tags are not at all semantically relevant. It knows you ran some policy at some point, but it doesn\u2019t know that its the conveyor or whatever
With a single trace you can\u2019t do anything other than trivial sequences/loops - no branches or conditions
With a demonstration, your workflow can\u2019t easily be impacted by \u201cunobservable\u201d stuff - like \u201cwaiting for a webhook\u201d or similar
What single shot deployment DOES do is break the need to launch with a fixed workflow. You can launch empty, and then have the robot doing something autonomously. Localization is a challenge with this runtime reloading though\u2026 A major issue with the current scheme is that because it re-uses the slam_toolbox used for our normal mapping stack, it requires data to be replayed through the stack. This was done \u201coffline\u201d from recorded mcap files for analysis and rapid iteration, but this also complicates things a bit and may not be necessary for the goals we have here.
Incremental Loop
Given some baseline, we want to be able to make edits to the workflow - at runtime. So not only do we need to load a workflow at runtime, we need to be able to reload that workflow at runtime. And we need to be able to do this while maintaining execution state (ideally) for rapid iteration. It isn\u2019t well-understood just how \u201cbig\u201d these edits can reliably be (or what the tradeoffs are between small edits maybe modifying a transition or inserting a new state vs larger edits which rework significant portions of the workflow). It may also be necessary to extend the map with new tags, which sort of blurs the line between what\u2019s done with the workflow and what\u2019s done with the map. This further bolsters the idea of potentially moving to a more \u201conline\u201d mapping system, but it could be challenging to reliably place tags if the map shifts or if localization is unreliable - especially in environments that can rapidly change over time.
So - we want to be able to get some baseline workflow (call it the 80% deployment), and then iterate the rest of the way incrementally via the agent loop using high-level intent/natural language. We don\u2019t care about the interface at this point - just this core loop. We could \u201clock\u201d the workflow or have some \u201cscene graph\u201d provided a priori, but this could be frustrating as the environment evolves, and its likely that this data will be stale. On the other hand, single shot deployment has its own downsides (tag names, unreliability, extending tags, etc).
Example Scenario
I\u2019m thinking something like we create a looping workflow for moving a stack of totes to the conveyor. This is easy to do with a looped single shot:
idle
navigate to stack
pick tote
navigate to conveyor
place tote
<loop>
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1803386953/Untitled%20Diagram-1774554199665.drawio.png?api=v2&version=2\")
Obviously this has a flaw - at some point the stack is empty and the robot gets stuck. We should be able to communicate something like: \u201cwhen you get a Destack work item, it will contain the total number of totes expected to move. stop after that many totes\u201d (or something like this). The robot can wait in some idle position, until this work item is injected into the workflow (via agent or cmdline or whatever) and then the robot will attempt to move that many pallets from the stack.
The good:
clear \u201cMVP\u201d for single-shot deployment that demonstrates the weaknesses of that approach
avoids need for perception or a priori knowledge of the scene (fragile) - instead we can inject this via a work-item/token in the coordinator (as we\u2019ve done for all our previous PoCs)
shows incremental addition of unobservable stuff
The bad:
need some way to trigger the workflow / inject the work
in \u201cediting mode\u201d it would be awesome if the agent could just trigger stuff for us. \u201chey, there\u2019s a stack of 5 totes on the pallet - time to move\u201d \u2192 injects Destack { count: 5 } to PendingStacks (or whatever)
need to keep track of counts as facts in the blackboard
new workflow extensions to decrement counters could make this pretty slick and easy/validate
map tags will be unintuitive/meaningless unless we have some way to re-name things
this is something we can cheat on a bit. for example, when we run policy we could provide a semantic label or something to back-channel this into the workflow/map as a tag name
The ugly:
we need to somehow patch the workflow and get the robot into some idle state online during/after editing - possible, but definitely challenging
I think this is a relatively simple loop that exposes the \u201cheart\u201d of Symphony without any major risk on the policy side. You can rapidly get some baseline given a configuration of the environment, and then iterate through the edge-cases to get a more sophisticated workflow. Over time as we get more capabilities, we can extend. this to get more and more complexity, but I think this is a good viable target for an initial integrated deliverable.
Anti-Goals
The above, even though its relatively simple, is probably going to be very difficult and full of weird shit making it harder than expected (beyond the more obvious stuff I\u2019ve identified). We\u2019ll want to maximize the amount of time we have to identify these issues without getting caught up in things that we don\u2019t need to address right now. For example, this minimal scenario is a single robot - so we don\u2019t need to worry about the cloud instances of these services, nor do we really need to persist things to disk. Having to redeploy repeatedly when we wipe switchboard could be annoying and frustrating, but its also a great forcing function to focus on the end-to-end loop and get many many many iterations trying this out.
We should resist temptation to address the details until all this scenario starts to come together. Once we can SOMETIMES make the above scenario work, we can start looking at the major challenges and prioritize a backlog to address. There are a number of improvements to agentic editing captured in Studio: Agentic Workflow Updates that could improve undo/history, context management, traceability of intent and the generation of human-level SOPs. These items too should all take a backseat to getting the basic loop above in place. Its 100 times easier to iterate on a working system than an incomplete system.
There will be some tough calls. I\u2019ll use the offline mapping as an example - its a big source of problems, but it kinda works. During the course of this, we may determine that a better approach is to use the slam toolbox lifelong mapping and just do tagging as we navigate in a map for deployment (rather than offline slam). Totally viable - many benefits. But it could be a 2-steps-back-3-steps-forward situation that we may not have time for. If we can get something working ASAP, we have plenty of time for these improvements. If we try and frontload these improvements, we risk not addressing unknown blockers and we may have enormous amounts of schedule crunch resulting in even MORE tech debt. On the other hand, if the offline mapping proves to be a blocker (like a real, nothing is going to happen, no workaround, no retry a few times) - then it must of course be addressed immediately in pursuit of the goal.
We all want a VERY reliable system - trust that these improvements will be prioritized. But the primary goal is to demonstrate something like the above scenario as soon as possible.
" }, { "id": "1637417000", "title": "AI Proxy", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1637417000", "last_modified": "2026-03-23T10:23:27.567Z", "created_at": "2026-02-12T13:56:05.684Z", "body_html": "Robots need controlled access to LLM APIs without carrying provider API keys on-device. The right shape is a single HMND endpoint that robots can call with the same device certificate they already use for Backbone, while the service itself holds provider credentials centrally and attaches usage to a concrete device_id. We do not need a billing platform in the first version, but we do want enough attribution and observability to understand how shared upstream keys are being used across devices.
The simplest useful design is to build a separate AI proxy service as a sibling to Backbone and use lm-proxy as the provider translation layer inside it. The service should live under hmnd_services/ai_proxy, and the corresponding deployment should live under hmnd_infra/modules/ai_proxy_stack. It should sit behind the same ALB mTLS pattern already used for Backbone device endpoints, reuse the same SPIFFE-based device identity extraction logic, and authorize requests against the existing devices table.
This is a better fit than a heavier gateway solution because it stays close to the shape of the current HMND stack. We already have the hard identity and certificate lifecycle pieces. What we need is a small, understandable egress layer for model traffic. lm-proxy is a good fit for that because it is lightweight, Python-native, OpenAI-compatible, and supports the provider set we care about without bringing in a much larger control plane.
Backbone should stay the control plane for device identity, enrollment, renewal, and authorization data. The AI proxy should be a separate service because LLM traffic has different operational behavior. Requests are larger, slower, and more failure-prone than enrollment or session heartbeats. Provider-specific outages, timeouts, and streaming edge cases should not threaten the device control plane. Splitting the services keeps rollout and scaling independent while still reusing the same identity model.
This also keeps the architecture honest. Backbone answers "who is this device and is it allowed." The AI proxy answers "how do I safely forward this model request and measure what happened." If we blur those roles, we turn Backbone into both a control-plane service and a data-plane service, which is not a good trade.
The proxy should expose an OpenAI-compatible HTTP surface at https://ai.core.<env>.hmnd.ai/v1/.... The first version should support POST /v1/chat/completions, POST /v1/responses, POST /v1/embeddings, and GET /v1/models. That matches the main HTTP patterns already used in the repo and gives robot applications a stable contract even when the upstream provider changes.
OpenAI Realtime WebSocket support should not be part of the initial scope. The repo does contain Realtime usage today, but Realtime is a different transport and a different portability problem. Trying to solve it in the first delivery would make the whole project harder and slower. The HTTP path should land first, and Realtime should be considered separately if it remains necessary.
lm-proxylm-proxy is attractive here because it is smaller and simpler than LiteLLM while still covering the essential requirement: an OpenAI-compatible interface over multiple providers such as OpenAI, Anthropic, and Google/Gemini. It is easier to inspect, easier to embed, and less likely to push the design toward a full-featured gateway platform than a larger tool would.
The tradeoff is that it does not bring the same breadth of built-in spend management, policy controls, or ecosystem depth that LiteLLM does. That is real, but it is not the main requirement here. HMND already has a stronger notion of client identity than most general-purpose AI gateways because our clients are robots authenticated by mTLS device certificates. The important thing is not whether the proxy has a sophisticated tenant-key system of its own. The important thing is that the HMND edge can attach device_id to every request, keep provider credentials off the robot, and emit enough structured usage data to be operationally useful.
The AI path should reuse the existing Backbone client certificate, private key, and chain. We should not invent a second robot credential or a second renewal mechanism. hmnd-backboned already manages the certificate lifecycle and stores the artifacts locally. The AI proxy should simply consume the same artifacts.
To make that practical, the robot side should get a small shared mTLS client helper. That helper should know the standard cert, key, chain, and server trust paths, construct an HTTP client with mTLS enabled, and reload credentials after renewal. Since the current AI callers are Python, a Python helper is enough for the first version. Without that helper, each application will end up reimplementing the certificate-loading problem differently.
Some OpenAI-style SDKs still expect an api_key argument even when real authentication is done with mTLS. That is manageable. Those clients can be given a placeholder value while the server ignores bearer auth entirely and relies on mTLS as the real credential. The important point is that no provider API key should live on the robot.
The AI proxy should authorize requests by reading the Backbone devices table directly rather than calling Backbone on every request. Once device_id is extracted from the client certificate, the service should perform a direct lookup in the device registry. If the device exists, the request proceeds. If it does not, the service returns 403 and does not call the upstream provider.
This keeps the hot path simple and avoids turning Backbone into a dependency on every model request. The AI proxy should receive its own database credentials with only the permissions it needs. In the first version, that likely means read access to the devices table and optional insert access if we later decide to persist usage events in SQL.
Provider credentials should live in AWS Secrets Manager and be scoped per provider. The service should load them through its task role, cache them in memory, and never expose them in logs or responses. The first version should focus on OpenAI and Anthropic. Gemini can be added once the basic path is stable, which avoids forcing an early decision about whether to start with Gemini API keys or go directly to Vertex AI credentials.
We do want to understand how shared upstream keys are being used across devices, but we do not need a full billing engine to get there. The best split is structured logs for per-request detail and low-cardinality metrics for service health.
Each request should emit a structured log event containing device_id, robot_id, robot_type, provider, requested model, resolved upstream model, status code, latency, request size, and token usage if the provider returns it. That is enough to answer the main operational question: which devices are using which provider and model, and with what error and latency profile.
Metrics should stay aggregate. Prometheus counters and histograms should record request counts, latency, token totals, and error totals by provider and coarse model family. We should not put device_id into metric labels because that creates unnecessary cardinality. If queryable historical usage becomes important later, that is the point to introduce a small ai_usage_events table keyed by device_id. Logs plus aggregate metrics are enough for the first version.
The biggest limitation of this design is that lm-proxy is smaller and less feature-rich than LiteLLM. That is why it is attractive for this use case, but it also means we should not expect a full platform\u2019s worth of built-in spend tracking, policy controls, or broad provider edge-case handling. If the system eventually grows into a very wide AI gateway, the simpler choice may stop being the better one.
The second limitation is that OpenAI compatibility is practical rather than perfect. The common HTTP paths can be normalized well enough for day-to-day use, but provider-specific features will still leak through at the edges. The third limitation is that Realtime remains out of scope for the first version. The fourth is that cert reuse only becomes easy if we actually ship the shared robot-side client helper. Without that helper, the design remains sound but adoption becomes awkward.
The first pass should stand up the new ai_proxy service and Terraform stack, wire the ALB mTLS edge, authorize against the devices table, and forward OpenAI-compatible HTTP requests to OpenAI first. That proves the core identity, secret custody, and attribution model.
The second pass should add Anthropic through lm-proxy, introduce simple model aliasing, and start emitting the structured usage logs and aggregate metrics we want. Gemini can follow once the team still wants it and the basic path is stable.
The third pass should introduce the shared robot-side mTLS client helper and migrate the existing robot-side HTTP AI callers to the proxy. At that point, the design is not only correct but also easy for application code to consume.
The fourth pass should harden the service with request-size limits, model allowlists, and coarse rate limiting. If queryable historical usage by device becomes important after that, we can add a dedicated SQL usage table.
The specific plan is to build a separate AI proxy service that sits behind the existing ALB mTLS device plane, reuses the Backbone certificate path on the robot, exposes an OpenAI-compatible HTTP contract, and uses lm-proxy internally as the provider adapter. This is simpler than a LiteLLM-centered design, better aligned with the current HMND Python and ECS stack, and still gives us the important properties we care about: centralized credential custody, per-device attribution, and useful visibility into how shared provider keys are being used.
Right now, mapping and tagging are done \u201con the side\u201d, separate from our normal data collection or workflow execution. The Spec: Single-Shot Deployment MVP spec outlines a way to resolve this a bit in the context of inferring a workflow, but here we can also think more broadly about this problem in particular.
When we start up our mapping stack, we will begin building a map (via slam_toolbox/nav2) immediately. During this time, we can save map poses as \u201ctags\u201d. When we save the map, we get an occupancy and some metadata describing the transform from pixel to map space, along with some map space tags. In the MVP above, we assume we will generate a tag for any policies we infer via sockpuppet.
We\u2019ll assume we have an initial map + tags from something like the single-shot MVP above.
Incremental Mapping and Tagging
Once you have an initial workflow, you have a map and some tags. How do we evolve these over time? We should be able to put the robot back into \u201cdeployment mode\u201d for example. Do we just bring up SLAM and allow modification and save if they look good? Or do we re-record and re-generate maps? With the former, we need to run quite a bit of stuff at runtime that we haven\u2019t typically been running, and this could interfere with efforts to run only localization. If we do the latter, we risk major issues with the origin shifting around depending on how the data is collected.
We also need to support adding new tags, shifting tags or re-naming tags. New tags would be needed if we added a new station or step to an existing workflow for example. We can shift or move tags by simply saving a new tag with the same name. Renaming tags is a bit trickier\u2026
We may want to rename tags if we want to support a voice-based deployment. An auto-generated tag during workflow inference like \u201crun_policy:place_2" is a pretty shitty identifier to have to speak. The challenge with a rename is that there is an ambiguity around creating a new tag vs renaming an existing tag. We can resolve this with a large-scale map editor or a list view of the tags - but this can be confusing if users have a poor grasp on the workflow or map. Another way to do this would be to have sockpuppet\u2019s deployment interface show a list of \u201ccurrent tags\u201d which could be renamed.
Saving and Reloading
In the event that we update the map or tags, we need to persist this. And reload it. Right now, this is frightful since the map + tags are stored in version control. So you have to actually make a PR, check the code in, re-release, re-deploy - then you can restart the mapping stack to get the new map. Of course we can simplify slightly for single-robot iteration in a dev context, but this scales horribly.
What we need to do is shift maps, workflows and tags from code to data. A robot should be able to publish an updated version of this data, and other robots should be able to poll for this data, pull the new map/tags and reload on the fly - without being delocalized.
" }, { "id": "1533018204", "title": "Spec: Data Recording and Upload", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1533018204", "last_modified": "2026-03-09T15:34:52.422Z", "created_at": "2026-01-20T15:10:46.362Z", "body_html": "Owner: Cody Griffin
Date:
Approvers:
Informed:
Related: \n\n \n\n \n\n\n\n\n \n HECO-2820\n -\n Getting issue details...\n STATUS\n \n
Today we run a \u201cdata_collection\u201d node along-side sockpuppet, which acts on user input to manage our recordings. These recordings are organized into packages/episodes, uploaded via rclone to S3 and then can be viewed or annotated with Console and used for training models.
With deployment by demonstration, we will need to introduce the ability to record mapping data during deployment, and there is also talk about uploading data from interventions (teleop) or even just while the demos are running.
The biggest issue with our current scheme is that its very stateful. If we have all these various components managing the state of recording, we run into two significant issues:
It will be easy to introduce performance regressions if we test while NOT recording data
It will be easy to have multiple components in the system disagreeing about the recording state or having issues if someone else currently holds the \u201caction goal\u201d
This system was largely based on what existed for Pre-Alpha, but moving forward we should think about data collection slightly differently. The goal here would be to:
Constantly record data in 1 minute chunks (with some upper bound on size just in case) to disk
Bound the storage on disk so we have a rotating buffer of the last N minutes of data
Upload events to the cloud to effectively index this data
Recording data constantly
How we partition/organize mcaps
Scheduling uploads
Trace logs
Video compression (see Spec: Video Compression for Cameras)
The main idea here is that we want to collect events from our robot applications, either during deployment or during execution. These are effectively \u201cTempo\u201d traces that we can view in Grafana, and form the \u201cbackbone\u201d of our collected data. What these trace spans do is describe what\u2019s happening during specific time intervals on the robot (hierarchically).
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1533018204/Untitled%20Diagram-1768922309178.drawio.png?api=v2&version=2\")
MCAP data, on the other hand, is just dumped flat into a directory on the robot where we keep the last N minutes of data. This will likely require a slightly custom version of the ros2 bag recording node (which isn\u2019t a bad thing since we could benefit from more control/monitoring here).
Our trace spans effectively point to 1 or more of these MCAP files - meaning we can easily associate robot data with the event stream. Instead of needing to manage the recording state or manually annotate packages or uploaded data, we simply upload events as we record.
Uploading Robot Data
Obviously we need to upload data sometimes. With finite storage on the robot, we have some period of time (N minutes - depends almost entirely on how we encode video) to schedule and complete uploads before data is potentially overridden by newer data.
Data can be triggered or marked for upload from the robot itself (data collection via sockpuppet or interventions), or by request from some Console-based interface (perhaps to upload an interesting demo).
Prod and cons of uploading all data:
Argument for all uploading:
1. Debugging/troubleshooting.
Argument for filtering out data on the edge / device before uploading:
Storage costs.
(Potential) Increased complexity of data infra (i.e. raw layer and cleaned data)
Increased Data Annotation Effort. At the moment, all data we collect, we collect to train manipulation policies. These policies expect data to be sliced on episodes.
Trimming MCAP Data
One downside of this approach is that the MCAP writer doesn\u2019t know or care about the more granular boundaries of our data. So while we can identify the 5 minute range of MCAP data which overlaps with the interesting span, it will potentially have some extra data at the beginning or end. Do we trim this on the robot before uploading (requires more processing/buffering on the robot), or do we trim these in the cloud? My preference would be cloud-side since we aren\u2019t as resource-constrained there.
Event Delivery Guarantees
At least once semantics would be required here. Note that there are specific challenges around handling power-off, since we can\u2019t guarantee that we\u2019ll send an event for this. For this reason each \u201csession\u201d (where we boot the robot) should be uniquely identified in the events so we may infer reboots later.
Operators start Sockpuppet, and emit trace spans for each episode we wish to record, annotated by the success/failure, package and any other relevant metadat. The current scheme records selectively, but it uploads everything that\u2019s recorded. This new scheme would flip this - we record everything but we\u2019re more selective about what\u2019s updated.
Here we toggle some deployment mode to emit a deployment event, perform the deployment work (potentially with more data collection for specific policies), and then emit a final event when we are done. We can then upload all the mapping data corresponding to the deployment span, and the policy data corresponding to any episodes collected while deploying for capability development and model training.
Sometimes we may experience something weird during a demo or PoC. If we can\u2019t reliably reproduce the issue, its nearly impossible to catch. But if we can identify the time window from logs fast enough, we can simply mark the MCAP data for upload because we\u2019re always recording.
Any time we fail to teleop during execution, we can emit a span and upload the resulting data.
" }, { "id": "1715994636", "title": "Studio: Agentic Workflow Updates", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1715994636", "last_modified": "2026-03-07T17:58:13.480Z", "created_at": "2026-03-07T14:02:59.213Z", "body_html": "While deployment from demonstration is a convenient way to infer simple workflows and anchor workflow states to key points on a map, it does have a few major drawbacks. Many aspects of a complex robot workflow are unobservable or only partially observable. Real robot workflows will also have many branching points to deal with exceptions or normal variation. Most critically, its not obvious how to update or edit them over time.
What makes deployment-by-demonstration so powerful is that it directly links the operator\u2019s intent to the resulting workflow. You don\u2019t need to think about Petri nets, graphs, states or guard statements. You can simply show the robot what you mean directly. Fortunately, AI gives us additional means to translate user intent into robot action. Here we will explore the idea of using AI to allow users to incrementally evolve complex workflows in a stable, predictable and interpretable/auditable way. Operators won\u2019t need to worry about modifying complex workflow representations. They can simply make updates in natural language by expressing the intent of their change, which can be reflect in one or both of a textual SOP document (for humans) and a structured workflow representation (for robots and other system 3 agents).
The naive approach to something like this is to simply take some old document, and apply the intent to get a new version of the document. You could do this for each of the documents separately, or you could update one and then from it regenerate the other. This does update the workflow and SOP, but it loses a lot of information in the process, reducing robustness and predictability.
Instead, we will keep an intent log from the user, from which we will generate patches against our documents and workflow representation, while also re-generating snapshots. The core agent loop here takes an older snapshot, the most recent patches and the most recent intents, and then generates a new patch. The model may make use of tools to make this process more reliably (validating the resulting workflow via analysis for example.
The Martur Fompak, Bosch and Robostore workflows were all developed through multi-turn conversations with Codex or Claude Code, providing some early signal about the viability of this overall approach.
In Scope
Intent log via text
Patch generation (the agentic part)
Snapshots generation from patches (deterministic)
Undo
Simple debug UI or CLI (possibly built on demo monitor or sockpuppet)
System 3 integrations (partially observable stuff)
Out of Scope
Multimodal intent (audio, images or video would be possible as a next step)
Polished Web UI (a separate spec once we validate the model here)
Mapping or navigation
Policy development
The architecture here is built around a persistent intent log which can be versioned and branched. As new intents are addeded to the log, our agent context is updated to generate the next patches, and then the patch is applied to update our snapshot. The full history of intent, generated patches and snapshots are kept, and full traceability is established across all of these. Meaning given the state of the workflow (the snapshot), we can see exactly which patches and thus which operator intents were used to generate it.
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1715994636/Untitled%20Diagram-1772893348471.drawio.png?api=v2&version=1\")
When a patch is generated, the resulting snapshot should be validated to ensure that the workflow remains valid. Because the workflow is built incrementally, it provides stability to the workflow over time, limiting surprises when changes are requested later when the workflow is more complex. Asking an agent to simply modify some workflow can work (at least Codex is more than capable of doing this), but without the persistent history it becomes difficult to experiment and undo changes. By including recent intent and patches, the agent can have some context about not just the state of the workflow but how that workflow has been evolving.
It should also be possible to converse with the agent without actually modifying the workflow (meaning we never call the \u201cnew patch\u201d tool). To support this usage, a small conversation history is probably included in the context as well, but the agent context will need to be managed carefully to support higher-complexity workflows in the future.
Initial work here should focus on the tools required to apply patches, generate snapshots and validate workflows via existing harnesses like Codex or Claude Code. This will allow rapid iteration in a full-featured agent which can also help to identify gaps in tooling. Once the role of the general purpose agent has been reduced to simple tool-use to manipulate our workflows, then a custom harness can be created (since deploying Claude Code or Codex into a larger system could be challenging for a number of reasons given the existing interfaces).
![\"image-20260307-175200.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1715994636/image-20260307-175200.png?version=1&modificationDate=1772905921723&cacheVersion=1&api=v2&width=647&height=385\")
In addition to updating the agent, some care needs to be given to how Switchboard will actually handle these updates. The easiest thing to do is simply reset the workflow on any change. But the smart thing would be to allow some sort of live migration. This could be looked at as a \u201cpatch\u201d to the existing petri marking and facts within Switchboard, though this could be fragile without a ton of validation and additional tooling.
Experience With Prior Workflows
AI-assisted workflow editing with prior PoCs allowed us to learn some tricks that help considerably:
Provide tools for inspecting the current state of the workflow (via Switchboard API calls). This allows asking questions about why certain transitions were taken or why transitions may be blocked.
Provide tools for modifying the workflow arbitraryil. In cases where you want the robot to do something slightly different, you sometimes need to \u201cundo\u201d a transition or to allow rapid exploration of the workflow without needing to reset the whole thing.
Provide explicit context about what you expect the robots to do, and the resulting outcomes. Our agent here should be sure to prompt for more information and to avoid making workflow edits that are ambiguous or unclear.
We should build up a suite of \u201ccanonical traces\u201d to serve as regression tests we can backtest changes against. Otherwise we risk slight regressions. A recent example of this was the issue with basket swap prioritization with the robostore and handling two items of the same time. We saw that these two features are coupled in surprising ways via the guard statements and priorities of items being dispatched, resulting in a series of regressions that could have been avoided with some testing/validation against previous \u201cgood\u201d runs.
Supporting Interfaces
While out of scope for this doc, it is important that this data model support visualization and multi-modal editing. For example, how do we render a response from a suggested patch to explain the changes to the operator? How do we show the evolution of the workflow, the snapshots, each delta. How do we rewind or fast-forward through history?
The better these interfaces are, the better our UIs and voice interfaces will be.
Once workflows are deployed, observed execution traces can be compared against the expected workflow model. Conformance checking identifies where observed behavior diverges from the intended process.
Because workflow versions are tied to the operation log, conformance analysis can reference the exact workflow state that produced a given trace. Deviations can therefore be traced back to specific workflow changes.
In addition to transforming user intent directly into workflow patches, we could also infer the intent from any deviations in the trace logs. As the robot executes these workflows, we can trace the robot\u2019s actions and compare them with the workflow. If the robot is deviating often (either because system 2 decides to improvise or because of human intervention or tele-operation), workflow conformance scores will decrease. If the score drops far enough, we could automatically trigger a process which takes the recent trace events and infers some workflow update (intent) to make the workflow more conformant.
Think of a scenario where an operator gets an alert about workflow conformance, and potentially a handful of possible actions that could be taken to reduce non-comformance in the future (suggested changes to the SOP and workflow).
Beyond simple syntax checks, workflows must satisfy structural and behavioral properties. These include detecting deadlocks, identifying unreachable states, enforcing resource constraints, and validating safety policies.
Future validation passes can analyze workflow graphs and execution semantics to detect these conditions automatically. Validation results become attached to the operation history so that workflow evolution remains observable and auditable.
Consider the scenario where a process has been carefully deployed to support 6 robots operating in parallel. Someone could make a change to the workflow which inadvertently reduces workflow parallelism, even if the overall workflow looks correct. These changes should be detected early, and one of the key benefits to a petri-net-based workflow representation is that analysis can be used to detect these scenarios.
The operation log forms a natural audit trail of workflow evolution. Each modification records its causal origin and the artifact changes it produced. This enables operators to understand why workflows changed and what operational consequences followed.
Future tools may visualize workflow evolution as a timeline of operations and artifact projections. This allows teams to review how workflows evolved and to relate operational incidents to the workflow modifications that preceded them. Or workflows could be visualized in purpose-built simulators or map renderings as they are editing.
While the initial system focuses on SOP documents and workflow YAML definitions, the architecture supports additional artifact projections. These may include environment maps, simulation models or scenarios, robot configuration and more.
Each artifact type becomes another projection derived from the operation log. Special agents calls are used to interpret operations relevant to that artifact and update its representation. Over time the system can become a unified framework for managing operational knowledge relevant to the fleet across documentation, simulation, and runtime behavior.
" }, { "id": "1485275162", "title": "Spec: Single-Shot Deployment MVP", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1485275162", "last_modified": "2026-03-06T15:38:07.467Z", "created_at": "2026-01-08T14:02:35.206Z", "body_html": "Owner: Cody Griffin
Date:
Approvers:
Informed:
Related: https://miro.com/app/board/uXjVIjzyrI0=/?moveToWidget=3458764652888984570&cot=14 , Spec: HMND Fleet Control Plane
Deploying a robot today is a chore. You must:
manually map the space and tag important locations as keypoints
collect data (100s of examples) to train policies
hand-code behavior trees
This approach is painful and doesn\u2019t scale. Its extremely difficult to adjust to changes in the environment, or to modify the behavior of the robot. Any changes to the deployment require a new software release.
Additionally, we have another integration point where customer systems like ERP or WMS need to tell our system about new work that may be arriving - so now there is another module of code that may need to be written/deployed/maintained to translate this work request into some specific sequence of actions.
Even if we could build 100 robots a day, we\u2019d only be able to deploy 1 robot every few weeks at this rate. This is a massive burden for our engineering, deployment and test teams and a show-stopper for any commercial ambitions.
In Scope:
Inferring simple linear/looping workflows from Sockpuppet demonstrations
Navigation
Policy Execution
Posing/Recovery
Executing these workflows on any robot without a software release
Fetching map
Hard-coded URLs for fleet endpoints
Out of Scope
Any 3D mapping
Semantics
Branching/Conditions
Updates from teleop
Multi-robot coordination (capacity/choke points/etc)
Currently, we have Sockpuppet working for data collection and tele-operation, simple hard-coded behavior tree-based applications executing our workflows and not much else. Coming online shortly we have some simple workflow coordination in the cloud and a new workflow execution app.
In the future, being able to physically push the robot around to demonstrate the workflow would be even better, as it would require less setup friction (and you can probably push a robot physically safer than you could teleoperate).
We can likely repurpose Sockpuppet to serve as a means to demonstrate workflows, in which case we are mostly missing the \u201cStudio\u201d component here, responsible for taking log data from Sockpuppet or Reverb and generating workflows which can be executed autonomously.
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1485275162/Untitled%20Diagram-1767883738137.drawio.png?api=v2&version=1\")
There are two pieces necessary for this right now:
Generating our map
Inferring the workflow YAML
In the future, this process could be much more involved, including iterating on the workflow within simulation or leveraging sophisticated agentic setup processes which can semantically break down the workflow to better capture user intent. This MVP is not that.
Simple Workflow
Let\u2019s consider a simple tote-moving workflow where:
robot starts in some origin position
navigates to a table A
picks up a tote
navigates to table B
puts down tote
returns to origin position
As mentioned earlier, the current means of deploying even this trivial workflow is pretty involved. We need to map the space. We need, while mapping, label explicitly these \u201ctags\u201d for the points of interest, we then need to remember these tag names and code up a behavior tree, and then we can maybe attempt to run the behavior tree autonomously and see the robot execute the workflow. This can take days.
Demonstration
So we will power on the robot and boot into Sockpuppet. We can then navigate to the origin position and enable our \u201cdeployment mode\u201d. Note that the user never needs to tag the location - its done implicitly here. The user then teleoperates the robot to table A, and then uses the Sockpuppet policy tab to run \u201cpick up tote\u201d. The robot will execute this policy, or if there is an issue (perhaps due to insufficient data collection), the operator can teleoperate the robot to pick the tote up and move things along. Similar to the origin, we automatically save the point where we run the policy as a tag - the user doesn\u2019t need to do this. Then we navigate to the other table, and run the \u201cplace tote\u201d policy. Then we navigate back to the origin and we \u201csave\u201d the workflow, closing the loop.
Logging
During the demonstration above, we need to log some data. To support mapping, we need to collect the lidar and odometry data necessary to generate a map offline. Sockpuppet can emit trace logs of the various tools being used (navigating, running policy, setting a pose), which can then be aligned with our odometry (by time) and placed into the generated map. These trace logs effectively become our points of interest - freeing our operator from the burden of naming these points themselves.
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1485275162/Untitled%20Diagram-1767900808095.drawio.png?api=v2&version=1\")
Inference
Now the fun - we need to take our lidar + odometry data and generate a map offline, and then we need to locate our Sockpuppet logs on this map to generate keypoints. Then we can run some very simple \u201cprocess mining\u201d to infer a petri-net from the logs. This is a relatively standard procedure in the world of data science, though it is old-fashioned and a bit crusty. In the future, this inference step is likely to be done by an LLM directly - but we can start with something much simpler for now.
The resulting workflow.yaml, occupancy grid map and mapping metadata (our inferred keypoints) can all be bundled up and saved to S3 as a workflow package.
Execution
Our coordinator can be loaded with a selected workflow package, and serve task assignments and map data to the robots. The robot\u2019s execution agent (reverb) can then progress through the workflow autonomously, moving from point to point and executing policies as specified.
Like with Sockpuppet, these logs are also kept and, in cases where operator intervention is needed, capture deviations or alternatives in the workflow. Given enough of these examples, we can potentially re-infer more complex workflows, this this \u201cupdate\u201d is out of scope for this MVP. The primary purpose of these logs will be for checking Workflow conformance (how well are we following the workflow).
SAP/MF PoC (done)
Coordinator: hard-coded workflow
Execution: fixed or agentic workflow \u201capp\u201d on robot
Offline mapping via Sockpuppet and Studio (tested in sim)
Deployment Mode: Record mapping data (lidars, odom) to mcap while tele-operating
Offline Mapping: Generate occupancy grid with origin tag and store in Studio
Workflow inference (tested in sim)
Sockpuppet Logs: Record logs from sockpuppet for various actions taken during deployment mode
Alignment: Use timestamps/odometry to place logs on the map for inferred keypoints
Workflow Package: Infer a workflow.yaml, combine with map data and metadata as a workflow package
Workflow execution (mostly tested in sim)
Workflow Selection: Choose which workflow package should be used by which robots, and download package (map/metadata specifically) to the robot.
Reverb Logs: Similar to Sockpuppet - should be same infrastructure, just tagged differently
Conformance: Check how closely workflow is being followed
Deployment
Start up sockpuppet
Toggle \u201cdeployment mode\u201d to enable extra logging
Perform the workflow via teleoperation
Save it.
Execution
Select workflow package ID when we start the fleet coordinator (Switchboard)
Point reverb at Switchboard to begin autonomous execution
Incremental Updates
The procedure here is great for the simplest workflows and can reliably handle simple linear sequences or \u201cloops\u201d of tasks. But to scale to more complex workflows, we need to be able to adjust things incrementally. There are two possibilities here we will likely be exploring later if this initial MVP works as-intended.
First, we can use additional data collected via teleoperation to allow the workflow to grow more complex over time. This would be enough to get us some simple branching logic perhaps. Second, we can use an editor within \u201cStudio\u201d to get more sophisticated editing capabilities, along with allowing Sockpuppet to incrementally edit the workflow sequence.
Multiple Robots
The coordinator should handle this, but I expect we may need to make further adjustments here (detecting other robots, handling congestion locally via policy).
Agentic Dispatch
Writing code to translate some incoming webhook or outgoing events to integrate with some 3rd party is also going to be an integration headache. In a perfect world, we have enough weight in the market to set standards and we just force them to integrate. We need to grow to get that weight though, and until then this is a major drag.
One idea here is to just use AI. We could have a prompt defined for a particular integration (\u201cYou\u2019ll receive a webhook looking like X, make sure to peel out fields A, B and C, and make sure to blah blah\u201d). Even with the rapid improvements in instruction following, its unlikely that this will be reliable enough - so after an initial deployment with this, we can collect enough data for the AI to effectively write a plugin - almost like a JIT.
The \u201cpose \u2192 run policy \u2192 reset \u2192 run next policy\u201d has been useful to de-risk data collection, policy development and our ongoing PoCs, but its starting to become a liability.
Right now each policy assumes a carefully staged starting configuration: robostore_table, robostore_shelf, and so on. Humans reset the robot to those configurations so the policy doesn\u2019t have to deal with the messy reality of arbitrary state. That simplifies training and debugging in the short term, there are some unfortunate downsides.
Every reset is a discontinuity in motion. The robot freezes, shifts into some canonical pose, then resumes policy or navigation. From a demo perspective it looks clumsy and stilted. From a systems perspective it is even worse: it encodes hidden operational knowledge into the deployment process. Someone has to know which pose corresponds to which policy and when to apply it. If they are wrong, the policy will fail or the robot may collide with the environment.
The robot only works because an engineer with intimate familiarity with the system scripted the right motion sequences. A scalable robotics product cannot rely on this process.
I am not the AI guy. This needs to be digested by the AI team as I suspect there are many intricacies here that are not being purposefully trivialized.
A more robust approach is to make policies responsible for their own approach. Instead of assuming a perfect starting pose, the policy learns how to move from the current configuration into a configuration where the policy can succeed (ideally while avoiding obstacles).
This \u201csmall\u201d (lol) tweak has major benefits:
First, motion becomes continuous. The robot flows from one behavior to the next instead of halting for resets. The result is smoother, more natural movement and dramatically better demonstrations of capability.
Second, deployment becomes simpler. Instead of a fragile sequence of manually curated poses, the operator triggers a task and the robot figures out how to get into position. The system carries its own assumptions rather than requiring humans to remember them.
Third, and most important, policies begin to generalize. When a policy can only operate from a narrow starting configuration, its effective operating envelope is tiny. The moment reality deviates (a slightly different arm position, a small disturbance, a prior task ending differently) the policy fails. Teaching policies to navigate into workable states expands that envelope. Over time the robot becomes tolerant of variation rather than allergic to it.
How does this compound?
We trade time spent toiling away in deployment, orchestrating poses and sequencing actions carefully for time spent in data collection. I think this is the correct trade - since we can re-use the trained policy over and over. Put another way, policy scales better than deployment. There are scaling laws for polices - get better at many tasks to get better at any task. There is no such scaling law for deployments.
Given the compounding nature of this work, the earlier we start making progress, the earlier we can start taking back time and re-investing into the product.
" }, { "id": "1711571015", "title": "Spec: Dev vs Release Images", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1711571015", "last_modified": "2026-03-06T11:35:32.675Z", "created_at": "2026-03-06T11:05:57.515Z", "body_html": "Owner:
Date:
Approvers:
Informed:
Related: Other OS-image-related docs?
We need one robot OS artifact that can behave like two different machines:
Release mode is for demos and data collection. It must be deterministic and self-healing. Any accidental system drift should evaporate on reboot.
Development mode is for engineering. Drift is allowed, expected, and persistent. Engineers need a live repo, mutable deps, and a controlled on-device workflow.
We will ship a single immutable base image. Mode is chosen at boot via a bootloader \u201cdirty bit.\u201d Converting release \u2192 dev is allowed. Converting dev \u2192 release is not allowed except via destructive reset that wipes dev state.
The key invariant is blunt:
A robot in release mode cannot retain root filesystem drift across reboot.
CLEAN means release mode.DIRTY means development mode.
Release services are the system-installed stack, launched by systemd units shipped with the image.
Dev services are repo-backed services, launched by systemd units that point into /data/dev/repo/....
We standardize all robot-owned configuration and state under consistent prefixes:
/etc/hmnd/ \u2014 configuration (shipped + runtime config overlays)
/var/lib/hmnd/ \u2014 persistent state (identity, provisioning state, databases, caches that must survive reboot)
/var/log/hmnd/ \u2014 logs (persistent in both modes)
/run/hmnd/ \u2014 runtime scratch (tmpfs)
/opt/hmnd/ \u2014 installed release binaries/assets (if not in /usr)
/data/ \u2014 persistent data volume (partition). Contains dev state when DIRTY.
This avoids the \u201crandom junk in /etc and /var\u201d entropy spiral.
These are required in both modes and must survive reboot:
/var/lib/hmnd/ (identity, provisioning, small DBs)
/var/log/hmnd/ (logs)
/etc/hmnd/certs/ (cert bundle and device keys; or a symlink into /var/lib/hmnd/certs\u2014pick one and stick to it)
/data/ (always mounted; used heavily in DIRTY)
Release mode is stateless for root drift, not for identity/logs.
Development state lives under:
/data/dev/upper \u2014 overlayfs upperdir
/data/dev/work \u2014 overlayfs workdir
/data/dev/repo/<project> \u2014 git checkout
/data/dev/build/ \u2014 build outputs (if needed)
/data/dev/state/ \u2014 stamps (last built rev, last good rev, dependency stamp)
The root filesystem is an immutable image (e.g., SquashFS) mounted read-only and layered with overlayfs.
CLEAN (release mode)
The overlay upperdir is tmpfs. Root drift cannot persist.
upperdir: /run/hmnd/overlay/upper (tmpfs)
workdir: /run/hmnd/overlay/work (tmpfs)
DIRTY (development mode)
The overlay upperdir is persistent disk at /data/dev.
upperdir: /data/dev/upper
workdir: /data/dev/work
In CLEAN: apt install might \u201cwork\u201d temporarily, but it dies on reboot. Good. That\u2019s the point.
In DIRTY: any mutation persists, and the box is explicitly non-release.
A bootloader-controlled flag determines mode at boot.
CLEAN boots release overlay and release services.
DIRTY boots dev overlay and allows dev services.
Export mode to userspace early (initramfs) as:
/run/hmnd/mode containing CLEAN or DIRTY
Mode must be reported to the control plane in device heartbeat.
Clearing DIRTY (returning to CLEAN) is only permitted if dev state is wiped. That means /data/dev is erased successfully.
No exceptions. No \u201cbut I uninstalled the packages.\u201d
Define two top-level targets:
hmnd-release.target \u2014 pulls in release services
hmnd-dev.target \u2014 pulls in dev services
CLEAN boots into hmnd-release.target.
DIRTY may run either, but dev switching is explicit (see below).
Release units:
hmnd-<svc>.service
Dev units:
hmnd-<svc>-dev.service
They must be disjoint. No unit should \u201csometimes\u201d run repo code. That\u2019s how lies happen.
Release services are shipped with the image and point to installed paths:
binaries in /usr/bin or /opt/hmnd/bin
config in /etc/hmnd/
state in /var/lib/hmnd/
logs in /var/log/hmnd/
Release services must be reproducible from the image build.
Dev services point into the repo checkout:
repo root: /data/dev/repo/<project>
config: typically /data/dev/repo/<project>/config/... (or still /etc/hmnd, but keep a clean story)
build outputs: /data/dev/build/... if applicable
Dev services are enabled/disabled by tooling, not by hand-editing symlinks.
robotctl responsibilitiesrobotctl is the only supported mechanism for changing mode and switching service profile.
Required commands:
robotctl enter-dev-mode
robotctl factory-reset
robotctl dev-sync
robotctl dev-build
robotctl dev-switch
robotctl status
robotctl enter-dev-mode performs:
Set bootloader dirty bit to DIRTY.
Ensure /data exists and create /data/dev/{upper,work,repo,build,state}.
Reboot.
After reboot (now DIRTY), the operator runs:
robotctl dev-sync \u2014 fetch/clone repo quickly
robotctl dev-build \u2014 build/install deps
robotctl dev-switch \u2014 disable release services, enable dev services, isolate hmnd-dev.target
This is intentionally a 2-step human-visible transition. You want a deliberate \u201cI am now leaving release land.\u201d
robotctl factory-reset performs:
Stop targets and kill dev processes.
Wipe /data/dev (or wipe whole /data per policy).
Clear bootloader dirty bit to CLEAN.
Reboot.
Clearing dirty bit without wipe is forbidden.
In DIRTY mode, we want a controlled \u201cconsole\u201d experience because:
it standardizes debugging,
it makes it obvious what\u2019s running,
it lets automation manage panes and watchers without playing whack-a-mole with user shells.
So: tmux is not optional in DIRTY.
In DIRTY mode, all SSH logins attach to the same tmux session (per user or per device\u2014choose one; per user is usually saner).
Policy:
If a session exists: attach.
If not: create it with the standard layout.
Prevent creating additional sessions by default (users can still do it if they\u2019re root and determined, but the \u201chappy path\u201d is one session).
Implementation is done via /etc/profile.d/hmnd-tmux.sh (DIRTY-gated) plus a thin wrapper hmnd-console.
Example gating:
# /etc/profile.d/hmnd-tmux.sh\nif [ -n "$SSH_CONNECTION" ] && [ -f /run/hmnd/mode ] && grep -qx DIRTY /run/hmnd/mode; then\n exec /usr/bin/hmnd-console\nfi\n
The console session should have multiple shells; Codex is one of them.
Example layout:
Window 0: \u201cops\u201d \u2014 tail dev target logs + status
Window 1: \u201crepo\u201d \u2014 shell in /data/dev/repo/<project>
Window 2: \u201ccodex\u201d \u2014 Codex CLI shell (interactive)
Window 3: \u201cbuild\u201d \u2014 build/watch output
Codex is present, but not worshipped.
When repo changes, dev services should refresh automatically. Whether you do hot reload or rebuild+restart depends on the stack; the contract is that the running system converges toward repo HEAD without manual babysitting.
Repo checkout at:
/data/dev/repo/<project>
Prefer systemd PathChanged= units over bespoke background daemons.
hmnd-dev-sync.path watches:
/data/dev/repo/<project>/.git/HEAD
/data/dev/repo/<project>/.git/refs/
or a REVISION stamp file written by robotctl dev-sync
hmnd-dev-sync.service runs:
robotctl dev-build
robotctl dev-restart (or systemctl restart hmnd-dev.target)
Rules:
Debounce updates (avoid restarting 30 times during a rebase).
If build fails: do not restart into failure; keep last good services running.
Write \u201clast good rev\u201d to /data/dev/state/last_good_rev.
This is the \u201cdo the thing\u201d layer.
cat /run/hmnd/mode # CLEAN or DIRTY\nrobotctl status # includes mode, target, repo rev, last good rev\n
robotctl enter-dev-mode\n# reboots\nrobotctl dev-sync --ref main\nrobotctl dev-build\nrobotctl dev-switch\n
robotctl factory-reset\n# wipes /data/dev, clears DIRTY, reboots into CLEAN + hmnd-release.target\n
Recommended defaults:
git clone --depth=1 <remote> /data/dev/repo/<project>\n# or maintain a mirror under /data/git-mirror and clone from file:// for speed\n
robotctl dev-build runs repo-owned, idempotent scripts, e.g.:
/data/dev/repo/<project>/dev/apt.txt
/data/dev/repo/<project>/dev/build.sh
/data/dev/repo/<project>/dev/install.sh
You stamp completion in /data/dev/state/ keyed by commit hash.
Targets:
/etc/systemd/system/hmnd-release.target
/etc/systemd/system/hmnd-dev.target
Release units live with the image.
Dev units can be shipped with the image but point to stable repo paths (/data/dev/repo/...). They should be inert unless DIRTY and switched.
Masking policy:
In CLEAN: hmnd-dev.target and hmnd-*-dev.service masked.
In DIRTY: allowed, but not necessarily default.
Codex CLI is installed on the image, but integrated as one tool in the dev console, not a monoculture.
Follow-ups:
Define codex-agent.service (DIRTY-only) if you want a persistent background agent.
Add a hmnd-codex tmux window that runs Codex CLI against /data/dev/repo/<project>.
Optionally wire Codex to robotctl commands (sync/build/switch/status) as a \u201ctoolbelt,\u201d not a replacement for normal ops.
This version nails the two big corrections:
filesystem hygiene (/etc/hmnd, /var/lib/hmnd, /var/log/hmnd, /run/hmnd, /data/dev), and
tmux as the controlled cockpit, with Codex as one instrument panel among several.
Next step is choosing whether dev units are shipped with the base image (preferred for stability) or generated from the repo (more flexible, more footguns).
" }, { "id": "1712324676", "title": "Spec: Dev Tooling & Agent Substrate", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1712324676", "last_modified": "2026-03-06T10:50:21.541Z", "created_at": "2026-03-06T10:42:44.117Z", "body_html": "Owner: Cody Griffin Oleg Sinavski
Date:
Approvers:
Informed:
Related:
The opportunity cost of NOT running agents is growing fast. Right now there is still quite a bit of toil involved - you have to clone a massive repo, you have to have the disk space, you have to have the CPU/RAM to do the builds, GPUs for simulation. Can we do better?
Cloud-based clones
Feedback/testing
Simulation
Evals/Benchmarks
Very high-level - but imagine an AI-powered sync engine that uses Git as the source of truth, but can render confluence specs (not unlike this one) and Jira tickets/updates as work is done by 24/7 agents.
The key idea is to push the bottleneck from development to merging. Ideally when making a release, someone can simply identify the ~10 or so agent-developed branches that they want to include.
Developers will need to document their feature along with the metrics and any necessary SOPs in a spec, which can then be broken down and implemented autonomously by coding agents. Developers (and others) can provide feedback within the spec or PR with the agents incorporating feedback autonomously. Agents keep the branches up to date with main, resolving conflicts until the spec is ready to be released.
At no point do developers need to clone the repo or run builds locally.
" }, { "id": "1672478731", "title": "Observability Stack (Grafana/Loki/Tempo/Mimir)", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1672478731", "last_modified": "2026-02-25T10:32:37.287Z", "created_at": "2026-02-25T10:31:18.698Z", "body_html": "Summary: Current \u201cPhase 1\u201d Grafana-based observability
Owner: Cody Griffin
Date: 02/25/2026
Approvers:
Informed: Core / Robot / Infra
Related: Spec: HMND Core Control Plane
We need a small, operationally simple observability stack for the Core Control Plane that:
Works \u201cday 1\u201d without a Kubernetes dependency.
Provides a single pane of glass (Grafana) for metrics, logs, and traces.
Offers durable object storage for long-term retention.
Is reproducible and environment-scoped via Terraform.
In-scope
Terraform module that provisions:
Public ALB with TLS termination.
Route53 records for service hostnames.
Single private EC2 instance running Grafana/Loki/Tempo/Mimir via systemd.
S3 bucket for long-term storage (blocks/chunks/traces + ruler/alert data).
A dedicated gp3 EBS volume mounted at /var/lib/observability for WAL/cache.
Basic auth at the edge (Nginx) for non-Grafana endpoints.
Grafana datasource provisioning so the UI is usable immediately.
Out-of-scope
High availability (multi-instance quorum, multi-AZ replication, automatic failover).
Fine-grained tenant separation / per-robot authZ.
\u201cProduction hardening\u201d (OIDC, mTLS, WAF, rate limiting, secret rotation workflows).
Managed scaling for high ingest rates.
ALB (public)
Listeners: :80 redirects to :443
:443 uses an ACM wildcard certificate for *.<domain_suffix> (example: *.core.dev.hmnd.ai)
Host-based listener rules forward to a single target group
EC2 instance (private subnet, no public IP)
Nginx reverse-proxy on :80
Services on localhost:
Grafana 127.0.0.1:3000
Mimir 127.0.0.1:8080
Loki 127.0.0.1:3100
Tempo 127.0.0.1:3200 (+ OTLP HTTP on 127.0.0.1:4318)
S3 bucket
Shared object storage backend for Loki/Mimir/Tempo persisted data
EBS gp3 cache volume
Mounted at /var/lib/observability for WALs and caches
Terraform creates Route53 A aliases pointing at the ALB (example domain suffix shown):
grafana.<domain_suffix> (e.g. grafana.core.dev.hmnd.ai) \u2192 Grafana UI
metrics.<domain_suffix> (e.g. metrics.core.dev.hmnd.ai) \u2192 Nginx \u2192 Mimir (basic auth)
logs.<domain_suffix> (e.g. logs.core.dev.hmnd.ai) \u2192 Nginx \u2192 Loki (basic auth)
traces.<domain_suffix> (e.g. traces.core.dev.hmnd.ai) \u2192 Nginx \u2192 Tempo (basic auth)
OTLP HTTP ingestion: https://traces.<domain_suffix>/v1/traces
Metrics: apps/agents remote_write \u2192 metrics.<domain> \u2192 Nginx (basic auth) \u2192 Mimir \u2192 S3 blocks
Logs: agents push \u2192 logs.<domain> \u2192 Nginx (basic auth) \u2192 Loki \u2192 S3 TSDB/chunks
Traces: OTLP HTTP \u2192 traces.<domain>/v1/traces \u2192 Nginx (basic auth) \u2192 Tempo \u2192 S3 blocks
Visualization: browser \u2192 grafana.<domain> \u2192 Grafana queries local Mimir/Loki/Tempo
S3 (observability_bucket_name, default hmnd-obs-dev)
Versioning enabled
Server-side encryption (SSE-S3 / AES256)
Public access blocked
Local disk (EBS)
WAL directories and caches under /var/lib/observability/*
hmnd_infra/variables.tf)Common knobs:
core_domain_suffix (default core.dev.hmnd.ai)
core_route53_zone_name (default dev.hmnd.ai)
core_route53_zone_id (optional)
observability_bucket_name
observability_instance_type
observability_allowed_ssh_cidrs / observability_ssh_key_name
observability_loki_version / observability_tempo_version / observability_mimir_version
observability_basic_auth_username / observability_basic_auth_password
observability_grafana_admin_password
EC2 is managed with AmazonSSMManagedInstanceCore attached to the instance role.
ALB target group health check hits GET /healthz served by Nginx.
Services are started via systemd (grafana-server, mimir, loki, tempo, nginx).
Grafana datasources are provisioned at boot for:
Local Mimir (default datasource)
Local Loki
Local Tempo
Plus optional external Prometheus datasources used today for lab clusters.
Single-instance / single-AZ deployment: an AZ or instance failure is a full outage.
Bootstrap security: defaults are weak until overridden; treat as non-production.
Grafana nightly package channel: increased upgrade/compat risk.
Multitenancy disabled in Mimir/Loki configs (fine for bootstrap; revisit for fleet scale).
HA design (multi-AZ, redundancy, automated failover).
Replace basic auth with OIDC and/or mTLS + per-device identity.
Alerting strategy (ruler/alertmanager durability, paging integrations, SLOs).
Move from single-node services to managed offerings or Kubernetes operators once needed.
Summary: PKI and secrets model for the current Backbone control-plane deployment.
Owner: Cody Griffin
Date: February 24, 2026
Approvers:
Informed:
Related: Spec: HMND Core Control Plane
We need a boring, auditable way to issue short-lived mTLS client certificates to robots, terminate mTLS at the edge for core services, and tie every request to a concrete device identity. For human-facing operations, we want SSO-backed identity with a minimal authorization model (group membership) and good operational ergonomics.
This implementation is intentionally AWS-managed (ACM PCA, ALB trust store, Cognito, Secrets Manager) rather than a self-hosted CA/auth stack. We avoid CRL/OCSP and instead rely on short certificate TTL plus immediate access removal by deleting or disabling the device record in the control plane.
The deployed implementation in hmnd_infra/modules/backbone_stack is the source of truth for edge behavior (ALBs, mTLS, Cognito wiring, WAF, secrets injection). The Backbone application defines the exact certificate profile and REST surface; the appendices below document the current intended contract and the parts of it that are enforced/protected by the infrastructure.
Backbone runs on ECS/Fargate behind two ALBs. The non-mTLS ALB provides the public browser and bearer-token entrypoints, and terminates standard TLS with ACM-issued certificates. It is exposed at backbone_public_domain_name (default public.core.dev.hmnd.ai) and backbone_api_domain_name (default api.core.dev.hmnd.ai).
Robots connect to a separate ALB when backbone_enable_mtls=true (default). That ALB is exposed at backbone_devices_domain_name (default devices.core.dev.hmnd.ai) and enforces mutual TLS at the listener. Concretely, the listener is configured with mutual_authentication.mode = verify, uses an ALB trust store sourced from an S3-published CA bundle (ca-bundle.pem), and does not ignore client certificate expiry.
Human identity is Cognito-backed. The stack creates a user pool and two groups (admin/operator names default to hmnd-core-<environment>-admins/operators, but are overrideable). The /v1/dashboard* paths are protected by ALB authenticate-cognito, which provides a simple browser login flow without the Backbone service having to run its own hosted UI.
At runtime, the service is configured for its auth adapters via environment variables (for example, BACKBONE_HUMAN_AUTH_MODE=alb_oidc_or_cognito_jwt and BACKBONE_DEVICE_AUTH_MODE=alb_mtls when the robots ALB is enabled). Certificate issuance is performed by the Backbone service using ACM PCA (BACKBONE_SIGNER=aws_pca and a resolved PCA ARN), with issuance permissions granted to the ECS task role.
The stack can either create a ROOT ACM PCA (default backbone_create_pca=true) or consume an existing PCA (backbone_create_pca=false and backbone_pca_arn set). The ALB trust store bundle is published to S3 as ca-bundle.pem. You can override the bundle explicitly with backbone_trust_store_ca_bundle_pem; otherwise it resolves from the current PCA certificate.
Robots present a client certificate issued from the trusted CA bundle. The ALB verifies that certificate chain before forwarding traffic to the service; the application then derives identity from the presented certificate and enforces authorization and device lifecycle via its backing store.
Backbone uses Secrets Manager for runtime configuration and credentials, and the ECS execution/task roles are granted secretsmanager:GetSecretValue scoped to the injected secret ARNs.
The module creates:
${name}/backbone/database_url (Aurora connection URL)
${name}/backbone/config (Backbone config JSON; currently includes BACKBONE_DB_MIGRATE=1)
optional ${name}/backbone/openai when backbone_openai_api_key is provided
It can also consume:
backbone_google_oauth_secret_arn (recommended when backbone_enable_google_idp=true)
backbone_extra_secret_arns (additional app-readable secrets)
Operational helper scripts in hmnd_infra:
upsert-backbone-secrets.sh bootstraps Google OAuth/OpenAI secrets
create-backbone-dev-user.sh creates/updates Cognito users and group membership
backbone-membership adds existing users to admin/operator groups (dev utility)
The modules/backbone_stack Terraform module provisions the full Backbone edge and stateful dependencies: ECR + ECS/Fargate service, a public ALB (plus an optional dedicated robots mTLS ALB), ACM certificates and Route53 records for the entrypoints, an ALB trust store backed by an S3 CA bundle object, Aurora PostgreSQL Serverless v2 (with pg_cron enabled), Cognito user pool + groups + hosted domain (plus optional Google IdP), WAF protection on the public ALB (including enrollment endpoint rate limiting), and Secrets Manager entries for DB/config and other application secrets.
The Backbone task definition wires configuration via environment variables and secret injection. The key parts are: (1) store selection (BACKBONE_STORE=postgres) backed by BACKBONE_DATABASE_URL from Secrets Manager, (2) certificate signer selection (BACKBONE_SIGNER=aws_pca with BACKBONE_PCA_ARN), and (3) auth adapter selection for humans vs robots (BACKBONE_HUMAN_AUTH_MODE and BACKBONE_DEVICE_AUTH_MODE).
Cognito integration is provided via BACKBONE_COGNITO_REGION, BACKBONE_COGNITO_USER_POOL_ID, and BACKBONE_COGNITO_APP_CLIENT_ID, along with group names (BACKBONE_ADMIN_GROUP and BACKBONE_OPERATOR_GROUP) that Backbone uses to implement a minimal authorization model (admin for write operations, operator for read-only).
This appendix describes the current intended REST surface for the Backbone control plane. Exact request/response shapes are defined by the Backbone service implementation, but the endpoints below reflect what the infrastructure is explicitly configured to protect and what clients are expected to use.
GET /health
Used by ALB target group health checks.
POST /v1/enrollments (unauthenticated; rate limited by WAF)
A robot begins enrollment by submitting metadata and a CSR. The control plane returns a short-lived enrollment record that the robot can poll.
Example request:
{\n "robot_id": "RBT-042",\n "robot_type": "alpha_wheelbase",\n "device_type": "rtpc",\n "csr_pem": "...",\n "nonce_hash": "..."\n}\nExample response:
{ "enrollment_id": "...", "expires_at": "..." }\nGET /v1/enrollments/{id}
While pending:
{ "status": "PENDING", "expires_at": "..." }\nOn completion, the response includes the issued certificate material:
{\n "status": "COMPLETE",\n "device_id": "...",\n "cert_pem": "...",\n "chain_pem": "...",\n "cert_expires_at": "..."\n}\nPOST /v1/devices (human-authenticated)
A human approves an enrollment by presenting the enrollment ID and the matching nonce. In the current model, write actions are expected to require admin group membership.
Example request:
{ "enrollment_id": "...", "nonce": "..." }\nTypical server behavior:
validate enrollment and nonce
issue certificate via ACM PCA
create or update the durable device record
delete the enrollment record
GET /v1/devices and GET /v1/devices/{device_id} (human-authenticated)
Read actions are expected to be permitted for operator group membership.
DELETE /v1/devices/{device_id} (human-authenticated)
Immediate blocking is implemented by removing/invalidating the device record so subsequent requests fail authorization.
POST /v1/sessions (robot-authenticated)
Robots periodically upsert last-seen presence and optional metadata. When mTLS is enabled, authentication is based on the client certificate verified at the ALB.
All state is stored in Aurora PostgreSQL. The infra enables pg_cron as a shared preload library and the intended design uses TTL-based cleanup for ephemeral tables. The exact schema is owned by the Backbone service migrations; the DDL below captures the intended structure.
devices (durable)
CREATE TABLE devices (\n device_id UUID PRIMARY KEY,\n robot_id TEXT NOT NULL,\n robot_type TEXT NOT NULL,\n device_type TEXT NOT NULL CHECK (device_type IN ('RTPC','ORIN','OTHER')),\n cert_fingerprint_sha256 TEXT NOT NULL UNIQUE,\n cert_expires_at TIMESTAMPTZ NOT NULL,\n created_by_iss TEXT NOT NULL,\n created_by_sub TEXT NOT NULL,\n created_at TIMESTAMPTZ NOT NULL DEFAULT now(),\n updated_at TIMESTAMPTZ NOT NULL DEFAULT now()\n);\n\nCREATE UNIQUE INDEX ON devices (robot_id, device_type);\nCREATE INDEX ON devices (robot_id);\nCREATE INDEX ON devices (cert_expires_at);\nenrollments (TTL)
CREATE TABLE enrollments (\n enrollment_id UUID PRIMARY KEY,\n robot_id TEXT NOT NULL,\n robot_type TEXT NOT NULL,\n device_type TEXT NOT NULL CHECK (device_type IN ('RTPC','ORIN','OTHER')),\n csr_pem TEXT NOT NULL,\n nonce_hash TEXT NOT NULL,\n expires_at TIMESTAMPTZ NOT NULL,\n created_at TIMESTAMPTZ NOT NULL DEFAULT now()\n);\n\nCREATE INDEX ON enrollments (expires_at);\nsessions (presence TTL)
CREATE TABLE sessions (\n device_id UUID PRIMARY KEY REFERENCES devices(device_id),\n last_seen_at TIMESTAMPTZ NOT NULL,\n data JSONB\n);\n\nCREATE INDEX ON sessions (last_seen_at);\n
TTL cleanup via pg_cron (example schedule: every 5 minutes)
DELETE FROM enrollments\nWHERE expires_at < now();\n\nDELETE FROM sessions\nWHERE last_seen_at < now() - interval '7 days';\n
A robot enrollment starts locally: the device generates a keypair and a CSR, generates a high-entropy nonce, and submits the CSR and a hash of the nonce to POST /v1/enrollments. The robot then shows the enrollment ID and the nonce out-of-band to a human operator.
The human completes approval by authenticating via Cognito (optionally federated to Google) and submitting POST /v1/devices with the enrollment ID and nonce. Backbone issues the certificate via ACM PCA, binds it to the device record, and the robot begins authenticating to the robots ALB using mTLS.
Renewal is expected to be a normal online operation prior to expiry, driven by a new CSR and resulting in an updated fingerprint/expiry on the same device identity. If a robot is offline past expiry and cannot authenticate to renew, recovery is treated as re-enrollment; the stable logical device identity is preserved in the registry while rotating keys and certificate.
The public ALB is protected by WAF with an IP-based rate limit on POST /v1/enrollments. The intended application-level controls include: CSR size limits, short enrollment TTL, and nonce hashing (nonce never stored in plaintext).
Spec: kexec-Based Ephemeral RootFS Boot for Robot Bringup
Owner: Cody Griffin
Date:
Approvers: \u2014
Flashing Jetson partitions and maintaining bootloader state is slow, brittle, and hostile to iteration. On x86, touching EFI/GRUB for every experiment is equally dumb.
We need a fast, reproducible way to boot a fully-built robot software stack (kernel + initramfs + rootfs) from inside a known-good host OS without repartitioning, bootloader changes, or permanent installation.
The system must support:
Jetson Orin AGX running JetPack (L4T), where we reuse host DTB/firmware.
Congatech x86 running an RT-patched kernel doing EtherCAT, where post-kexec EtherCAT behavior is the biggest risk.
Boot a versioned \u201cguest OS\u201d built by CI/Bazel as an immutable SquashFS image.
Transition from host \u2192 guest via kexec in seconds.
Keep a small set of state/config persistent across guest boots (e.g., /etc/hmnd, /var/lib/hmnd).
Make failure modes diagnosable from serial/console without needing network.
Preserve a clean escape hatch back to the host baseline.
Replacing the platform bootchain (UEFI/MB1/MB2/GRUB) or changing partition layouts.
Full hardware reinitialization guarantees equivalent to power-cycle.
Secure boot / measured boot / attestation (future work).
Long-term \u201cproduction boot\u201d path (this is a development and field-debugging boot path first).
Host-side launcher that loads guest kernel/initramfs (from SquashFS) and executes kexec.
Guest initramfs that assembles the final root from:
lowerdir: immutable SquashFS rootfs
persistence: bind mounts from host root partition
optional overlay upperdir: tmpfs (ephemeral) or persistent dir on host partition
Platform support:
Jetson Orin AGX (JetPack host; reuse host DTB; firmware via host bind or guest packaging)
Congatech x86 (RT kernel host; EtherCAT driver behavior under kexec explicitly tested)
Packaging into the robot image (systemd service + CLI + initramfs + docs).
Remote boot selection.
Automatic fallback boot integration with firmware boot menus.
Hot-swapping DTBs on Jetson with bootloader-equivalent fixups (we explicitly avoid this).
Host OS (baseline, \u201clauncher\u201d)
Known-good OS installed normally (JetPack on Orin; RT Linux on x86).
Provides storage, firmware blobs (optional), and persistence directories.
Runs hmnd-kexec launcher to transition to guest.
Guest OS (ephemeral, \u201cimage\u201d)
Built artifacts:
vmlinuz (guest kernel)
initramfs.img (guest initramfs)
root.squashfs (immutable rootfs)
manifest.json (version metadata, kernel ABI, modules hash, etc.)
Boots via kexec into guest initramfs.
Guest initramfs mounts/assembles final / and switch_roots into it.
Host mounts root.squashfs read-only (loop mount).
Host copies vmlinuz + initramfs.img to tmpfs (avoid loop device dependency during handoff).
Host runs:
kexec -l <vmlinuz> --initrd <initramfs> --command-line "<cmdline>"
kexec -e
Guest kernel starts, executes initramfs /init.
Guest initramfs:
mounts proc/sys/dev
mounts the host root partition read-only at /host (from root= / UUID)
loop-mounts SquashFS image at /squash (from hmnd.squash=)
creates /newroot using one of:
RO mode: bind mount /squash as /newroot
Overlay mode (recommended): overlayfs with lowerdir=/squash, upperdir=tmpfs (or persistent)
bind-mounts persistence paths from /host \u2192 /newroot
switch_root /newroot /sbin/init
Guest kernel cmdline includes:
root= host partition identifier (UUID/LABEL) used only as a source mount in initramfs
hmnd.squash= absolute path to SquashFS on host FS (e.g., /ssd_2tb/images/hmnd-root.squashfs)
hmnd.persist= comma-separated list of host paths to bind into guest (rw)
hmnd.host_ro= comma-separated list of host paths to bind into guest (ro) for vendor firmware/libs
hmnd.overlay= tmpfs|persistent|off
hmnd.overlay_dir= (if persistent) where upper/work live on host partition
hmnd.debug= enables initramfs shell on failure
Example:
console=tty0 root=UUID=<host-root-uuid> ro rootwait \\\nhmnd.squash=/ssd_2tb/images/hmnd-root.squashfs \\\nhmnd.persist=/etc/hmnd,/var/lib/hmnd \\\nhmnd.host_ro=/lib/firmware \\\nhmnd.overlay=tmpfs hmnd.debug=1\n\n
DTB policy
Default: reuse running DTB (no --dtb passed to kexec).
Rationale: Jetson boot firmware applies fixups/carveouts; reusing avoids mismatch and \u201cGPU explodes later\u201d scenarios.
Firmware policy
Default: bind-mount /lib/firmware from host into guest (ro).
Optional: ship firmware in SquashFS if we pin a known-good bundle.
Kernel policy
Phase 1: use host-matched kernel ABI (or same kernel build) to reduce driver surprises.
Phase 2: allow guest kernel divergence once stable; require explicit compatibility validation.
Known risks
Hardware not reset; devices may be in non-default states.
Any NVIDIA stack mismatch between kernel/userspace/firmware becomes \u201cfun\u201d quickly.
Kernel policy
Guest kernel must be RT-patched and configured consistent with EtherCAT requirements.
Keep NIC driver, PTP, and any real-time settings in the guest image, not inherited magically.
EtherCAT risk focus
EtherCAT masters/drivers often assume clean device init and deterministic timing.
kexec may leave NIC/PCIe/interrupt state \u201cwarm\u201d, causing:
missed link negotiation edge cases
IRQ affinity drift
DMA ring state confusion
cycle jitter or DC sync instability
Mitigations
Before kexec (host):
stop EtherCAT master cleanly
unload EtherCAT modules (if applicable)
bring down relevant NIC(s)
optional: reset NIC via ethtool -r / driver reload
In guest initramfs / early boot:
set IRQ affinities and CPU isolation deterministically
re-init NIC link state before starting EtherCAT
enforce RT sysctls / chrt policies
Verification gates are stricter for x86 target (see below).
/usr/local/sbin/hmnd-kexec (launcher CLI)
systemd unit(s):
hmnd-kexec.target (optional)
hmnd-kexec.service (runs kexec when requested)
Optional \u201creturn-to-host\u201d helper:
a guest-side reboot --bootloader-entry= is not assumed
simplest: guest triggers plain reboot and firmware returns to host by default
initramfs /init implementing the mount/overlay/bind/switch_root logic
minimal debug tools in initramfs (busybox shell, mount, blkid, losetup, dmesg)
/etc/hmnd/ and /var/lib/hmnd/ treated as persistent bind mounts
Build produces a versioned bundle:
hmnd-guest-<platform>-<version>.squashfs
vmlinuz, initramfs.img, manifest.json
CI publishes artifacts to the robot host storage location.
Launcher selects bundle by version tag, \u201clatest\u201d, or explicit path.
Boot time: host \u2192 guest switch_root within target budget (e.g., < 20s).
Persistence: values written to /etc/hmnd and /var/lib/hmnd survive across guest boots.
Host cleanliness: after rebooting back to host, no filesystem corruption; services resume.
GPU sanity: run a minimal CUDA/NVDEC smoke test (or equivalent) post-boot.
Firmware availability: confirm expected firmware files resolved from bound /lib/firmware.
Device stability across repeated cycles: N cycles of kexec without requiring power-cycle.
Determinism: measure cycle jitter under load (CPU stress + network bulk).
DC sync stability: no drift beyond threshold over duration.
Link reinit reliability: repeated kexec cycles do not degrade link bring-up success rate.
IRQ and CPU pinning: validated post-boot (expected affinities, isolated CPUs, RT scheduling).
Initramfs logs to console with explicit step markers.
On failure: drop to initramfs shell if hmnd.debug=1.
Capture:
dmesg around kexec transition
journalctl -b in guest
on x86: cyclictest, EtherCAT master logs, NIC driver stats
M1 \u2013 Prototype (both platforms)
Deliverable: manual kexec into guest; initramfs mounts SquashFS and boots to systemd
Verification: repeated boot works; persistence bind mounts functional
M2 \u2013 Automation
Deliverable: host launcher + systemd integration; versioned image selection
Verification: kexec invoked reliably; logs available; no manual mount steps
M3 \u2013 Jetson Stabilization
Deliverable: DTB reuse path locked; firmware policy decided (bind vs ship)
Verification: GPU/codec smoke test passes; N-cycle kexec stability
M4 \u2013 x86 EtherCAT Hardening
Deliverable: pre-kexec teardown + guest early-boot NIC/RT setup; test harness
Verification: EtherCAT jitter/DC metrics within thresholds across N cycles
M5 \u2013 Packaging & Rollout
Deliverable: packaged into robot host images; docs; \u201cknown bad\u201d safeguards
Verification: field units can boot guest and report health; rollback path documented
kexec does not reset hardware; some peripherals will occasionally wedge without a power cycle.
Jetson DTB/carveouts: passing a custom DTB risks silent incompatibility; default is DTB reuse.
NVIDIA userspace/kernel mismatch: binding host libs/firmware into guest can become ABI roulette if not pinned.
x86 EtherCAT determinism: kexec warm state may break assumptions; requires aggressive verification and teardown/reinit.
Developers can boot a full robot stack as an immutable image in seconds without touching bootloaders or partitions. Jetson uses the host\u2019s DTB/firmware to minimize platform weirdness. x86 prioritizes determinism and EtherCAT correctness, with explicit teardown/reinit and hard gates on jitter and sync stability.
" }, { "id": "1177911301", "title": "Spec: Wifi QoS", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1177911301", "last_modified": "2025-12-11T09:18:49.237Z", "created_at": "2025-10-22T21:45:34.884Z", "body_html": "Spec: Robot Network QoS and Metrics Integration
Owner: Cody Griffin
Date:
Approvers: \u2014
Robot control and telemetry share the same uplink.
Interactive sessions (SSH, UI) and realtime traffic (ROS 2 / teleop) compete with bulk data (logs, updates, downloads).
This causes unpredictable latency spikes\u2014visible as control lag and WebRTC hiccups\u2014especially when bulk transfers saturate the link.
We need deterministic local egress scheduling so critical flows retain low latency regardless of other activity.
Additionally, these queues must be observable through our existing telemetry pipeline (Vector \u2192 Mimir/Grafana).
In scope
Egress shaping on the robot\u2019s external interface (eth0 / wlan0)
Local traffic classification into three service classes:
Realtime \u2013 ROS 2 UDP, WebRTC/teleop
Interactive \u2013 SSH, Web UIs, WebSockets
Bulk \u2013 everything else
Deployment on Ubuntu systems managed by NetworkManager
Export of queue utilization/drops to Vector for observability
Out of scope
Ingress shaping (handled later via IFB if required)
Preservation of DSCP markings beyond the host
Per-container/pod shaping (handled by upstream orchestration)
Remote or cloud QoS enforcement
Architecture
Classification (nftables)
Chain inet qos mark_out in OUTPUT hook.
Marks packets:
UDP 7400\u20137550 + multicast \u2192 fwmark 10 (realtime)
TCP 22 \u2192 fwmark 20 (interactive)
All others \u2192 fwmark 30 (bulk)
Scheduling (tc/HTB)
tc-qos.sh creates three HTB classes on the external interface:
1:10 Realtime (10 Mbit min / full ceil)
1:20 Interactive (10 Mbit min / full ceil)
1:30 Bulk (5 Mbit min / full ceil, lowest prio)
Each class uses fq_codel for sub-queue fairness.
Filters match fwmark values to direct packets to proper class.
Activation
/etc/NetworkManager/dispatcher.d/70-qos executes tc-qos.sh on up event for the external NIC.
nftables.service ensures mark_out chain is loaded at boot.
Observability (Vector exec source)
Vector runs tc -s qdisc show dev eth0 every 30 s.
A transform parses counters (Sent, dropped) into Prometheus-style metrics:
tc_packets_total{class="10"}
tc_drops_total{class="10"} etc.
Mimir/Grafana dashboards visualize queue load and drop rates per class.
Phase | Deliverable | Verification |
|---|---|---|
M1 \u2013 Prototype | Working manual setup of nftables + tc on dev robot | Ping RTT under 50 ms with bulk flood active |
M2 \u2013 Automation | Dispatcher script auto-applies rules on interface up |
|
M3 \u2013 Observability | Vector collects and forwards metrics | Metrics visible in Grafana dashboard |
M4 \u2013 Rollout | Systemd/NM integration packaged in robot image | All field units report queue metrics |
M5 \u2013 Optimization | Tune rates per bandwidth profile | Realtime jitter < 20 ms under 80% link load |
Functional Tests
Bulk transfer (e.g., iperf3) should not elevate RTT of ROS 2 UDP pings.
SSH remains responsive under heavy download/upload.
Metrics
tc_packets_total and tc_drops_total increase monotonically.
Drop rate of realtime class \u2248 0; bulk may drop under saturation.
Dashboards
Grafana panels: per-class throughput, latency histograms, drop counters.
Alert when realtime queue drops > 0 or delay > 50 ms.
Logs
Vector logs confirm successful exec scrape every 30 s.
nft list ruleset and tc -s qdisc exported for troubleshooting.
table inet qos {\n chain mark_out {\n type route hook output priority mangle; policy accept;\n\n # ROS 2 / DDS UDP ports\n udp dport 7400-7550 meta mark set 10\n udp sport 7400-7550 meta mark set 10\n ip daddr 224.0.0.0/4 udp meta mark set 10\n ip6 daddr ff00::/8 udp meta mark set 10\n\n # SSH\n tcp dport 22 meta mark set 20\n tcp sport 22 meta mark set 20\n\n # Default\n meta mark 0 meta mark set 30\n }\n}\n\n#!/usr/bin/env bash\nset -euxo pipefail\nIF=$1\nUP=100mbit\n\ntc qdisc del dev "$IF" root || true\ntc qdisc add dev "$IF" root handle 1: htb default 30\ntc class add dev "$IF" parent 1: classid 1:1 htb rate $UP\n\nfor class prio rate in "10 0 10mbit" "20 1 10mbit" "30 2 5mbit"; do\n read c p r <<<"$class prio rate"\n tc class add dev "$IF" parent 1:1 classid 1:$c htb rate $r ceil $UP prio $p\n tc qdisc add dev "$IF" parent 1:$c handle ${c}: fq_codel\ndone\n\nfor proto in ip ipv6; do\n for h in 10 20 30; do\n tc filter add dev "$IF" parent 1: protocol $proto handle $h fw flowid 1:$h\n done\ndone\n\n#!/usr/bin/env bash\nIF="$1"; ACTION="$2"\nif [[ "$IF" == "eth0" && "$ACTION" == "up" ]]; then\n /usr/local/sbin/tc-qos.sh "$IF"\nfi\n\n
[sources.tc]\ntype = "exec"\ncommand = ["/usr/sbin/tc", "-s", "qdisc", "show", "dev", "eth0"]\ninterval_secs = 30\n\n
watch -n 1 'tc -s qdisc show dev eth0'\n\n
Outcome:
The robot now enforces deterministic outbound traffic priority. ROS 2 and teleop stay crisp even during bulk uploads. Queue stats flow into Vector, making QoS both visible and tunable without SSHing into the robot.
We\u2019ve had a number of discussions around how to deploy robots. This is my attempt to put together a strawman end-to-end deployment scenario. This will be heavily influenced by my prior experience with deploying AMRs into commercial spaces, but looking forward torwards a solution that is more agentic and AI-driven.
Deployment
What is deployment? There is the physical aspect: shipping the robot, uncrating it, charging it for the first time. There is the security aspect: adding certificates, keys, identities. We\u2019ll ignore these (very real, still difficult) problems for now and focus on the actual \u201csetup\u201d portion. This involves a few key things:
How does the robot learn its environment?
How does the robot know what to do?
How does the robot know where to do it?
Deployment Anti-Pattern
I don\u2019t know why, but the usual pattern for deployment is to create some sort of sprawling interface. Something generates a map, and then operators are expected to draw paths and waypoints on some GUI, draw no-go zones, etc on the interface to adjust the robot\u2019s workflow.
This is kinda miserable in practice. For one, operators are manipulating an abstract model of the robot\u2019s environment/workflow and it takes a lot of specialized skill to be able to do this correctly. Consider a scenario where a robot takes a corner \u201cimproperly\u201d. Maybe it\u2019s going too wide or too sharp - the details are unimportant for this example. We want to adjust the path.
The \u201cclassic\u201d way this is done is to draw a new waypoint. But this is easier said than done. You see the robot do a thing physically, but now you need to move to an abstract representation. You need to know where to click to add the waypoint. You may need to adjust/re-orient the waypoint if you need directional constraints. If the workflow is complicated, you need to be mindful about topology - adding a new waypoint into the middle of an existing path can be difficult when there are many overlapping/looping paths in the area. Finally, once the workflow is adjusted you need to go BACK to the robot to re-execute and hopefully it works. Miserable.
Humanoid Deployment
From prior experience, I\u2019ve learned its best to close the gap between deployment and operation of the robot as much as possible. Having operators shifting between real-world context and abstract \u201cUI\u201d context of waypoints and no-go zones is extremely challenging in practice. So let\u2019s make some assumptions:
we have teleoperation - we can move and manipulate via teleoperation
we have SLAM - we can map a space (and get an initial mapping trajectory - important for later)
we have policy execution
We have 2 of these 3 accessible from \u201cSockpuppet\u201d - so let\u2019s pretend for a moment that our mapping is available there too. Now we have a \u201cdeployment tool\u201d. Note that This assumes we have some sort of teleoperation setup available - but another approach would be a local display and a torque-assisted robot that allowed you to physically move it through the space manually. We\u2019ll consider this an implementation detail. What\u2019s important is that we have a way to control robots manually in the actual environment.
Mapping and Setup
When we first show up in a new environment, we should position the robot at some initial pose and beginning the mapping process. The robot should then traverse the environment (hopefully uncovering the space) and then we should return back to that initial pose to save the map. This is a very important detail, as it gives us a free loop closure constraint.
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1372160012/Untitled%20Diagram-1764767332555.drawio.png?api=v2&version=1\")
Now we have a map, and we have an initial spline loop where both the beginning and end pose are our known initial pose.
Editing
So our robot starts at our initial pose, and let\u2019s assume for this example we need to take totes from the ingress table (on the right) and place them on the egress table (on the left). We can start the robot at its initial position, and then enter \u201cdeployment mode\u201d and teleop the robot to the table on the left and add a new waypoint. Now we have a looping spline between these two waypoints.
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1372160012/1764767747998-Untitled%20Diagram-1764767332555.drawio.png?api=v2&version=3\")
Now we can start the robot at its initial pose, and we can expect the robot to navigate to the ingress table, and then return back to our initial pose. We have two waypoints now - w0 (our initial pose) and w1 (the ingress table). We can choose to label or name a waypoint to turn it into a \u201ckeypoint\u201d (point of interest) which we can use within our workflow. Keypoints are \u201clocked\u201d which we will see is important for the next step.
When we teleoperate the robot to the keypoint and save it, we can also tell the robot to run a policy - \u201cpick up the tote\u201d. This saves a workflow transition in addition to our mapping data, so that the robot knows that when its at the ingress keypoint its supposed to run the policy.
Then we can teleoperate the robot to the egress table, save a keypoint, and tell the robot to \u201cput down the tote\u201d. but here, something else happens. You could see how if we add many keypoints and waypoints, our path will become complex and unpredictable. Editing will get more and more difficult. We want to keep a light touch. Our initial pose is still \u201cjust\u201d a waypoint. An implicit one at that. When we drop a new waypoint at the egress table, we can look at any waypoints close by in the path and remove them, simplifying the path and keeping things manageable over time. If we don\u2019t want a waypoint to be dropped, we simply lock it by giving it a name (turning it into a keypoint).
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1372160012/1764767747998-Untitled%20Diagram-1764767332555.drawio.png?api=v2\")
The radius/distance that we clear waypoints can be configurable of course. But at this point, without ever opening up a map editor or clicking, we\u2019ve managed to tell the robot about two key points and that it should run policies at each point. Under the hood this involves inferring our map waypoints/keypoints, and inferring some states in the workflow (before policy/after policy) and transitions (our pick/place policies and navigation).
Rewind and Constraints
Let\u2019s assume that we add some new static obstacle to the environment. Its positioned in a way that SOMETIMES the planner tries to navigate above it, which risks blocking access and causing robots to get stuck. The robot could be stopped during this (go into teleoperation/manual mode) and then we could \u201crewind\u201d the robot by moving backward through the trajectory until the robot is back at the table. Then we could teleoperate the robot to the proper side of the obstacle, add a waypoint, and then resume. Now the robot will reliably and consistently pass on the proper side of the obstacle.
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1372160012/1764770954698-1764767747998-Untitled%20Diagram-1764767332555.drawio.png?api=v2\")
In my opinion, this \u201crewind\u201d is a killer deployment feature. Without it, you have to somehow remember what the robot did incorrectly, then navigate to your map editing tool, somehow make the fix, then re-run the workflow. With it, you can scrub back and forth along the workflow. This scrubbing can be used as a quick check of new waypoints immediately (add the waypoint, move backward to see how it constrains the approach, and through it to see how it constraints the follow-through)
Snap and Edit and Versioning
As the robot progresses through its workflow, we can show the current waypoint/workflow state/transition. If the robot starts doing something wrong (perhaps its following an older, stale waypoint or its generated some unconstrained path), the operator can enter deployment mode and scrub forward or backward to see which waypoints are influencing the current action. Those waypoints can then be modified (their properties, not their pose) or deleted right there with full context.
There are two orders of waypoints that are important to keep in mind. The version (basically ordered by timestamp) and the index (basically ordered by arclength through the spline). If we keep a full log of all edits, operators can easily version control the workflow and undo changes. By relying on the robot\u2019s position to act as a cursor through the index (its currently position along the workflow), its easier to make sure that waypoints are \u201cinserted\u201d into the existing path while preserving the topology.
Over time our map may need to be regenerated. As long as we can maintain some number of fixed loop closures (most of our workflows will be cyclic in some way), we should be able to regenerate maps. This is a whole area of research, and will depend on the underlying mapping/localization stack, but I just want to highlight the possibility that this is handled automatically.
Branching and Looping
More sophisticated workflows may involve the robot needing to \u201cbranch\u201d from a waypoint. For example, some inspection-related policy may result in good or bad results, with good being placed to the left, and bad being taken to some bin on the right. When entering deployment mode at a waypoint, the robot can simply be operated in another direction and the new waypoint saved, with the branch being inferred.
This is probably the area where this \u201cimplicit\u201d approach becomes most difficult - we need to capture the branching/looping nature in the workflow but its not clear how this could be done purely with a local interface like this. What\u2019s most important to capture with the robot is the physical pose of the waypoints to ensure that the various constraints are created, and to allow the incremental editing of the existing paths. Its possible that more complex looping or branching-type workflows will require another pass within the fleet mgmt editor itself. Who knows?
Policy and Agentic Deployment
As mentioned in some other places, there is a tension that exists between more policy-driven approaches and more \u201cclassical\u201d constrained approaches. Classical approaches are more predictable, and can fit into existing rigid workflows more easily, but don\u2019t adapt to exceptional special cases very well. Policy-driven approaches are extremely adaptable and flexible but may not always do what customers need or may be difficult to integrate with other more rigid processes. Hopefully here we can split the difference.
We should be able to allow deployment operators to stitch together complex end-to-end workflows from well-supported policies and behaviors. In turn, the end-to-end execution of these workflows can be used to train longer-horizon policies which incorporate the necessary constraints. The hope (and it is just a hope) would be that very dense, complex workflows and deployment schemes could be made less granular over time as policies become more capable. Instead of \u201cgo to ingress\u201d, \u201cpick up tote\u201d, \u201cgo to egress\u201d, \u201cplace tote\u201d we should be able to run \u201cpick up totes from the ingress table and take them to the egress table\u201d. Easier said than done. Similarly we could do RL on a reasoning model based on these workflow traces to get more predictable workflows.
Another possible use of AI during deployment would be scan documents describing SOPs and building the initial scaffold of the workflow. The keypoints could be extracted from the SOP so that the interface has a list of them ready to go, along the deployment tool to guide the operator through the setup with a bit less friction.
Summary
I think what\u2019s outlined here is a viable deployment scenario that allows operators with full situational awareness (via manual or teleoperation) to incrementally evolve the workflow without being burdened by the challenges of rapidly flipping between real operations and an abstract map view of the workflow. Operators can scrub forward and backward through time to insert new waypoints to constrain navigation so that we can run policies and navigate predictably through customer sites.
\u201cBut Cody - we won\u2019t be using lidar maps or stone-age SLAM methods forever\u201d. Yep. But I think we must provide an easy way for deployment teams to constrain navigation in order to reliably work in real-world environments. A deployment tool that\u2019s very \u201cmap centric\u201d will effectively REQUIRE some sort of map view to be edited, which I think is more constraining to our navigation and perception efforts. An implicit deployment scenario outlined here doesn\u2019t really care about specifics - only that the robot knows where it is (some map pose), and which waypoints need to be satisfied in which order (index order not necessarily the order the waypoints were created). Obviously we\u2019d need to think about this in the context of VIO and more policy-driven approaches to navigation but that\u2019s why I\u2019m writing this down.
Questions
- How would you propose we mark speed zones or would this be automatic? - Jack Smith
-How would we mark no go zones? - Jack Smith
Summary: Local Fleet-centric Control Plane for Robot Coordination
Owner: Cody Griffin
Date:
Approvers:
Informed:
Related:
When many robots are operating within a single operational zone, perhaps collaborating on the same workflow, the existing behavior-tree model of our applications begin to fall short:
Maps only exist local to the robot - we have no way to share them.
We have no way to coordinate behavior tree progress between robots.
Behavior trees cannot really be analyzed to exploit concurrency or identify bottlenecks.
Defining behavior trees requires modifying application code and lengthy build-test-release cycles.
Even just a few robots operating in a small area can cause things to get clogged up. There may only be space at a table for one robot to pick up one tote. Or there may only be space in one aisle for one robot to pass at a time. We need shared maps and workflows within which robots can coordinate.
We also need our robots to possibly integrate w/ customer-managed systems like WMS, point-of-sale/orders or anything else that may require additional context for how robots ought to coordinate their work. We need to be able to accept work items from some external system, and report progress events to these same external systems.
Another major challenge (maybe specific to short-term realities within Humanoid) is running demos across a small pool of Alpha hardware. For a demo which needs to run across multiple robots, it will become unbearably difficult to manually recover/setup each robot before demos. This will become even more painful if these robots need to be used for other demos or PoCs, resulting in significant development and configuration churn.
Why not a completely agentic fleet manager?
In a perfect world, business operations would be scripted by simple rules and nothing would ever go wrong. Just move the items from A to B. Forever. These rules are easy to describe (and thus specify in some business process). But everything fails all the time.
Instead, we can write the happy path as some rigid workflow, and then call out to an agent or any other 3rd party API to keep things moving when exceptional scenarios arrive. Exceptional doesn\u2019t mean rare, so very powerful agents could significantly improve process throughput.
Finally, agents often require fine-tuning to reach viable performance beyond simple demonstrations. The trace logs could be used for RL (score = work item throughput for example). The rate of work items (robot charging, order fulfillment, machine feeding, tote handling) is maximized. We can use the workflow to string together more complicated prompts and longer-horizon actions for our VLA.
In Scope:
Fleet Enrollment and Site Assignment (non-PKI - simple identifiers)
Simple one-way map sharing (static)
Workflow setup (static)
including points of interest on our static map
Shared workflow execution (static)
Dynamic workflow assignment (add robots / remove robots from a static workflow)
Simple \u201cworkflow-following\u201d application running on robot
Out of Scope
Fleet-local PKI (see Spec: HMND Core Control Plane)
Dynamic mapping or mapping updates
Dynamic workflows or workflow updates
Multi-map workflows.
Training data or data compliance (still relying on existing humanoid console + ML buckets)
no jurisdictions yet
Data plane - teleop still done \u201crobot-local\u201d (sockpuppet hosted by robot) rather than fleet-local
At the top-most level we have our global control plane, which includes every unit we\u2019ve ever manufactured. For clarity we won\u2019t refer to this as a \u201cfleet\u201d and we\u2019ll reserve that terminology for robots that exist within some customer-specific context (meaning they have sites, maps and workflows). We\u2019ll call this top-level grouping the \u201cfield\u201d, as this goes along with \u201cfield replacements\u201d and \u201cfield population\u201d terminology which isn\u2019t uncommon. This name can be changed if something better is proposed.
Within the field, we may have one more or more fleets, which are robots which have been assigned to some customer context. In the future this will include RaaS-specific things like billing along with concerns like jurisdictions or data sovereignty - but for now its mostly a way to group robots independently of our global control plane. Fleets are largely a logical grouping of robots, meaning there is no expectation that robots within a fleet will share maps or workflows.
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1170341889/Untitled%20Diagram-1761039594597.drawio.png?api=v2&version=1\")
The more physical grouping of robots will occur at the site-level. A site is a physical location where robots may be assigned and where these robots are expected to work. Maps and workflows will be scoped to a site for example.
For the moment we will assume that robot fleet enrollments are static - meaning we will manually onboard a robot to a specific fleet. This allows us to build the data model now without paying for the complexity of dynamic assignments. In the future, especially under a RaaS model, we should expect robots to be transferred/re-enrolled between different fleets.
To see how this all fits together - let\u2019s imagine Humanoid as a fleet. We\u2019d have 3 sites - one in Vancouver, one in Boston and one in London - each with its own maps and demo areas/workflows. Robots are assigned to these specific sites. A robot may be assigned to London as its built there, but if its later shipped to another site we\u2019d simply re-assign the current site (leaving the fleet enrollment alone).
In the scope of a Fleet, we may have some number of Operations, the top-level container for organizing work. An example of this may be something like \u201cOrder Fulfillment Operations\u201d for a large distribution company. At the moment it isn\u2019t clear how much functionality will actually exist at this level, so we could treat this as simply a tag or a label to help organize things. In the short-term this layer is almost certainly unnecessary, but if we start versioning or inferring workflows from demonstration it could become much more important.
The real meat here would be the workflow which is a description of a set of tasks which can be performed in sequence to accomplish some operational goal, within the context of a specific site and map. A workflow can encode the various states and transitions robots may take to complete the work (waiting for a package, filling a tote, carrying a payload through a series of waypoints). By representing the workflow as a \u201cPetri-Net\u201d, we can scale the number of robots within a workflow to increase parallelism in addition to identifying bottlenecks. In the future, we can even discover or adapt these petri-net workflows from event logs meaning that robots can learn workflows from demonstration.
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1170341889/Untitled%20Diagram-1761049273775.drawio.png?api=v2&version=4\")
Tasks represent behaviors that can be performed by the robot autonomously. For example, navigating to some point of interest on the map or executing some specific policy. In addition to actually executing the task, the robot should verify pre/post-conditions as part of task execution. Meaning that the robot should verify that there is a tote in front of it before executing the \u201cpick up tote\u201d policy, or that there is a global path available before executing a navigation policy. After picking up a tote, the robot should validate that its successfully grasped the tote or after navigating that it has reached the PoI within some margin. This core loop represents the new application that robots will run in order to execute these fleet-owned workflows. Roughly:
Get new task lease - returns any transitions that the robot is permitted to make (expiring after some time)
Validate task preconditions
Execute task - potentially \u201crenewing\u201d leases if its taking a long time to avoid spurious timeouts/churn
Validate task post-conditions
The exceptional cases above (failing validate or execute) can be handled via local tele-operation initially, but we may be able to incorporate more agentic capabilities into the fleet manager itself to better handle these issues at the \u201csystem 3\u201d level (avoiding the need for teleoperation ideally).
In addition to the actual dependencies/ordering of tasks, the workflow also contains information about the capacity of individual \u201cplaces\u201d in the petri-net. This ensures that only one robot at a time is occupying spaces like the end of a conveyor belt or a table for staging totes.
Oracles & Middleware
The workflow is a relatively simple description of how robots and goods are expected to progress through various tasks. Each edge is guarded by a number of conditions. For example, the robot may need to be close enough to SKU #XYZ before it can be expected to pick the item to progress the workflow. How does the workflow know?
None of this information can really be kept inside the workflow itself, as it would cause an explosion in the state-space and render the entire thing computationally intractable. Instead, the workflow can consult a number of different \u201coracles\u201d about the state of the world. Because Oracle is a well-known ERP, we probably shouldn\u2019t call these oracles - and they basically function like integration middleware or plugins so we\u2019ll stick with middleware for now.
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1170341889/Untitled%20Diagram-1764534673454.drawio.png?api=v2&version=1\")
For example, we could have a simple \u201cspatial middleware\u201d running which knows the map and the locations of some static keypoints. A \u201crobot middleware\u201d could tell us where robots are in that map (current pose) along with charge levels and other relevant facts. The customer\u2019s PoS, WMS or other ERP systems could be integrated as middleware, so that the workflow can know how to resolve order_ids, or locate individual SKUs.
A more extreme example of an middleware would be an agentic VLM capable of taking arbitrary information/context about the fleet and making decisions. Keeping these systems outside of the workflow itself means we can keep things verifiable and understandable, while opening up endless possibilities for expansion and integration. Robots never talk to middleware directly, so the workflow can scale to many integration points without adding complexity to the robot software itself. Integrated systems never communicate with anything other than middleware, which helps to limit interactions and keep things simple.
Concurrency Control
One of the main challenges of a multi-robot workflow is managing congestion/conflict. How do we make sure that only one robot at a time passes through a choke point or approaches a tote? Especially in a noisy environment with poor network connectivity?
In cases where the robot needs to perform some task (navigating, running a policy, etc) the robot should acquire a lease from the workflow coordinator. Leases are exclusive permissions to perform some task, and the coordinator can guarantee that only one robot will have a valid lease at any given time. Leases should expire after a short-period of time (though robots could renew if necessary). The workflow itself cannot know whether or not it is safe to grant a lease, but it can make sure that only one robot at a time has the lease. The rough separation of responsibilities is thus:
Robot \u2192 responsible for knowing that it should obtain a lease before performing a transition. If the robot performs a transition without a lease, that\u2019s an implementation bug.
Coordinator \u2192 responsible for granting up to K leases (depending on the capacity)
Middleware \u2192 responsible for determining wether or not the lease can actually be granted
Python simulation of PoC Workflow (TBD) using off the shelf petri-net library/DES tooling \n\n \n\n \n\n\n\n\n \n HECO-1736\n -\n Getting issue details...\n STATUS\n \n
demonstrate multiple robots making progress concurrently within the workflow
demonstrate adaptability as robots are removed/added to the workflow (1, 2 or 3)
Wrap PoC workflow in simple coordinator API \n\n \n\n \n\n\n\n\n \n HECO-1737\n -\n Getting issue details...\n STATUS\n \n
Drive petri net (from milestone 1) with API calls, returning task assignments/leases as appropriate
Just do everything in RAM (small) and checkpoint to S3 periodically if necessary (mostly for debug)
Create simple map and robot status oracles
robot status tells us which robots are around and what their current pose is (updated periodically by robots directly) \n\n \n\n \n\n\n\n\n \n HECO-1739\n -\n Getting issue details...\n STATUS\n \n
map oracle stores our map + key points \n\n \n\n \n\n\n\n\n \n HECO-1738\n -\n Getting issue details...\n STATUS\n \n
Create robot behavior tree (app) for executing workflows \n\n \n\n \n\n\n\n\n \n HECO-1740\n -\n Getting issue details...\n STATUS\n \n
App should make API calls to coordinator to get task assignments
App should execute task assignments (no fallback yet)
Workflow Visualization (stretch) \n\n \n\n \n\n\n\n\n \n HECO-1741\n -\n Getting issue details...\n STATUS\n \n
Since we have the static map saved under the site, we should be able to render along with robot telemetry
TBD - lots of different ways to achieve this
Fleet Enrollment - for this initial MVP, fleet enrollment will be setting an HMND_FLEET_TOKEN somewhere in the environment. In the future we will use PKI and mint fleet certificates (similar to how we mint provisioning certificates in the global control plane)
Site Mapping - a static map (plus some information about origin pose) is saved during mapping. An operator uploads this map to an endpoint.
Site Assignment - an API call to the fleet manager will allow setting a robot\u2019s current site. A robot will download the static map for the configured site.
Workflow Definition - a declarative representation of the workflow (petri-net, PNML, YAML - lots of options) is uploaded to an endpoint.
Workflow Assignment - a robot is added to the workflow via an endpoint.
Petri-Nets are a formalization of processes comprising \u201cplaces\u201d and \u201ctransitions\u201d. For this reason they are sometimes also called PT-nets. Each place can contain some number of \u201ctokens\u201d, which can move through transitions to go from place to place. A bit unintuitive, but places don't correspond to physical places (like locations in the map), but are are more conditions or constraints which the tokens satisfy. Because we are dealing with discrete sets, we typically don't encode poses directly in the workflow. We simply \u2018ask' the map questions about the spatial relationships in our transitions\u2019 guards. This also saves us from needing to re-configure poses we embedded into the workflow of our map updates.
Petri nets allow for non-deterministic execution, meaning they are well-suited for concurrent distributed systems like a multi-robot fleet workflow. In these applications, robots are one of the types of tokens that will transition through the places in the net, but we can also represent payloads like orders or totes or assemblies - anything that must move through the workflow.
There are all sorts of fancy formalized variations of petri nets that are relevant for us. For example, maybe a place can only have so many tokens. Maybe only 1 robot at a time can stand in front of the conveyor belt. These places are k-bounded, and these are bounded petri nets. When tokens aren\u2019t fungible, then \u201ccolored\u201d petri nets may be appropriate. If we want to model the timing of transitions we may want to look at timed or stochastic petri nets. There are tons of sources on these things and a rich ecosystem of algorithms.
Inferring or Updating Workflows
A really interesting area of research is known as \u201cprocess mining\u201d, which is the goal of inferring something like a petri net from the traces generated by some process. Using a technique like alpha mining or inductive mining, you can take the logs from some arbitrary process, and then reverse engineer the petri net that would produce them. Although not in scope for this initial demo, this has very interesting ramifications for learning workflows from demonstration and for adapting workflows to changes via intervention.
Workflows - A directed graph of states connected by tasks
States - Named buckets of work items or units satisfying some constraints.
Tasks - State changes with guards (CEL expressions) and optional timing. Two kinds:
internal: coordinator-only bookkeeping; no robot action.
robot: requires exclusive lease granted to a robot; the robot executes a local policy/behavior and makes claims about the status.
Work Item/Unit - Units of work. Each has a token_id, a place_id, and a color (typed payload map). Work Items flow from state to state via tasks, possibly being transformed in the process.
Middleware - A service which provides external context - issuing tickets or providing facts relevant to the workflow
Coordinator - The service responsible for executing a workflow. Its goal is to \u2018push\u2019 work items along the workflow.
Facts - Site-scoped, versioned, TTL\u2019d objects the coordinator caches from oracles (e.g., maps.poi/BIN-A17, order.line/OL-42, robot.status/R14). Guards read facts; no network in the hot path.
Leases - A temporary binding of inputs (satisfying guards) and tokens which agents may use to perform specific tasks.
Claims - The result of any task executed.
States
min/max/average of how long work items were in state
token throughput
Tasks
success rate
duration
token throughput
Leases
average hold time
conflicts/rejections?
Goal is to make looking at things like \u201cUPH\u201d or \u201csuccess rates\u201d intrinsic to the workflow - measuring this stuff should pretty much be \u201cfree\u201d.
Many collaborating agents (robots, chargers, 3rd party ERP systems) may work together via the coordinator API to advance work items through the workflow as quickly and reliably as possible.
Agents will make requests to the Coordinator for leases to perform tasks which will advance work items through the workflow. Agents will then return any claims (that these tasks succeeded or were retried or failed or whatever) in the request for a new lease.
The coordinator will, based on the claims, move the work items between states in the workflow. The rate (how many work items are transitioned to the \u201cdone\u201d state per unit time or UPH) and reliability (how many work items that were accepted were transitioned to the \u201cdone\u201d state) can be produced directly as KPIs. Or agent availability (time not spent in some \u201cdown\u201d state).
Our coordinator needs to get a lot of external context to evaluate task guards or verify claims. Agents communicate with middleware services which may have side effects (communicating with external services or running some manipulation or navigation policy) while they communicate with the coordinator to determine what to do next.
A robot agent could periodically update a robot middleware with the current pose. The coordinator may lookup this robot\u2019s pose to see if a guard has been satisfied (\u201cdistance(r1.pose, poi1) < 0.1\u201d), which will poll for this pose and cache it for a bit.
An \u201cSAP\u201d agent could feed new work items (totes arriving at some ingress stack) and subscribe to events about the workflow which can be used to update records in SAP or to add additional work items.
claims = []\nwhile True:\n claims, leases = coordinator.get_leases(claims)\n task = choose(leases)\n claims.append(run_policy(task))\n # run_policy may have side-effects and may call middleware with updates?\n
" }, { "id": "1039335426", "title": "How To Slack", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1039335426", "last_modified": "2025-11-13T17:45:13.897Z", "created_at": "2025-08-13T18:56:35.187Z", "body_html": "
Slack is an amazing tool for coordinating and communicating, but it's not always obvious how to get the best use out of it. What follows is a rought attempt to provide some guidance on how to work effectively in Slack.
Short, generic names (#software, #build, #ai, #random) should be the busiest places. Broad channels build network effects - more eyes, more serendipity.
People learn that their queries will be answered in these channels because they are active. Niche channels are fine, but the default should be to post in the broadest relevant one. Make it as easy as possible for people to decide where to post.
Post where it will have the highest chance of being seen by the right people, even if it feels slightly off-topic.
Avoid scattering related discussions across multiple tiny channels - centralize instead.
Keep channels open by default.
Transparency builds shared context and avoids \u201cprivate knowledge silos.\u201d
If it must be private, at least announce or summarizes what\u2019s happening somewhere. For example, the weekly TLT digest serves this purpose.
Don\u2019t @channel or @here unless it\u2019s genuinely urgent for everyone.
Direct @-mentions are fine, but only for people who truly need to respond.
Think before pinging: \u201cDo they need this now, or can they find it when they check in?\u201d
Always reply in threads unless you are starting a new conversation.
Threads keep channels scannable and make context easier to follow.
If a thread becomes high-value, summarize it back to the channel.
Don\u2019t expect immediate replies - Slack is not an on-call pager.
Multiple conversations can run in parallel across different timescales.
You can leave a thread for hours or days and still pick it back up.
Provide enough detail in your initial message so others can respond without hunting for backstory.
Link to docs, code, or prior threads rather than re-explaining from scratch.
Summarize outcomes so lurkers still learn from the conversation.
Due to limited retention, record major decisions somewhere more deliberate (confluence or git)
Let's take a look specifically at the engineering channels for Humanoid as a whole. We have approximately 20 team_eng-* channels (though not all are in active use), and roughly 10 robot_ipt-* and a few other integration channels.
The intent (I believe) was to provide places for functional area/team discussion in additional to cross-functional/cross-team integration.
When a developer wants to make a post, they need to almost digest and process our entire org chart in order to understand where to post. Is this an integration request? Which team? Which subsystem?
The root issue here is that this is a tough decision and we don't have any 'tie-breaker' logic. So communication gets spread around and we lose network effects and ultimately the utility of the tool. We just end up with dozens of nearly dead channels.
Let's imagine we allow developers to endlessly create new channels - what prevents us from just smearing out communications? Don't we need to lock everything down? Believe it or not - you don't!
We have already started the process of consolidating some channels (thanks to all who have helped make this happen) which has I believe resulted in more effective use of Slack. But we can do better.
I suggest we drop our naming scheme entirely to help reduce the noise of channel names and to make defaults more obvious.
Instead of:
```
#team_eng-software
#team_eng-hmnd-robot
#chapter-software
```
Just `#software`.
At some point, channels will become too busy. If the main page scrolls so fast that no one can follow or if ~50% of the messages are pure noise to any one participant, then chances are the channel is too broad and too busy and it's worth splitting something out.
For an example, let's consider #software. Assume we have one giant software channel. At some point, it will be flooded with questions from developers about random build errors, CI issues, submodules mismatches etc etc. Eventually it may make sense to move build support conversations to a separate #build thread.
The general goal should be to start with few channels, building critical mass and then splitting off new channels when it makes sense. This more organic growth with ensure good utilization of the tool while keeping things organized.
" }, { "id": "1186627652", "title": "Spec: Secure Thor + OTA", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1186627652", "last_modified": "2025-10-27T14:21:42.153Z", "created_at": "2025-10-25T01:28:56.112Z", "body_html": "Spec: Secure OTA, Encrypted Rootfs, and Verified Boot on Thor
Owner: \u2014
Date: \u2014
Approvers: \u2014
We\u2019re running robots on Nvidia Thor-class compute. We want to ship new OS images over the air (OTA), atomically upgrade them, roll back safely if things go sideways, and prevent anyone (including with physical access) from tampering with or spying on the filesystem.
Right now we have none of that.
Pain today:
Flashing new software means invasive manual re-image and chasing revisions
Root filesystem is readable if you pull the NVMe.
Kernel / boot chain are not actually locked.
There\u2019s no A/B slotting, so upgrades are high-risk (in-place).
We need:
A deterministic way to build a full stack image (bootloader \u2192 kernel \u2192 userland).
A way to boot our new stack temporarily on a live robot without bricking it.
A way to lay down an encrypted disk image with A/B rootfs slots.
A way to verify what boots, with hardware-backed trust.
In scope
Bazel-built Linux for Tegra (L4T) kernel, modules, and firmware blobs as a versioned artifact.
Bazel-built rootfs (Ubuntu 24 base + our services) as a versioned artifact.
Capture of Nvidia proprietary bits (CUDA, graphics, Holoscan, TensorRT, etc.) into two .deb packages we control:
nvidia-kernel.deb
nvidia-userland.deb
Live boot via kexec into that Bazel-built rootfs, without flashing yet. (intermediate step)
Full disk encryption via LUKS on NVMe.
A/B rootfs layout on NVMe for OTA + rollback.
Boot chain hardening: fuse burn, measured / trusted boot.
Out of scope
Secure comms channel for OTA delivery (we assume we can already pull artifacts from a trusted endpoint over TLS/mTLS).
User data partitioning / log persistence design - assume we have a shared data partition to mount overlay
Recovery via physical UART/U-Boot interposer
We\u2019re going to walk from volatile to permanent:
We build images.
We boot them temporarily.
We lay down encrypted disk with A/B slots.
We flip the boot target.
We lock the silicon.
We produce three outputs from CI, all immutable and content-addressed.
Kernel bundle (thor-kernel.tar.zst)
Cross-compiled L4T kernel for Thor, plus kernel modules, device tree blobs (DTBs), and firmware blobs.
Built under Bazel, not JetPack. We vendor the Nvidia kernel sources / defconfigs / out-of-tree modules into the repo, and we make Bazel rules that:
apply our patches (PREEMPT_RT knobs, IRQ affinity, missing drivers for our PCIe devices, etc.),
spit out:
/boot/Image (or Image.gz/zImage depending on chain),
/boot/*.dtb,
/lib/modules/<kernel-ver>/**,
proprietary firmware blobs for GPU, ISP, etc.
This bundle is later repackaged as nvidia-kernel.deb.
Userland bundle (thor-userland.tar.zst)
Ubuntu 24 rootfs we own.
Our daemons, our QoS stack, ROS2, teleop stack, inference services, etc.
No Nvidia proprietary runtime yet.
Built root-first (Bazel pkg_tar / deb_image style), so we get a reproducible / tree.
This becomes the canonical \u201crobot OS userspace.\u201d
Nvidia proprietary split We don\u2019t want to be hostage to JetPack, but we still need CUDA, Vulkan/Wayland accel, display stack, TensorRT, Holoscan, codecs.
So we snapshot those from a reference L4T/JetPack-style install on a sacrificial dev box and repack them into two internal Debian packages:
nvidia-kernel.deb
Kernel-adjacent binaries:
GPU kernel modules (if they\u2019re out-of-tree)
Firmware blobs not already baked
Any initramfs hooks that must run early (power rail bring-up, GPU clocks, etc.)
UAPI headers/so files that tightly couple to kernel version
Installed into /lib/firmware, /lib/modules/<ver>/extra, /usr/lib/modprobe.d, possibly /etc/initramfs-tools/hooks/nvidia-*.
nvidia-userland.deb
User-mode acceleration stack:
CUDA libs
TensorRT
V4L2 / ISP userspace bits
Wayland/EGL/OpenGL/Vulkan user libraries for display
Holoscan runtime/SDK pieces that assume Nvidia SoC layout
GStreamer accel plugins
Any compositor bits we need for kiosk mode / HUD
Installs into /usr/lib/*, /opt/nvidia/*, /usr/share/holoscan/*, etc.
These two packages are versioned against:
specific Thor silicon rev
specific kernel ABI
specific CUDA/TensorRT ABI
CI enforces that thor-userland.tar.zst + nvidia-userland.deb + nvidia-kernel.deb + thor-kernel.tar.zst all agree on ABI versions. If not, build fails.
Result: we can apt install/upgrade Nvidia stuff like normal packages, instead of re-running JetPack voodoo.
Deliverable: thor-os.sqfs (squashfs rootfs image) assembled from:
base Ubuntu 24
our services
nvidia-userland.deb
nvidia-kernel.deb
kernel bundle staged under /boot and /lib/modules
This thor-os.sqfs is the payload we\u2019ll roll out OTA.
Philosophically: Bazel is the forge, .deb is the distribution format, squashfs is the shipping crate.
We do not immediately overwrite what\u2019s running in the field. That\u2019s suicide.
Instead, we:
Push new artifacts (thor-os.sqfs, kernel Image+dtb, initramfs) to the robot into a staging dir on the current rootfs.
Use kexec to jump directly into the new kernel + initramfs without touching bootloader or disk layout.
Mechanics:
Generate an initramfs for the candidate image that:
mounts thor-os.sqfs read-only as root,
brings up minimal GPU stack (Wayland kiosk splash) so we know graphics init works,
runs a health probe:
enumerates PCIe / I2C devices we depend on,
validates /dev nodes for actuators/sensors,
runs a tiny inference through TensorRT to prove CUDA path,
runs a Holoscan pipeline (camera \u2192 passthrough \u2192 sink) to prove sensor graph alive.
kexec -l <Image> --initrd <initramfs> --command-line "<our cmdline>"
Then kexec -e to enter it.
This gives us an ephemeral boot of \u201cthe new world\u201d in RAM. We haven\u2019t touched NVMe yet. If it blows up, we reboot and we\u2019re back to the old world.
This is Milestone M1 health check, see milestones below.
Why this matters: we prove our Bazel-built kernel and userland can actually run on that physical Thor SKU, with its weird daughterboards and timing quirks, before we commit the update to disk encryption hell.
Once a candidate image passes kexec validation, we lay down the permanent on-disk layout.
Disk is NVMe (/dev/nvme0n1). Layout target (final form):
Partition 1: EFI / bootloader / boot control (unencrypted, signed content only)
Partition 2: LUKS container A
Partition 3: LUKS container B
Partition 4: /data (may be separate LUKS container later; not mandatory in v1)
Inside each LUKS slot (A or B):
One ext4 (or btrfs, calm down) rootfs.
That rootfs contains our system, basically thor-os.sqfs exploded or mounted as lower layer + overlay for mutable bits like /etc, /var.
We alternate:
Active rootfs = A
Next OTA gets staged into B
Flip
Repeat
Rollback is \u201cboot the other slot.\u201d
Encryption:
Each LUKS container uses dm-crypt with strong cipher + integrity mode.
Keys are not sitting loose. Keys are sealed to the device using a TPM-like hardware root (Thor has fuse-backed secrets; if we don\u2019t get that, we use an encrypted keyfile in boot partition, itself wrapped with a device-unique secret burned into fuses / keyslots). The idea: if someone steals the NVMe and plugs it into a laptop, they get ciphertext.
Write process for OTA:
Pick inactive slot (say slot B).
Format/refresh its LUKS container.
Unlock and mount it as /mnt/newroot.
Lay down the new rootfs version into /mnt/newroot:
install thor-os.sqfs or fully untarred tree,
install versioned /boot with the new kernel + dtb + initramfs for that slot,
stamp metadata: version string, build hash, signature.
Update boot control vars (see below) to \u201cboot slot B next boot.\u201d
Reboot normally via bootloader.
If new slot boots and stays alive long enough to pass health checks, mark it \u201cgood\u201d and keep it as active. If it fails early boot health, bootloader or early userspace reverts to previous slot.
This is classic A/B update flow like Android system partitions, but for Thor.
Important: both A and B live inside full-disk crypto. You don\u2019t get a free look at either slot just by pulling the drive.
We close the loop with trust. The steps go like this:
Boot chain of trust:
We rely on Thor\u2019s secure boot hardware (fuse-backed keys) to enforce that only signed bootloader and kernel images can execute.
We burn fuses to lock the signing chain. After this point, you can't boot unsigned junk without hardware glitching.
Bootloader responsibilities:
Validate signature on kernel + initramfs + DTB before loading.
Decide whether to boot slot A or slot B based on:
last known good,
pending update flag,
fallback counter.
Kernel/initramfs responsibilities:
Unlock correct LUKS slot using a key sealed to device identity.
This is the moment where disk confidentiality relies on that hardware secret.
Mount rootfs.
Hand off to userspace.
Tamper resistance story:
You unscrew the robot, yank NVMe, plug into your box? You see dm-crypt noise.
You try to alter the rootfs on-disk? Signature mismatch or dm-verity-style integrity check kills boot / drops to recovery.
You try to boot your own kernel to bypass all this? Secure boot refuses because fuses say \u201conly keys we signed.\u201d
This prevents:
Offline IP theft of ML models / perception code.
Silent insertion of backdoored daemons.
Random \u201cfield tech\u201d debugging with vim in /usr/bin on a production unit.
We are now a real robot company, not a loose pile of tarballs in a trench coat.
\u2013\u2013\u2013
M1 \u2013 Ephemeral boot (kexec smoke test)
Deliverable: We can kexec from the factory Thor image into our Bazel-built kernel + initramfs + thor-os.sqfs, run health probes (GPU init, Holoscan pass-through, one-shot inference).
Verification: robot survives, teleop overlay draws, CUDA tensor multiply returns sane result.
M2 \u2013 Packaging (nvidia-*.deb split complete)
Deliverable: nvidia-kernel.deb + nvidia-userland.deb built and installed into thor-os.sqfs purely via dpkg -i, no JetPack installer.
Verification: clean Bazel rootfs + those two .debs still pass M1 health probe.
M3 \u2013 Disk provisioning + A/B layout
Deliverable: Script that:
Partitions NVMe,
Creates LUKS A/B,
Populates slot A from thor-os.sqfs,
Boots natively from encrypted slot A (not kexec).
Verification: Cold boot from NVMe, rootfs mounted from dm-crypt, robot comes up and runs control stack.
M4 \u2013 OTA update & rollback
Deliverable: Update service that:
Downloads a new signed thor-os.sqfs,
Writes it into inactive slot,
Flips boot target,
Reboots,
Marks success/failure,
Auto-reverts on failure.
Verification: We intentionally ship a \u201cbad\u201d image; box reverts to previous slot automatically.
M5 \u2013 Secure boot + fuse lockdown
Deliverable: Secure boot enforced. Bootloader, kernel, initramfs signed. Fuses burned for prod units so unsigned images will not boot. LUKS keys sealed to that hardware identity, not just a plaintext file.
Verification: Altering rootfs on the NVMe from an external machine leads to a non-booting system. Attempting to boot unsigned kernel fails before handoff.
We have to prove this machinery works, with metrics, not vibes.
Functional checks:
From running system, report:
active slot (A or B),
OS version hash,
secure boot status (enforced / not enforced),
disk encryption status (LUKS active yes/no),
rollback pending yes/no.
From updater service logs:
write time of new slot,
verification steps it ran,
signature validation result.
We\u2019ll export these as Prometheus-style gauges/counters the same way we\u2019re doing tc/qdisc stats in the QoS spec.
Concretely:
thor_slot_active{slot="A"} = 1
thor_secure_boot_enforced = 1
thor_root_encrypted = 1
thor_update_success_total counter
thor_update_rollback_total counter
thor_update_healthcheck_seconds histogram of how long validation took
Vector will scrape a small local CLI (call it thor-status --metrics), which prints these, Prometheus text format.
Vector forwards to Mimir. Grafana panels show fleet compliance: who\u2019s on newest hash, who\u2019s drifted, who rolled back.
Debug surfaces:
journalctl -b -1 | grep thor-update shipped via Vector to Loki, so we can read previous-boot failure context without SSH.
A \u201cwhy did I revert\u201d code path that logs the bootloader decision (slot B failed healthcheck N times, fell back to slot A).
This makes deployment auditable and lets us spot units that are stuck on old software because they keep failing health.
Robot is running \u201cslot A normal world.\u201d
OTA agent downloads:
Image (kernel),
initramfs-new,
thor-os.sqfs,
signatures/metadata.
Agent runs:
signature check (our signing key),
ABI check (kernel \u2194 CUDA \u2194 TensorRT).
Agent loads: kexec -l ./Image --initrd=./initramfs-new --command-line="root=/dev/ram0 init=/sbin/thor-probe quiet splash"
Agent executes kexec -e.
Probe environment boots from RAM:
mounts thor-os.sqfs,
runs GPU bring-up,
runs Holoscan graph test,
runs inference smoke (e.g. 1-layer conv),
validates basic IO to real robot peripherals.
Probe writes result to a known handoff location (e.g. tmpfs shared via kexec handoff or persisted to the old root via a tmp mount).
If probe passes, we\u2019re allowed to install to inactive slot and schedule reboot.
We do not touch NVMe, boot slots, LUKS headers, or fuses until the probe passes.
This is our airbag.
Pseudocode-ish provisioning script (first-time install or full-disk refresh):
#!/usr/bin/env bash\nset -euxo pipefail\n\nDISK=/dev/nvme0n1\n\n# 1. Partition table:\n# p1: EFI/boot (256M)\n# p2: slotA (rest/2)\n# p3: slotB (rest/2)\n# (p4 later for data)\nparted --script "$DISK" \\\n mklabel gpt \\\n mkpart boot fat32 1MiB 257MiB \\\n set 1 esp on \\\n mkpart slotA 257MiB 50% \\\n mkpart slotB 50% 100%\n\n# 2. LUKS format slotA / slotB\ncryptsetup luksFormat ${DISK}p2 --key-file /root/thor.key\ncryptsetup luksFormat ${DISK}p3 --key-file /root/thor.key\n\ncryptsetup open ${DISK}p2 thorA --key-file /root/thor.key\nmkfs.ext4 /dev/mapper/thorA\nmount /dev/mapper/thorA /mnt/newroot\n\n# 3. Populate rootfs\ntar -C /mnt/newroot -xpf thor-os.tar # thor-os.sqfs exploded or equivalent\n# install boot artifacts for slotA, write version metadata, etc.\n\numount /mnt/newroot\ncryptsetup close thorA\n\nIn production, /root/thor.key is not a plaintext file. The unlock key is derived/loaded from hardware-sealed storage. That detail lands in Milestone M5.
We maintain a tiny boot-control struct, ideally in the unencrypted boot partition (so bootloader can read it early), something like:
ACTIVE=A\nPENDING=B\nA_GOOD=1\nB_GOOD=0\nRETRIES_LEFT=3\nVERSION_A=2025.10.24.abc123\nVERSION_B=2025.10.25.def456\nSIG_B_OK=1\n\n
Bootloader logic:
If PENDING is set and RETRIES_LEFT>0, try booting that slot.
On first userspace start, systemd service thor-boot-good.service marks that slot GOOD and clears PENDING.
If the new slot fails to come up and report GOOD, RETRIES_LEFT decrements. Eventually we fall back.
This is how rollback becomes automatic and boring.
Final manufacturing / field-hardening step:
Generate signing keys for boot images and kernel. Guard them like crown jewels.
Program Thor\u2019s fuse bank / key slots so only images signed with that key will execute.
Write bootloader configured to:
verify kernel+initramfs signature,
verify DTB signature (prevents DTB attacks that remap I/O or disable watchdogs),
refuse unsigned.
Seal the LUKS decryption secret using that same root of trust (hardware-unique secret \u2192 unwrap disk key).
Permanently burn fuses so unsigned boot is no longer allowed.
Now:
Attacker with physical NVMe learns nothing (LUKS).
Attacker trying to inject malware kernel is blocked (secure boot).
Attacker trying to downgrade to an older vulnerable image is blocked if we enforce anti-rollback counters (monotonic version in boot metadata signed by us).
This is the \u201cyou can steal it off the loading dock, but you can\u2019t turn it into your surveillance robot\u201d guarantee investors love and ops needs.
We ship a tiny /usr/bin/thor-status binary/script with two modes:
Human-readable summary:
Slot active / pending
Version hash
Secure boot enforced Y/N
LUKS active Y/N
Last update result
Rollback flag
Prometheus text mode (thor-status --metrics):
thor_slot_active{slot="A"} 1
thor_os_version{slot="A",version="2025.10.25.def456"} 1
thor_secure_boot_enforced 1
thor_root_encrypted 1
thor_update_success_total 9
thor_update_rollback_total 1
Vector scrapes that via exec every 30s (same model as tc/qdisc scrape from the QoS spec) and forwards to Mimir. Grafana gives:
fleet-wide version map,
% of robots on latest signed build,
who rolled back last night,
who\u2019s unencrypted (red alert),
who\u2019s not fuse-locked yet (yellow alert; dev hardware).
This folds OS integrity into the same monitoring fabric as network QoS, so ops sees not just \u201cis it online\u201d but \u201cis it running something we actually trust.\u201d
\u2013\u2013\u2013
Outcome
Thor boxes graduate from \u201cSD card with dreams\u201d to an updatable, encrypted, signed software appliance:
We can stage a new OS, verify it in RAM via kexec without risking the unit.
We can rotate the rootfs on an encrypted NVMe LUKS slot, atomically, with A/B rollback.
We can enforce secure boot at the silicon level so nobody can boot unsigned junk or exfiltrate our models.
We get per-robot attestation and health metrics in Grafana instead of blind faith.
This gives us safe OTA, post-compromise recovery, IP protection, and auditable configuration drift control \u2014 the grown-up stack you need before you start shipping fleets into angry warehouses.
" }, { "id": "1177878534", "title": "Spec: CPU Pinning and Slices", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1177878534", "last_modified": "2025-10-22T22:08:24.812Z", "created_at": "2025-10-22T22:07:00.907Z", "body_html": "Realtime control loops, human-facing UIs, and sludge like compiles/backups are fist-fighting on the same CPUs. Linux is great, but \u201cbest effort\u201d isn\u2019t a strategy. We need explicit CPU partitioning so:
RT control meets jitter budgets.
Interactive (SSH, Grafana, WebRTC signaling) stays snappy.
Bulk (logging, compression, downloads, background jobs) doesn\u2019t poison latency.
Goal: deterministic CPU isolation and priority with zero \u201cwhoops, irqbalance moved it.\u201d
In scope
CPU partitioning via systemd slices (rt.slice, interactive.slice, bulk.slice)
Kernel boot isolation and IRQ steering for NIC/WireGuard
RPS/XPS and workqueue pinning
Service placement (drop-ins) for core daemons (control, sshd, vector, docker, nginx, wg-quick)
Basic observability (PSI, per-CPU usage, softirq/IRQ stats) exposed to Vector
Out of scope
Hard real-time kernels (PREEMPT_RT tuning beyond policy & affinity)
NUMA-aware placement on >1 socket
Container orchestrator-level policies (K8s cpusets, etc.)\u2014this spec is host/systemd centric
RT cores: 2\u20133 (tight, deterministic)
General cores: 0\u20131 (interactive + bulk + IRQs + crypto)
Adjust numbers to your box; the pattern holds.
Pin kernel noise off RT cores and bias IRQs to general cores:
isolcpus=managed,2-3 nohz_full=2-3 rcu_nocbs=2-3 irqaffinity=0-1\n\n
isolcpus=managed: lets systemd cpuset manage those cores while keeping scheduler background out.
nohz_full: ticks off on RT cores (less jitter).
rcu_nocbs: push RCU callbacks away.
irqaffinity=0-1: initial IRQ placement.
We define three slices with cpusets and priorities:
rt.slice
AllowedCPUs=2-3
CPUSchedulingPolicy=fifo
CPUSchedulingPriority=90 (or whatever your stack needs)
CPUWeight= not relevant for RT, but keep defaults
AllowedMemoryNodes=0 if single NUMA node (otherwise omit)
interactive.slice
AllowedCPUs=0-1
CPUWeight=900 (favored over bulk)
IOSchedulingClass=best-effort
IOSchedulingPriority=2
Nice=-5 (optional; don\u2019t fight RT)
bulk.slice
AllowedCPUs=0-1
CPUWeight=100
IOSchedulingClass=idle
Nice=10
NIC IRQs \u2192 CPUs 0\u20131
Write smp_affinity masks for eth0 (and friends).
RPS/XPS \u2192 CPUs 0\u20131
Keep NAPI processing off RT cores.
Workqueues (/sys/devices/virtual/workqueue/cpumask) \u2192 0\u20131
Crypto/compress/kworkers won\u2019t trespass onto RT cores.
WireGuard
Kernel datapath obeys IRQ/RPS; userspace wg-quick@wg0.service runs in interactive.slice (or bulk.slice if you prefer).
Control loop / robot RT stack \u2192 rt.slice
sshd, nginx, grafana-agent, wg-quick, Vector \u2192 interactive.slice
docker, package updates, backups, compiles, log processors \u2192 bulk.slice
Everything else defaults to bulk.slice unless promoted.
Phase | Deliverable | Verification |
|---|---|---|
M1 \u2013 Kernel & slices | Boot with isolcpus/nohz_full/rcu_nocbs; slices defined |
|
M2 \u2013 IRQ/RPS/XPS/workqueues | IRQ masks + RPS/XPS + workqueue cpumask persisted |
|
M3 \u2013 Service placement | Drop-ins applied for core services |
|
M4 \u2013 Observability | Vector ingest of CPU/PSI/softirq and IRQ stats | Grafana shows stable jitter; alerts wired |
M5 \u2013 Load test & tune | iperf/WebRTC storms don\u2019t degrade RT jitter | Control loop jitter < target under stress |
Metrics to export (low overhead, exec via Vector):
CPU PSI: /proc/pressure/cpu \u2192 psi_cpu_some, psi_cpu_full (avg10/60/300)
Per-CPU utilization: deltas from /proc/stat \u2192 cpu_seconds_total{cpu="N",mode="user|system|irq|softirq|idle|steal"}
IRQ distribution: /proc/interrupts per-IRQ deltas for NIC/eth0 & wg; verify affinity
Softirq pressure: /proc/net/softnet_stat \u2192 dropped/squeezed counters
cgroup CPU usage per slice: /sys/fs/cgroup/<slice>/cpu.stat (cgroup v2: usage_usec, user_usec, system_usec)
Sched latency (optional): /proc/schedstat deltas to estimate runqueue wait (only if needed)
Acceptance checks
Under bulk upload + docker build, RT loop period and 99.9th percentile latency remain within budget.
ksoftirqd/* threads run on CPUs 0\u20131 during load (ps -eLo pid,psr,comm | grep ksoftirqd).
NIC IRQ counters climb only on CPUs 0\u20131.
PSI cpu some for rt.slice \u2248 0 under steady-state (means it isn\u2019t starved).
/etc/systemd/system/rt.slice
[Slice]\nAllowedCPUs=2-3\nAllowedMemoryNodes=0\n\n
/etc/systemd/system/interactive.slice
[Slice]\nAllowedCPUs=0-1\nCPUWeight=900\nIOSchedulingClass=best-effort\nIOSchedulingPriority=2\n\n
/etc/systemd/system/bulk.slice
[Slice]\nAllowedCPUs=0-1\nCPUWeight=100\nIOSchedulingClass=idle\n\n
Reload: systemctl daemon-reload
RT control /etc/systemd/system/robot-rt.service.d/policy.conf
[Service]\nSlice=rt.slice\nCPUSchedulingPolicy=fifo\nCPUSchedulingPriority=90\nLimitRTPRIO=95\nLimitMEMLOCK=infinity\n\n
WireGuard /etc/systemd/system/wg-quick@wg0.service.d/cpu.conf
[Service]\nSlice=interactive.slice\n\n
Vector /etc/systemd/system/vector.service.d/cpu.conf
[Service]\nSlice=interactive.slice\nNice=-2\n\n
Docker /etc/systemd/system/docker.service.d/cpu.conf
[Service]\nSlice=bulk.slice\n\n
/usr/local/sbin/net_affinity.sh
#!/usr/bin/env bash\nset -euxo pipefail\nIF="${1:-eth0}"\nGEN_MASK_HEX=3 # CPUs 0-1\n\n# Pin NIC IRQs\nfor IRQ in $(grep -w "$IF" /proc/interrupts | awk -F: '{print $1}'); do\n printf "%x" "$GEN_MASK_HEX" | tee "/proc/irq/$IRQ/smp_affinity" >/dev/null\ndone\n\n# RPS/XPS\nMASK_HEX=$(printf "%x" $GEN_MASK_HEX)\nfor q in /sys/class/net/$IF/queues/rx-*; do echo "$MASK_HEX" | tee "$q/rps_cpus" >/dev/null; done\nfor q in /sys/class/net/$IF/queues/tx-*; do echo "$MASK_HEX" | tee "$q/xps_cpus" >/dev/null; done\n\n# Workqueues\necho "$MASK_HEX" | tee /sys/devices/virtual/workqueue/cpumask >/dev/null\n\n/etc/systemd/system/net-affinity@.service
[Unit]\nDescription=Pin IRQ/RPS/XPS/workqueues for %I\nAfter=network-online.target\nRequires=network-online.target\n\n[Service]\nType=oneshot\nExecStart=/usr/local/sbin/net_affinity.sh %I\nRemainAfterExit=yes\n\n[Install]\nWantedBy=multi-user.target\n\n
Enable: systemctl enable net-affinity@eth0
/usr/local/sbin/net_offloads.sh
#!/usr/bin/env bash\nIF="${1:-eth0}"\nethtool -K "$IF" gro off gso off tso off lro off || true\nsysctl -w net.core.netdev_max_backlog=1000\n\n/etc/systemd/system/net-offloads@.service
[Unit]\nDescription=Disable latency-harmful offloads for %I\nAfter=network-online.target\n\n[Service]\nType=oneshot\nExecStart=/usr/local/sbin/net_offloads.sh %I\nRemainAfterExit=yes\n\n[Install]\nWantedBy=multi-user.target\n\n
/usr/local/sbin/cpu_metrics.sh
#!/usr/bin/env bash\nset -euo pipefail\n\n# PSI\nawk '/some/ {print "psi_cpu_some_avg10 " $2 "\\npsi_cpu_some_avg60 " $3 "\\npsi_cpu_some_avg300 " $4}\n /full/ {print "psi_cpu_full_avg10 " $2 "\\npsi_cpu_full_avg60 " $3 "\\npsi_cpu_full_avg300 " $4}' \\\n /proc/pressure/cpu 2>/dev/null\n\n# /proc/stat (per-CPU) \u2014 raw counters\nawk '/^cpu[0-9]+ /{\n cpu=$1; user=$2; nice=$3; system=$4; idle=$5; iowait=$6; irq=$7; softirq=$8; steal=$9; guest=$10; guest_nice=$11;\n printf "cpu_seconds_total{cpu=\\"%s\\",mode=\\"user\\"} %f\\n", cpu, user/100;\n printf "cpu_seconds_total{cpu=\\"%s\\",mode=\\"system\\"} %f\\n", cpu, system/100;\n printf "cpu_seconds_total{cpu=\\"%s\\",mode=\\"idle\\"} %f\\n", cpu, idle/100;\n printf "cpu_seconds_total{cpu=\\"%s\\",mode=\\"irq\\"} %f\\n", cpu, irq/100;\n printf "cpu_seconds_total{cpu=\\"%s\\",mode=\\"softirq\\"} %f\\n", cpu, softirq/100;\n printf "cpu_seconds_total{cpu=\\"%s\\",mode=\\"steal\\"} %f\\n", cpu, steal/100;\n}' /proc/stat\n\n# softnet_stat drops/squeezes (sum over CPUs)\nawk '{d+=strtonum("0x"$2); s+=strtonum("0x"$3)} END{\n print "softnet_dropped_total " d; print "softnet_squeezed_total " s\n}' /proc/net/softnet_stat 2>/dev/null\n\n# cgroup v2 cpu.stat per slice (if present)\nfor S in rt.slice interactive.slice bulk.slice; do\n P="/sys/fs/cgroup/$S/cpu.stat"\n [ -f "$P" ] || continue\n awk -v s="$S" '/usage_usec|user_usec|system_usec/{\n gsub("_usec","",$1); printf "cgroup_cpu_usec_total{slice=\\"%s\\",mode=\\"%s\\"} %s\\n", s,$1,$2\n }' "$P"\ndone\n\nVector source:
[sources.cpu_metrics]\ntype = "exec"\ncommand = ["/usr/local/sbin/cpu_metrics.sh"]\ninterval_secs = 15\ntimeout_secs = 3\n\n
Slices & placement: systemd-cgtop / systemctl status <svc>
IRQs pinned: grep -E "eth0|mlx|ixgbe" /proc/interrupts
ksoftirqd on 0\u20131: ps -eLo pid,psr,comm | grep ksoftirqd
Cores in use by services: taskset -pc $(pidof <proc>)
RT policy active: chrt -p $(pidof robot-rt)
Three CPU lanes with teeth. RT threads live on clean cores under FIFO policy; interactive stays responsive; bulk can grind without wrecking latency. IRQs, softirq, and workers are shoved off the RT island. Metrics prove it, not vibes.
" }, { "id": "1151303693", "title": "Spec: HMND Core Control Plane", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1151303693", "last_modified": "2025-10-21T12:52:24.355Z", "created_at": "2025-10-11T00:05:36.975Z", "body_html": "Summary: Production-grade Global Control Plane for Humanoid robots
Owner: Cody Griffin
Date:
Approvers: Adam Kelsall Leo Lesin (Deactivated) Yiannis Levantis
Informed: most people\u2026
While we currently have a small development fleet, we are already have 3 models spread across 3 engineering sites. In the future, we\u2019ll have thousands of robots across hundreds of customer sites. We have the opportunity now to start thinking about a growing fleet while starting small and relatively lean.
Our current processes involve individual developers setting hostnames and adding lines the the \u201cenv_vars\u201d script which is sourced when you activate a Pixi environment on the robot. Or setting static IPs in our Unify gear. Honestly fine for less than a dozen robots, but moving forward we need to start thinking longer term about how we identify, authenticate and authorize users and robots to our cloud infrastructure.
While for Alpha, we have largely been able to survive with ad-hoc infrastructure and processes, for Beta we should put in place some global \u201ccore\u201d infrastructure supporting:
robot identity and provisioning
software updates and release channels,
observability - metrics, logs, traces
auditing (future)
events (future)
jurisdiction-scoped concerns like data sovereignty (future)
site-local realtime services like robot teleoperation (future scope)
We should maintain our existing \u201clab\u201d infrastructure as-is - the feature-set is different (data collection and training primarily). Our Core infrastructure will live in London, probably with backups in another region. At some point we may need to support multiple jurisdictions, but for now we\u2019ll treat our existing lab infrastructure as a default jurisdiction. Our initial \u201csites\u201d will be London, Boston and Vancouver.
When robots are built, they will be provisioned with their static identity: model, serial number, date, origin - anything we want to track for the lifetime of the robot. When robots are deployed, they will be provisioned with a dynamic identity or lease: site, jurisdiction, account, current software version - things that can change over time as a robot may be transferred repeatedly before decommissioning.
The key points of this review before we commit to this path:
Can we leverage a shared, global control plane for all Humanoid robots? This is a major win operationally for us, and I think its a pre-requisite.
What are the reliability requirements? Arbitrarily I\u2019m shooting for 99.9% to start.
Certificate rotations? How long do we expect robots to exist without connectivity?
In-scope
CI-driven AWS infrastructure using Terraform via GitHub OIDC.
Core identity, release, and metrics services for fleet management.
PKI for static and dynamic identity.
Automated provisioning/claiming and update distribution.
Observability stack (Grafana / Loki / Tempo / Mimir / ClickHouse).
High availability across multiple AZs.
Blue/green deploys and safe schema migrations.
Operational visibility even when application Kubernetes clusters are down.
Out-of-scope
Teleoperation or SSH access data plane: control traffic only for now
Customer-facing web apps beyond a simple status page (humanoid console should be unaffected).
Multi-tenancy or non-anonymized data (currently the scope of existing data pipeline)
At the top-most level, we have our AWS root account. We already have an environment for our lab - we should create a new account in AWS for each core environment we want to support. We will probably have dev and prod, maybe staging, maybe previews per developer branch.
Each environment is completely isolated, and we\u2019ll be starting w/ just a core.dev.hmnd.ai for development. The expectation is we simply clone the infrastructure later when we need a production environment. Site-specific services can be hung off this core for data sovereignty or latency-related reasons (perhaps in one or more other, customer-centric accounts). The intent of the core stack is to provide a single, global control plane for Humanoid\u2019s robots. Its a single pane of glass for Humanoid to manage a fleet of robots, without getting tied up with data sovereignty or performance concerns.
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1151303693/Untitled%20Diagram-1760271223159.drawio.png?api=v2\")
Root/Management account: hosts hmnd.ai , delegates DNS, manages AWS Org + SSO.
No \u201creal\u201d infrastructure.
Break glass - if we need access to our accounts outside of CI or existing roles.
Dev and Prod accounts: core workload environments.
Google Workspace provides SSO via AWS IAM Identity Center.
sync google groups w/ ssosync or similar
GitHub Actions assume per-env terraform deployment role via OIDC; no long-lived keys.
mTLS and OIDC ALBs for HTTP access to core services
Break glass - some manual roles that could be assumed which can bounce instances/services or make administrative changes (no manual infra changes)
PKI for robot identity
Root CA for each environment - hopefully AWS ACM so we aren\u2019t managing re-issue ceremonies ourselves
Intermediate CAs
Global for provisioning robots, signing updates
Jurisdiction or customer enrollment for smaller scope (future)
Mint certificates for static identity - basically a robot birth certificate
Mint certificates for dynamic identity - basically deployment paperwork (future)
Mint certificates to be imported into ACM for Robot ALB
Bootstrap (CloudFormation) \u2014 creates tfstate S3, DynamoDB lock, initial terraform deployment role.
The only \u201cmanual\u201d deployment.
Terraform (hmnd-core) \u2014 defines all VPCs, ECS clusters, RDS, S3, Vault, observability, etc.
No manual deployments.
CI/CD \u2014 PR to plan infra (merge to apply), deploy preview service?
hmnd/hmnd_cloud/core - terraform infra
slightly different layout from existing hmnd_cloud infra, but we can consolidate later after getting some good habits
api.core.$env.cloud.hmnd.ai \u2014 registry API (provision, claim, update, events)
vault.core.$env.cloud.hmnd.ai \u2014 CA + cert issuance
status.core.$env.cloud.hmnd.ai \u2014 internal UI (OIDC protected)
List of units, model + serial, current version, current site
{grafana, logs, metrics, traces}.core.$env.cloud.hmnd.ai \u2014 observability stack
These endpoints represent what humans would see via the OIDC ALB - robots authenticating w/ mTLS would see slightly different domains for security reasons.
Note - we specifically deploy our observability stack on something lower-level than k8s so that we can monitor our k8s clusters more reliably.
The data model is relatively standard SQL. A few tables, a few constraints. A couple indices. Maybe the odd materialized view. Rows are written, updated periodically and queried in batches.
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1151303693/Untitled%20Diagram-1760350354211.drawio.png?api=v2\")
Large artifacts stored in S3 (certificates, images) but referenced via sha256 in postgres.
images/ - signed release images (multiple gigabytes)
certs/ - certificate bundles (current and future)
crl/... - revocation lists (maybe)
Robot uses certificate to establish mTLS to ALB, which can then be used to write events to our observability stack.
Operators authenticates via OIDC; groups control dashboards.
Vault PKI - replicated across AZs on EC2, backed up to S3 via snapshots. Rolling upgrades (1 node at a time).
High quality EBS volumes ideally - doing host volumes on ECS defeats the purpose. No need for autoscaling.
If we out-grow a single PKI instance in the global core, we can create additional intermediate CAs later
API services - scaled out on ECS (fargate) across AZs. Blue/Green. No state (easy)
ECR + containers in scope, but give us local testing more easily than lambdas
RDS Aurora Postgres - MultiAZ, AWS-managed upgrades. Backups to S3.
Grafana Stack - EC2 + local storage for writes, flush to S3. Rolling restarts (though HA may not be necessary at first)
S3 - replicate sensitive data to another bucket
TLS 1.3 everywhere; mTLS robots, OIDC humans.
IAM least-privilege roles per service.
KMS-encrypted buckets, RDS, EBS.
Secrets in AWS Secrets Manager / Vault.
Access logs and audit to CloudTrail + S3.
\u201cEverything encrypted in transit. Everything encrypted at rest.\u201d - Compliance people love this.
GH Actions \u2192 AWS OIDC \u2192 TerraformDeployRole.
Plan on PR, Apply on Merge.
All terraform changes serialized via CI
Environment approvals for prod.
Bootstrap complete - CloudFormation stack, CI role
\n\n \n\n \n\n\n\n\n \n HECO-1358\n -\n Getting issue details...\n STATUS\n \n
Github action\u2019s terraform apply is a no-op
Core spine (dev) - API, Database, PKI
ALBs (mTLS + OIDC)
Observability live
\n\n \n\n \n\n\n\n\n \n HECO-1363\n -\n Getting issue details...\n STATUS\n \n
Robots can publish metrics via ALB w/ mTLS
Operators can manage dashboards via OIDC login
Release channels flow
\n\n \n\n \n\n\n\n\n \n HECO-1365\n -\n Getting issue details...\n STATUS\n \n
Operators (or CI) can publish new releases - signed, stashed images in S3, row in database
Operators can promote releases within channels
Robots can fetch releases from assigned channel
HA drills
Survive AZ down
No downtime updates
Certificate renewals automated
Restore from backups (frequent on dev)
Estimated: 4\u20138 weeks with 1 core engineer. Doesn\u2019t need to happen all at once.
This infrastructure will require some additional SOPs to be documented.
Provisioning
Operator runs a tool which POSTs to api.core.{env}.hmndi.ai/units with model, serial and other static metadata (like a CSR). API mints certificate and writes to database before responding. The tool can then write the resulting trust bundles to the robot\u2019s certificate store.
At this point the Operator has \u201cbootstrapped\u201d the robot\u2019s identity, and now the robot can access core services using mTLS.
Release Publishing
Operator or CI runs a tool which POSTS to api.core.{env}.hmnd.ai/releases with the release image data in the POST body (and release metadata as headers). The image data is written to S3 while the metadata is written to the DB (with some care being taken to avoid orphaned images). The image data is written with a suffix containing the signature, and the resulting release manifest is returned in the response.
When an image is published it may optionally be set to the current channel version. All robots assigned to the channel will then pull the image and perform an OTA update.
Channel Promotion
Typically newly-published images are added to a \u201cdevelopment\u201d channel, and later promoted to a \u201cstable\u201d channel.
Metrics should be labeled by current release.
Cert Rotation
ALB? Hopefully we can pick up from S3 or mirror from ACM? Robots can be re-keyed transparently from the ALB\u2019s perspective, but the intermediate CA or root CA would be difficult. Ideally overlap.
Robots? Probably initiate rekey periodically. This bounds how long robots can be offline before keys expire. For dev we will do this very frequently. For production, probably less often.
Database Migrations
Online safe pattern: add nullable column \u2192 backfill \u2192 enforce \u2192 drop old.
Rollback = drop column if unused.
Upgrades / Rollouts
ECS rolling updates (100% healthy min)
https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/USER_UpgradeDBInstance.PostgreSQL.html
Incident Response
Logs in Loki; traces in Tempo.
Event lag alarms via Mimir.
Rollback releases within a channel (opposite of promoting).
Teleop plane: OIDC humans \u2194 mTLS robots via regional relays (WebRTC / TURN).
Multi-region replication: ClickHouse shards per region; Vault intermediate per region.
Tenant isolation: scoped prefixes in CAS + group-based policy in API.
SSH Bastion: access robots via SSH, tied to authz + auditing
This appendix defines the public key infrastructure (PKI) architecture used to authenticate and authorize robots, operators, and cloud components. The design leverages AWS Certificate Manager (ACM) as the root of trust, with HashiCorp Vault operating as an intermediate certificate authority (CA). Certificates issued from Vault are used to establish mutual TLS (mTLS) between robots and backend services, and to secure communication across all control-plane endpoints.
AWS ACM serves as the system\u2019s root CA.
It is created once and used solely to sign the Vault intermediate. This avoids exposing Vault-generated roots to long-term compromise and provides central visibility and revocation capabilities through AWS.
ACM\u2019s root certificate is:
Offline by design: keys are never exported or accessible outside AWS.
Stored under an AWS account with restricted administrative access.
Used only to sign the Vault intermediate CA certificate (no end-entity certificates).
Exported artifacts:
Root certificate (public only) for inclusion in Vault\u2019s trust chain.
Optional: export the trust bundle (root + intermediates) for ALB and internal service consumption.
Vault runs on an EC2 instance within a private subnet across multiple Availability Zones.
The service uses integrated storage and auto-unseal (via AWS KMS) for HA operation. TLS termination is handled by ALB (with OIDC for human access), but mTLS validation happens internally for robots.
Vault configuration:
pki_int mount for issuing short-lived leaf certificates.
Intermediate CSR generated from Vault and signed by ACM root.
Resulting signed certificate chain imported back into Vault.
TTLs are intentionally short (e.g., 30 days) with automated rotation.
Vault EC2 instances:
Spread across \u22652 AZs using an ASG or manual multi-AZ deployment.
Bound to a security group permitting only ALB ingress and internal AWS access.
IAM instance profiles limited to KMS (for auto-unseal) and CloudWatch (for audit/logging).
Each robot model (e.g., alpha-wheelbase-v0) has a unique serial number (e.g., a1-00123).
Each physical robot includes two compute targets\u2014for example:
a1-00123-orin
a1-00123-rtpc
Certificates are minted separately for each compute node.
Both share the same authorization scope (same robot identity) but have unique private keys and certificates for better compartmentalization.
Certificate Subject naming convention:
CN = robot.serial\nO = model\nOU = compute_target\n
Example:
CN = a1-00123\nO = alpha-wheelbase\nOU = orin\n
Additional metadata (serial/model/compute_target) may also be stored in Vault\u2019s metadata for search and audit.
Each robot\u2019s certificate is created through a static provisioning workflow:
Robot generates its own private key and CSR locally.
CSR includes robot metadata and is submitted to a provisioning endpoint.
Vault signs and returns the certificate chain (cert + intermediate).
Private key never leaves the robot.
Renewal is handled by Vault\u2019s PKI lease mechanism or an explicit renewal request from the robot.
Renewal policies:
Robots renew certificates automatically within 7 days of expiry.
If renewal fails or the lease is revoked, the robot reverts to the provisioning workflow.
Expired or revoked certificates are invalidated in CRL/OCSP endpoints published by Vault and mirrored to S3 for ALB consumption.
Automated renewal can use:
Vault Agent with auto_auth and template stanzas (preferred for static deployments).
Lightweight CLI wrapper in the robot OS that runs as a systemd timer.
Private keys are always generated locally on the robot compute node\u2014either by OpenSSL, Rust/Python crypto libraries, or a TPM/HSM backend if available.
Process summary:
Generate private key (openssl genpkey or library equivalent).
Generate CSR (openssl req -new -key <key> -subj ...).
Submit CSR to Vault API /v1/pki_int/sign/robot.
Receive signed certificate (PEM chain) and store locally.
Vault is never trusted with private keys, only CSRs.
This ensures key material cannot be leaked even if Vault is compromised.
Vault uses a structured naming convention for clarity and auditability:
pki/\n \u251c\u2500\u2500 root/ # ACM Root (read-only reference)\n \u251c\u2500\u2500 pki_int/ # Intermediate CA mount\n \u2502 \u251c\u2500\u2500 roles/\n \u2502 \u2502 \u251c\u2500\u2500 robot\n \u2502 \u2502 \u251c\u2500\u2500 human\n \u2502 \u2502 \u2514\u2500\u2500 service\n \u2502 \u251c\u2500\u2500 certs/\n \u2502 \u2514\u2500\u2500 crl/\n \u2514\u2500\u2500 metadata/\n \u251c\u2500\u2500 robots/\n \u2502 \u2514\u2500\u2500 <serial>/<compute_target>\n \u251c\u2500\u2500 humans/\n \u2514\u2500\u2500 services/\n
Role configuration (pki_int/roles/robot):
allowed_domains = "robots.dev.hmnd.ai"
allow_subdomains = true
server_flag = true
client_flag = true
max_ttl = "720h"
require_cn = true
key_type = "ec"
key_bits = 256
Common Name pattern: *.robots.dev.hmnd.ai (robots signed per compute target).
The ALB serving robot APIs (e.g. api.core.dev.hmnd.ai) enforces mTLS for inbound robot connections.
Setup:
Import ACM trust bundle (root + Vault intermediate).
Enable \u201cRequire client certificate\u201d on the ALB listener.
ALB validates the client certificate chain against this trust bundle.
Downstream targets (e.g., FastAPI or Grafana Mimir) trust ALB headers indicating verified certificate subject.
AWS CLI example:
aws acm import-certificate \\\n --certificate-chain fileb://vault_chain.pem \\\n --certificate fileb://root_cert.pem \\\n --private-key fileb://dummy_key.pem\n
ALB listener configuration:
HTTPS (443)
TLS policy enforcing TLS1.2+
Client authentication: required
Trust store: ACM bundle (Vault intermediate + root)
Forward headers: X-Client-Cert-Subject, X-Client-Cert-Issuer
ALB validation is sufficient for access control\u2014robots need no credentials beyond their valid client certificates.
" }, { "id": "1154548100", "title": "Fleets", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1154548100", "last_modified": "2025-10-14T16:55:57.244Z", "created_at": "2025-10-14T16:21:09.185Z", "body_html": "Many of our core business plans rely on selling a fleet of robots rather than individual robots. This makes sense - building a single robot that can operate with enough uptime to satisfy industrial use-cases is impossible - but selling enough robots so that the aggregate uptime matches is much more feasible. We won\u2019t really be viable shipping a single robot, and so we shouldn\u2019t worry about selling just a single robot \u201cfleet\u201d.
![\"image-20251014-162423.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1154548100/image-20251014-162423.png?version=1&modificationDate=1760459065160&cacheVersion=1&api=v2&width=434&height=218\")
But we won\u2019t have a fleet of robots until we manufacture a lot of robots. And we won\u2019t be able to manufacture a lot of robots until we have a good robot. The path to a scalable robotic fleet is through a really really good robot. I\u2019m going to make the case here that we should probably focus on building a really good robot, and that a good fleet will follow.
\u201cThe whole is greater than the sum of its parts\u201d. Notice that greater doesn\u2019t really have a valence associated. If we build a really good robot, I think its reasonable that a fleet of these robots will be even better. And if we build a shitty robot, we should only expect that a bunch of them will be even shittier. In this way, a fleet is effectively an amplifier.
So with this in mind - what does a really good robot look like? Thinking as abstractly as possible, a really good robot is probably one that:
does what its asked (and nothing more)
provides insights into its status and history (how did it get here)
provides predictions about its future (where is it going)
The first point is really about solving the commercially viable work. The second two points are more about ensuring that some system can effectively \u201ccollaborate\u201d (non-safety context here). If the robot is a perfectly-executing blackbox, it becomes impossible for humans or other robots to operate around it as there is no real idea of where it is going or what it may have already done. So we need to expose this context. This context is important for both HRI and fleet management. Notice that I didn\u2019t include anything about reliability, performance, flexibility/adaptability - this is trying to just be as high-level as possible. Of course these other items are important, but I\u2019d argue that they will largely fall under point 1 here (the robot can\u2019t do what its told if its broken or delocalized or whatever).
So - let\u2019s assume we build a \u201creally good robot\u201d . What next?
The overall logic behind fleet management rhymes a lot with the general history of the scalable infrastructure. Instead of building giant reliable mainframes with 99.999% uptime, just run hundreds of cheap servers behind a load balancer. Instead of building a hard disk that never fails, just store many copies of your data across a bunch of cheap disks and get the overall reliability up. We are hoping to do the same thing - horizontally scaling our unit of work (the robot) to provide some sort of aggregate performance to the customer. The key idea is we build a reverse proxy as a facade which provides a single interface over many individual backend components.
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1154548100/Untitled%20Diagram-1760460408316.drawio.png?api=v2&version=1\")
We can do this by effectively thinking of the Fleet Manager as a proxy to a number of robots. Let\u2019s suppose for a moment that we have a simple \u201clocal\u201d API on each robot for implementing the 3 criteria from above. We can use this API to task the robot and to retrieve status/history/upcoming tasks. This API knows nothing about other robots. We can treat our Fleet API almost like a reverse proxy to this robot API. The Fleet Manager may need to provide some coordination (don\u2019t give robots the same tasks or the same chargers), but generally these behaviors would be built on top of the baseline robot - just as a large web service is built on top of individual server components.
Before we can build a great Fleet, we need to build a great robot. And before we build a great Fleet API, we need to build a great Robot API. It will be so much easier to scale a really good robot into a massive fleet than to scale a bad robot. We should be thinking more about how we deploy and task individual robots with the confidence that layering coordination and group-management is largely a solved problem. At least compared to building and deploying agentic robots.
" }, { "id": "1134395462", "title": "Wireguard: hmnd-tunnel", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1134395462", "last_modified": "2025-10-13T11:49:50.594Z", "created_at": "2025-10-01T11:59:18.386Z", "body_html": "Our robots have a mini-network onboard across the various computers and sensors, but occasionally we want to extend this network to a developer laptop. For pre-alpha robots, this was done using NAT where the Jetson computer could communicate with a developer laptop by way of the NUC. The NUC would simply forward all the traffic to the Jetson. Easy.
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1134395462/Untitled%20Diagram-1759320225919.drawio.png?api=v2&version=1\")
For Alpha this isn\u2019t so trivial, as we need BOTH the x86 RTPC and the Jetson/Orin to be accessible from the laptop at the same time. If we want to use DDS, its extremely cumbersome to maintain this NAT since some DDS ports would need to be routed locally to the RTPC and some forwarded to the Orin.
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1134395462/Untitled%20Diagram-1759320719959.drawio.png?api=v2&version=1\")
The root issue here is that to the laptop, both the Orin and Jetson have the same IP (whatever the IP of the robot is), and so how can DDS establish who is who? Any ROS node running on the Jetson has to advertise itself as something on the external IP of the robot. Any ROS node on the RTPC must bind to both the external and internal interfaces, but it doesn\u2019t know how to resolve the ambiguities involved with the NAT. And even worse - we need to somehow know which of the hundreds of ports get bound for each participant. This is solvable, but removes a lot of flexibility. Echoing topics or running debugging tools will create new DDS participants which require new ports - do we really expect developers to be tweaking firewall rules and DDS config just to echo topics? Probably not\u2026
Wireguard Tunnels
Enter Wireguard: a simple VPN solution native to Linux where we can tunnel the developer\u2019s laptop over some external interface (WiFi) and then the laptop basically appears on the same routable network as the computers in the robot. This makes it much easier to setup DDS and such for development use-cases.
To simplify setup and management of wireguard, robots have a service running on port 1111 on the RTPC which provides a service for allocating tunnels. Many developers can all connect and can have visibility to which tunnels are active via the tunnel webpage at port 1111:
![\"image-20251001-122025.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1134395462/image-20251001-122025.png?version=1&modificationDate=1759321226837&cacheVersion=1&api=v2&width=857&height=336\")
Now we can configure all DDS participants to be identical across all robots. Because of transitive discovery, we should be able to get DDS configuration identical across even the laptops (though this is still a work in progress).
To setup a tunnel, simply run the hmnd-tunnel script from hmnd/tools and point it at the robot:
user@laptop:git/hmnd/tools$ ./hmnd-tunnel 10.44.10.10:1111\nBringing tunnel up: 1c6bbb560a (hmnd-1c6bbb560a via /etc/wireguard)\n[#] ip link add hmnd-1c6bbb560a type wireguard\n[#] wg setconf hmnd-1c6bbb560a /dev/fd/63\n[#] ip -4 address add 192.168.111.2/24 dev hmnd-1c6bbb560a\n[#] ip link set mtu 1420 up dev hmnd-1c6bbb560a\n[#] ip -4 route add 192.168.1.0/24 dev hmnd-1c6bbb560a\n{"id":"1c6bbb560a","who":"","client_ip":"192.168.111.2","rx_bytes":180,"tx_bytes":124,"last_handshake_unix":1759321190,"age_s":0,"ttl_s":299}\n{"id":"1c6bbb560a","who":"","client_ip":"192.168.111.2","rx_bytes":180,"tx_bytes":124,"last_handshake_unix":1759321190,"age_s":10,"ttl_s":289}\n{"id":"1c6bbb560a","who":"","client_ip":"192.168.111.2","rx_bytes":180,"tx_bytes":124,"last_handshake_unix":1759321190,"age_s":20,"ttl_s":279}\n{"id":"1c6bbb560a","who":"","client_ip":"192.168.111.2","rx_bytes":180,"tx_bytes":156,"last_handshake_unix":1759321190,"age_s":30,"ttl_s":269}\n{"id":"1c6bbb560a","who":"","client_ip":"192.168.111.2","rx_bytes":180,"tx_bytes":156,"last_handshake_unix":1759321190,"age_s":40,"ttl_s":259}\n{"id":"1c6bbb560a","who":"","client_ip":"192.168.111.2","rx_bytes":180,"tx_bytes":188,"last_handshake_unix":1759321190,"age_s":51,"ttl_s":248}\n{"id":"1c6bbb560a","who":"","client_ip":"192.168.111.2","rx_bytes":180,"tx_bytes":188,"last_handshake_unix":1759321190,"age_s":61,"ttl_s":238}\n{"id":"1c6bbb560a","who":"","client_ip":"192.168.111.2","rx_bytes":180,"tx_bytes":188,"last_handshake_unix":1759321190,"age_s":71,"ttl_s":228}\n{"id":"1c6bbb560a","who":"","client_ip":"192.168.111.2","rx_bytes":180,"tx_bytes":220,"last_handshake_unix":1759321190,"age_s":81,"ttl_s":218}\n{"id":"1c6bbb560a","who":"","client_ip":"192.168.111.2","rx_bytes":180,"tx_bytes":220,"last_handshake_unix":1759321190,"age_s":91,"ttl_s":208}\n{"id":"1c6bbb560a","who":"","client_ip":"192.168.111.2","rx_bytes":180,"tx_bytes":252,"last_handshake_unix":1759321190,"age_s":101,"ttl_s":198}\n{"id":"1c6bbb560a","who":"","client_ip":"192.168.111.2","rx_bytes":180,"tx_bytes":252,"last_handshake_unix":1759321190,"age_s":111,"ttl_s":188}\n{"id":"1c6bbb560a","who":"","client_ip":"192.168.111.2","rx_bytes":180,"tx_bytes":252,"last_handshake_unix":1759321190,"age_s":121,"ttl_s":178}\nAs long as this runs, you will be able to access the RTPC and Orin on their internal IPs (192.168.1.1 and 192.168.1.2 respectively). This should work on both MacOS and Linux (and maybe Windows via WSL2).
DDS Config
With hmnd-tunnel we can now use a \u201cuniversal\u201d DDS configuration. The previous DDS config often required substituting the robot\u2019s external IP (due to NATing on PreAlpha) into the configuration files, or manually configuring some wireguard-related IP on Alpha. This setup is toilsome and at times frustrating.
./tools/hmnd-provision $robot_type $robot_number can be used to write a simple \u201cprovisioning\u201d file on each computer:
$ cat /etc/hmnd/hmnd.env\nexport ROBOT=alpha\nexport ROBOT_ID=6\n
Now, if we connect a tunnel to the robot and pixi shell, we can have a full-mesh DDS setup. When pixi activates the environment, it will either:
see sw-internal and configure CycloneDDS with the robot-side configuration (sourcing additional info from the provisioning file above)
see hmnd-tunnel and configure CycloneDDS with the laptop-side configuration.
In both cases, the ROS_DOMAIN_ID is set to 200 - which is somewhat arbitrary.
" }, { "id": "1140654136", "title": "Making Jira Tolerable", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1140654136", "last_modified": "2025-10-03T16:21:47.210Z", "created_at": "2025-10-03T13:42:58.665Z", "body_html": "Jira. Few words instill more fear or paranoia into developers. Burn down charts, velocities, sprints, releases, backlogs, linked tickets, epics, stories, tasks, subtasks, initiatives blah blah blah. \u201cJust let me code\u201d. As much as we\u2019d love this to be the path, stakeholders DO need some visibility into what\u2019s actually going on. The only thing worse than not doing anything at all is putting effort into the wrong things - and how can stakeholders be sure that our efforts are going to the right things if there is no visibility?
Sure - we could ask them to look at GitHub PRs. But git is highly optimized to the needs of developers at the expense of more managerial types. And usually Jira is setup for the benefit of managers at the expense of developer workflows. Unlike Git or Github though, Jira is endless configurable enterprise slop software. Can we make it work for developers too?
Because developers are already working with GitHub, we should first explore how to take advantage of automation here. Using the GitHub/Jira Connector developers can link their branches/PRs with Jira by simply including the ticket number in the branch name or PR description. codg/xyz-123-work-description would be linked to XYZ-123 for example.
Automation can be added so that when a PR is opened for a branch, the ticket moves to in-progress, and when the PR is merged, the ticket is resolved as done. If the PR is closed, you could resolve as canceled, and if the branch is re-opened in another PR, the ticket could be resurrected.
Just these basic bits of automation can help to meet developers in the middle, making it easier to get accurate information within Jira for management types. If you really want to make things more strict, you can also add a pre-commit check that ensures that the commit or branch name reference a valid ticket.
As a developer, using Jira itself is often frustrating. I don\u2019t care about sprint boards or organizing things or linking epics or any of this nonsense. I know that I need to write code and solve problems. I think a big part of this gap is that Jira\u2019s minimal unit is the \u201cissue\u201d: a task, a bug, an epic, etc. So developer throughput becomes issue throughput: creating, updating, closing. creating, updating closing. This is wildly inefficient, and doesn\u2019t really represent normal developer workflows.
Making matters worse is trying to manage releases or sprints in Jira. This is cognitively a lot for developers to deal with while trying to make sure that everything else (coding, testing, git, etc) are actually happening. Few things are more frustrating than creating a new ticket, and then not being able to find it since you\u2019re looking at the current sprint and its somewhere in the backlog even though you\u2019re working on it now.
Developers ought to be using Jira as a work log, with the primary view being their assigned/reported issues. They should be able to see all of these issues, sorted in some reasonable way, and manage them easily with as little friction as possible. Here\u2019s an example:
![\"image-20251003-160816.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1140654136/image-20251003-160816.png?version=1&modificationDate=1759507698125&cacheVersion=1&api=v2&width=875&height=818\")
This is a dashboard representing the following JQL query in a filtered issue list:
project in (10097) AND (assignee = currentUser() OR reporter = currentUser())\nORDER BY resolution DESC, status DESC, updatedDate DESC\n
Not bad, but there are some gaps:
Can\u2019t re-order easily. You could ORDER BY priority, and change the piority, but this brings the second weakness\u2026
No detail issue view or \u201cquick edits\u201d. You can click on an issue but this context shift is annoying and can slow you down.
No ability to \u201csplit issues\u201d. Making it easy to split issues means that developers can be encouraged to commit what they have and then save the rest for later. Here we have no split issue option available\u2026
Some of these features are only available on a Kanban board, which would be sort of ideal if we could control it a bit better. I\u2019m not sure what addons or configuration changes might be necessary to get us there, but to summarize we should be looking at how to give devs a streamlined URL they can bookmark to see their issues. They should be able to manage their work like a log as much as possible, with as little thought as possible going to managing issues.
The view above is also useful for managers and leads to see what individuals are focused on - but we need a bit more. Managers and leads need a bird\u2019s eye view across engineers to plan work effectively. Fortunately Jira has the tools necessary for this, and usually they are setup by default.
For example, in the \u201cmanagement\u201d view, I\u2019d like to be able to assign issues to specific releases for release planning. I want to be able to see the progress of epics and all the normal stuff. But 95% of the time, engineers don\u2019t care about any of this and its just getting in the way.
![\"image-20251003-161958.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1140654136/image-20251003-161958.png?version=1&modificationDate=1759508400301&cacheVersion=1&api=v2&width=875&height=902\")
Jira is better at bottom-up communication than top-down communication. We shouldn\u2019t expect engineers to do stuff when issues are created or modified: that\u2019s what automation is for. But if we make it really easy for developers to keep Jira up-to-date, we can provide managers with an accurate picture of where effort is being spent. Managers can then \u201cclose the loop\u201d directly with individuals or with the team either on Slack or in meetings.
Individual teams can create their own boards or custom dashboards. I strongly recommend creating dedicated dashboards with issue filters for each member of the team which can be quickly reviewed at standup meetings or during 1:1s. This is usually faster and easier than trying to adjust the filter in the view shown above.
" }, { "id": "1135345776", "title": "Sockpuppet: Teleoperation Simplified", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1135345776", "last_modified": "2025-10-02T16:59:25.254Z", "created_at": "2025-10-01T20:46:29.422Z", "body_html": "Sockpuppet is an easy-to-setup application for remote teleoperation. It's adapted from the existing quest_teleop modules in ROS, but with some nice simplifications:
UI, including camera streams, is hosted by the robot and accessible with a modern web browser.
Standard web interfaces mean we can punch through NATs more easily.
No need to setup DDS or ROS on client machines
No need to pair laptops with specific robots
Overview
Sockpuppet contains a few individual pieces working together:
A lightweight proxy running on the laptop to access the Meta Quest
The impedance and whole-body controllers running on the x86 RTPC
Cameras running on the Orin
Data collection on the Orin
The sockpuppet server and quest_teleop
Setup Laptop
First you need to prepare the Meta Quest itself:
Once the Quest has been configured, you can install and run the local quest proxy with:
# install UV (this doesn't require a full pixi build on the laptop)\ncurl -LsSf https://astral.sh/uv/install.sh | sh\n\n# launch the proxy\n./hmnd_robot/apps/sockpuppet/sockpuppet/sockpuppet_quest_proxy.py\n\n# or if you don't have a quest:\nMOCK=1 ./hmnd_robot/apps/sockpuppet/sockpuppet/sockpuppet_quest_proxy.py\n
At this point you are setup. You can connect to and teleoperate any robots currently running Sockpuppet.
The MOCK=1 proxy will generate \u201crandom\u201d transformations and should probably not be attached to a real robot (though the motions are small enough that the teleop server will successfully calibrate the headset\u2026 usually)
Controller Bringup
Hopefully automatic
Sockpuppet Bringup
On the Orin:
pixi run sockpuppet_alpha_wheelbase\n
Connecting to Robot
Goto http://robot:7777 (robot IPs link)
** screenshot - show panels
Verify the quest proxy is connected
Verify calibration
Toggle teleoperation
Collect data
Have fun!
Launching Alternatives
Mock and mock quest proxy
Cameras only
Implementation Notes
The proxy is a simple websocket server that reads states from the controllers via oculus_reader/adb and sends haptic commands hosted at localhost:7778
The Sockpuppet server has a ROS nodes which acts as a bridge between FastAPI/HTTP and ROS. The controller data is a websocket endpoint, and the images are sent as JPEG streams which constantly replace the contents of img tags.
The UI is a simple web page which will connect to the Sockpuppet server and attempt to connect to the local quest proxy. If the connection to the local proxy is established, another websocket to the Sockpuppet server is opened and the data is forwarded both ways.
Data collection works because the pedals are just a fancy keyboard device which will trigger service calls via the ROS bridge along with the package and episode IDs.
The Sockpuppet server+quest proxy basically replaces the existing 'start_trackers' node, the quest_teleop node is largely untouched and the teleop_gui is replaced with the web page.
Future Improvements
We should pipe some of the forward kinematics or TF frames back to the browser so we can do a few things:
Rotate the wrist camera images to keep a consistent horizon
Render controller poses from the proxy locally in the camera's frame for immediate operator feedback.
This would require more significant refactoring - the interface via HTTP wouldn't be Quest poses anymore, but Cartesian targets (computed locally either via the proxy or the browser)
Consolidate quest_teleop and sockpuppet server nodes - simplifies a few things
Remove use of parameters as state
WebRTC instead of websockets
Lower latency, no head-of-line blocking
Requires STUN or TURN infra - not hard just annoying
One of the most vexing problems in robotics is structure-from-motion (SfM) - how do we estimate the structure of a 3D environment from a series of 2D images? SfM is one of the pillars of computer vision research, and rightfully so. There have been a number of attempts to side-step this problem. The root of the challenge is that for any given image produced by a camera, there are an infinite number of scenes that could result in that image.
![\"image-20250901-142817.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1071513687/image-20250901-142817.png?version=1&modificationDate=1756736899347&cacheVersion=1&api=v2&width=384&height=287\")
This has been used by VFX artists for decades to perform various bits of trickery, but its a major confounding variable for computer vision. More recently, sensors have been developed which can sense depth directly. \u201cSparse\u201d depth sensors like LIDARs (or ultrasonics or radar) can probe the environment directly for depth information albeit at relatively poor spatial resolution, and \u201cdense\u201d depth cameras using structured light or time-of-flight which can infer depth directly from an IR image at much higher resolutions.
Using these depth sensors, the common approach has been to populate some sort of voxel-based occupancy grid. Each pixel on the depth sensor providing two pieces of information: 1) the voxel corresponding to (x, y, z) is occupied and 2) the ray from the camera TO the voxel at (x, y, z) is clear. SLAM-based approaches can help to stitch all of this into a global map, though global voxel-based solutions are EXREMELY memory-heavy without a lot of additional heuristics and datastructures like sparse-voxel octrees or other accelerating datastructures (used commonly in Voxel-based games like Minecraft for example).
Great - we can sense depth directly and now SfM is solved and we don\u2019t have to worry about it, right? Well - not quite. In my experience, depth sensors are an enormous pain in the arse. They have extremely nasty and fickle edge-cases that will ultimately dominate your development time. Things like glass doors or polished floors causing \u201cghost voxels\u201d to appear in front of the robot, or interference between neighboring sensors, robots or even ambient sources like heat lamps or the sun can result in erroneous readings.
Another major challenge is that the resulting voxel-based quantization makes marking and clearing an obstacle map extremely challenging. Using something like 3D Bresenham\u2019s to clear voxels through the grid results in many \u201cstray\u201d voxels that don\u2019t get cleared since everything is being quantized to voxel space, but the robot usually needs to maneuver much more precisely - rotations in particular will result in various bits and pieces being left behind, resulting in false positive obstacles.
To get around these various issues, developers will spend man-years developing and tuning filters. How do you filter the \u201cghost voxel\u201d or clear the \u201cstray voxel\u201d without having the robot running into a pole or some other obstacle?
Finally, interpreting depth images alone is challenging as they usually lack relevant semantic info for labeling or understanding the scene. This means that depth images often need to be aligned with RGB images anyway - which limits sensor selection and positioning.
![\"image-20250902-092407.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1071513687/image-20250902-092407.png?version=1&modificationDate=1756805049117&cacheVersion=1&api=v2&width=600&height=338\")
There are some reasonable solutions to these problems (for example nvblox: https://nvidia-isaac-ros.github.io/concepts/scene_reconstruction/nvblox/index.html?utm_source=chatgpt.com ) but it still requires a whole new modality and eats up a significant amount of processing in addition to the base-level vision processing required for our VLA to work.
Researchers have forever been guided by the knowledge that humans don\u2019t need depth-specific modalities. A pirate wearing an eyepatch can swashbuckle and navigate around the ship without much issue - so even stereo vision seems more like a nice-to-have in some contexts. Can we benefit? (note - a pirate also benefits from an enormous amount of priors and other sensor modalities).
Going from RGB images to some sort of dense scene representation like a voxel occupancy grid is very challenging, but also probably not what we want to do (given the issues above with voxel-based representations). The state of the art here revolves around NERF and Splats.
![\"image-20250902-092856.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1071513687/image-20250902-092856.png?version=1&modificationDate=1756805337980&cacheVersion=1&api=v2&width=656&height=365\")
These techniques infer the \u201cradiance\u201d of the environment (meaning the actual lighting - not just the color of an object) using some sort of optimization scheme. NERF uses differentiable rendering techniques to essentially encode the scene in the weights of a neural network. This means that learning a new scene means training a new neural network which kinda sucks. Splats are a significant improvement on this so we\u2019ll focus on this approach.
Gaussian splatting basically initializing each image with a grid of ellipsoids. We know that the pixel we see is from somewhere along a ray from the camera, so we can fit a gaussian ellipsoid (the \u201csplat\u201d) along that ray (relatively low variance in the x-y direction, high variance in the z/depth direction). We can then run a fancy optimizer to solve for camera poses and splats. Some splats (typically around occluded features) may need to be split into multiple new splats, while others (typically features viewed from many images) will be collapsed into a single splat. The end result is a solution of the camera\u2019s trajectory through some world space, and all the gaussian splats that best \u201cexplain\u201d the image data. For a primitive demo you can see here: https://webgl-gaussian-splatting.vercel.app/
Splats enable some extremely powerful capabilities.
Because they aren\u2019t just learning the pixels but the view-dependent lighting (fancy anisotropic gaussians) of the scene at each point, you can render the scene from arbitrary viewpoints not in the original dataset. A robot with a splat-based map could in theory \u201cimagine\u201d what actions it may want to take and choose the best one by playing forward rendered images from an imagined hypothetical trajectory.
It may also help to bridge the sim-to-real gap - if we only trained on imagery rendered from gaussian splats, we may be able to eliminate sensitivity to camera calibration/placement, scene lighting and more from our training (in the same way that our whole body controller eliminates some kinematic or calibration-related dependencies on the control side). The artifacts from splat-based rendering can be significant though - and it remains to be seen whether or not we can train or fine-tune image models on this data.
One of the major challenges with \u201copen-loop\u201d testing is that you often work with recorded datasets. This puts the robot \u201con rails\u201d, and any changes to the code which take the robot from the recorded trajectories will invalidate all the future recorded data after that change. NVS allows some amount of variation from the path in these cases, since you can render new viewpoints from the generated model. It becomes possible to record a canonical behavior of the robot and then randomize it at runtime - without blowing up the amount of data required.
Splats are extremely impressive in demos, but creating a product based on these is probably very difficult right now. You could spend enormous amounts of time collecting huge amounts of video data, all to find that the scene never converges. These issues exist in general SLAM solutions (lidar slam, VSLAM) but because of the difficult of solving a splat scene its generally much more painful to iterate and tweak until a solution is found.
The bigger issue is that splats are mostly an offline technology - you need all images from the past, present and future at the same time to solve the world. This greatly limits their usefulness in the context of realtime mapping and perception. So you need a huge data and compute commitment just to try to solve.
There has been some research toward incrementally building splats, but running this in realtime is a major challenge since the solver can take wildly unpredictable amounts of time.
There\u2019s a lot of current research in splats attempting to build feed-forward neural networks that can handle the \u201cfrontend\u201d portion of the problem - basically going from an image to an array of reasonably bounded splats in a deterministic amount of time. Because these guesses are better than the naive initialization used in offline approaches, incrementally optimizing/solving becomes much more tractable. (https://github.com/nickwzk/StreamSplat )
The StreamSplat approach linked above does this by using DINOv2, which is a self-supervised image foundation model that excels at the dense/local tasks like segmentation and depth estimation as opposed to the sparse/global tasks like more general scene understanding that are used to power our VLA. DINOv3 was recently announced, and I\u2019m confident that teams will be racing to apply it to these problems.
We should assume that we will need both global scene understanding w/ language at lower temporal resolution (something like Gemma or existing VLMs) in addition to something with local feature understanding at a much higher resolution. We could use the local models to generate splats, depth or even to do visual servo-ing, while we use the global models to drive the language-conditioned task. The DINOv3 encoding could also provide other data that could be useful in the context of robots - think reading human poses from vision.
The splat-based representation provides a sparse model of the world that is both spatial AND semantic in ways that a simple voxel grid is not. Given some local region of a splat-based map, we can construct a voxel grid from the splat data. While voxel grids aren\u2019t a great ground-truth datastructure for the world, they are a good accelerated datastructure for spatial queries (planning and obstacle avoidance being the biggest examples).
One of the major challenges with voxel grids is updating them over time - especially due to quantization or as the robot moves. Splats don\u2019t have this issue - they are not quantized in any meaningful way, and can be rotated freely. Planning a path in this space is miserable though. For navigation-related tasks (global navigation specifically) and other spatial queries, we should process the splat data periodically into a flat \u201cnavigation grid\u201d where we can run something like A* or classic planners to get from point A to point B.
Most splat engines will already be organizing the splats within some spatial datastructure like an octree or a bounded-volume hierarchy. This same representation can be used to efficiently generate voxel occupancy grids or navigation grids for more efficient spatial querying while still preserving the splats as the \u201cground truth\u201d.
If we look at all the things the robot needs to do piecewise, things get very expensive, very quickly. Different sensors for obstacle detection vs semantic scene understanding, different GPU kernels, algorithms or neural networks - it all adds up very quickly.
Vision allows us to get economies of scale. We already NEED vision to do the tasks we want to do. It just so happens that we can ALSO do a bunch of other stuff with vision alone. To do this, we need to get much fancier in our AI architecture. We need to consider language embeddings, vision embeddings, action embeddings/tokenizers - all running at potentially different rates - to make the best use of our compute hardware.
" }, { "id": "1069973507", "title": "Swinging Big", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1069973507", "last_modified": "2025-09-01T09:53:19.775Z", "created_at": "2025-08-30T21:44:23.371Z", "body_html": "There is a classic debate raging through most startups. On one side you have Paul Graham's 'incrementalism' - you iterate rapidly with customers in the loop guiding you until you get product-market fit and then scale the business as fast as possible. On the other side you have Peter Thiel suggesting that you invent the future and create entirely new markets. The truth is that to succeed a company needs to do a bit of both, but I think a humanoid robotics company should be erring on the side of going big.
![\"image-20250901-092507.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1069973507/image-20250901-092507.png?version=1&modificationDate=1756718708701&cacheVersion=1&api=v2&width=257&height=171\")
The whole point of building humanoid robots is that the general form factor allows us to tackle massive scale because the world is already built for human proportions and morphology. We should double and triple down on this. This is such an enormous vision by itself that it almost disqualifies doing anything other than building the future. We can't incrementally build humanoids, because we will probably go bankrupt or be disrupted before we actually get there.
As a starting point, we should assume that we want our robots to be standalone, autonomous agents. Every additional widget, cable or box we ship adds friction to our growth and ultimately takes time away from building the best possible robot. We should aim to replace as much physical infrastructure with robot intelligence/behavior - because for the first time in history this is actually sorta possible.
If we ship networking infrastructure, servers, teleoperator terminals, tablets or dongles we shift away from providing a scalable robotics product toward providing systems integrations. We become IBM providing professional services and sub-contracting warehouse integrations. This is probably a bad business given our ambitions. Probably the difference between 40% growth and 400% growth YoY. All of this extra infrastructure would gobble up our margins and spread our attention and focus.
Now of course there will be extra infrastructure - but we should start from a default of zero additional infrastructure and only add when there are no other solutions. Customers don't want some vendor ops teams lingering around. They want plug-and-play labor, with an obvious ROI.
Another major challenge in robotics is about accountability. Who is actually responsible for our robots doing their job and how do we make sure they want us to succeed? Consider what happens if our robots get stuck or impede access in congested areas within some hypothetical factory:
If employees have to wait for some teleoperator to notice and solve the issue, they will HATE our product and want it to fail. They may even actively sabotage it. We would be effectively stealing accountability from them and shifting it to ourselves - leaving them frustrated and powerless. This doesn't scale at all. We would be fronting enormous amounts of operational risk with little upside (at least compared to alternatives).
Okay - so we will have them login to an app or a tablet to move the robot or resolve the issue. And again - they will despise us for this. This is a miserable interaction and we are making it difficult (if not impossible) for them to be successful.
If we layer apps, web interfaces or remote teleoperators between the robots and the people attempting to work alongside them, we are just a really shitty temp agency with extra steps and more complex pricing. Why would employees want our robots to succeed (they don\u2019t see the benefits). Why would teleoperators care about the impact to workers in the factory? The incentives are wildly misaligned and we shouldn\u2019t expect for things to magically work out in our favor.
Instead of trying to take on all the operational risks ourselves, we should be looking at how to push this accountability to the factory floor where it belongs. We are transforming a labor problem into a management problem. So how do we provide the best tools for managing these robots? I doubt it looks like someone playing Starcraft in the back room.
Our potential customers simply don't know or understand the current trajectory of AI as it applies to robotics. The capabilities are evolving at a dizzying rate, and the future that we are building cannot even be imagined by customers so we should be sure to put customer feedback into the correct context.
![\"image-20250901-095250.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1069973507/image-20250901-095250.png?version=1&modificationDate=1756720372381&cacheVersion=1&api=v2&width=354&height=289\")
Customers will be all too happy to tell us that they just want the industrial robotics pitches of the last decade to deliver what was actually promised - some fleet manager, feeding tasks from some WMS or other software system, with human-in-the-loop teleoperation for recovering from faults and keeping their operations moving along.
Our vision should be onboarding robots not as tools and infrastructure but as agentic workers. We should assume that models will become increasingly capable and that new techniques will allow us to cram more capabilities into our footprint. We should be building with enormous thermal and power margins, assuming that our robot's motion will become much faster and more aggressive over time.
We can't let the industry hold us back if we want to build the future. We need to pull them along, kicking and screaming. Or someone else will.
So what should this look like? I'm going to outline a strawman robot without any regard to safety, hardware or software complexity or even feasibility - the idea is to be bold and ambitious, and our engineers will have an easy(-ish) time dialing it back to make it cost-effective or feasible.
The robot is smart and autonomous. Petaflops of embedded compute, 360 degree RGB vision and two reasonably dextrous arms with grippers at the end. All on a wheel base that can drive around.
The robot should have 5G so it can phone home with status, get updates or push novel episodes to expand the skill repertoire - all without any additional infrastructure or setup. The robot doesn\u2019t need 100% connectivity, because its smart and autonomous - but it knows how to navigate to different areas of the environment with service when needed.
The robot should have a touchscreen display so that users can interact with it locally with some reasonable bandwidth. Voice is probably not feasible in an industrial setting, but this touchscreen could be used to bootstrap other forms of connectivity (pairing a Bluetooth headset, configuring local wifi). In addition to setup, the touchscreen allows our interfaces to evolve over time. Just because we can build the future doesn\u2019t mean we know what it looks like right now.
The robot should have a way to be moved or directed physically. Pushing the robot around with some handles would be natural and low-friction compared to literally anything else.
![\"image-20250901-094035.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1069973507/image-20250901-094035.png?version=1&modificationDate=1756719637305&cacheVersion=1&api=v2&width=300&height=168\")
The robot should be able to plug itself into the wall for charging. No infrastructure needed. If you want to get crazy, let robots plug into other charging robots.
The robot should be able to hot-swap end-effectors. Grippers, hands, probes, screwdrivers - who knows what capabilities we want in 2028 or 2030. Plus we probably need to do this for field service anyway.
The customer receives a new robot and uncrates it. Its powered on, and the touchscreen guides them through minimal setup and then they should be able to push the robot through the environment, annotating and labeling via the touch screen.
A manager should be able to show the robot's workspace and describe its job in natural language. If we have some sort of in-context learning, a demonstration of the task could be used. Ideally the robot is capable of learning from human demonstration rather than only teleoperation. Put items into totes like this. Unload totes onto conveyors like this. Sort boxes like this.
For very large deployments a shortcut should be possible to quickly onboard individual robots, perhaps by showing the robot a QR code and entering a pin on the touchscreen to provision. For example, the first robot is setup the \u201cclassic\u201d way, but subsequent robots are registered via fleet management to receive the map and all the necessary context.
The robot then plugs itself into the wall to charge.
The robot is notified via it's touchscreen by a manager that it's time to get back to work. The jobs it knows are saved and can be selected on the screen. The manager selects the job for filling and stacking totes.
The robot navigates to the appropriate work station for this job and begins loading items from a conveyor into totes. When the totes are full they are carried and stacked on a pallet. At various times during the shift the robot may have difficulty loading or stacking totes. It should improvise to keep working and possibly come back to problematic situations later. For example, instead of asking for help 5 separate times during a shift, can the robot have a strategy where it makes progress and defers help until the end where one intervention can address all the issues?
Sometimes new skills may be demonstrated to the robot. These should be stored via some sort of RAG-like scheme so that the robot can make use of new schemes using in-context learning rather than requiring additional fine tuning. Managers and employees know that if they continue demonstrating the tasks, the robot will get better fast enough to make it worthwhile (they don't need to wait weeks for a new model). Periodically these behaviors will be added to training runs to improve the foundation model\u2019s performance.
With enough robots, some sort of coordination is necessary to avoid congestion and balance work across robots. This should be an optimization rather than core to basic interactions and behaviors of the robot.
Robots will need to be aware of one another to avoid congestion around charging locations. Robots may need to communicate about bulk uploads/downloads to avoid congesting the network. But generally fleet management is about modulating existing robot behaviors rather than creating entirely new behaviors.
Yes - there are ID concerns, and safety and cost. Maybe outlets can't deliver enough power to enough robots. Or they are inaccessible. Maybe we need some markers to help label the world physically because SLAM isn't perfect. Maybe we need some additional network solution because 5G isn't available at all locations.
The point isn't to dogmatically build this minimal robot. The point is to imagine it - because that is the future we actually want to build. It's easy to get distracted by the realities of today, diluting our vision and ultimately tying an anchor to our ambitions.
If we can avoid shipping any extra hardware by making the robot smarter, this is probably a good trade. Infrastructure is not getting easier to deploy at the rate that AI is improving.
If we can shift operational accountability from Humanoid to the factory floor, we should do this. It aligns incentives, limits our risk and helps us scale faster.
We should weigh customer feedback as if they are unaware of the rapid AI progress occurring and as if they are over-fit towards older solutions to automation. They don't know what's possible, and they don't know specifically what they want. They want to save money. They want uptime. They want transparency. But they don't know how to make those things happen. If Steve Jobs had asked every executive in 2005 how important a physical keyboard was on a smartphone, how many would have been fine with a full screen touch screen and virtual keyboard?
We are building a category-defining product and we should have the conviction to follow a bold vision.
" }, { "id": "1052508214", "title": "Wireless Networking Sucks", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1052508214", "last_modified": "2025-08-26T22:15:02.381Z", "created_at": "2025-08-21T09:51:28.098Z", "body_html": "My time at Viasat (working on Satcom networks) and Cloudflare (more traditional Internet infra) has led me to believe that wireless generally sucks. We should minimize how much we depend on it. This doesn\u2019t mean tethered cables or sneaker-net robots - but I think it does involve acknowledging the strengths and weaknesses of wireless technologies and building a solution that fits with reality.
WiFi
Wifi was originally created as simply wireless ethernet. The air is a hub, with modems chirping when its time to send data. If multiple modems chirp data at the same time on a channel, exponential backoff is used to ensure collision-free communication. It works, but it introduces a lot of tail latencies.
More modern WiFi fundamentally works the same way, but is better about packing more collision-free chirps within the allotted bandwidth. There are a few paths to doing this:
MIMO - take a faster datastream, break it into two lower-rate streams which are modulated with different codes (separated) on the same channel. This helps to better utilize the available spectrum (packing more chirps per channel)
Beamforming - blasting energy in all directions is wildly inefficient from a power and spectrum perspective. Every chirp sees every other chirp as interference. Better if you can direct all the energy toward where it will be received. This reduces the amount of collisions on channels and allows better utilization of the spectrum (less backoff - more chirps per channel)
Higher frequencies - allow for more data to be pushed in the channel.
Modern WiFi is impressive (especially WiFi7) - but for industrial robots there are still some dark corners here. Beamforming and higher-frequencies mean non line-of-sight (NLOS) use-cases suffer greatly. What if there is a giant piece of industrial equipment between the AP and your robot? What if the robot is moving through a space where reflections and equipment change the path rapidly requiring beamforming to repeatedly re-acquire the signal?
Typically to cover the space you need to have many APs with the robots roaming around between them, or routing WiFi between the robots themselves\u2026
PTP
Time-multiplexed ptp links are also possible (see Practical Evaluation of Low-Latency TDMA-based 60 GHz PtP Link for Robot Control ), but here we just trade-off collision backoff for acquisition delays. And we have more strict requirements about line-of-sight. Probably not terribly helpful.
Mesh Networking
Mesh networking is one of the great lies of the 21st century. The idea is that instead of having some hub and spoke topology, you instead of a dynamic routing network of peers that can discover and adjust over time. In some sense, the Internet is a mesh network powered by modern IP routing algorithms like BGP. Doing this with wireless though? Madness.
You\u2019re basically taking all the challenges of WiFi and piling them into a full system. It would be extremely fragile with extreme tail latencies and lots of intermittent failures. Each hop in the network means taking on queuing delays and re-transmission delays.
Any time the topology changes (frequent when you consider the issues above w/ beamforming and acquisition), you have additional latencies and loss - typically not bounded at all.
5G
At least modern 5G was built for mobility. It has a scheduled MAC, which keeps collisions low and avoids the random backoff latencies, it has QoS and it supports hand-over natively.
5G also supports private spectrum/buildout - but this is probably more of a lift than WiFi even.
What Could We Do?
WiFi7. Many small APs. Low power. Maybe a redundant link if we really want to be open, but I\u2019ll try and make the case below that we shouldn\u2019t bother.
WiFi7 should allow us to get relatively seemless handover between cells.
We should probably put the robot\u2019s WiFi antennas on the head. This places them relatively far away from the chonkier metal parts of the robot, but we still run into the situation where the antenna will be coupled to the robot\u2019s body and thus pose. Meaning that as the robot moves through poses, it will affect the beamforming and overall wireless performance. I think this would be minimized at the head (a lot more plastic, can get a few cm away from the big metal bits), but this is going to be an issue no matter what.
APs on the ceiling, antennas on the head would probably give the best performance for commodity WiFi but that would need to be verified. Perhaps the approach is just to put an AP above every workspace.
A 5G modem (or slot for one) would be nice\u2026 In theory we could sell standalone robots to areas that need no additional infra (think deployments <6 robots). This is not a small lift (RF, I/O ports, antennas).
If we need backhaul from a customer site, we should consider something like Starlink where we can own more of our network infra (as opposed to trying into their network).
Its obvious why we would want some perfect wireless link to our robots. It would make a lot of things easier. Unfortunately, my experience tells me this is a never-ending quagmire and ultimately will bottleneck the growth of our fleet. I\u2019m confident we could build robots faster than we could perform site surveys and deploy wireless infra for our robots.
The primary reason for high-quality, reliable wireless comms is to remotely control the robot. Either via teleoperation or via some remote inference server. If we move forward with these approaches, we\u2019ll need to solve networking at a very high level - and I think this problem is on the same order of complexity as deploying the robot itself.
Crappier Networking
So what if our network is crappier? Loss of connection, tail latencies \u2192 what does this mean? If we are running teleoperation, it could lead to operators becoming confused or being unable to accurately control the robot. If we are running inference, the policy is less likely to \u201cbe\u201d confused (as it experiences time differently than a human would), but the performance of the robot would suffer and generally it would look like crap.
But a crappier network is still good for non-realtime control. We could send tasks, get feedback, diagnostics, update software, upload data. No problem.
Dealing with Latency
The problem with latency is that its unpredictable. Even high predictable latency is problematic for realtime control - ever try to SSH to a server over a satellite connection with 0.5s of latency? This is a silly example, but I think there is maybe a useful technique here. In remote terminals with high latency, the trick is to locally echo the characters being typed to provide immediate feedback to the user while the characters are being transferred over the buffer. Typical user interactions with a remote shell are broken into natural chunks - commands, confirmations, etc etc. Each of these can be handled easily by locally echoing the characters and then the user can wait for the next step to continue. Not ideal, but robust.
Could we do the same? Could we provide some way to locally confirm teleop commands in a way that decouples the user\u2019s interactive experience from the robot while still allowing us to do the work? Consider a teleop monitor showing the robot\u2019s view being fed over a less-than-perfect network. The latency is usually say 100ms, but can spike to 400 or more randomly. When these spikes happen while the user is still, no problem. If it happens while the user is controlling some trajectory, it can result in confusion and the user stopping or making unpredictable motions (thus producing garbage data\u2026)
What if we overlaid a semi-transparent rendering of where we believe the robot should be based on the teleoperation commands? If not the actual robot positions (challenging with cartesian control running on the robot), we could at least show the targets being controlled in screen space. This way the operator can get immediate feedback about their inputs even if the robot is susceptible to some issues due to latency. This may be a much cheaper way to paper over some of the issues with crappier networking without resorting to some complicated networking infrastructure.
Avoiding the Network
Of course, we should also consider avoiding the network entirely. For policy execution, we should be running on the robot\u2019s GPU anyway. But what about teleoperation?
One dumb possibility is to put the the controller into LIMP mode where the joints are back-drivable and have the operator manually operate the robot. The reason this is dumb is because it brings a lot of challenges:
safety - you are asking an operator to directly manipulate robot joints in certain modes. this could be challenging to certify\u2026
data - the data being generated this way isn\u2019t really going to be the same as data being produced by feeding cartesian targets to the controller.
For the second point, maybe you can just manipulate the end-effectors as targets and run the rest of the thing IK - but again, this produces an interesting safety risk. Is solving this safety problem more difficult than solving the networking challenge?
Another possibility is to have the robot watch a demonstration directly. We have 360 vision around the robot\u2019s head, and only the front/wrist cameras are used for teleoperation and policies currently. Would it be possible for an operator to stand behind or to the side of the robot looking at a display to control the robot via motion capture from video? There are obvious challenges with this approach too:
depth - depending on the demonstration position relative to the robot, capturing depth could be challenging and result in poor-quality data
data - tracking these trajectories in mocap is also going to result in some different data characteristics that could be challenging to learn from.
One major benefit to the \u201cdemonstration\u201d aspect is that it sorta generalizes beyond just teleoperation. If we were to crack in-context, one-shot learning it would map nicely to that. Or perhaps a robot working among people could observe others working and learn. This is all scifi - but so is a perfect wireless network.
I\u2019m not sure what the point of this document really is. Mostly to get out some beliefs around how much wireless networking sucks and what a quagmire it can be. Maybe a little bit of poking/prodding to see if we can think of ways to meet our product objectives without requiring so much infrastructure and performance out of this network.
Maybe the robot knows how to plug itself into the wall for tele-op via a cable. Maybe some fancy ptp laser link. I really don\u2019t know but there must be some other options here.
" }, { "id": "1052377106", "title": "Hardware Abstraction", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1052377106", "last_modified": "2025-08-21T09:49:34.517Z", "created_at": "2025-08-21T09:19:36.447Z", "body_html": "One of the fun challenges about robotics software in particular is the hardware dependency. Unlike your email or note-taking application, the point of a robot to do something physically in the real world. To push on the world, and see the changes as photons hitting a sensor as interactively as possible.
Software is a pain in the arse, but it has some benefits that are worth acknowledging. You can make perfect copies of software and version it exactly for almost zero cost. Hardware engineers would love to have this capability, but instead they are plagued with supply chain nonsense, tolerance nonsense and things breaking. The world of atoms is far less forgiving than the world of bits.
When we build hundreds or thousands of different robots, these minute differences in hardware \u201cpush\u201d on our software in interesting ways that can wreak havoc if we are not deliberate about our design and architecture. Tiny configuration changes, and branches can litter your code making it brittle and expensive to maintain or change.
Most platforms that sit on top of mass-produced hardware include something known as a hardware abstraction layer or HAL. Windows 95 had one. Android has one. Linux has one (although its fragments under different names). All of these platforms run on very diverse hardware, yet want application developers to have a stable platform to target for end-users. When you boot up Chrome to visit websites, you don\u2019t want the website to care about which CPU you have, or how much memory, or which disk vendor or motherboard or ethernet adapters you are using. The browser only cares about the platform interfaces: threads, buffers, sockets, files.
![\"image-20250821-094827.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1052377106/image-20250821-094827.png?version=1&modificationDate=1755769708399&cacheVersion=1&api=v2&width=333&height=151\")
For a robot, the problem is both more bounded and more difficult. We don\u2019t necessarily expect robots to plug-and-play different actuators, sensors or be configured in arbitrary morphologies - but we do have a much larger surface area of hardware abstraction than more traditional compute platforms.
Hardware is fundamentally concurrent and asynchronous. Usually hardware peripherals have their own computers, running software on their own clocks and sending messages on their own time. The environment that the robot operates in is also largely non-deterministic.
Non-determinism stinks - it makes it extremely difficult to reproduce faults or to debug the system. It makes testing difficult. It makes reasoning about the system difficult. What\u2019s the point of building a bunch of nice deterministic software modules that can be well-tested if we\u2019re just going to bolt them on top of a firehose of random data?
One way to view something like a HAL is it acts as the boundary between the non-deterministic world of sensors and actuators and the more deterministic world of our software. By recording the data at this layer, we can reproduce faults and behaviors more-or-less exactly.
So let\u2019s imagine for a moment that we abstract all the hardware behind a platform-specific HAL which can come in several variants. One implementation may take recorded data and simply replay it, perfectly reproducing the unique scenario that was captured with running hardware. Another may run a simulator like Isaac or Mujoco. And of course we have to have an implementation backed by real hardware drivers running on the target devices (otherwise what\u2019s the point?!)
The goal would be to cut the HAL interface such that all the software on top runs without a care in the world about which of the variants it is running on. There shouldn\u2019t be \u201cif simulation\u201d or \u201cif testing\u201d flags littering our software stack - those differences should be on the other side of the HAL.
To support multiple platforms, we\u2019ll likely be re-composing a number of shared elements including things like power mgmt interfaces, actuator interfaces, sensor interfaces and more. Different application will depend on different platforms, though nothing prevents any specific application from supporting multiple platforms.
Right now we have a significant safety risk where developers can SSH to robots to perform work while others are nearby. Or perhaps connect to the wrong robot for teleoperation.
I think the best approach here is physical interlocks/keys which force a human to take accountability for a specific robot before accessing it.
Goals:
Robots should have at most 1 operator with exclusive permission to move the robot
Robots shouldn\u2019t physically move out of sight from the operator
impossible to enforce in full - but we can patch w/ process
Software Options
We could do something to limit the number of concurrent SSH connections on the robot, but this does\u2019t really cover the case where an \u201cidle\u201d robot in the lab suddenly comes to life. We could ensure that robots are powered off when not in use, but it is often the case that some level of access is needed for development work meaning the robots would be powered on and on the network.
There isn\u2019t really a good way in pure software - what we want is to effectively force an operator to go to the robot to \u201cclaim\u201d it.
Hardware Options
Imagine every developer/operator who is expected to operate a robot has a yubikey. This key has a unique ID and allows for some cryptographic functions - but the primary purpose is that we can prevent the controller stack from operating (for Alpha we could do this all the way down at the ethercat driver layer which should change relatively rarely) and issue a challenge to the key.
Using a physical key means that a developer has to actually go to the robot to claim it. If someone else has already claimed the robot, there will already be a key inserted. Given a key, it should be possible to trace who it was issued to.
Implementation Plan
ensure there is a single accessible USB port dedicated for these keys (we can ensure only this port can be used for the locking functions as defined above)
issue keys to developers and register each one - at some point we may want to do this for other reasons, so its good to build habits now
implement a lockout mechanism in the controller stack:
libfido2 can be used from C++ to challenge the key and unlock the stack
this isn\u2019t trivial, but its also not impossible
we can put a time limit on it - so developers have to return to the robot and touch the key every so often to restore access
Alpha E-Stop
This plan above isn\u2019t so different from the Alpha e-stop - which is a physical lock-out paired to the robot needed to operate the robot. If this isn\u2019t sufficient we can look at some of the options above for more traceability, but I\u2019m not sure we have the engineering bandwidth for this if the wireless e-stop does the trick.
" }, { "id": "1052442642", "title": "The Case For (Some) Hard Realtime", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1052442642", "last_modified": "2025-08-21T08:46:55.021Z", "created_at": "2025-08-21T08:15:26.438Z", "body_html": "Our Alpha robot has a \u201crealtime\u201d PC running x86 with a preempt-able Linux kernel, along with an nVidia Orin which handles the bulk of our higher level compute. Our usage of EtherCAT for 1000Hz impedance control at the joint level requires us to have low jitter and low latency, so the RT-patched kernel allows us to run this on Linux using commodity Ethernet adapters instead of bulky peripherals. We can then run the joint-level controller within ROS and communicate via DDS with the Orin.
The solution has a ton of complexity, without too much benefit. The realtime Linux isn\u2019t REALLY realtime - but its probably good enough for what we want to do (latencies in the 10s of microseconds) - as long as you don\u2019t have any dodgy drivers that aren\u2019t preempt-able. But for what we\u2019re trying to do the RTPC is wildly overpowered and requires us to distribute our ROS stack over multiple computers. This has troubling implications for software updates and core properties of the system like determinism (how can we be sure we aren\u2019t dropping joint states over that network\u2026)
One core tenet going into Beta is to simplify the compute architecture, hopefully using a single nVidia Thor. The Thor is an absolute beast of a machine, and its possible that we can isolate some CPU core just for impedance control and meet our 1000Hz control rates. Especially if we assume that we have compatible adapters and all that (not a given, nor is it a given that EtherCAT drivers will build on Orin - but for the purposes of this discussion we\u2019ll assume they can).
But do we even want to close a 1000Hz control loop from Linux with 40-some odd actuators? Maybe not.
I\u2019m not convinced we can move impedance control down to the joints or actuators since we are controlling cartesian targets and we rely on complex feed-forward dynamics. Basically this would require transmitting kinematic and dynamic state (in addition to actuator control) to each actuator which blows up complexity in a different way. But there is a benefit here - if hypothetically we COULD run impedance control within the actuator itself, then Linux would only need to close a 100Hz loop. This is much more tolerable.
We will already need some hard real-time compute on the system to support functional safety. Currently the scope of this is limited to reacting to data coming over black channels, but this does require strict upper bounds to remain safe. What if implemented impedance control in our safety controller? For the moment we\u2019ll ignore the challenges of certification - we\u2019ll look at this only as a compute/software problem.
Modern MCUs have pretty generous amounts of compute - and if there is hardware floating-point it should be feasible to run impedance control for all our joints at 1000Hz in a hard-realtime setting. There are some challenges related to getting configuration to the controller, but there are some benefits - some near-term and some further afield:
The Orin can close a 100Hz loop easily with a safety controller over standard Ethernet
The safety controller can bound our latencies and isolate faults better than the Orin ever could under Linux
The safety functions can directly control state machines of our impedance control - modulating forces and such as-needed (going into limp mode during certain faults for example)
We can force our (soft realtime) whole body controller and VLA to operate within an envelope
For future biped platforms, this architecture makes much more sense, as we\u2019ll need the tighter guarantees more and it will be difficult (IMO) to avoid certification of something like impedance control - though we certainly don\u2019t need to do this now for Alpha
In fact, it may even make sense to run a portion of the whole body controller in the safety MCU, but this is a bit out of scope.
We should lean into some of our hard real-time constraints, and see if we can build a more robust, safer system that can get us through beta and future bipeds. The \u201csafety biped\u201d effort is a totally independent thing at this point, but I\u2019d be surprised if it doesn\u2019t tilt this direction anyway - and even if the underlying implementation is entirely different, setting up this architecture can mean more commonality across platforms and reduce software complexity over the long run.
We should keep Linux soft real-time (even with pre-empt RT - especially as we add sketchy drivers). We should have a home for hard real-time, and we should avoid closing >100Hz loops from Linux if at all possible. This allows each compute platform to play to its strengths, isolate faults and opens the door to more sophisticated functional safety down the road.
" }, { "id": "1051361456", "title": "The Case for Two Networks on Beta", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1051361456", "last_modified": "2025-08-20T18:46:36.263Z", "created_at": "2025-08-20T16:48:47.200Z", "body_html": "There is a lot of discussion around the network topology for Beta and the discussion has mostly been dominated by the decision around whether or not to use EtherCAT. I\u2019m not going to bother with this here. Instead I want to look at the network more generically. We can assume we are using Ethernet everywhere for this discussion.
In networking, we often talk about \u201cmice flows\u201d and \u201celephant flows\u201d because mixing these different flows in the same network is generally a very difficult problem. Its why playing a fast-twitch FPS while your significant other streams Netflix results in a bad time.
On the Internet, the problem is phrased a little differently: 90% of the FLOWS are mice flows (a flow being a unique TCP connection) and 90% of the BYTES belong to elephant flows (since they are so much bigger). This results in challenges where the elephant flows unfairly eat up all the queues, causing our little mice flows to get congestion controlled into oblivion. Whoops.
In all the discussions about network topology, no one (thankfully) has mentioned TCP. So why does this even matter? Even if we don\u2019t use TCP, we still have a similar problem.
Our elephant flows are going to belong to our sensors, and because of our plans to use nVidia Thor, we\u2019ll be using RDMA over Ethernet. RDMA is just a fancy way to write directly to the Thor\u2019s GPU memory from a remote device - meaning that camera buffers magically show up where they are needed without any CPU or kernel overhead.
Our mice flows are going to belong to our actuators, and because of our current plans to leverage a 1000Hz control loop (sticking with this assumption for now given we want high-performance impedance control), this isn\u2019t very forgiving about latencies or jitter. Because we are controlling actuator chains, the jitter and latency of these packets can be very important, as errors at the end effector will grow non-linearly with those timing issues resulting in difficulty with precision control.
RDMA wants lossless ethernet, as it needs to blast megabytes of data across thousands of packets to write even a single camera frame to the GPU. To reduce overhead and maximize throughput, you typically want to enable jumbo frames (where ethernet packets can grow to 9000 bytes). To support lossless ethernet, RDMA typically wants you to configure some sort of congestion control (DCQCN if you like alphabet soup).
While we don\u2019t want to lose control packets, with torque-controlled actuators dropping a packet is probably preferable to being late. Being late is very very bad - and we want to have bounded latency so we can even reason about our control loop performance. To minimize serialization-related latencies, we want to keep these control packets as small as possible. Do you see the conflict yet?
If we have a single Ethernet network, we will be forced to mix giant, lossless RDMA flows for sensors with tiny, bounded control flows for actuators. I don\u2019t think this problem is worth solving.
If datacenters can solve the \u201cmice and elephants\u201d problem, surely we can too? Probably\u2026
In the datacenter the approach is usually to throw VLANs at the problem. One VLAN for sensor data, another VLAN for control data. We trunk these VLANs through shared links, and break out access ports for our peripherals in the switches when needed (assuming that peripherals don\u2019t support VLAN) or trunk directly to the devices (if they do support tags).
There are a number of features that can be configured at the switch level to maybe get what we want. We want DCQCN for the sensor VLAN so we get explicit congestion notifications and lossless ethernet, while we want our control packets to be marked as strict-priority so that these packets are processed before we move to our sensor flows.
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1051361456/Untitled%20Diagram-1755710935439.drawio.png?api=v2&version=1\")
There is a small issue remaining of serialization latency - along a chain of ~6 actuators in the arm for example, we could possibly need to wait for 6 jumbo frames to transmit before we can get our control packets through. A 9000 byte jumbo frame would take about 70us to be serialized on a standard 1Gbps link for example, so our worst-cast jitter would be about half a millisecond. Maybe this is a problem? I\u2019m not really qualified to say one way or another, but its definitely not really deterministic as there will be a lot of variability on this jitter.
Some switches allow pre-emption, where you effectively interrupt a jumbo frame to send the smaller packet. In some cases this can discard/drop the packet, but most 802.1Qbu (TSN-related) allows the serialization to be resumed later - but only if every single path in the network understands this.
Some Specifics
As always, we\u2019ll focus on the arms since this is where these points are more clear, though they also apply at the base (drive motors and lidars - though lidars aren\u2019t RDMA so probably no problem there) and the head (neck actuators + cameras).
A single wrist camera could be producing 16-bit 1080p feeds at 30FPS - which is about 0.99Gbps. So already this would push us over 1Gbps. Are we really going to run 10Gbps? Do we have NBASE-T support in the switches to do 2.5Gbps or 5Gbps?
Its important to note that we want raw frames from these cameras to show up in the GPU of the Thor, because ultimately we need to process that raw data. Sending compressed JPEGs is problematic for a few reasons:
We\u2019d have to take the JPEG and decode it - which okay, this probably isn\u2019t the end of the world\u2026
We\u2019d have to allocate buffers for JPEG images since we\u2019re blasting these into the RDMA buffer\u2026. How big do we allocate?
The second point here is a big issue - what is the upper bound for the sizeof a compressed frame? It will depend on the JPEG quality as well as the scene itself. A black frame would be very small, while a frame of noise could be very large. RDMA wants to know how big the buffer should be on the receiving side, but our sender doesn\u2019t know until AFTER we decode - meaning we are suffering additional latency here. You can also have \u201cpathological\u201d frames which could be LARGER than the actual raw image - meaning we would be over-provisioning buffers or implementing some crazy quality backoff scheme that adds more latency.
Put simply, we want 1Gbps raw streams from the cameras, and adding JPEG compression w/ RDMA is a bit of a challenge that adds significant latency and complexity to an already complex scheme. Fixed-size, raw frames are easy, but push us over 1Gbps which could complicate the switching infra of our single fabric.
Just run the cables and have two independent networks. The system will be easier to understand, easier to troubleshoot and easier to debug. We don\u2019t actually need a fully generic topology because we aren\u2019t building a research platform.
We can break out two physical interfaces on the Thor - one for sensors and one for control. We can trunk these to two switches: one configured for RDMA for sensors, and one configured for more deterministic control. This trades some of the difficult-to-quantify development risk and general \u201cconfiguration risk\u201d of the VLAN solution for a more quantifiable \u201ccable risk\u201d of the extra network.
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1051361456/Untitled%20Diagram-1755711283362.drawio.png?api=v2\")
If we have some hypotheses about the true need for determinism or the flexibility of Ethernet, we can prove them out on the control network without worrying about mucking with the RDMA configurations for our sensor stack. We could even cross-wire them later to prove out that there no more risks and that a unified topology is viable - basically we just stop populating that cable or use it as a redundant trunk or something in later revisions.
I can already hear it: \u201cCables are evil! We will be increasing the failure rates of our robot and increasing connectors/wires!\u201d
Yep. But this is a long-term problem as we scale up (more robots, more scale, more failures). In the short term, I\u2019d argue its much more important to de-risk development and produce a system that\u2019s easier for us to understand.
Let\u2019s assume that configuring all of these switches to make RDMA and our actuators happy takes 3 months (could be fewer, could be more). How many failed cables is that worth at the tail end of the program? The answer is definitely not 0, since 3 months of engineering/integration time is very expensive now.
Imagine the sort of integration issues we\u2019ll run into during development: actuators stalling out, loops being delayed, sensor data being dropped/corrupted. By design, a single shared Ethernet fabric would be at the heart of the entire system - do we really expect that every engineer interfacing with it would fully understand the alphabet soup required to troubleshoot or debug problems? Or would this all be coming back to a very small group of people? Contrast this with a world where we have two networks and we can much more quickly root-cause and iterate.
Finally - we have a path to removing the cable risk later if we do figure this all out. If we go all-in on a single topology up front, there is a chance we blow our schedule due to the extra complexity and overhead.
" }, { "id": "1048969226", "title": "Simplifying Teleoperation", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1048969226", "last_modified": "2025-08-20T12:19:37.159Z", "created_at": "2025-08-19T15:19:45.052Z", "body_html": "Our existing tele-operation system is very complex, especially on the Pre-Alpha setup. We have a number of distinct challenges here:
RemoteSHM - Our realsense cameras are streamed from the robot to the laptop using multi-link wifi, custom transport and shared memory to minimize latency as much as possible (pretty important given how much latency there is with a realsense camera)
Laptop + Robot need to have CycloneDDS configured so they can communicate ROS bidirectionally.
Need to run/sequence ROS nodes across both the laptop and robot to make sure the setup is stable
For VR, also need to attach quest w/ \u201cadb\u201d so the oculus_reader library can read controller poses to be published
This results in a fairly tight coupling between the laptop and the robot, meaning that setting up a teleoperation environment is relatively time-consuming and and difficult to scale.
![\"image-20250819-152241.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1048969226/image-20250819-152241.png?version=1&modificationDate=1755616963564&cacheVersion=1&api=v2&width=751&height=568\")
I think the root of complexity here is that there is no clear \u201cserver\u201d - the robot and the data collection/teleop laptop form a network of bidirectional peers.
Another major concern I have is once we start having Alpha and PreAlpha robots - how do we guarantee that someone attaching a laptop configured for PreAlpha data collection to an Alpha robot doesn\u2019t create some sort of unsafe situation? Or perhaps the other way around?
Let\u2019s start with a simplifying assumption: lets turn the robot into a server and make it as smart as possible, and turn the laptop into a client and make it as dumb as possible. The idea is that any client can connect to any server without mucking with CycloneDDS or custom UDP transport IPs or ports.
For the sake of example, we\u2019ll go so far as to assume we can do this in the browser (some problems with this will be outlined below). So our desired user experience is something like:
Power up robot
Launch teleoperation \u201capp\u201d which exposes our server API
Connect laptop to lab network and go to http://${robot_ip}:${api_port} to load the interface
At this point the laptop should be connected to the robot, and ready to show camera images and accept poses/commands. Easy right?
With our shift to cartesian targets via the whole-body controller, we are leveraging Meta Quest VR headsets + controllers to tele-operate the robots. Due to concerns around comfort, we aren\u2019t using the headset display at the moment. Continuing with the example client-> server above, we can leverage WebXR which allows a quest device\u2019s browser to attach to a server to provide a VR experience.
Our existing VR teleop makes use of an oculus_reader library, which relies on the headset being enabled for debug mode w/ adb (Meta headsets are just fancy Android devices). This library is used to read tracked poses and button events from the attached headset which are then published via ROS to the robot.
Here, we can have a javascript \u201cclient\u201d which is hosted by our robot server which runs in the Quest browser. The client gets a WebXR session handle, and can provide a GUI (replacing the tele-op gui from above) which can have various state or show camera images from the robot to enable tele-operation.
For data collection specifically, we send \u201cpedal\u201d events to start and stop recording - here our pedal events would be handled as keyboard events by the quest\u2019s client, which would then call endpoints on the server API hosted by the robot.
![\"image-20250819-154253.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1048969226/image-20250819-154253.png?version=1&modificationDate=1755618174312&cacheVersion=1&api=v2&width=743&height=574\")
Note - WebXR is really nice, but I think this may actually be a challenge for us if we don\u2019t want to use the Quest displays. Mirroring the display/GUI to the laptop is possible, but configuring this appears to be involved - which sorta defeats the whole purpose of simplifying the setup.
An alternative approach here is a local daemon running along-side the quest to serve poses via HTTP so the robot-served client can tie it all together. I\u2019ll evaluate this too.
There are a number of existing \u201cROS bridges\u201d out there which will re-publish your data on websockets - but we\u2019ll assume we can do this manually right now so that we can have a \u201ccrisp\u201d API that makes a bit more sense semantically. We\u2019ll have a uvicorn FastAPI HTTP server running inside a ROS node on the robot which can accept HTTP requests and publish/subscribe to ROS messages, and the client (running on the Quest in this case) only needs to worry about speaking HTTP:
GET /mjpeg/wrist_left/compressed - return camera MJPEG as multipart/x-mixed-replace streams.
POST /recordings/start - initialize a recording episode
GET /recordings/$id - retrieve websocket for recording feedback
PUT /recordings/$id/save - save the recording (and end it)
PUT /recordings/$id/cancel - cancel the recording (and discard it)
GET /teleop/tracking - intialize a websocket for VR control. VR controller will be enabled as long as this websocket is alive. If the websocket is dead, the robot isn\u2019t controllable.
POST /teleop/reset - reset the robot
The client running in the Quest would have a simple GUI (or use pedals/controller buttons) to call these endpoints. When its time to tele-operate the robot, the tracking websocket can be created and the current WebXR session can read tracked poses from the quest controllers and send them up the websocket the robot. The robot can in turn send information like haptics (WBC solve status) and other feedback down the websocket for the client to process.
Each camera can be retrieved as a low-latency MJPEG stream which can be displayed in an <img> tag without any re-encoding overhead.
You can see just the camera path (https://github.com/HumanoidTeam/hmnd/pull/703 ) as an example.
On the robot side, our ROS bridge will maintain the necessary subscriptions and state to feed the client and accept commands. Incoming VR poses from the tracking websocket can be converted into WBC targets and published using the VR \u201cpuppet\u201d, and mapped buttons can trigger other endpoints which may call services or start actions as needed.
While this approach does introduce a second transport protocol (HTTP), this cost is paid once but the benefits of reduced setup friction are realized over and over again. Especially as we scale.
In the example above, I show how we could use WebXR and have the Quest browser act as our client. This works great as long as our input devices are supported by the Quest. Or that we are using a Quest at all. But what if we decide to do something else entirely?
Fortunately, HTTP clients are far more pervasive and well-supported than ROS clients, so I don\u2019t expect there to be any issues here. In fact, we could build a tele-op client in whatever language we wanted using whatever tools we wanted because HTTP is supported by practically everything.
For example, if we decided to move to Vibe wrist-tracking - we could use a PCVR-type setup with a Windows laptop (since we don\u2019t need to worry about ROS networking or anything like that).
" }, { "id": "1048543355", "title": "Configuring Robots", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1048543355", "last_modified": "2025-08-20T09:08:54.687Z", "created_at": "2025-08-19T17:29:06.655Z", "body_html": "Currently our robot stack is configured via soup of YAML files, ROS params and URDF files scattered across our git repository. As we scale up more platforms and more units, we will need to be more deliberate about how we build and track configuration. The key goal is that we should have a single unified view of configuration for a given robot, so that we can completely capture the configured state of the robot for data collection activities (in addition to the version of software running on the robot).
rosparam, the venerable approach to injecting configuration into your nodes, is not exactly what we are going to want. ROS has no way to validate configuration and with ROS2 there is no longer a global source of truth for params (the old roscore did this), and instead params are managed by each node individually. One benefit to this approach is that it avoids having nodes reaching across to \u201cunrelated\u201d parameters and coupling the system together in fragile ways. But the major downside is that there is no good way to capture the \u201creal\u201d state of the system after loading all parameters and applying overrides so that we can track configuration along with our software version - very important for data collection and evaluation activities.
To start with, we\u2019ll consider a simple world where we have a few platforms (alpha_wheelbase and alpha_biped) and a few units of each one.
The Canonical Configuration represents the \u201cstandard\u201d configuration of any given platform. If we could manufacture with 0 tolerance and everything was exactly the same - this would be our robots' configurations across the fleet. Typically this will be coming from CAD and various defaults set in version control for a given robot platform. For example, the Alpha wheelbase may have some default stiffness of its impedance controllers or scaling factors for its actuators.
Of course, we don\u2019t live in some perfect world - we should expect that each robot will need to be tweaked individually so that the robots behave the same even though their hardware is slightly different. Camera intrinsics/extrinsics, IMU drift, actuator offsets/scaling, frictions - all of these could be different for each robot.
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1048543355/Untitled%20Diagram-1755678554978.drawio.png?api=v2\")
For each robot when we load the software stack, we should load the platform configuration for the robot and then apply the calibration as an override to get our single robot-specific configuration. This configuration can then be saved when we perform data collection activities to ensure we have full traceability of potential differences per robot.
We won\u2019t just have two layers of configuration though. We may have various controller or perception parameters tuned differently to support some ongoing A/B experimentation or perhaps just because some environment or use-case benefits from slight tweaks to configuration.
So really our configuration is some canonical base, then we layer per-robot calibration (potentially derived from service records or calibration-related activities - but this is out of scope here), then we can layer on additional configuration overrides as needed until we get our final configuration.
Right now its difficult to think of configuration as anything other than the YAML or XML files on disk. But we should only consider these as rendered views of our system\u2019s configuration. We can import from YAML or XML (if necessary), and we can export to these files similarly (or something else).
![\"image-20250819-174150.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1048543355/image-20250819-174150.png?version=1&modificationDate=1755625311922&cacheVersion=1&api=v2&width=265&height=190\")
Ideally we use something like Pydantic define our configuration schema in python - so we know not just default values but types, expected ranges, optionality so that we can more effectively validate our configuration. We can catch typos in keys and ensure that our final configuration is complete.
When we import configuration from various sources, we can track the provenance of those keys in our internal model so that we know where a given parameter came from. If some parameter is causing issues with data collection, it should be clear that this parameter is being set from version control or whether it is some override being applied somewhere else.
At some point, we may be writing all of this configuration to some large database to be managed for the fleet. You could imagine a world where field service operations are replacing components like actuators and performing partial re-calibration - we\u2019d want these new configuration parameters to be associated with that activity and tracked over time so we can get a full history of the robot configuration.
Like ROS2, we want to encourage modules to own their local configuration, but at some point these various keys need to be brought under a global namespace so we can manage the whole thing. Storing hierarchical configuration in a database would be way more complexity than necessary, and so we should consider our paths \u201cflattened\u201d. Different modules would then be responsible for different \u201cprefixes\u201d in the configuration, which means we can still avoid many of the problems that ROS2 tried to avoid, while still having a flat global namespace for tracking our configuration over time.
For example, we may import the URDF for alpha wheelbase under hmnd.alpha_wheelbase.description, with description having top-level keys for joints, transmissions, etc. At some later point, we may apply some alpha-specific simulation overrides to hmnd.alpha_wheelbase.description (some sort of partial URDF or YAML fragment), and then we can ask the configuration system to export hmnd.alpha_wheelbase.description (including overrides) as a new URDF.
So this all sounds great in theory - how do we make this happen? I expect this section to get really fuzzy as we capture various concerns. Off the top of my head:
Some nodes load YAML directly in cpp (ethercat, hmnd_vision)
Some nodes load params from ROS2
Some nodes get params from launch files
xacros are used to customize and reduce duplication across a number of XML configs for controllers
We probably need to prioritize calibration over \u201cglobal snapshots\u201d for data collection, so we\u2019ll likely want to start with how we load YAML for actuators and ros2 control that vary per robot.
" }, { "id": "1037304324", "title": "Robot Puppetry", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1037304324", "last_modified": "2025-08-13T13:21:51.804Z", "created_at": "2025-08-13T12:12:27.928Z", "body_html": "Ideally, we have a single interface which our various tele-operation schemes can use and which our learned policies can use to control our robot controller. Our new whole body controller (WBC) accepts high-level cartesian targets which it will use to generate joint trajectories which satisfy a number of constraints. But there are some slight differences to these interfaces which can make things challenging.
![\"image-20250813-121306.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1037304324/image-20250813-121306.png?version=1&modificationDate=1755087187957&cacheVersion=1&api=v2&width=523&height=412\")
The policy for example runs slower than our WBC and when its running off-board has some unpredictable latencies. For this reason (and that it generally improves predictions), policies learn to predict \u201cchunks\u201d which are effectively future targets for the WBC. Tele-operation on the other hand relies on human operators controlling the robot in realtime, meaning we want responsiveness. Human operators cannot predict or communicate future intent the same way a policy can, yet want the behavior of the trained policy to match the behavior of the demonstrated behavior - even though we often have some \u201chard-coded\u201d mappings in the middle here that can be tricky to untangle.
Further complicating matters is that these aren\u2019t the only possibilities. We may have leader arms which directly control joint poses. Or we may have a number of discrete preset targets we want to use (for example driving simulation), perhaps with some randomization added.
Imagine for a moment that the robot is a puppet. We may connect strings to it and pull on the various limbs (each string being like a cartesian target), or we may think of it like a sculptor\u2019s manikin which can be posed directly. Or maybe its a socket puppet where you jam your hand up its backside to control it. The point is that there are a number of different modalities or mappings from intent to the desired action.
In reality, this is another layer of control - but naming this a \u201ccontroller\u201d I think confuses terminology too much since this isn\u2019t really a controller in the classical sense since we rely on something \u201cabove\u201d to close the loop. Instead, we should create a new interface in the system and for fun we will call it Puppet.
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1037304324/Untitled%20Diagram-1755088321691.drawio.png?api=v2&version=2\")
A Puppet is simply an interface for a module that converts puppet-specific intents into Cartesian targets suitable for the WBC. The idea is that the WBC doesn\u2019t care about the specifics - any Puppet can control the WBC and use whatever input is appropriate.
The primary benefit here is to create a dedicated place for this logic that can be tested, rather than burying it inside the respective higher-level behavior layer code. Different puppets may perform different amounts of smoothing, interpolation or blending before issuing the targets to the WBC, and this logic is ideally easily identifiable and testable. Puppets may adjust the mode of the WBC in addition to just sending targets, so the logic here can be quite stateful/complex - the more we can separate this \u201cconversion\u201d logic from the WBC or the actual Policy/Sim/Teleop, the easier it is to test and compose things.
VR Puppet
The obvious example here is a VR puppet, which takes VR-specific intent from the Oculus controller (poses and transforms of the controllers, button states) and maps these into Cartesian targets for the WBC. This may involve setting specific modes in the WBC, specifying constraints or a number of other possible actions - but the goal is that an operator can use the VR controllers to effectively control the robot. We already have this today, but the logic is buried inside of the QuestTeleopServer.
Leader Puppet
For Pre-Alpha we may consider the use of a leader-arm puppet which instead of taking VR-specific intent from a controller, takes leader arm poses from a connected device and maps these to robot joint states. In this example we\u2019ll assume that the WBC allows for some form of joint-level control (perhaps maintaining constraints and avoiding self-collisions), but we are more-or-less attempting to control each joint individually.
Simulation or Test Puppet
For simulation or test scenarios, we may want to control the WBC much more directly as we have ground-truth information from the test or simulation environment. For example, we could have a puppet which knows how to interpolate the end effector targets from some random initialization pose to the sides of a tote. Or perhaps we can choose from a number of discrete pre-recorded cartesian targets to feed the WBC. The point here is that these intents can be arbitrarily sparse or simple, while the output of the puppet interface should be roughly equivalent across implementations for the WBC.
Policy Puppet
An interesting twist here is to consider the chunks of the VLA as intent for a Policy Puppet, which is responsible for feeding the chunks to the WBC at the correct rate. It may also be that the Policy is an alternative to the puppet (something that feeds targets to the WBC), but I\u2019m including it here since its not the craziest idea.
The gist here is we can create a dedicated layer in the stack to serve as the interface to the WBC where we can place logic (and test it) which doesn\u2019t necessarily belong in the higher-level modules or within the WBC itself. We certainly don\u2019t want the WBC being concerned about Oculusisms or Policyisms - but at the same time, we may not want to couple the logic for how we arrive at these Cartesian targets inside of these very complex policies or teleoperation schemes. So given something like a VR setup we may have the same Oculus controllers and the same overall software stack, but we may swap out the puppets for different use-cases. Sure - we could just pile all of this into the QuestTeleop server - but we can probably benefit by pulling this logic out and testing it independently. We could have puppets for more novice operators performing some simple/directed data collection activities, while having puppets for more expert users that have much more flexibility and responsiveness.
" }, { "id": "1026130021", "title": "Building Software", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1026130021", "last_modified": "2025-08-08T16:13:23.575Z", "created_at": "2025-08-05T15:49:44.808Z", "body_html": "A build is a process by which our many source artifacts (the shit we\u2019ve checked into GitHub) are transformed into a number of intermediate and final build artifacts. \u00a0 Sometimes source artifacts aren\u2019t transformed at all (Python scripts, pre-compiled binaries) while others are compiled, templated, linked or archived into different outputs.\u00a0 The build is one of the most important aspects of the software development lifecycle, and will make or break your system as you scale.\u00a0 So what makes a good build?
Hermetic - Builds are isolated and unaffected by the local environment, ensuring consistent results across machines and CI.
Incremental - Only rebuild or test what has actually changed, making extensive use of caching to speed things up.
Traceable - All artifacts, configurations, and dependencies can be traced thru a single build graph which explains what depends on what, allowing builds to be run in parallel or even distributed.
Extensible - It should be possible for any engineers writing software to extend the build as-needed (new dependencies, new targets, new artifacts).
Reproducible - multiple builds of the same commit are identical. (this is really hard)
A good build allows developers to focus on building robots rather than fighting environments or dependencies.
A modern robot is extremely complex, so we\u2019ll introduce some additional terminology to help explain things:
Platform - a specific robot configuration.\u00a0 Pre-Alpha, Alpha-Wheelbase, Alpha-Biped, Alpha-SpecialSauce, etc
Target - a specific computer within the platform.\u00a0 Including firmware images, SOMs, FPGAs, safety or even 3rd party sensors.
Bundle - artifacts for all the targets of a platform in a single archive, versioned together
During the course of development, there will be dozens of distinct platforms.\u00a0 For a given program phase (Alpha or Beta) we should expect to support builds running on a number of different variations.\u00a0 The build should be flexible enough to represent all of this in a single build graph so that we can leverage as much commonality as possible across targets.
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1026130021/Untitled%20Diagram-1754556058604.drawio.png?api=v2&version=2\")
When a build is produced, a number of artifacts may be produced across a number of target devices for the platform.\u00a0 These should be bundled together during the build so that there is a single versioned artifact.\u00a0 This artifact can then be deployed to the robot, which should take care of ensuring that all of the target devices are updated correctly.\u00a0 With many target devices, this process is anything but simple, but for now we\u2019ll consider the specifics of OTA out of scope, and only consider how we produce these release bundles.
To get hermetic builds, it's important to consider the environment in which software is built.\u00a0 We can borrow some terminology from cross-compilation:
Host - the environment actually running the top-level build.\u00a0 Usually this will be your laptop or some CI runner in the cloud. We should expect quite a bit of variation here (since we can\u2019t really control the host environment, as much as we\u2019d like to).
Build - the environment which actually runs the build.\u00a0 For hermetic builds, we want this to be distinct from the host.\u00a0 Generally no two host environments are the same, and so having a separate build environment provides a way to get consistency across different hosts. This would be toolchains and libraries.
Target - the runtime environment on our target hardware or simulation.\u00a0 We separate this from the build environment because we typically only need a small subset of the artifacts to actually be deployed. We don\u2019t need to ship gcc or clang to the robot (usually\u2026)
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1026130021/Untitled%20Diagram-1754555655678.drawio.png?api=v2&version=3\")
We should be able to simply bootstrap the process on any host, which then produces a hermetic build environment which can then be used to produce a number of artifacts.\u00a0 These artifacts would then be packed into some target-specific artifacts (disk images, firmware images) which can all be bundled together depending on the platform and deployed to some specific robot.
Our current build is not really much of a build. There is no real separation between source and build artifacts and no real separation between build and runtime artifacts.
Pixi is a tool which does help to isolate the host to a degree by providing a way to manage Conda environments using some more modern tooling. These environments are pretty fragile though, and can't easily be shipped around.
Within this environment, ROS's colcon tool is used to kick off a ROS-flavored CMake build, which in turn actually generates a ninja build which actually produces our ROS build artifacts.
The Pixi environment is stored under a .pixi folder, the ROS artifacts are stored under an install/ folder and some extra bits and pieces are placed under the build/ folder.
While it's a decent tool for developers managing their local workspaces, Pixi is not the best choice for a good build since it's not truly hermetic (relying on environment variables and rpath hacks), it's not very traceable or incremental (the caching is wildly unreliable), but it is pretty extensible for developers and so it's not the worst possible choice in the short-term as we rush to bring up Alpha and support our PoCs.
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1026130021/Untitled%20Diagram-1754552661651.drawio.png?api=v2&version=2\")
To mitigate some of the downsides of Pixi, we can actually build the environment in a hermetic rootfs using Bazel. This allows the image to be archived as a release and shipped to robots instead of rebuilding the Pixi environment for every deployment.
A number of custom Bazel rules have been produced to allow rootfs images to be produced which can contain arbitrary packages or run arbitrary build steps. This allows us to make use of Bazel as a foundation for a more proper build without giving up pre-built binaries and existing packages (especially those from Nvidia). A rootfs image is like a Docker container, only slightly more general. Eventually the hope is that we can boot these rootfs images produced by the build directly on our target hardware.
The Pixi environment has some major downsides as we scale. First, it's not very granular or composable. Unless we subdivide our entire codebase into individual Pixi packages, it's difficult to reuse pieces between different environments like training pipelines and robots. This lack of granularity also leads to a second major issue - the build cannot be cached effectively. The Pixi cache is far too coarse and cannot easily be cached in CI or across many machines without introducing the risk of a poisoned cache. Instead of caching an entire environment, we should be caching individual build artifacts.
We should move from Conda dependencies to Apt and PyPI, and rely on a proper hermetic environment built by Bazel. This environment can be much more granular and compostable with dependencies being assigned not to the environment as a whole, but individual modules within the environment. Conda is a fantastic tool for quickly spinning up R&D environments, but the runtime environment on our nVidia-provided compute hardware (like the Orin or future Thor) can make this very difficult as nVidia frequently provides binaries which expect an Ubuntu ABI or specific Python versions. Given the strategic importance and complexity of the nVidia stack, it makes sense to \u201clean in\u201d and accept these constraints. We will use the Jetpack-supported versions of Ubuntu, Python, Numpy, Pytorch, etc as much as possible since these versions are largely pre-integrated and widely supported. Any deviations here (such as those from Conda-built packages) are a significantly liabiltiy and risk subtle ABI-breakage and wasting enormous amounts of developer time troubleshooting and debugging.
This will still require some additional tooling to support quick iteration and development locally. One simple solution to this would be to embrace remote SSH workflows via vscode, with Bazel producing and updating a developer image that can be 'booted' into vscode. Helper scripts can be used to add dependencies in a way that get tracked in the build (similar to pixi add).
When this phase is complete, we should no longer have a Pixi virtual environment, but we still have entangled build and runtime environments.
Most of our apt and PyPI-related dependencies exist in public HTTP repos which are not controlled by us. This represents a risk for build reproducibility and availability. What happens if someone sneaks a breaking change into the repo? What if a repo becomes unavailable for some reason, breaking all of our builds? We should be able to easily setup a read-thru cache for these repositories using S3. The basic idea is we point our mirror at the upstream repository (ports.ubuntu.com for example) and an S3 bucket. When requests come to the mirror, we first look in the bucket for the asset. If its not there, we proxy the request to the upstream repository, and write it to S3 while we serve it to the user. We may need to do some locking w/ dynamo to prevent conflicts while writing to S3, but this is a relatively lightweight system that will end up with all of our dependencies \u201ccached\u201d in S3. If the upstream repo goes offline, then we still have the assets in S3. If something upstream changes, we can still go back to previous versions since we have everything versioned in S3. This should give us the benefits of leveraging public repositories without many of the downsides (until some day we standup dedicated repositories ourselves).
The next steps would be to consolidate our ROS packages, and to get to a single pass build. Colcon (and ament, the underlying ROS2 build framework) rely on package.xml metadata to order dependencies and run multiple passes over the build. Meaning that package A must be built before package B\u2019s \u201cfind_package\u201d CMake-isms can successfully locate the package. This complicates things significantly and means that integrating new dependencies often requires wrapping them up as ROS packages and further couples our stack to ROS. Our VLA focus will lead to fewer ROS nodes anyway, so it makes sense to simplify here. A flat, single-pass build can then be expressed purely as CMake without any colcon or ROS2 specifics. Rather than relying on topologically sorting packages and using \u201cfind_package\u201d and friends to resolve dependencies, CMake can simply \u201cadd_subdirectory\u201d to build up a single unified build containing all our dependencies. Some complexity may pop up for certain build dependencies that must exist before we actually run CMake itself (think of something like compiling protobuf from source) but we\u2019ll manage that if it comes up.
Finally at this point, the CMake targets can be ported to Bazel or wrapped as foreign builds. This brings the entire granular build graph into Bazel and allows for artifacts to be cached much more effectively.
At this point we can finally start to peel apart our build and runtime environments. Bazel and it's toolchains can produce a build environment with any build dependencies (multi-stage potentially, but ideally not), and then this final build image can be used to produce our runtime artifacts which we can install into our target environments. Because we want to use Ubuntu-compatible packages, we can treat the build image as a sysroot for compilation - and this could even allow us to build cross-arch, this this isn\u2019t an explicit goal at this point.
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1026130021/Untitled%20Diagram-1754558565374.drawio.png?api=v2&version=1\")
This means we can have 100s of GB of development and build assets while only installing a couple GB on the robot itself. This optimization will be critical as we start scaling and performing OTA updates.
The more granular build graph makes it much easier to leverage build caching across many different branches, versions, hosts, architectures, etc to accelerate builds. The dependencies can be individually traced through the build graph making it easier to share artifacts (libraries, binaries or other tools) across multiple projects. For example, the ML transformations running on the robot can specify exactly their local dependencies within Bazel and be shared across a number of target configurations including the data pipeline running in the cloud.
The broad theme here is to go from a coarse-grained build with an R&D environment toward a more fine-grained build graph which is more cacheable, more composable and more traceable. This is going to be a long-term effort, but incrementally we can get there. This end-state is going to be necessary if we want to ship and support 100s or thousands of robots.
" }, { "id": "1028981407", "title": "Erecting the Monolith", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1028981407", "last_modified": "2025-08-08T15:59:22.051Z", "created_at": "2025-08-07T12:33:50.465Z", "body_html": "The current robot architecture inherits quite a bit from the pre-alpha scramble for early PoCs. Many ROS nodes, multiple controllers, behavior trees, configuration spread over a a number of YAML and XML files (with templating/macros of course). For more traditional robotics stacks, this is generally how things get started:
You have a number of launch files, which coordinate how the various nodes are started and configured, and then these nodes publish and subscribe to various topics to wire themselves into a graph. Data flows through this Graph, and can be inspected at various points using command line tooling like ros2 topic echo and friends. In https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1010335765/Robot+Middleware?atlOrigin=eyJpIjoiYjY0YjBhMTc1MzM2NDM4Y2I4N2FjYThkNmZkOGM1YjYiLCJwIjoiYyJ9 I attempted to make the case that this probably isn\u2019t what we want to do, and so in this document we\u2019ll focus more on the \u201chow\u201d rather than the \u201cwhy\u201d.
![\"image-20250807-123523.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1028981407/image-20250807-123523.png?version=1&modificationDate=1754570126067&cacheVersion=1&api=v2&width=1027&height=465\")
In addition to the points made in the doc referenced above, we should also just simply acknowledge where robotics is going and try to align our long-term roadmap with this trend. Instead of having a number of hand-crafted modules like \u201cgrasping\u201d, \u201cmanipulation\u201d, \u201cplanning\u201d, etc these systems are overwhelmingly moving toward learned approaches which are defined more by the data than the code. In this world, having a graph of independent functions doesn\u2019t make much sense, since some cross-attention layer somewhere is going to be integrating all of this in a massive neural network. While we will still have other auxiliary processes in addition to the \u201ccore\u201d models used to execute policies on the robot, we should be mindful that the capabilities of these models will grow over time, and the stuff \u201coutside\u201d the model (related to robot behaviors at least) will shrink.
As a final point before we get started, the goal isn\u2019t so much to remove ROS entirely, but to remove ROS from the \u201ccore\u201d of the system. We should expect that we\u2019ll be publishing/recording ROS-compatible data even after we complete the transition as described here.
The initial controller stack for alpha has been born from a much more chaotic controller stack from the pre-alpha, which relies on the Rainbow SDK for low-level control and operates with a patchwork of nodes, controllers etc. We\u2019ll zoom in here with what we have for Alpha today and look at where we should take things in the near future.
Alpha Wheelbase Controller Stack
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1028981407/Untitled%20Diagram-1754571182500.drawio.png?api=v2&version=1\")
This is a snapshot of the Alpha wheelbase controller stack (part of it) - there are about 15 nodes (across only a couple processes) and about 32 topics here. For reference, this is just from booting up the mock controllers in my workspace. There is a LOT of additional complexity here that is not necessary for our future path - basically a whole-body controller (WBC) driven by a high-level large behavior model like a VLA. We don\u2019t need to swap controllers, we don\u2019t need to have all these separate interfaces within ROS. It\u2019s extremely difficult to see the dataflow here, and its very difficult to test individual pieces or to identify the true \u201cboundary\u201d between our hardware interfaces and our core logic (since a lot of this stack depends on what platform/description we launch with). As a simple example, how would we trace which joint states/commands were associated with which controller? How would we deal with continuity issues when swapping controllers \u201cunder load\u201d?
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1028981407/Untitled%20Diagram-1754573671690.drawio.png?api=v2&version=3\")
Here we can still wrap our control system in a node (and publish data like TF, actuator states, joint states, etc) - but we should try to consolidate into a few clean layers that can be more easily parameterized using the robot model from the URDF. Many of the individual controllers from the ros2_control_node are holdovers from PreAlpha, and so will be folded into the WBC. The joint control mechanism is ideally an impedance controller, but we may need to swap in an admittance controller for certain scenarios (low-fidelity simulators like Isaac for example - but ideally we can avoid this with better actuator models).
We also consider two additional interface layers to consider: the HAL and a potential safety boundary. In the HAL, our actuator control is really just a bag of actuators which may be configured with limits and encoder scaling values and such, but there is no real \u201csemantic\u201d meaning. Its just a bag of hardware. Once we get to joint and WBC, the morphology of the robot becomes more important and so these actuators are assigned to specific joints and those joints are arranged in a specific way with inertias, weights, stiffnesses, etc. In an ideal world, the ONLY thing that changes between simulation and real hardware is below the hardware boundary: same URDF, same controllers - but we have an interface which abstracts over the simulated actuators and the real, EtherCAT actuators.
there is also a potential safety boundary in here if we consider the future biped. We may need to run actuator control, impedance control and some portion of WBC in a safety-certifiable way (at least the tasks responsible for keeping the robot from losing balance for example). Having the safety interface \u201cfurther up\u201d the control stack is more expensive as it increases the scope of the system which must be certified, but because control loops are looser towards the the top it can be easier to manage communication between our soft real-time Linux environment and our hard real-time safety environment if we end up moving this direction. Put another way, its a bit easier to close a 50Hz loop between Linux and a safety MCU than to close a 1000Hz loop.
We\u2019ll continue with this \u201csimple\u201d example of the system, since it includes full feedback loops from joint states and commands across a number of time-scales. Without ROS, how do we communicate between our actuator control, joint control and WBC modules? How do we manage the fact that these things will be operating at different frequencies?
Because of the 1000Hz update loop, the coupling between the joint control and actuator control is very tight. In ros2_control this doesn\u2019t actually rely on any topic (thankfully), instead the joint controller \u201creads\u201d the actuator states via the ethercat plugin, and then writes the commands via the same plugin. This depends on both the EtherCAT master and the ros2_controller running at the same rate (1000Hz), though they aren\u2019t actively synchronized. The joint controller simply polls the state of the actuators at 1000Hz, and these states are updated at the same rate via the EtherCAT master and latched in some internal buffers. Similarly, the joint controller will issue commands at 1000Hz, which are polled by the EtherCAT master in its 1000Hz loop and sent to the slaves.
With the ROS datapath, we have callbacks executing at the rate of other nodes' publishers, along with some \u201clocal\u201d update loop and we need to synchronize all of this. This makes it difficult to control timing, as other (presumably misbehaving) nodes can flood a topic and impact the CPU usage of other nodes. This can make it impossible to even do things like report diagnostics or fail in some safe/predictable manner. With multi-threaded executors, these callbacks can all be firing concurrently in many different threads, resulting in significant amounts of lock contention (wasting CPU) and significant risk of sneaky race conditions and subtle bugs. On top of all that, its impossible to control the order that these things happen resulting in nondeterministic behavior - you will never be able to reconstruct the internal state of that node from recorded data or logs since its fundamentally concurrent. You are at the mercy of the Linux scheduler and DDS QoS.
We don\u2019t need to fire some subscriber callback at 1000Hz in the WBC to respond to joint states. Instead, the joint controller should write its updates a bounded circular buffer as a queue. This bounds the latency of the system and memory, while providing a means to decouple the timing of our fast impedance control loop and the slower WBC. So the joint controller scrapes the HAL for its actuator states at 1000Hz and the WBC scrapes the joint controller\u2019s queue at ~50Hz for its joint states (each time it scrapes it will get a buffer of ~20 states if we operate at 1000Hz). You could get a similar effect by actually having the subscriber writing to a circular buffer in the WBC, but it would still be a thread in the WBC process doing this and so you still run the risk of starving out other tasks (timers, callbacks) on the executor and you still have all the overhead of messaging w/ DDS.
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1028981407/Untitled%20Diagram-1754579766891.drawio.png?api=v2&version=2\")
With the alternative datapath, the core logic looks very similar - but we make the queues within ROS explicit at the producer and the consumer scrapes these queues on their own time. In the example above, the WBC runs at 50Hz and simply drains the buffer of the joint controller every time it loops resulting in approximately 20 items in the buffer each iteration. Because the queue is explicit we get a number of other benefits:
deterministic execution - the WBC controls the rate at which it drains and can do this predictably compared to a number of concurrent callbacks
explicit queue semantics - if the WBC is slow, the joint controller overflows the queue which allows us to trigger faults easily. if the WBC is fast (or the joint controller is slow), the WBC will underflow the queue which can trigger other behaviors.
only one queue per producer - much easier to reason about reliability and latency.
each module runs at a consistent rate - the WBC code runs at 50Hz and the Joint Controller runs at 1000Hz - no mixing/matching - also easier to reason about latencies and performance.
The primary downside of this approach is that fan-out is slightly more difficult (meaning having multiple \u201cdownstream\u201d consumers of the joint state queue). In practice this can be mitigated easily by consolidating to a linear datapath which avoids fanning out in the first place. In the case of logging data, you could either log the data which is produced as its written to the joint state queue, or you could dump the buffer as its consumed after its used to update the internals of the WBC. But even better is to log only at the HAL boundary which is the interface between the software stack and the outside world and rely on a deterministic stack to ensure we can reconstruct any intermediate state purely from this HAL data. So in this example we\u2019d log the actuator states, and pass them to our joint controller during replay to reconstruct the joint states. This is much easier said than done, and so there is no problem logging intermediate streams as an alternative (I/O budgets permitting). In that case, logging the consumer side of the queue within the WBC is preferred for two reasons:
its done much slower compared to the update - so we can log at 50Hz instead of 1000Hz
it happens AFTER any drops (overflows) in the queue, meaning we are recording exactly what the WBC is seeing rather than what it SHOULD be seeing - which is super useful for trouble-shooting edge-cases in the future.
By far the biggest win here comes with testing. With a ROS node, testing logic can be difficult due to the interaction and timing of the various subscriber threads and callbacks. You can \u201cspin\u201d the node manually, but this really fights against what ROS is trying to abstract by even introducing the executors and callbacks in the first place.
Consider testing the WBC. We have some JointStates, and we want to make sure that the WBC\u2019s internal state updates correctly. With ROS, we\u2019d have to spin up some sort of mock environment or a real ROS runtime and publish our JointStates so that we can execute the subscriber callback logic. All the while, our WBC is firing at some other rate (nominally 50Hz) in this example. Does this mean we are limited to testing our node in real-time only? Or perhaps we drive the WBC loops from only the subscriber callback of the JointStates (calling some internal \u201cupdate\u201d every 20ms). This is a lot of complexity and overhead for the task at hand.
Instead we can simply assume our WBC has some update function that takes a vector of time-stamped joint states. We should be able to take our test data, and feed it in 20 items at a time, and assert on the state produced after each update. No need to reason about callbacks, synchronization, topics, test publishers - just call the logic in a loop. These tests are much faster (can be run faster than real-time)
Explicit Goal: Synchronous, functional components. Give inputs, assert on outputs.
We should begin giving more consideration about how we are organizing our software for the benefit of AI coding agents like Claude Code or OpenAI\u2019s Codex. In a ROS system with many nodes, callbacks, topics it will be difficult for the agent to have enough context to meaningfully contribute, as so much of the behavior of the system can only be inferred by looking at the whole thing. For example, looking at just the WBC ROS node (in this contrived example above), it would be impossible to know that the joint state subscriber callback would be firing at 1000Hz without also having full context of the joint controller and its configured rates.
The good news is that organizing code for agents ALSO makes it easier for humans to understand, and also makes it easier to support testing and such due to solid abstractions and module boundaries.
Explicit Goal: Flat structure. Modular. Local-Context. Easy to reason about the part separately from the whole.
Another challenge we need to overcome is tracking and managing configuration. ROS typically organizes configuration in a way that makes sense for the ROS graph, but generally makes things very difficult for the organization as a whole. Consider the problem of training pipelines:
We will have many robots operating across different platforms and many different software versions. How do we trace these changes back to the data pipeline? We can version the code of course with a single commit (already in progresshttps://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1005027354/Integration+and+the+Monorepo?atlOrigin=eyJpIjoiNzk4YTkxNzY3OWY1NDllYjkxMmU3NDQ4OGIwN2MyYjEiLCJwIjoiYyJ9 ) and this is necessary, but not sufficient. What about \u201cextra\u201d parameters coming from configuration management? What about overrides provided on the command line? We should expect there to be configuration changes outside of version control for things like A/B testing and experimentation.
In this case, having a soup of YAML files, launch files and XML files spread across many nodes is very difficult to check. ROS also introduces \u201cdynamic reconfiguration\u201d which allows parameters to be changed at runtime - we should generally just forbid this. We should strive to have a relatively unified configuration architecture across all platforms which, together with the versioned code, perfectly captures the behavior of the robot.
With a more monolithic approach (fewer ROS nodes), this means having some per-platform URDF (static, version-controlled), some HAL-related configuration (static, per hardware environment like mocks, simulation, real hardware), some configuration params (all the various parameters in the system which we want to control outside of code) and overrides. At runtime, we should be able to render this into a single consistent view which can be archived during data collection.
Explicit Goal: Unified, data-driven configuration parameters. Easy to trace configuration differences even with same software version.
![\"image-20250808-102616.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1028981407/image-20250808-102616.png?version=1&modificationDate=1754648778085&cacheVersion=1&api=v2&width=318&height=159\")
Phase 1: Consolidation and Organization
Both Alpha and the PreAlpha system have a number of launch files in the monorepo which can boot up nodes for controllers, perception, or policy execution. The more nodes we have, the more junk we have between them. If we reduce the number of nodes, we reduce the complexity of trying to coordinate them. Think of it as reducing the ratio of surface area to volume.
Things that are PreAlpha-specific, should be marked as such and eliminated from the Alpha stack. Superfluous nodes and configurations should be removed. Code and config related to low-level control should be \u201cclose\u201d together in the repository but distinct from code related to actuator control drivers or WBC.
ros/\n control/\n actuator_control/\n ethercat\n mocks\n issac\n joint_control/\n impedance\n admittance\n whole_body_control/\n wbc_solver\n "stray" controllers (left arm, right arm, head, torso, etc)\n platforms/\n alpha/\n alpha_configuration\n
This is a slight simplification, but the main idea is that even within ROS, we should be able to organize things to reflect the sort of organization we want to see longer-term, and we should begin better separating code from configuration. More specifically, the joint controllers and WBC ought to be separate from the configuration which is going to be more platform-specific.
We should also consider renaming/organizing some of our existing core/ functionality too. For instance, hmndlib_pinocchio is more than just a pinocchio wrapper \u2013 its basically the beginnings of our core WBC (with pinocchio just being one of many possible dependencies used to solve some of the rigid body dynamics needed in a WBC solution).
The primary result of this phase is less ROS to deal with and slightly better organization to help us reason about the next steps.
Phase 2: Isolation and Separation
We should be able to start separating our core functionality from the ros2_control framework and internals at this point, resulting in much more testable and understandable code.
ros/\n control/\n ROS-specific controller stuff\n platforms/\n alpha/\n alpha_configuration\n\ncore/\n hmndlib_hal/\n actuator_control/\n ethercat\n mocks\n issac\n\n hmndlib_control/\n joint_control/\n impedance\n admittance\n whole_body_control/\n wbc_controller\n
The main shift here is that instead of having the core logic embedded into ROS nodes and controllers (and all the problems from above), we have isolated the core functionality into more functional (input \u2192 output) modules under the core/ folder - free of any ROS-related messaging or threading.
For example, you should be able to instantiate an ImpedanceJointController with some parameters, and then update it synchronously within a test. Even better, this core interface should be wrapped w/ Python bindings so we can test w/ pytest and along with performing analysis with Python tooling like notebooks. If we need to keep the ros2_control framework around a bit longer, it should simply wrap the core function. Then we can at least be sure that our core implementations have test coverage (though we ideally have SOME end-to-end coverage over ROS as well to sanity check the integration into the framework).
Phase 3: Collapsing the Graph
At this point we should have some core (non-ROS) functionality for actuator control, joint control and whole-body control, along with a means for configuring them and booting them within ROS. We should have test coverage for our individual core pieces and hopefully some sort of end-to-end integration test (perhaps some sort of simulation) for the stack as a whole.
It should be fairly obvious that we should be able to simply connect the basic functionality directly together, without any middleware or frameworks at all.
from hmndlib_common import RingBuffer\nfrom hmndlib_hal import EthercatActuatorDriver # has .poll() -> ActuatorState, .send(cmd)\nfrom hmndlib_control import ImpedanceJointController # .update(traj?, state) -> (JointState, ActuatorCommand)\nfrom hmndlib_control import WholeBodyController # .update(task?, joint) -> (TaskState, JointTrajectory)\nfrom hmndlib_control import ActuatorState, ActuatorCommand\nfrom hmndlib_control import JointState, JointTrajectory\nfrom hmndlib_control import TaskState, TaskCommand\n\njoint_states = RingBuffer(100) # joint ctrl -> WBC\njoint_trajectories = RingBuffer(100) # WBC -> joint ctrl\ntask_commands = RingBuffer(10) # upstream app -> WBC\ntask_states = RingBuffer(10) # WBC -> downstream app\n\ndriver = EthercatActuatorDriver()\njointctl = ImpedanceJointController()\nwbc = WholeBodyController()\n\n# just for illustration - running a 1KHz control loop in python is... not okay\n\ndef joint_loop(period_s: float = 0.001):\n next_t = time.perf_counter()\n while True:\n traj = join_trajectories.pop_latest()\n act_state = driver.poll()\n # underflow logic here\n\n # our core joint controller update\n js, act_cmd = jointctl.update(traj, act_state)\n\n # overflow logic here\n joint_states.push(js)\n driver.send(act_cmd)\n\n next_t += period_s\n time.sleep(max(0, next_t - time.perf_counter()))\n\n\ndef wbc_loop(period_s: float = 0.02):\n next_t = time.perf_counter()\n while True:\n task_cmd = task_commands.pop_latest()\n js = joint_states.pop_latest()\n # underflow logic here\n\n # our core WBC update\n task_state, traj = wbc.update(task_cmd, js)\n\n # overflow logic here\n task_states.push(task_state)\n joint_trajectories.push(traj)\n\n next_t += period_s\n time.sleep(max(0, next_t - time.perf_counter()))\n\n\nif __name__ == "__main__":\n threading.Thread(target=joint_loop, daemon=True).start()\n threading.Thread(target=wbc_loop, daemon=True).start()\n\n while True:\n time.sleep(1) # main thread parked for this example\n
This is of course a simplified example, and we are taking some significant liberties by mixing the loops in this way. But this does show roughly how we can re-compose our individual core building blocks without ROS. Of course, we could still publish all of the data from these queues, but this isn\u2019t demonstrated above.
In the example above, the layer writing TaskCommands to the queue could be our trained policy or it could be some teleoperation app.
TF Nonsense
The ROS2 joint controller accepts joint trajectories at a rate significantly slower than the joint controller itself runs, and reports JointStates at 1000Hz. The robot_state_publisher will subscribe to these JointState messages and publish transforms for all the joints so that other parts of the system can convert between the various coordinate frames. This is where a lot of the complexity of the ROS solution comes into play - we have 14 DoF in the arms alone being updated at 1Khz - this is 14,000 updates per second to the robot_state_publisher callback internally which then get broadcast (I think at 20Hz) so that all so that other nodes (via TransformListener) can reconstruct the state that already exists inside the WBC.
This is so dumb, and a perfect example of why smashing a robot into a distributed system is a bad idea. Why would we re-compute the forward kinematics AGAIN outside of our controller, with lower-resolution data? Why would we worry about the realtime clock and bag replay issues while doing this?
If we were in the same process, we could simply \u201cask\u201d the WBC to transform coordinates for us, since it already has all this information (and much more). The rate of these queries will be far lower than the rate of updates necessary attempting to keep TF state clean across TransformListeners and multiple processes.
" }, { "id": "1027899618", "title": "MacOS Linux Development - Lima + VSCode Remote SSH", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1027899618", "last_modified": "2025-08-06T14:30:04.993Z", "created_at": "2025-08-06T14:04:24.838Z", "body_html": "This guide lets you use Lima + VSCode\u2019s \u201cRemote - SSH\u201d extension to treat an x86_64 or ARM64 VM like a remote dev machine. Clean, fast, reproducible builds. No Docker hell.
Isolated, reproducible build envs.
Works with VSCode like it\u2019s a remote machine.
Use your normal editor, not weird container shells.
Install:
limactl
VSCode
VSCode extension: Remote - SSH
# install lima\nbrew install lima\n\n# create VM\nlimactl create --name ubuntu-arm64 --arch aarch64 --cpus 8 --memory 16 --disk 100 --vm-type vz --rosetta --mount-writable template://ubuntu\n\n# start VM\nlimactl start ubuntu-arm64\n\n# get a shell\nlimactl shell ubuntu-arm64\n
Now you have a reasonably fast/performance Linux system on your Mac.
cat ~/.lima/ubuntu-arm64/ssh.config\nHost lima-ubuntu-arm64\n <contents dumped here>\n
\u201cSplice\u201d the contents of this file into your ~/.ssh/config so that you can SSH to your Lima VM with:
ssh lima-ubuntu-arm64\n
Then from VSCode:
Cmd+Shift+P \u2192 Remote-SSH: Connect to Host...
Select lima-ubuntu-arm64
Done. You\u2019re in.
" }, { "id": "1027506232", "title": "Bushwhacking Software", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1027506232", "last_modified": "2025-08-06T09:17:39.727Z", "created_at": "2025-08-06T08:45:01.786Z", "body_html": "Planning software projects is hard. Really really hard. Typical project plans attempt to break down some large deliverable into pieces, organize those pieces on a timeline, assign resources and probably also divide time into large-scale phases. In reality, software development requires doing all of these things concurrently, constantly. There is no line between development and testing. There is no line between development and integration. Planning activities for software shouldn\u2019t focus on timelines or work breakdowns as much as they should seek to clarify goals, ownership and define feedback.
Hardware development is traditionally expensive, slow, and physically constrained. You have limited shots, each cycle costs real money, and you have real-world constraints - gravity, thermals, friction, EMI, supply chains - all providing neat boundaries. Sure, there\u2019s complexity, but it\u2019s a bounded complexity. Each iteration can take weeks or months and has significant material costs. Because of this, extensive planning, careful requirements gathering, and rigorous V-model methodologies really do help reduce your risk.
Software is fundamentally different. It has no such guardrails, no material costs to speak of (ignoring training costs), and iteration speed can be made arbitrarily fast. Without the constraints of physics, complexity grows combinatorially. Threads, shared memory, distributed systems, configuration - it's an explosion of dimensionality. You can't realistically predict every interaction or failure mode ahead of time. It\u2019s pure chaos, unbounded.
In short:
Hardware: Iterations are expensive and necessarily slower \u2192 Planning pays off.
Software: Iterations are cheap and can be faster \u2192 Planning yields diminishing returns quickly.
Planning is the best way to mitigate hardware development risks. Fast iteration is the best way to mitigate software development risks.
Planning is still valuable in software, but its primary role is to clarify goals and expectations, not to micromanage every implementation detail. If your software plans look like hardware plans, you\u2019ve probably gone too far. At some point, you're just fooling yourself into thinking complexity can be perfectly predicted and managed upfront - it just can\u2019t.
Instead, your plan should focus on three things:
Clearly defined goals - what are we doing?
Well-structured, rapid feedback loops - how do we know we are succeeding at the goal?
Ownership - who is accountable for the goal?
If your team knows what the software is supposed to achieve (goal clarity), and you have solid feedback mechanisms (data, metrics, observable results), you have everything you need. Excessive upfront specification - trying to capture every requirement before iteration - just wastes time, creating a false sense of certainty.
Its also important to consider staffing of course. How can you reason about how many engineers should be working toward a goal? During planning activities, owners should be assigned to goals and these owners are responsible for end-to-end progress toward the goal and the resulting feedback signals. This is less of an exercise of breaking down work items and assigning to individuals on a timeline, and more about giving owners the authority and accountability to delegate and rapidly adjusting priorities within the team. There will never be enough staff to do 100% of what we want to do, and so this is a constant negotiation as priorities shift and new lessons are learned over time.
Think of it like inverse kinematics:
For a simple, planar two-joint arm, sure, you can easily solve analytically. You have clear constraints, no redundancy, and an obvious solution. Now consider a full-body humanoid robot with 40-some redundant joints. The analytical approach is madness - you\u2019d waste years just formulating the problem, and for what? Infinite solutions exist. You instead solve numerically. You pick a small timestep (fast iteration), clear constraints (goals), and use strong gradients (feedback from reality) to quickly converge on a workable solution.
Software is exactly this: a hyper-redundant , high degree-of-freedom system with infinite possible solutions. You don\u2019t win by meticulously planning every motion; you win by iterating faster and better than your competition.
Planning isn't useless - it\u2019s just limited. Do just enough to define your goals clearly and set up strong, fast feedback loops. Then get out of your team's way and let them iterate rapidly. They\u2019ll reach the right solution faster, and the solutions they find will be more robust precisely because they're grounded in real data rather than brittle, outdated assumptions.
Plan just enough to frame the problem - then iterate as fast as humanly possible.
" }, { "id": "1020264502", "title": "Whole Body Control for Dummies", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1020264502", "last_modified": "2025-08-05T09:45:23.097Z", "created_at": "2025-08-01T15:07:13.822Z", "body_html": "In part 1 (https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1017905442/Actuator+Control+for+Dummies?atlOrigin=eyJpIjoiOWNmZTY4MzM1YzJiNGQ4NTg0NWY5ZmVlNzc5NDA2MDAiLCJwIjoiYyJ9), we looked at how standard actuators are controlled. In part 2 (https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1019543565/Joint+Control+for+Dummies?atlOrigin=eyJpIjoiYWUzMmQwOWI3NGNlNGZmNWIyZDRlZGRmNTQ3MzU4YzAiLCJwIjoiYyJ9), we look at how we can better control not just the actuator, but the interactions the actuator has with its environment. Here in part 3 we will look at how we can control many joints simultaneously in order to achieve specific tasks while maintaining a prioritized list of constraints.
Once we start arranging joints in a chain, interesting things start to happen. In addition to the Coriolis effects we need to compensate for in impedance control, we need to also propagate reference frames up the chain in order to compute where the end effector (usually the end of the chain) will be located.
Let\u2019s consider a simple 2D robot arm where we have a shoulder and elbow, with the end effector being at the point of the wrist (though no articulation here).
![](\"https://sklvc.atlassian.net/wiki/download/attachments/1020264502/Untitled%20Diagram-1754061178160.drawio.png?api=v2&version=1\")
We won\u2019t be digging too far into the math here, but the rough shape of the challenge with robotics is given some robot configuration (the lengths of the links L1 and L2) and the angles of the joints (the red dots with angles \u03b81 and \u03b82), how do we compute where the end effector at the end of the arm will be located with respect to our base coordinate system?
The basic strategy is that we treat this as a chain of transformations. The rotation \u03b81 and translation L1 allow us to place the location of the second joint with respect to the first joint. Then the rotation \u03b82 and L2 allow us to do the same for the end effector with respect to the location of the second joint. To get the position of the end effector with respect to our base coordinate system, we just need to apply these transformations one after the other. Obviously this gets a bit more complex once we have 7DoF, differential joints and end effector orientations to worry about, but the basic math is just a chain of transformations.
Forward kinematics is useful, but the real magic starts when you consider the inverse problem: given some desired end effector location, what joint angles do we need? This is unfortunately a much more difficult to problem to solve, as there usually isn\u2019t a clean analytical solution for this.
The location of our end effector is what we would refer to as our task space and in this example above it has 2 degrees of freedom (x and y). The configuration of the angles is referred to as our joint space and also has 2 degrees of freedom (\u03b81 and \u03b82). In this example, the degrees of freedom of the joint space and the task space are the same, meaning that the robot isn\u2019t \u201credundant\u201d (though there may be multiple solutions to the IK problem such as \u201celbow-up\u201d or \u201celbow-down\u201d. This sorta like solving a quadratic equation and choosing which of the two solutions to use.
If we have more degrees of freedom in our joint space than our task space (such as adding another link in the arm above), our system becomes redundant and there are now an infinite number of solutions to the IK problem. At this point analytical approaches break down entirely and we must use solvers or optimizers. A major challenge here is solving the IK solution for each time-step while ensuring motion remains smooth and continuous.
We also need to worry about singularities - solutions where the robot loses degrees of freedom. In the double-linkage arm above, if the arm is fully extended or fully-collapsed, the links become co-linear and so we lose a degree of freedom. This can cause major challenges when planning trajectories, and so usually we need to do a lot of trickery to avoid singularities.
When we have redundancy, things get much more fun. Redundancy is what allows you to keep your elbows out of the way when trying to use a screwdriver in a tight-space, or why you don\u2019t tip over when you try to pick a sock up off the floor. You effectively operate within two spaces: what is the task you are trying to solve, and what \u201cother\u201d degrees of freedom do you have to avoid any nasty side-effects like tipping over or smacking your elbows.
When we introduce redundancy, we can \u201cfix\u201d the end effector (the thing we are trying to move to accomplish some specific task) which then gives us some null space which is a continuous manifold of \u201cother shit\u201d we can do without impacting the actual task. Hold your handout in front of you on the wall, and then move your whole body around while keeping your hand fixed: that\u2019s your null space for that task.
So our robot needs to move its arms in front to pick an object up. This represents about 12 degrees of freedom (position + orientation of 2 end effectors in 3d space). The robot also has degrees of freedom in the base, the bendy leg/legs and some extra in the arm to create a pretty complicated null space where we can optimize for some secondary or tertiary tasks.
In the most simple and crude terms, whole-body control allows the robot to prioritize these tasks and plan within the null space of each one until we use all our degrees of freedom (or run out of tasks). A biped for example may try to control where its center of mass lies on the ground to maintain balance. This only requires a few degrees of freedom, leaving significant amounts left for the task to be solved. There may also be tasks related to maintaining constraints (both arms picking up some rigid object), optimizing joints (keeping them from limits or distributing torques over more joints) or optimizing manipulability (ensuring that the robot doesn\u2019t maneuver near singularities).
" }, { "id": "1022394482", "title": "Welcome to the Jungle", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1022394482", "last_modified": "2025-08-04T12:17:26.957Z", "created_at": "2025-08-04T09:57:39.166Z", "body_html": "Sure, testing proves your code actually works. Functional correctness, requirements coverage - blah, blah, we've heard it before. But in a monorepo shared by dozens developers, your tests aren\u2019t just correctness checks. They are your way of communicating expectations and guarding your code against well-meaning but clueless invaders.
![\"image-20250804-121602.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1022394482/image-20250804-121602.png?version=1&modificationDate=1754309764306&cacheVersion=1&api=v2&width=259&height=194\")
Let\u2019s reframe testing for a second. In a monorepo:
Tests aren't just checks - they're communication channels.
Tests show your teammates (clearly, loudly, sometimes aggressively) how your code expects to be treated.
Tests document the assumptions you made about inputs, outputs, and behavior, making your implicit expectations explicit.
In other words, testing is your defense against someone else\u2019s bad day becoming your bad day.
When writing tests, think about your fellow developers:
Readable tests: If someone new lands in your module, your tests should scream, \u201cHey! Here\u2019s exactly how this thing is supposed to behave:\u201d
Explicit assertions: Leave no room for guesswork. Every test is a strong opinion about your component\u2019s behavior.
Obvious failure modes: If your test fails, it shouldn\u2019t whisper - it should loudly and clearly point to exactly what's wrong or what assumptions have been violated.
Your tests are part of your module\u2019s API. Write them like a good README: opinionated, concise, and impossible to misinterpret.
The moment you merge your code, it enters the jungle. Other developers, rushing towards deadlines, might unknowingly stomp through your delicate assumptions. Your testing strategy should:
Be integrated into the CI pipeline - run tests early and often.
Leverage incremental test execution - only run the tests that are impacted by code changes (because no one has time for full runs every commit).
Fail loudly and immediately - make sure breakages get attention fast.
If someone breaks your shit, make sure everyone knows it quickly.
Each test type has a role to play:
Unit - \u201cTouch this function incorrectly, and I will immediately complain.\u201d
Integration - \u201cYou broke the communication between these components - fix your wiring!\u201d
Smoke/Acceptance - \u201cIf this fails, something is seriously wrong - come fix it ASAP.\u201d
This layering isn\u2019t just about catching bugs - it\u2019s about clear communication to your teammates. If an integration test fails, someone immediately knows they broke a contract.
Monorepos can turn into chaos without clear testing strategies. If tests are flaky, slow, or cryptic, people will stop respecting them. Keep your tests:
Fast enough to run frequently.
Reliable enough that they\u2019re trusted and provide high signal.
Clear enough to quickly pinpoint responsibility.
If you don\u2019t trust your tests, neither will anyone else - a recipe for chaos.
Testing in a monorepo isn\u2019t so much about quality control as it is about survival. Make your assumptions explicit, your expectations clear, and your defenses strong.
Welcome to the jungle.
" }, { "id": "1007976535", "title": "Software", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1007976535", "last_modified": "2025-08-04T09:30:23.848Z", "created_at": "2025-07-25T15:54:36.728Z", "body_html": "This is meant to be a dumping ground for all things software at Humanoid. Specifically, I\u2019ll be trying to capture more general software guidance here, including things related to how we work, what we are doing and where we are going.
https://miro.com/app/board/uXjVIjzyrI0=/
" }, { "id": "1019543565", "title": "Joint Control for Dummies", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1019543565", "last_modified": "2025-08-01T10:25:26.878Z", "created_at": "2025-08-01T08:22:00.665Z", "body_html": "In part 1, https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1017905442/Actuator+Control+for+Dummies?atlOrigin=eyJpIjoiNmQxZGI5ZDA4NDJjNDRlYTkzNjYwYjFhZTY5ZWNhZWMiLCJwIjoiYyJ9, we looked at how our off-the-shelf actuators allow for position, velocity and torque control over standardized \u201cCANopen\u201d interfaces using EtherCAT. At the very end, I mentioned how this isn\u2019t quite good enough.
With a motor sitting on the workbench, the actuator control is probably good enough for anything you\u2019d want to do with the motor. But once you actually put a load on the motor (especially if that load exerts some force via moment arm), we need to be concerned with how these actuators interact with the environment. In robotics we often call this \u201ccompliant control\u201d.
Compliant control allows us to model the interaction between the robot and the environment as if there is some virtual spring with some configurable stiffness. A very compliant (low stiffness) controller will be noodle-y and limp, yielding to the environment easily and exerting very little force. A less compliant (high stiffness) controller will move quickly and with precision, but may exert much more force on the environment in order to do so. Compliant controllers often require mode sophisticated dynamics models to work, since they are attempting to control a virtual \u201cdynamic\u201d system (the spring dynamics mentioned earlier).
In electrical systems, we say that impedance is a circuit's resistance to current (its a slightly more general form of resistance that can include capacitance or inductance for AC or RF circuits but we\u2019ll ignore this for now). If you remember Ohm\u2019s law, V = IR, you can basically see how impedance works if you pretend the R is a Z and ignore the imaginary numbers involved.
Because a lot of controls engineers have electrical backgrounds, they couldn\u2019t resist (lol) the analogy here where the impedance controller is attempting to \u201cresist\u201d some mechanical motion given some force. We can crudely re-write Ohm\u2019s law for our motor with something like Force = Movement * Impedance.
So given some desired movement (a position and velocity target perhaps along a trajectory), an impedance controller will compute an error from the currently measured movement, and from this compute some inferred torque as if this were the displacement in some spring. This tells the controller how hard to push back so that the error goes to zero. The higher the error, the more the controller will push back (just like a spring).
Again, electrical engineers can\u2019t resist the temptation, and so the inverse of impedance in electrical systems is known as admittance and is represented with a Y. This is analogous to the \u201cconductance\u201d of the circuit, or how easily current can move.
Instead of giving the controller a movement and generating a torque, an admittance controller will take a force and determine a movement. In order to close the loop, admittance control will need some way to sense force - either an external force sensor or perhaps a crude approximation by measuring motor currents.
So we measure the external forces/torques, and then compute the expected displacement of our virtual spring (gross oversimplification - sorry) and apply this to the actuator using something like CSP mode. We tell the actuator how to move in response to some external force.
Admittance control is useful in scenarios where you have extremely high-friction, non-backdriveable actuators that you want to simulate some sort of compliance with. You can slap on a torque or force sensor and use admittance control to allow the system to be pushed around in a more compliant mode.
Let\u2019s imagine a robot arm being held straight out in front of the robot, being impedance controlled with a very low stiffness. Gravity will be pulling the arm down, generating some pretty hefty torques on the shoulder joint and resulting in sagging. Or imagine the robot swinging its arm out to pick something up, the resulting Coriolis forces would be generating torques for many of the joints that are along the chain.
These external torques can cause some major issues with impedance control, making precision control very difficult if not accounted for. Unlike the simpler actuator control, impedance control requires much better models of the robot\u2019s dynamics and morphology so that we can properly cancel out these dynamics when controlling our robot.
" }, { "id": "1017905442", "title": "Actuator Control for Dummies", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1017905442", "last_modified": "2025-07-31T18:06:19.321Z", "created_at": "2025-07-31T17:32:44.778Z", "body_html": "This is part 1 of a series of posts here about control. I\u2019m not a controls engineer, so a lot of this is to help me organize my thoughts on matters, but also to help other SWEs unfamiliar with the details of controls to get up to speed.
Many of us probably attached a DC motor directly to a battery at some point as kids to make it spin. The battery energizes the electromagnetic windings of motor and generates a torque which spins the motor. The higher the voltage, the more current and thus more torque generated. In this fashion, the motor is operating in an open loop configuration (without feedback), and so its difficult (impossible) to control for precise torques.
With closed-loop feedback, we measure torque (or rather infer it from actual current consumption) and use some PID loop to feed this back and control the input so that we get some target target. Easy right? Well - in practice this is actually pretty challenging, usually requiring some fancy \u201cfield-oriented control\u201d because the motors we use are typically brushless and much more efficient that the simple DC motors we hooked up to batteries as children. These torque loops operate at 10s - 100s of KHz for smooth control, because it needs to keep up with the inductance and switching frequencies required to drive the motor windings (which requires a very high speed and low-latency). For these reasons, torque control is often implemented directly within the actuator itsef.
Once you have a core torque loop, you can wrap it with a velocity controller which attempts to match some specific speed. You can use an encoder to get the current speed of the motor, and adjust the target torque up or down until you get the desired speed. This can run much slower than the torque loop, 100s of Hz to KHz range, because unlike torque control the feedback here is the mechanical spinning of the motor itself rather than the actual electrical properties of the windings and such.
Finally, you can wrap this velocity controller with a position controller which given some specific position will command the velocity required to get there. This will operate typically around the same speed as the velocity control loop (maybe a little slower but not much).
![\"image-20250731-180330.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1017905442/image-20250731-180330.png?version=1&modificationDate=1753985011847&cacheVersion=1&api=v2&width=664&height=334\")
This stack of controllers is called a \u201ccascade controller\u201d, and it lies at the heart of the actuators used on our robots. If you\u2019re controlling position, it will go to the position controller which will generate a velocity target, which goes to the velocity controller which generates a torque target which goes to the torque controller which ultimately controls the actuator with FOC. If you\u2019re controlling velocity, you can set the velocity target directly and skip the position controller. If you\u2019re controlling torque you only drive the last layer in the controller.
In the world of industrial automation, there is no shortage of demand for precision actuation. Conveyor belts, servos, robot arms, doors, wheels - endless possibilities. To help manufacturers out, some open standards were created.
DS402 defines the device profile for a generic actuator using the cascade controller above. These devices have a standardized state machine, control words/addresses and modes including CSP, CSV and CST (cyclic synchronous position, velocity and torque respectively) which allow different applications with the same profile. You can also configure limits, homing and all that, but we\u2019ll conveniently forget about all this for now.
Originally the communication bus was CAN (controller area network) which was originally used in automotive for networking the components of the car. CAN has robust cabling, simple integration but (compared to more modern stacks at least) low data rates. You can set velocities or positions over CAN, but anything force sensitive requiring direct torque control is going to be impossible\u2026
So some clever industrial folks decided to lift the protocols from the limited CAN bus to Ethernet and called it EtherCAT. Orders of magnitude more bandwidth, deterministic switching and microsecond latencies allow for multi-KHz update loops for dozens of actuators.
Having solid actuator control is critical to building a humanoid robot, but its not enough. For example, compliant interfaces or force-sensitive tasks like picking up fruit or other fragile objects requires driving the torque directly and skipping the CSP or CSV modes (usually - there are exceptions of course).
This means we need ANOTHER layer of control sitting above our actuator cascade control operating at the KHz range to make sure our motions are smooth and safe.
" }, { "id": "1010335765", "title": "\ud83e\udea4 Robot Middleware", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1010335765", "last_modified": "2025-07-28T14:23:07.664Z", "created_at": "2025-07-28T14:05:26.077Z", "body_html": "If you\u2019ve built real robots, you\u2019ve probably tripped over the same architectural pothole: middleware. ROS, DDS, some custom pub-sub nonsense - take your pick. Each promises modularity, scalability, and discoverability. But in reality? You get a mess of launch files, mismatched message definitions, time sync gremlins, and mysterious data black holes between nodes that should be talking.
Robotics middleware like ROS can be useful early in prototyping as researchers and engineers cobble together pre-existing modules, but it quickly becomes a liability under the pressures of shipping a real product.
![\"image-20250728-142158.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1010335765/image-20250728-142158.png?version=1&modificationDate=1753712519607&cacheVersion=1&api=v2&width=460&height=337\")
It doesn\u2019t take long before the magic middleware is fighting you and teams are swimming upstream to ship anything at all.\u00a0 At some point, the question isn\u2019t \u201cwhich middleware should we be using?\u201d It\u2019s: Why are we using middleware at all?
Middleware exists to \u201cpre-solve\u201d integration. Every module is nicely encapsulated and exposes a strict set of messages that can be used to communicate with it.\u00a0 You can find research labs who have published ready-to-go modules and put everything together like Lego and get a working robot.\u00a0 You can swap out controllers or camera nodes or SLAM nodes with minimal impact to the rest of the system.\u00a0 As long as each node adheres to the middleware interface, everything fits together.\u00a0 Very Nice!
The most common sort of middleware with robotics systems is some sort of message bus.\u00a0 You hang various nodes covering different functions of the robot on this bus, and they communicate by publishing messages on various topics.\u00a0 This sort of architecture has been proven to be effective in large-scale data systems in the cloud (event sourcing basically), but is it the correct choice for a robot?
In cloud-based systems, message buses are almost exclusively used for unidirectional communication.\u00a0 System A notifies system B that something happened, which then triggers some action and then it notifies some downstream System C.\u00a0 These systems are usually very large scale, with billions of messages spread over hundreds or thousands of hosts.
Automobiles also make extensive use of a message bus, though this is the CAN interface.\u00a0 It operates across dozens of nodes and has many orders of magnitude fewer messages.\u00a0 Because of the way that automotive supply chains have been distributed over the last couple decades, these interfaces are fairly standardized and largely plug & play.
So if a message bus can work for a large-scale cloud system and a smaller-scale automobile (or fly-by-wire aircraft or a number of other vehicles), why not for a robot?
What makes middleware challenging for robotics is that the state of the system doesn\u2019t really partition naturally.\u00a0 At some point, you need to build a world model.\u00a0 And at some point, you need to decide how the robot as a whole will act given this model.\u00a0 These are fundamentally global things, and partitioning this state over many separate nodes communicating via pubsub (or any other messaging interface) means that you spend an enormous amount of time trying to reconstruct that global state from these smaller fragments.
Concurrency and Non-Determinism
By smashing nodes up and decoupling everything, you effectively transform your robotics stack into a distributed system.\u00a0 This means that every node will not be able to agree on a common global \u201chistory\u201d of events within the system.\u00a0 Different nodes may see events in different orders (across topics) or different sets of events (within a topic) due to scheduling differences or QoS.
Consider camera data - usually \u201cold\u201d data is not useful, and so to avoid increasing latency when the system is temporarily overwhelmed you want to enable the KEEP_LAST QoS policy which means that subscribers will keep the last N messages only, bounding the queues (and thus latency).\u00a0 What this means is that while a message may be published on some camera topic, there is no guarantee that all the subscribers would see that message.\u00a0 You may have a camera message that isn\u2019t seen by your perception stack, but it IS logged by data collection.\u00a0 Or perhaps the opposite scenario. In these cases, it is difficult to reason about the data that has passed through each node without logging everything within each node and doing extensive post-processing.
This makes it impossible to replay an episode from recorded data and guarantee that you can reproduce the same behavior.\u00a0 When it comes to tracking down long-tail reliability issues, this is a major impediment to progress.\u00a0 In order to reconstruct the same behavior, you need to be able to reconstruct the same global state.\u00a0 But if that global state is distributed over independent processes which may see different subsets of data, it's nearly hopeless.\u00a0 This is made worse by the fact that robots are inherently chaotic in that tiny changes to the input can result in significant changes to the output.\u00a0\u00a0\u00a0
Observability
One benefit of middleware is that it often provides a way to \u201csniff\u201d the message bus, making it possible to observe the communication between nodes.\u00a0 But only a tiny amount of the state of the system is actually exchanged on these topics (at least that\u2019s the hope - otherwise the system would never work as you\u2019d quickly overwhelm the message bus).\u00a0 But which history are you seeing?\u00a0 When you subscribe to a topic to see what\u2019s going on, this is NOT the same data stream that other nodes are seeing.\u00a0 They could be dropping or queuing those messages internally in ways that are not directly observable from your temporary subscriber.
And what about performance analysis, profiling or debugging the actual nodes themselves?\u00a0 This can get challenging - as attaching a debugger to a single node often changes the performance of the node and can trigger cascading failures around the system.\u00a0 You often need to dump the message bus AND collect profiling data from all the nodes and then do some sort of extensive analysis to line everything up to see how the system as a whole is performing.\u00a0\u00a0
Related to the above, if a random node crashes or segfaults, it can be fiendishly difficult to reproduce this behavior in isolation since you can\u2019t be guaranteed that you can reproduce exactly the internal state.\u00a0 You can maybe grab crash dumps, but loading that dump into a single node and trying to figure out what happened without the context of the rest of the system is extremely difficult.
ROS2 does indeed offer a ton of useful tooling including visualizers and data logging - but we don\u2019t actually need to leverage ROS2 as core middleware to make use of these things.\u00a0 What would a system look like if we chose NOT to use something like ROS2?
![\"image-20250728-142227.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1010335765/image-20250728-142227.png?version=1&modificationDate=1753712548303&cacheVersion=1&api=v2&width=662&height=168\")
First, consider that each topic is really a publish queue attached to N subscriber queues with a \u201clossy channel\u201d in the middle (the topic).\u00a0 The lossiness of the channel is determined by how QoS is configured within DDS (and ultimately depends on the latency vs reliability tradeoffs necessary for the topic or nodes).\u00a0 Consider a case where one subscriber runs much slower than the others - there are only a couple possibilities:
Apply backpressure all the way to the publisher, slowing everything down - this almost ensures that any data seen at the publisher is seen by all subscribers, but it means that the slowest subscriber will control the pace of all data on the topic.\u00a0 Say goodbye to performance.
Drop messages at the slowest subscribers - this keeps the publish rate up, but now we are deliberately choosing a scenario where some nodes see a different set of data than others.\u00a0 Say goodbye to determinism.
Instead of leaving the management of these queues up to some abstract middleware, we should own them.\u00a0 Instead of forcing everything to be serialized and transported over a socket, we should just enqueue native data and call functions.\u00a0 If we are going to introduce IPC, it should not be the default - it should only be introduced for a good (and specific) reason.\u00a0 For example, we may need to communicate between multiple hosts or language runtimes, or we may want to isolate some I/O heavy workloads like data logging from the actual core autonomy stack to avoid any sort of interference.
Imagine that instead of trying to shoot our data all over various topics and nodes, we instead have a monolithic application where we queue up all of our sensor data with a unified and integrated QoS policy and we log exactly the messages that are processed:
![\"image-20250728-142254.png\"](\"https://sklvc.atlassian.net/wiki/download/thumbnails/1010335765/image-20250728-142254.png?version=1&modificationDate=1753712575855&cacheVersion=1&api=v2&width=662&height=357\")
This more linear datapath forces a total ordering of all data in the system, which brings us closer to deterministic execution.\u00a0 If you play the same log of sensor messages into the system, you should get the same actions.\u00a0 This isn\u2019t trivial, but it's a worthwhile goal since storing all of the state is intractable - but if we can reliably recreate the state from recorded observations then we can have many of the same capabilities.
Just because this linear datapath exists doesn\u2019t mean that the whole loop has to operate at the same frequency.\u00a0 There may be high-frequency and low-frequency paths in here, but as long as the data itself can be ordered reliably, then we should be able to reproduce those states.
The logging above could in fact still be ROS2, meaning we can still leverage rviz and other ROS2 tooling to help accelerate development.\u00a0 The point isn\u2019t to avoid ROS2 entirely - it's to avoid fragmenting our system up into many pieces and obfuscating the overall state and dataflow in the system.\u00a0 It\u2019s to avoid introducing tight, bi-directional coupling over a message bus when we can instead have modules integrated more directly using simpler primitives like threads and queues.\u00a0\u00a0
You want to swap perception modules, pipe planning into control, log data without polluting your control loop. Abstractions sound helpful - until they start abstracting away determinism, observability, and sanity. The minute you insert a message bus between two components, you stop owning their behavior. Timestamps drift. Buffers overrun. QoS policies silently drop your data.\u00a0 Nodes may not agree on what data has been sent through the system, and thus the true global state of the system becomes more difficult to pin down.
That\u2019s not a bug in your system. That is your system.\u00a0 Distributed systems are fundamentally this way, and we choose to build them because oftentimes the tradeoff is worthwhile.\u00a0 Maybe all the algorithms don\u2019t fit in a single computer.\u00a0 Or maybe we need to isolate something for safety or performance reasons - the point is that when we choose to distribute state, it should be for a good reason rather than being the default choice as it often is when we choose some middleware.
The problem isn\u2019t that ROS2 is bad (though it has its quirks). The problem is what middleware encourages: fragmentation disguised as flexibility. You\u2019re forced to build around the middleware\u2019s assumptions - types, lifecycles, transports - until your architecture is shaped more by the message format than by what the robot actually needs to do.\u00a0 The architecture doesn\u2019t serve the product - it serves the middleware you\u2019ve chosen.
And sure, you could write your own lightweight protocol. Many do. But that\u2019s not the real choice either. The deeper point is: every middleware stack is a solution to problems created by introducing middleware in the first place.
" }, { "id": "1005027354", "title": "Integration and the Monorepo", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1005027354", "last_modified": "2025-07-25T16:01:03.615Z", "created_at": "2025-07-24T08:55:59.870Z", "body_html": "TL;DR - Your code goes here: https://github.com/HumanoidTeam/hmnd
In robotics, integration is everything.\u00a0 Without integration, you have a number of interesting parallel research tracks.\u00a0 But to get to a working robot that does commercially viable work, you need to painstakingly integrate the dozens of different subcomponents and subsystems.\u00a0 In almost every facet, the whole is much much greater than the sum of its parts:
Most of your time will be spent integrating
Most of your bugs will be found during integration
Most of the value is created by integrating
Integration represents a massive risk for the project - how can we be sure that all these individual pieces will fit together correctly at some point in the future?\u00a0 You can write ICDs, interface specs, APIs all you want, but until you actually put it all together you can\u2019t really be sure.
Integration therefore should be front-and-center during development.\u00a0 Each change should be integrated with the entire system as soon as possible.\u00a0 The longer we go without integrating, the more costly it ultimately becomes.\u00a0 The way we organize, build and release our software should be oriented around delivering an integrated robot rather than individual team productivity.
Simply placing all components in the same repository puts us into an \u201cintegration-by-default\u201d environment.\u00a0 Of course there are many tradeoffs involved.\u00a0 With a multi-repo environment, each team is free to move very quickly, unburdened by others\u2019 modules.\u00a0 Each repo is smaller and therefore can be cloned and tested more quickly.\u00a0 Each repo is fundamentally more reliable (fewer tests being run means fewer chances for flakiness to derail CI).\u00a0 But the tradeoff here is that tying all of these separate repositories together is much more difficult.\u00a0 Given how important integration is to developing a robot, is this the right tradeoff?\u00a0 Does it make sense to defer all this coordination overhead until some later date?
The integration could happen within source control via git submodules and some sort of integration repository, but this requires multiple pull requests (PRs) across multiple repositories to make a change.\u00a0 Coordinating interface changes across multiple components could involve changes to 3 or more repositories, and this is extraordinarily cumbersome with git submodules as they don\u2019t track branches easily. \u00a0 This is so painful that developers will often delay necessary refactors or integration changes until the future (where inevitably they are even more expensive).
Integrating later (fetching pre-built artifacts at build or deployment time) has many of the same downsides as using submodules, but also introduces a lot of additional versioning and artifact management overhead.\u00a0 Each component is versioned and every interface between components needs to be re-integrated over and over and over again.\u00a0 Integration generally becomes much more expensive.
A single repository can produce a single set of artifacts versioned by a single commit.\u00a0 It becomes trivial to reason about compatibility, as every interface can be tested immediately within the repo.\u00a0 Every interface can be refactored in a single PR.\u00a0 Any changes across modules can be reviewed and merged as a single PR.\u00a0
A decision to use a monorepo is a deliberate choice to accept a set of tradeoffs where individual teams may have a bit more friction, but end-to-end integration is faster and easier.\u00a0 The key assumption here is that because programs typically spend most of their time in integration, this more than pays for itself as integration occurs in tiny increments continuously.
Most of the downsides (larger repo size, tons of \u201csuperfluous\u201d artifacts, longer CI times) can actually be mitigated by better infrastructure.\u00a0 I\u2019m not aware of any such \u201cshort-cuts\u201d for integration as a whole, other than simply doing it frequently and often.\u00a0 Sparse clones can be leveraged to clone only a small portion of the repository.\u00a0 Bazel (if used) can query a set of build targets to determine which are affected by a given git diff.\u00a0 Robot software can be updated as deltas to avoid needing to schlepp multi-GB disk images around.\u00a0 Large non-code assets can be stored in some sort of content-addressable bucket in S3 (addressed by hash) if Git LFS becomes too cumbersome.
Note that the decision to use a monorepo has nothing to do with which modules we choose to have - only that these modules are versioned together in the same repository.\u00a0 In order to keep the monorepo manageable, it will still be important to layout the repo to ensure that sparse clones and selective CI testing are viable.\u00a0 A monorepo will punish you for poor organization.
The scope of what should be versioned in this repo should include as much as possible.\u00a0 Not just the source code or the artifacts built from that code, but the entire root filesystem, all of the various 3rd party dependencies, drivers, kernels and even the bootloaders would ideally be versioned by a single commit.
When the whole stack - firmware, infrastructure code, dependencies - is under unified version control in a monorepo, the state of the system is explicit and fully reproducible. This clarity allows a large team to share limited hardware efficiently, since you no longer worry about mismatched dependencies or stale builds causing conflicts. Instead of managing scattered, fragile setups across multiple environments, each developer confidently deploys a known-good version, maximizing hardware utilization and minimizing downtime or debugging overhead.
The confidence that you gain in deploying this way means\u00a0 you can focus on the real work, rather than wasting time on setup and managing dependencies/configuration.\u00a0 It also means that we can trivially trace our collected datasets to a single set of artifacts which produced the data.\u00a0\u00a0
Trunk-based development on main
Feature branches off main (feature/short-description)
Squash/Rebase on trunk, no merge commits
Release branches (release/x.y)
Hotfixes should be applied to both release and trunk (this can be automated)
Each commit that lands on trunk or a release branch must be green. Each build can be traced back to a single SHA that defines the system state across all layers.
PRs should be squashed and rebased, which ensures that the history of the commits on the trunk reflect tested/gated changes rather than intermediate commits.\u00a0 It's difficult to enforce rules like every commit should build clean without otherwise blowing up CI costs (running CI for every commit in the PR) or trusting developers to run CI locally for every commit, and having a linear history makes bisecting to track down tricky bugs in the future much simpler.
Lint everything, always (fast, automatic, enforced)
Incremental testing runs only affected builds/tests
Changed ROS node? Run its integration tests.
Changed TF? Run terraform plan.
Changed models/weights? Run evals.
Changed URDF or CAD?\u00a0 Run sim.
Goal is to gate PRs by tests impacted by the change.
Tag the latest commit, and kick off a full build (non-incremental) including all tests
Potentially include more expensive evals or simulation that are not feasible for gating PRs
When appropriate, a nightly may be selected as a release candidate build.
It may be worth building the tip of main more frequently than every 24 hours, depending on how large the gap gets between PR test coverage and nightly test coverage.
In order to keep PR test times manageable, we may run a smaller subset of tests.\u00a0 This means that occasionally issues may slip through resulting in a broken nightly.\u00a0 Teams should respond urgently to fix these issues to ensure no one else is blocked on broken builds (and ideally any identified holes in the PR CI process should be patched).
The idea is to strike a balance where feature branches can be merged quickly into main and integrated (long tests means huge merge queue times and lots of rebasing), and ensuring that our integrated code is tested as thoroughly as possible.\u00a0\u00a0
To create a release, a \u201crelease manager\u201d should tag and branch off a commit on main.\u00a0 The release branch should be release/$version and the tag $version.\u00a0 Any PR against a release branch (as opposed to main) may trigger a different set of release-gating tests, and merging a PR into a release branch should automatically open a PR into main with the same changes (which may need to have conflicts resolved or other modifications before being merged).
All changes to a release branch should be approved by a release manager.
"Won\u2019t this slow teams down?"
Only if you don\u2019t count integration as part of your timeline.
"What if I just want to work on my piece?"
You can. You own your modules. But once it lands on the trunk, it has to work with the rest of the system. That\u2019s not overhead - it\u2019s the entire point.
"Does this remove team autonomy?"
No. It replaces implicit, fragile team autonomy with explicit coordination, tracked by Git and tested by CI.
"What if we need long-lived experiments?"
Branch away. Pin infra.\u00a0 Comment out all the tests on your branch.\u00a0 But when it's time to ship, it goes through the same path as everything else: trunk \u2192 release \u2192 artifact.
Our monorepo provides a single commit to version our entire stack from platform to application (and eventually may include cloud infra as well).\u00a0 This drastically simplifies integration, but there are still some challenges that need to be addressed in order to ship code effectively.\u00a0 Having multiple platforms and multiple applications all being developed in one place and needing to be released at different times at different levels of maturity is a major challenge.\u00a0\u00a0
We\u2019ll try to address this with a few principles:
Never break the build - every commit on main should be deployable.\u00a0 If your change is dead-on-arrival, you have failed.
Branches should be short-lived - merge pain grows quickly with branch lifetimes.
Release late - do the bulk of the work on main, release as late as possible for stabilization.
Commit incrementally - extensive use of feature flags allows long-running research efforts to be integrated before they are complete, minimizing merge pain.
The trunk (our main branch) is the single source of truth for the state of the system.\u00a0 It captures all dependencies and all integrations across all supported platforms and applications.\u00a0 It is much cheaper to keep all of this constantly updated through small, frequent changes than to attempt to release and track compatibility across a matrix of platform variations, demos, PoCs, etc.\u00a0\u00a0
When developers want to create new functionality or make changes to the code, it's often necessary to branch off the trunk to provide a \u201csafe space\u201d to make these changes without impacting others.\u00a0 Some more extreme variations of Trunk-based development insist on developers committing everything directly to main (not unlike older non-distributed VCS like perforce or CVS), but the GitHub PR workflow provides a lot of additional benefits around reviews and CI that we should want to keep in place.
Ideally, every single commit on the trunk should be deployable.\u00a0 It should always be possible to load the latest commit from the main branch onto a robot to see the latest and greatest changes.\u00a0 Of course this requires extensive investment in CI and automated testing, but this is the goal.\u00a0 A compromise is that \u201cthe happy path\u201d should always work.\u00a0 What this means is that you should be able to load up a build and perform normal, expected behaviors without any major issues.\u00a0 There may be edge-cases or regressions in some areas that are under-tested, but a developer who merges code which is dead-on-arrival has broken the contract and brings great shame upon themselves.\u00a0 These breaking changes should be reverted immediately and fixed as soon as possible to keep the trunk healthy for everyone.
Every branch is a liability, accruing interest over time.\u00a0 When you have a branch, every other developer at the company becomes a liability as they merge changes and update dependencies, increasing the amount of work you need to do to \u201cfinish\u201d your branch and merge back to main.\u00a0 To every other developer, your branch is a ticking time bomb, accumulating changes that they aren\u2019t aware of that may cause issues with their own changes.\u00a0 For these reasons, branches should live for as short a time as possible.\u00a0 Generally speaking, the more \u201ccentral\u201d the changes are (the more of the system impacted), the shorter the branch should exist to reduce merge issues and deferred/accumulated integration costs.
Another way to think about this is that when you have a branch, you\u2019re the only one responsible for keeping that branch up-to-date.\u00a0 When you merge to main, you effectively have a contract with the org that this addition is to be supported by everyone (no one is allowed to break the build after all).\u00a0 For this reason, developers should be incentivized to merge their branches within hours or days rather than weeks - even if that work is incomplete (which we\u2019ll cover shortly).
These short branch-cycles improve the velocity, reduce merge-related overhead and serve as effective communication within the team about who is making what changes.\u00a0 Small changes, rapid integration.
In an ideal world, trunk commits could instantly deploy everywhere. In reality, a little stabilization might be necessary. \u201cLate branching\u201d from the trunk for releases ensures minimal divergence. Hotfixes for stabilization should always land back into the trunk (and automation is your friend here).\u00a0 Late branching means that once the functionality has been merged into the main trunk, a release branch can be created strictly for stabilization/hotfixes.
When working to stabilize a release branch, PRs should simply be made against the release branch in git.\u00a0 When that PR is merged to the release branch (where presumably the stabilization is needed first), a second PR should be opened against main with the changes automatically.\u00a0 Depending on the nature of development, it\u2019s possible that the merge to main may have conflicts that didn\u2019t exist on the release branch.\u00a0 This new PR provides the space to resolve those conflicts with the team and get the improvements back to the trunk.
Work on a release branch should follow the same guidance as work on the trunk - short-lived branches and quick merges.\u00a0 These changes will be reflected on the trunk after-all, so making sure to break things up will ensure things merge more smoothly and keep overhead low.
Sometimes, especially in research and infrastructure work, there are long-running efforts that can take months or even years to complete.\u00a0 This work should be broken up into reasonable increments and integrated regularly, even if incomplete.\u00a0 Ideally these incremental merges would be partially usable, but at a minimum they should be logically reviewable and testable.
This incomplete functionality can be behind a feature flag to protect the system from these incomplete changes.\u00a0 If you\u2019re working on a new policy, merge it behind a \u201cuse_fancy_new_policy\u201d flag which defaults to false.\u00a0 At some point in the future you can enable this flag when the feature is complete, and then set it as the new default and remove the configuration.
In addition to allowing incremental development and integration, these feature flags enable better data-driven development.\u00a0 You can toggle features within a single build and evaluate performance to get a better idea of the impact.\u00a0 This provides much better data than trying to compare two different builds with two different feature sets, as there could be many other confounding variables affecting the metrics.
"What if my fancy research effort takes 2-3 months?"
Use feature flags. Integrate early, hiding new functionality behind a flag, toggle off by default, and avoid integration nightmares later.
"What if I\u2019m re-writing our entire perception system?"
Embrace parallel development.\u00a0 Keep the old system intact and add the new system gradually in parallel behind feature flags.\u00a0 You\u2019ll want to do this anyway so you can properly compare and evaluate both systems within the same version of software to control for other variables.
"I need to prepare a customer PoC or demo - won\u2019t all these changes be risky?"
We should be writing tests to protect our core functionality, but obviously this isn\u2019t enough when days or weeks matter.\u00a0 A late release branch allows you to isolate functionality from the trunk-based churn, allowing you to focus on the demo without worrying about new changes (unless you want/need them)
"What about hotfixes - isn\u2019t this all complicated and messy?"
As part of stabilization, hotfixes will be made against release branches and with the power of GitHub automation we can automatically open PRs against main with these same fixes so we don\u2019t lose track.
" }, { "id": "1004732467", "title": "Move Fast and Merge Things", "url": "https://sklvc.atlassian.net/wiki/spaces/ROBOT/pages/1004732467", "last_modified": "2025-07-24T12:28:25.037Z", "created_at": "2025-07-24T08:54:53.384Z", "body_html": "In robotics, speed is everything. But too often, code reviews become a bottleneck, slowing down the team and killing momentum. Here at Humanoid, we've embraced trunk-based development on a monorepo precisely to avoid these pitfalls.
Branches are like fresh produce: great when fresh, trouble if they linger. The longer a branch exists, the bigger the headaches when it merges. So make branches short-lived. Every branch merged quickly means fewer conflicts, less stale code, and happier developers.
Part of keeping branch lifetimes short is making sure PRs are reviewed in a timely manner.\u00a0 We need to weigh the benefits of being strict in our reviews with the costs of delaying progress and extending branch lifetimes.
Our automated CI pipeline is the first and most thorough reviewer. It checks tests, lints code, verifies builds, and catches the vast majority of potential issues. Trust it. Let automation set you free. Human reviewers aren't there to nitpick code style or enforce trivial rules. Your CI should be ruthless so your reviewers don\u2019t have to be.
If your style quibbles cannot be covered by formatting or linting rules, they aren\u2019t worth enforcing.
Human code reviewers act as guardrails, not gates. They protect the repo from genuinely scary problems - huge, breaking changes, accidental disabling of tests, security pitfalls or safety-related issues. Reviews should be quick sanity checks, not perfectionist deep dives. If reviewers start obsessing over every line of code, something's gone wrong (and code is not being shipped).
That said - if you identify patterns or longer-term problems accumulating, it's worth bringing these to the team.\u00a0 But consider whether or not these concerns should block the immediate changes being made.
Code reviews are also about visibility. Reviewing code forces developers to see what's changing, improving team-wide awareness and helping everyone understand the state of the project. Perfection isn't the goal - awareness is.
We also probably can't rely only on automated GitHub notifications to summon reviewers.\u00a0 These will be noisy and likely increasingly ignored.\u00a0 A fundamental challenge here is the bystander effect - if everyone can review, who is going to take the action to do so?
This requires a culture shift where everyone wants to see things reviewed and merged faster.\u00a0 Developers should advertise their ready-to-merge PRs in relevant slack channels, and it should be expected that one or more reviews would be received.\u00a0 Developers should be incentivized to take part in this process because they will benefit when it is their turn to have a PR reviewed.
While we obviously encourage AI tooling, it\u2019s important to disclose that your PR was authored with AI tools.\u00a0 In addition, you as the creator of a PR are solely responsible for any AI-generated code.\u00a0 Disclosure is important since the types of mistakes and trickery that an AI coding agent may make can be quite different and surprising, and so reviewers should be made aware up front.
Don\u2019t hold code hostage in endless review loops. Better to merge fast and fix quickly in small follow-up PRs than strive for perfection before merging. Incremental, rapid iterations keep momentum high, frustration low, and innovation flowing.
At the end of the day, our philosophy is simple: trust CI, trust your teammates, and keep moving forward. Velocity is your friend, perfection is the enemy, and momentum is everything.
" } ] } }, "wan": { "sites": { "lab": { "site_desc": null, "networks": [], "vpn_tunnels": [], "devices": [] }, "1-triton": { "gw_cpu": 4.2, "gw_mem": 29.9, "wan_latency": 1, "wan_availability": 100.0, "internet_latency": 1, "vpn_status": "error", "site_desc": "Default", "networks": [ { "name": "hmnd-native", "subnet": "192.168.1.0/24", "gateway": "192.168.1.1", "vlan": null, "enabled": true }, { "name": "hmnd-infra", "subnet": "10.10.60.0/23", "gateway": "10.10.60.1", "vlan": 60, "enabled": true }, { "name": "hmnd-humans", "subnet": "10.10.40.0/23", "gateway": "10.10.40.1", "vlan": 40, "enabled": true }, { "name": "hmnd-iot", "subnet": "10.10.70.0/24", "gateway": "10.10.70.1", "vlan": 70, "enabled": true }, { "name": "hmnd-strangers", "subnet": "10.10.90.0/24", "gateway": "10.10.90.1", "vlan": 90, "enabled": true }, { "name": "hmnd-madness", "subnet": "10.10.80.0/24", "gateway": "10.10.80.1", "vlan": 80, "enabled": true }, { "name": "hmnd-robots", "subnet": "10.10.50.0/23", "gateway": "10.10.50.1", "vlan": 50, "enabled": true }, { "name": "hmnd-alpha", "subnet": "10.10.100.0/24", "gateway": "10.10.100.1", "vlan": 100, "enabled": true }, { "name": "hmnd-corp", "subnet": "10.10.30.0/23", "gateway": "10.10.30.1", "vlan": 30, "enabled": true }, { "name": "hmnd-mgmt", "subnet": "10.10.10.0/24", "gateway": "10.10.10.1", "vlan": 10, "enabled": true }, { "name": "hmnd-experiments", "subnet": "10.10.250.0/24", "gateway": "10.10.250.1", "vlan": 250, "enabled": true } ], "vpn_tunnels": [ { "name": "London Lab", "enabled": true, "ipsec_peer_addr": null, "ipsec_local_ip": null, "ipsec_profile": null, "ipsec_key_exchange": null, "ipsec_ike_version": null, "ipsec_esp_dh_group": null, "ipsec_dpd_enabled": null, "ipsec_dpd_interval": null, "remote_subnets": [] }, { "name": "Deploy 99 | Luke", "enabled": true, "ipsec_peer_addr": null, "ipsec_local_ip": null, "ipsec_profile": null, "ipsec_key_exchange": null, "ipsec_ike_version": null, "ipsec_esp_dh_group": null, "ipsec_dpd_enabled": null, "ipsec_dpd_interval": null, "remote_subnets": [] }, { "name": "hmnd-aws-1", "enabled": true, "ipsec_peer_addr": null, "ipsec_local_ip": "165.65.48.198", "ipsec_profile": null, "ipsec_key_exchange": "ikev2", "ipsec_ike_version": null, "ipsec_esp_dh_group": 14, "ipsec_dpd_enabled": null, "ipsec_dpd_interval": null, "remote_subnets": [] }, { "name": "hmnd-aws-2", "enabled": true, "ipsec_peer_addr": null, "ipsec_local_ip": "165.65.48.198", "ipsec_profile": null, "ipsec_key_exchange": "ikev2", "ipsec_ike_version": null, "ipsec_esp_dh_group": 14, "ipsec_dpd_enabled": null, "ipsec_dpd_interval": null, "remote_subnets": [] } ], "gw_name": "1 Triton", "gw_model": "UDMEA4B", "gw_version": "5.1.19.33549", "gw_uptime": 2581144, "gw_temp_cpu": 39.437, "wan_rx_bytes": 6098010567055, "wan_tx_bytes": 5716080457382, "wan_speed": 10000, "wan_rx_24h": 720444974368.7908, "wan_tx_24h": 1007517684017.4651, "devices": [ { "type": "usw", "name": "ECS 48 PoE | 4", "model": "USWF069", "serial": "8CEDE1C238EF", "mac": "8c:ed:e1:c2:38:ef", "ip": "10.10.10.10", "version": "7.4.1.16850", "uptime": 2830784, "state": 1, "adopted": true, "num_port": 52, "poe_power": 2150, "uplink_mac": "94:2a:6f:40:dc:6a", "uplink_name": "ECS Aggregation South" }, { "type": "usw", "name": "ECS 48 PoE | 6", "model": "USWF069", "serial": "8CEDE1C23ADE", "mac": "8c:ed:e1:c2:3a:de", "ip": "10.10.10.217", "version": "7.4.1.16850", "uptime": 869641, "state": 1, "adopted": true, "num_port": 52, "poe_power": 2150, "uplink_mac": "94:2a:6f:40:dc:6a", "uplink_name": "ECS Aggregation South" }, { "type": "usw", "name": "USW Pro Max 16 PoE | 18 (Desk)", "model": "USPM16P", "serial": "1C6A1B684E64", "mac": "1c:6a:1b:68:4e:64", "ip": "10.10.10.183", "version": "7.4.1.16850", "uptime": 785227, "state": 1, "adopted": true, "num_port": 18, "poe_power": 180, "uplink_mac": "8c:ed:e1:c2:3b:15", "uplink_name": "ECS 48 PoE | 15" }, { "type": "usw", "name": "ECS 48 PoE | 15", "model": "USWF069", "serial": "8CEDE1C23B15", "mac": "8c:ed:e1:c2:3b:15", "ip": "10.10.10.128", "version": "7.4.1.16850", "uptime": 1882325, "state": 1, "adopted": true, "num_port": 52, "poe_power": 2150, "uplink_mac": "94:2a:6f:40:e5:d2", "uplink_name": "ECS Aggregation North" }, { "type": "uap", "name": "E7 | Office (Middle Rear)", "model": "UAPA697", "serial": "8CEDE100EF2D", "mac": "8c:ed:e1:00:ef:2d", "ip": "10.10.10.160", "version": "8.6.11.18870", "uptime": 652521, "state": 1, "adopted": true, "num_sta": 6, "satisfaction": 100, "radios": [ { "band": "ng", "channel": 1, "ht": 20 }, { "band": "na", "channel": 149, "ht": "20" }, { "band": "6e", "channel": 37, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:37:6e", "uplink_name": "ECS 48 PoE | 14" }, { "type": "usw", "name": "UPS Tower", "model": "USWDA24", "serial": "1C0B8B3C157B", "mac": "1c:0b:8b:3c:15:7b", "ip": "10.10.10.71", "version": "1.5.0.378", "uptime": 573879, "state": 1, "adopted": true, "num_port": 1, "uplink_mac": "84:78:48:5c:84:e1", "uplink_name": "ECS 24 PoE | 16 (Demo DC Lab) " }, { "type": "usw", "name": "ECS 48 PoE | 5", "model": "USWF069", "serial": "8CEDE1C23DE0", "mac": "8c:ed:e1:c2:3d:e0", "ip": "10.10.10.57", "version": "7.4.1.16850", "uptime": 1184016, "state": 1, "adopted": true, "num_port": 52, "poe_power": 2150, "uplink_mac": "94:2a:6f:40:dc:6a", "uplink_name": "ECS Aggregation South" }, { "type": "udm", "name": "1 Triton", "model": "UDMEA4B", "serial": "A89C6CF4D2B8", "mac": "a8:9c:6c:f4:d2:b8", "ip": "165.65.48.198", "version": "5.1.19.33549", "uptime": 2581148, "state": 1, "adopted": true, "temp_cpu": 39.375, "uplink_mac": "8c:ed:e1:c2:3c:5f", "uplink_name": "ECS 48 PoE | 11 Artum" }, { "type": "uap", "name": "U7 Pro XG Wall | TEC-11 Cab", "model": "UAPA6BA", "serial": "9041B25243B0", "mac": "90:41:b2:52:43:b0", "ip": "10.10.10.253", "version": "8.6.11.18870", "uptime": 1035025, "state": 1, "adopted": true, "num_sta": 0, "satisfaction": -1, "radios": [ { "band": "ng", "channel": "auto", "ht": 20 }, { "band": "na", "channel": 104, "ht": "20" }, { "band": "6e", "channel": 41, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:3c:5f", "uplink_name": "ECS 48 PoE | 11 Artum" }, { "type": "usw", "name": "ECS 48 PoE | 11 Artum", "model": "USWF069", "serial": "8CEDE1C23C5F", "mac": "8c:ed:e1:c2:3c:5f", "ip": "10.10.10.65", "version": "7.4.1.16850", "uptime": 2225836, "state": 1, "adopted": true, "num_port": 52, "poe_power": 2150, "uplink_mac": "a8:9c:6c:f4:d2:b8", "uplink_name": "1 Triton" }, { "type": "uap", "name": "E7 | CEO (Right - Desk)", "model": "UAPA697", "serial": "8CEDE100D41D", "mac": "8c:ed:e1:00:d4:1d", "ip": "10.10.10.107", "version": "8.6.11.18870", "uptime": 1035261, "state": 1, "adopted": true, "num_sta": 2, "satisfaction": 98, "radios": [ { "band": "ng", "channel": 1, "ht": 20 }, { "band": "na", "channel": 64, "ht": "20" }, { "band": "6e", "channel": 29, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:3c:5f", "uplink_name": "ECS 48 PoE | 11 Artum" }, { "type": "uap", "name": "U7 Pro XG Wall | NE SER", "model": "UAPA6BA", "serial": "9041B2525D38", "mac": "90:41:b2:52:5d:38", "ip": "10.10.10.58", "version": "8.6.11.18870", "uptime": 0, "state": 0, "adopted": true, "num_sta": 0, "satisfaction": null, "radios": [ { "band": "na", "channel": 120, "ht": "20" }, { "band": "6e", "channel": "auto", "ht": "20" }, { "band": "ng", "channel": 5, "ht": "20" } ], "uplink_mac": "84:78:48:5a:39:8c", "uplink_name": "ECS 24 PoE | NE | MGMT " }, { "type": "uap", "name": "U7 Pro XG Wall | TEC-4 Cab", "model": "UAPA6BA", "serial": "9041B25247F4", "mac": "90:41:b2:52:47:f4", "ip": "10.10.10.22", "version": "8.6.11.18870", "uptime": 1183289, "state": 1, "adopted": true, "num_sta": 0, "satisfaction": -1, "radios": [ { "band": "ng", "channel": 13, "ht": 20 }, { "band": "na", "channel": 161, "ht": "20" }, { "band": "6e", "channel": 77, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:38:ef", "uplink_name": "ECS 48 PoE | 4" }, { "type": "uap", "name": "U7 Pro XG Wall | TEC-6 Cab", "model": "UAPA6BA", "serial": "9041B2524458", "mac": "90:41:b2:52:44:58", "ip": "10.10.10.28", "version": "8.6.11.18870", "uptime": 771164, "state": 1, "adopted": true, "num_sta": 0, "satisfaction": -1, "radios": [ { "band": "ng", "channel": 11, "ht": 20 }, { "band": "na", "channel": 64, "ht": "20" }, { "band": "6e", "channel": 85, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:3a:de", "uplink_name": "ECS 48 PoE | 6" }, { "type": "uap", "name": "E7 | Build Lab (front middle)", "model": "UAPA697", "serial": "8CEDE100CF25", "mac": "8c:ed:e1:00:cf:25", "ip": "10.10.10.53", "version": "8.6.11.18870", "uptime": 771186, "state": 1, "adopted": true, "num_sta": 1, "satisfaction": 100, "radios": [ { "band": "ng", "channel": 7, "ht": 20 }, { "band": "na", "channel": 157, "ht": "20" }, { "band": "6e", "channel": 57, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:3a:de", "uplink_name": "ECS 48 PoE | 6" }, { "type": "uap", "name": "U7 Pro XG Wall | TEC-2 Cab", "model": "UAPA6BA", "serial": "9041B2524BFC", "mac": "90:41:b2:52:4b:fc", "ip": "10.10.10.230", "version": "8.6.11.18870", "uptime": 235928, "state": 1, "adopted": true, "num_sta": 0, "satisfaction": -1, "radios": [ { "band": "ng", "channel": "auto", "ht": 20 }, { "band": "na", "channel": 124, "ht": "20" }, { "band": "6e", "channel": 49, "ht": "20" } ], "uplink_mac": "8c:30:66:94:40:92", "uplink_name": "USW Pro XG 24 PoE | 2 | Replacement" }, { "type": "uap", "name": "E7 | Build (Rear Right)", "model": "UAPA697", "serial": "8CEDE100EDD9", "mac": "8c:ed:e1:00:ed:d9", "ip": "10.10.10.108", "version": "8.6.11.18870", "uptime": 1175630, "state": 1, "adopted": true, "num_sta": 0, "satisfaction": -1, "radios": [ { "band": "ng", "channel": 7, "ht": 20 }, { "band": "na", "channel": 136, "ht": "20" }, { "band": "6e", "channel": 65, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:3e:f3", "uplink_name": "ECS 48 PoE | 7" }, { "type": "usw", "name": "ECS 24 PoE | SW | MGMT | 1", "model": "USWF067", "serial": "8478485A36D4", "mac": "84:78:48:5a:36:d4", "ip": "10.10.10.54", "version": "7.4.1.16850", "uptime": 3006272, "state": 1, "adopted": true, "num_port": 26, "poe_power": 1050, "uplink_mac": "94:2a:6f:40:dc:6a", "uplink_name": "ECS Aggregation South" }, { "type": "uap", "name": "E7 | Eng Lab (Front)", "model": "UAPA697", "serial": "8CEDE100D645", "mac": "8c:ed:e1:00:d6:45", "ip": "10.10.10.123", "version": "8.6.11.18870", "uptime": 1024921, "state": 1, "adopted": true, "num_sta": 0, "satisfaction": -1, "radios": [ { "band": "ng", "channel": 4, "ht": 20 }, { "band": "na", "channel": 157, "ht": "20" }, { "band": "6e", "channel": 61, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:3b:83", "uplink_name": "ECS 48 PoE | 12" }, { "type": "uap", "name": "U7 Pro XG Wall | TEC-15 Cab", "model": "UAPA6BA", "serial": "9041B2524510", "mac": "90:41:b2:52:45:10", "ip": "10.10.10.70", "version": "8.6.11.18870", "uptime": 757201, "state": 1, "adopted": true, "num_sta": 3, "satisfaction": 100, "radios": [ { "band": "ng", "channel": 11, "ht": 20 }, { "band": "na", "channel": 136, "ht": "20" }, { "band": "6e", "channel": 61, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:3b:15", "uplink_name": "ECS 48 PoE | 15" }, { "type": "uap", "name": "E7 | Office (Middle Front)", "model": "UAPA697", "serial": "1C0B8B00463D", "mac": "1c:0b:8b:00:46:3d", "ip": "10.10.10.243", "version": "8.6.11.18870", "uptime": 485211, "state": 1, "adopted": true, "num_sta": 10, "satisfaction": 100, "radios": [ { "band": "ng", "channel": 6, "ht": 20 }, { "band": "na", "channel": 40, "ht": "20" }, { "band": "6e", "channel": 89, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:37:6e", "uplink_name": "ECS 48 PoE | 14" }, { "type": "uap", "name": "E7 | Creative", "model": "UAPA697", "serial": "1C0B8B0047C5", "mac": "1c:0b:8b:00:47:c5", "ip": "10.10.10.242", "version": "8.6.11.18870", "uptime": 91181, "state": 1, "adopted": true, "num_sta": 0, "satisfaction": -1, "radios": [ { "band": "ng", "channel": "auto", "ht": 20 }, { "band": "na", "channel": 128, "ht": "20" }, { "band": "6e", "channel": 37, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:3e:f3", "uplink_name": "ECS 48 PoE | 7" }, { "type": "usw", "name": "ECS 48 PoE | 8", "model": "USWF069", "serial": "8CEDE1C23994", "mac": "8c:ed:e1:c2:39:94", "ip": "10.10.10.31", "version": "7.4.1.16850", "uptime": 2245898, "state": 1, "adopted": true, "num_port": 52, "poe_power": 2150, "uplink_mac": "94:2a:6f:40:dc:6a", "uplink_name": "ECS Aggregation South" }, { "type": "uap", "name": "E7 | Eng Lab (Middle Front NL3 )", "model": "UAPA697", "serial": "8CEDE100F459", "mac": "8c:ed:e1:00:f4:59", "ip": "10.10.10.16", "version": "8.6.11.18870", "uptime": 1035244, "state": 1, "adopted": true, "num_sta": 1, "satisfaction": 100, "radios": [ { "band": "ng", "channel": "auto", "ht": 20 }, { "band": "na", "channel": 165, "ht": 20 }, { "band": "6e", "channel": "auto", "ht": 20 } ], "uplink_mac": "8c:ed:e1:c2:3c:5f", "uplink_name": "ECS 48 PoE | 11 Artum" }, { "type": "uap", "name": "E7 | Eng Lab (Left Front)", "model": "UAPA697", "serial": "8CEDE100ED1D", "mac": "8c:ed:e1:00:ed:1d", "ip": "10.10.10.168", "version": "8.6.11.18870", "uptime": 1116056, "state": 1, "adopted": true, "num_sta": 1, "satisfaction": 96, "radios": [ { "band": "ng", "channel": 1, "ht": 20 }, { "band": "na", "channel": 144, "ht": "20" }, { "band": "6e", "channel": 65, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:39:cb", "uplink_name": "ECS 48 PoE | 9" }, { "type": "usw", "name": "UPS Tower | 16 ", "model": "USWDA24", "serial": "1C0B8B3C1425", "mac": "1c:0b:8b:3c:14:25", "ip": "10.10.10.121", "version": "1.5.0.378", "uptime": 918607, "state": 1, "adopted": true, "num_port": 1, "uplink_mac": "84:78:48:5c:84:e1", "uplink_name": "ECS 24 PoE | 16 (Demo DC Lab) " }, { "type": "uap", "name": "E7 | Office (Right Front)", "model": "UAPA697", "serial": "8CEDE100D40D", "mac": "8c:ed:e1:00:d4:0d", "ip": "10.10.10.91", "version": "8.6.11.18870", "uptime": 485238, "state": 1, "adopted": true, "num_sta": 1, "satisfaction": 95, "radios": [ { "band": "ng", "channel": 4, "ht": 20 }, { "band": "na", "channel": 108, "ht": "20" }, { "band": "6e", "channel": 53, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:3b:15", "uplink_name": "ECS 48 PoE | 15" }, { "type": "usw", "name": "ECS 48 PoE | 14", "model": "USWF069", "serial": "8CEDE1C2376E", "mac": "8c:ed:e1:c2:37:6e", "ip": "10.10.10.18", "version": "7.4.1.16850", "uptime": 1884407, "state": 1, "adopted": true, "num_port": 52, "poe_power": 2150, "uplink_mac": "94:2a:6f:40:e5:d2", "uplink_name": "ECS Aggregation North" }, { "type": "usw", "name": "UPS Tower", "model": "USWDA24", "serial": "1C0B8B3C1589", "mac": "1c:0b:8b:3c:15:89", "ip": "10.10.10.84", "version": "1.5.0.378", "uptime": 573977, "state": 1, "adopted": true, "num_port": 1, "uplink_mac": "84:78:48:5c:84:e1", "uplink_name": "ECS 24 PoE | 16 (Demo DC Lab) " }, { "type": "uap", "name": "U7 Pro XG Wall | TEC-12 Cab", "model": "UAPA6BA", "serial": "9041B2524AC8", "mac": "90:41:b2:52:4a:c8", "ip": "10.10.10.66", "version": "8.6.11.18870", "uptime": 1033187, "state": 1, "adopted": true, "num_sta": 0, "satisfaction": -1, "radios": [ { "band": "ng", "channel": 8, "ht": 20 }, { "band": "na", "channel": 128, "ht": "20" }, { "band": "6e", "channel": 53, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:3b:83", "uplink_name": "ECS 48 PoE | 12" }, { "type": "uap", "name": "E7 Campus | Float", "model": "UAPA6B1", "serial": "8C3066A2D8C7", "mac": "8c:30:66:a2:d8:c7", "ip": "192.168.1.209", "version": "8.6.11.18870", "uptime": 0, "state": 0, "adopted": true, "num_sta": 0, "satisfaction": null, "radios": [ { "band": "ng", "channel": "auto", "ht": 20 }, { "band": "na", "channel": "auto", "ht": 80 }, { "band": "na", "channel": "auto", "ht": 80 } ], "uplink_mac": "84:78:48:5a:3c:61", "uplink_name": "ECS 24 PoE | SW | 1" }, { "type": "uap", "name": "E7 | Build Lab (Rear Middle)", "model": "UAPA697", "serial": "8CEDE100E4C5", "mac": "8c:ed:e1:00:e4:c5", "ip": "10.10.10.76", "version": "8.6.11.18870", "uptime": 309061, "state": 1, "adopted": true, "num_sta": 0, "satisfaction": -1, "radios": [ { "band": "ng", "channel": 10, "ht": 20 }, { "band": "na", "channel": 104, "ht": "20" }, { "band": "6e", "channel": 9, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:3a:de", "uplink_name": "ECS 48 PoE | 6" }, { "type": "uap", "name": "E7 | Board Room (left)", "model": "UAPA697", "serial": "8CEDE100D569", "mac": "8c:ed:e1:00:d5:69", "ip": "10.10.10.46", "version": "8.6.11.18870", "uptime": 309048, "state": 1, "adopted": true, "num_sta": 0, "satisfaction": -1, "radios": [ { "band": "ng", "channel": 7, "ht": 20 }, { "band": "na", "channel": 52, "ht": "20" }, { "band": "6e", "channel": 65, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:3c:5f", "uplink_name": "ECS 48 PoE | 11 Artum" }, { "type": "uacc", "name": "AirWire", "model": "UAPEA07", "serial": "A89C6CA074C8", "mac": "a8:9c:6c:a0:74:c8", "ip": "", "version": "8.5.21.18681", "uptime": 0, "state": 2, "adopted": true, "uplink_mac": null, "uplink_name": null }, { "type": "uap", "name": "U7 Pro XG Wall | TEC-10 Cab", "model": "UAPA6BA", "serial": "9041B2525E74", "mac": "90:41:b2:52:5e:74", "ip": "10.10.10.229", "version": "8.6.11.18870", "uptime": 2571106, "state": 1, "adopted": true, "num_sta": 0, "satisfaction": -1, "radios": [ { "band": "ng", "channel": 5, "ht": 20 }, { "band": "na", "channel": 136, "ht": "20" }, { "band": "6e", "channel": 81, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:3e:4e", "uplink_name": "ECS 48 PoE | 10" }, { "type": "uap", "name": "U7 Pro XG Wall | TEC-1 Cab", "model": "UAPA6BA", "serial": "9041B2525070", "mac": "90:41:b2:52:50:70", "ip": "10.10.10.151", "version": "8.6.11.18870", "uptime": 220434, "state": 1, "adopted": true, "num_sta": 1, "satisfaction": 95, "radios": [ { "band": "ng", "channel": "auto", "ht": 20 }, { "band": "na", "channel": 100, "ht": "20" }, { "band": "6e", "channel": 93, "ht": "20" } ], "uplink_mac": "1c:0b:8b:08:d7:b3", "uplink_name": "USW Pro XG 8 PoE | 1 Replacement" }, { "type": "uap", "name": "E7 | Eng Lab (Front)", "model": "UAPA697", "serial": "8CEDE100E575", "mac": "8c:ed:e1:00:e5:75", "ip": "10.10.10.109", "version": "8.6.11.18870", "uptime": 485201, "state": 1, "adopted": true, "num_sta": 3, "satisfaction": 97, "radios": [ { "band": "ng", "channel": 12, "ht": 20 }, { "band": "na", "channel": 116, "ht": "20" }, { "band": "6e", "channel": 29, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:3e:4e", "uplink_name": "ECS 48 PoE | 10" }, { "type": "uap", "name": "E7 | Kitchen", "model": "UAPA697", "serial": "1C0B8B0046ED", "mac": "1c:0b:8b:00:46:ed", "ip": "10.10.10.171", "version": "8.6.11.18870", "uptime": 219230, "state": 1, "adopted": true, "num_sta": 2, "satisfaction": 100, "radios": [ { "band": "ng", "channel": "auto", "ht": 20 }, { "band": "na", "channel": 153, "ht": "20" }, { "band": "6e", "channel": 33, "ht": "20" } ], "uplink_mac": "90:41:b2:40:8a:0e", "uplink_name": "EAV XG 24 PoE | NE SER " }, { "type": "uap", "name": "E7 | Phone Booth (middle)", "model": "UAPA697", "serial": "8CEDE100D5FD", "mac": "8c:ed:e1:00:d5:fd", "ip": "10.10.10.194", "version": "8.6.11.18870", "uptime": 1181932, "state": 1, "adopted": true, "num_sta": 8, "satisfaction": 98, "radios": [ { "band": "ng", "channel": 3, "ht": 20 }, { "band": "na", "channel": 120, "ht": "20" }, { "band": "6e", "channel": 45, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:3a:de", "uplink_name": "ECS 48 PoE | 6" }, { "type": "uap", "name": "U7 Pro XG Wall | Meeting Room B", "model": "UAPA6BA", "serial": "9041B2524B40", "mac": "90:41:b2:52:4b:40", "ip": "10.10.10.41", "version": "8.6.11.18870", "uptime": 579806, "state": 1, "adopted": true, "num_sta": 1, "satisfaction": 100, "radios": [ { "band": "ng", "channel": 1, "ht": 20 }, { "band": "na", "channel": 48, "ht": "20" }, { "band": "6e", "channel": 45, "ht": "20" } ], "uplink_mac": "84:78:48:5a:3c:61", "uplink_name": "ECS 24 PoE | SW | Access | 1" }, { "type": "usw", "name": "EAV XG 24 PoE | SW SER", "model": "USWF001", "serial": "9041B2408C6E", "mac": "90:41:b2:40:8c:6e", "ip": "10.10.10.89", "version": "7.5.2.16967", "uptime": 3005302, "state": 1, "adopted": true, "num_port": 29, "poe_power": 2150, "uplink_mac": "94:2a:6f:40:dc:6a", "uplink_name": "ECS Aggregation South" }, { "type": "usw", "name": "ECS 48 PoE | 13", "model": "USWF069", "serial": "8CEDE1C23C96", "mac": "8c:ed:e1:c2:3c:96", "ip": "10.10.10.120", "version": "7.4.1.16850", "uptime": 1124417, "state": 1, "adopted": true, "num_port": 52, "poe_power": 2150, "uplink_mac": "94:2a:6f:40:e5:d2", "uplink_name": "ECS Aggregation North" }, { "type": "uap", "name": "E7 | Build Lab (Right Rear)", "model": "UAPA697", "serial": "8CEDE100EADD", "mac": "8c:ed:e1:00:ea:dd", "ip": "10.10.10.25", "version": "8.6.11.18870", "uptime": 333927, "state": 1, "adopted": true, "num_sta": 0, "satisfaction": -1, "radios": [ { "band": "ng", "channel": 2, "ht": 20 }, { "band": "na", "channel": 149, "ht": "20" }, { "band": "6e", "channel": 73, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:39:94", "uplink_name": "ECS 48 PoE | 8" }, { "type": "uap", "name": "U7 Pro XG Wall | Meet Room D", "model": "UAPA6BA", "serial": "9041B2524244", "mac": "90:41:b2:52:42:44", "ip": "10.10.10.33", "version": "8.6.11.18870", "uptime": 484789, "state": 1, "adopted": true, "num_sta": 0, "satisfaction": -1, "radios": [ { "band": "ng", "channel": 11, "ht": 20 }, { "band": "na", "channel": 132, "ht": "20" }, { "band": "6e", "channel": 81, "ht": "20" } ], "uplink_mac": "84:78:48:5a:3c:61", "uplink_name": "ECS 24 PoE | SW | Access | 1" }, { "type": "uap", "name": "E7 Lift Lobby ", "model": "UAPA697", "serial": "8CEDE100E63D", "mac": "8c:ed:e1:00:e6:3d", "ip": "10.10.10.163", "version": "8.6.11.18870", "uptime": 219191, "state": 1, "adopted": true, "num_sta": 2, "satisfaction": 99, "radios": [ { "band": "ng", "channel": "auto", "ht": 20 }, { "band": "na", "channel": 140, "ht": "20" }, { "band": "6e", "channel": 41, "ht": "20" } ], "uplink_mac": "90:41:b2:40:8a:0e", "uplink_name": "EAV XG 24 PoE | NE SER " }, { "type": "usw", "name": "USW Flex 2.5G 8 PoE | 17 (3D Printers)", "model": "USWED37", "serial": "942A6FFE5357", "mac": "94:2a:6f:fe:53:57", "ip": "10.10.10.193", "version": "2.1.8.971", "uptime": 614720, "state": 1, "adopted": true, "num_port": 10, "poe_power": 76, "uplink_mac": "8c:ed:e1:c2:3b:83", "uplink_name": "ECS 48 PoE | 12" }, { "type": "uap", "name": "E7 | Office (Left Front)", "model": "UAPA697", "serial": "8CEDE100F039", "mac": "8c:ed:e1:00:f0:39", "ip": "10.10.10.34", "version": "8.6.11.18870", "uptime": 253285, "state": 1, "adopted": true, "num_sta": 7, "satisfaction": 98, "radios": [ { "band": "ng", "channel": 6, "ht": 20 }, { "band": "na", "channel": 52, "ht": "20" }, { "band": "6e", "channel": 17, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:3c:96", "uplink_name": "ECS 48 PoE | 13" }, { "type": "uap", "name": "E7 | Reception", "model": "UAPA697", "serial": "8CEDE100BFDD", "mac": "8c:ed:e1:00:bf:dd", "ip": "10.10.10.189", "version": "8.6.11.18870", "uptime": 219751, "state": 1, "adopted": true, "num_sta": 0, "satisfaction": -1, "radios": [ { "band": "ng", "channel": "auto", "ht": 20 }, { "band": "na", "channel": 116, "ht": "20" }, { "band": "6e", "channel": 25, "ht": "20" } ], "uplink_mac": "1c:0b:8b:08:d7:b3", "uplink_name": "USW Pro XG 8 PoE | 1 Replacement" }, { "type": "uap", "name": "U7 Pro XG Wall | TEC-5 Cab", "model": "UAPA6BA", "serial": "9041B25242F0", "mac": "90:41:b2:52:42:f0", "ip": "10.10.10.206", "version": "8.6.11.18870", "uptime": 1183836, "state": 1, "adopted": true, "num_sta": 2, "satisfaction": 97, "radios": [ { "band": "ng", "channel": 7, "ht": 20 }, { "band": "na", "channel": 100, "ht": "20" }, { "band": "6e", "channel": 61, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:3d:e0", "uplink_name": "ECS 48 PoE | 5" }, { "type": "usw", "name": "USW Flex 2.5G 8 PoE", "model": "USWED37", "serial": "8C3066CC78AE", "mac": "8c:30:66:cc:78:ae", "ip": "192.168.1.106", "version": "2.1.8.971", "uptime": 0, "state": 0, "adopted": true, "num_port": 10, "uplink_mac": "8c:ed:e1:c2:37:6e", "uplink_name": "ECS 48 PoE | 14" }, { "type": "uap", "name": "E7 | Board Room (Right)", "model": "UAPA697", "serial": "8CEDE100D46D", "mac": "8c:ed:e1:00:d4:6d", "ip": "10.10.10.187", "version": "8.6.11.18870", "uptime": 1035328, "state": 1, "adopted": true, "num_sta": 0, "satisfaction": -1, "radios": [ { "band": "ng", "channel": 8, "ht": 20 }, { "band": "na", "channel": 153, "ht": "20" }, { "band": "6e", "channel": 85, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:3c:5f", "uplink_name": "ECS 48 PoE | 11 Artum" }, { "type": "uap", "name": "E7 | Demo DC Lab (Left Front)", "model": "UAPA697", "serial": "8CEDE100D261", "mac": "8c:ed:e1:00:d2:61", "ip": "10.10.10.200", "version": "8.6.11.18870", "uptime": 235949, "state": 1, "adopted": true, "num_sta": 6, "satisfaction": 100, "radios": [ { "band": "ng", "channel": 3, "ht": 20 }, { "band": "na", "channel": 132, "ht": "20" }, { "band": "6e", "channel": 25, "ht": "20" } ], "uplink_mac": "8c:30:66:94:40:92", "uplink_name": "USW Pro XG 24 PoE | 2 | Replacement" }, { "type": "uap", "name": "E7 | NE SER", "model": "UAPA697", "serial": "8CEDE100E64D", "mac": "8c:ed:e1:00:e6:4d", "ip": "10.10.10.179", "version": "8.6.11.18870", "uptime": 0, "state": 0, "adopted": true, "num_sta": 0, "satisfaction": null, "radios": [ { "band": "6e", "channel": "auto", "ht": "20" }, { "band": "ng", "channel": 13, "ht": "20" }, { "band": "na", "channel": 64, "ht": "40" } ], "uplink_mac": "84:78:48:5a:39:8c", "uplink_name": "ECS 24 PoE | NE | MGMT " }, { "type": "uap", "name": "U7 Pro XG Wall | TEC-9 Cab", "model": "UAPA6BA", "serial": "9041B2526130", "mac": "90:41:b2:52:61:30", "ip": "10.10.10.248", "version": "8.6.11.18870", "uptime": 2082533, "state": 1, "adopted": true, "num_sta": 0, "satisfaction": -1, "radios": [ { "band": "ng", "channel": 9, "ht": 20 }, { "band": "na", "channel": 56, "ht": "20" }, { "band": "6e", "channel": 17, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:39:cb", "uplink_name": "ECS 48 PoE | 9" }, { "type": "uap", "name": "E7 | Office (Right Rear)", "model": "UAPA697", "serial": "8CEDE100E885", "mac": "8c:ed:e1:00:e8:85", "ip": "10.10.10.210", "version": "8.6.11.18870", "uptime": 757223, "state": 1, "adopted": true, "num_sta": 2, "satisfaction": 100, "radios": [ { "band": "ng", "channel": 6, "ht": 20 }, { "band": "na", "channel": 128, "ht": "20" }, { "band": "6e", "channel": 73, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:3b:15", "uplink_name": "ECS 48 PoE | 15" }, { "type": "usw", "name": "ECS 48 PoE | 7", "model": "USWF069", "serial": "8CEDE1C23EF3", "mac": "8c:ed:e1:c2:3e:f3", "ip": "10.10.10.188", "version": "7.4.1.16850", "uptime": 2314872, "state": 1, "adopted": true, "num_port": 52, "poe_power": 2150, "uplink_mac": "94:2a:6f:40:dc:6a", "uplink_name": "ECS Aggregation South" }, { "type": "usw", "name": "USW Pro XG 24 PoE | 2 | Replacement", "model": "USWED44", "serial": "8C3066944092", "mac": "8c:30:66:94:40:92", "ip": "10.10.10.241", "version": "7.4.1.16850", "uptime": 236104, "state": 1, "adopted": true, "num_port": 26, "poe_power": 720, "uplink_mac": "94:2a:6f:40:dc:6a", "uplink_name": "ECS Aggregation South" }, { "type": "udb", "name": "UDB Switch | CDR ", "model": "UDBA69F", "serial": "8CEDE11C2E20", "mac": "8c:ed:e1:1c:2e:20", "ip": "10.10.10.88", "version": "8.5.21.18681", "uptime": 0, "state": 0, "adopted": true, "uplink_mac": "8c:ed:e1:c2:39:94", "uplink_name": "ECS 48 PoE | 8" }, { "type": "uap", "name": "U7 Pro XG Wall | Testing", "model": "UAPA6BA", "serial": "9041B252491C", "mac": "90:41:b2:52:49:1c", "ip": "10.10.10.32", "version": "8.6.11.18870", "uptime": 0, "state": 0, "adopted": true, "num_sta": 0, "satisfaction": null, "radios": [ { "band": "ng", "channel": "auto", "ht": 20 }, { "band": "na", "channel": 140, "ht": "20" }, { "band": "6e", "channel": 73, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:3a:de", "uplink_name": "ECS 48 PoE | 6" }, { "type": "uap", "name": "U7 Pro XG Wall | TEC-8 Cab", "model": "UAPA6BA", "serial": "9041B2526064", "mac": "90:41:b2:52:60:64", "ip": "10.10.10.190", "version": "8.6.11.18870", "uptime": 2245342, "state": 1, "adopted": true, "num_sta": 2, "satisfaction": 100, "radios": [ { "band": "ng", "channel": 13, "ht": 20 }, { "band": "na", "channel": 48, "ht": "20" }, { "band": "6e", "channel": 1, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:39:94", "uplink_name": "ECS 48 PoE | 8" }, { "type": "uap", "name": "E7 | Meeting Room A", "model": "UAPA697", "serial": "1C0B8B004745", "mac": "1c:0b:8b:00:47:45", "ip": "10.10.10.114", "version": "8.6.11.18870", "uptime": 606164, "state": 1, "adopted": true, "num_sta": 0, "satisfaction": -1, "radios": [ { "band": "ng", "channel": 5, "ht": 20 }, { "band": "na", "channel": 60, "ht": "20" }, { "band": "6e", "channel": 13, "ht": "20" } ], "uplink_mac": "84:78:48:5a:3c:61", "uplink_name": "ECS 24 PoE | SW | Access | 1" }, { "type": "uap", "name": "E7 | Demo DC 1 (Left Middle)", "model": "UAPA697", "serial": "8CEDE100D3C1", "mac": "8c:ed:e1:00:d3:c1", "ip": "10.10.10.159", "version": "8.6.11.18870", "uptime": 167056, "state": 1, "adopted": true, "num_sta": 8, "satisfaction": 95, "radios": [ { "band": "ng", "channel": "auto", "ht": 20 }, { "band": "na", "channel": 144, "ht": "20" }, { "band": "6e", "channel": 53, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:38:ef", "uplink_name": "ECS 48 PoE | 4" }, { "type": "uap", "name": "E7 | Product Deployment", "model": "UAPA697", "serial": "1C0B8B005065", "mac": "1c:0b:8b:00:50:65", "ip": "10.10.10.158", "version": "8.6.11.18870", "uptime": 90196, "state": 1, "adopted": true, "num_sta": 1, "satisfaction": 100, "radios": [ { "band": "ng", "channel": "auto", "ht": 20 }, { "band": "na", "channel": "auto", "ht": 20 }, { "band": "6e", "channel": "auto", "ht": 20 } ], "uplink_mac": "8c:ed:e1:c2:3e:f3", "uplink_name": "ECS 48 PoE | 7" }, { "type": "uap", "name": "E7 | Demo DC (Gantry Middle)", "model": "UAPA697", "serial": "8CEDE100E599", "mac": "8c:ed:e1:00:e5:99", "ip": "10.10.10.143", "version": "8.6.11.18870", "uptime": 57114, "state": 1, "adopted": true, "num_sta": 2, "satisfaction": 100, "radios": [ { "band": "ng", "channel": "auto", "ht": 20 }, { "band": "na", "channel": 52, "ht": "20" }, { "band": "6e", "channel": 85, "ht": "20" } ], "uplink_mac": "8c:30:66:94:40:92", "uplink_name": "USW Pro XG 24 PoE | 2 | Replacement" }, { "type": "uap", "name": "E7 | Demo DC (Front Middle)", "model": "UAPA697", "serial": "8CEDE100D541", "mac": "8c:ed:e1:00:d5:41", "ip": "10.10.10.6", "version": "8.6.11.18870", "uptime": 1186641, "state": 1, "adopted": true, "num_sta": 1, "satisfaction": 100, "radios": [ { "band": "ng", "channel": 12, "ht": 20 }, { "band": "na", "channel": 128, "ht": "20" }, { "band": "6e", "channel": 65, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:3d:a9", "uplink_name": "ECS 48 PoE | 3" }, { "type": "usw", "name": "ECS 48 PoE | 3", "model": "USWF069", "serial": "8CEDE1C23DA9", "mac": "8c:ed:e1:c2:3d:a9", "ip": "10.10.10.251", "version": "7.4.1.16850", "uptime": 1190955, "state": 1, "adopted": true, "num_port": 52, "poe_power": 2150, "uplink_mac": "94:2a:6f:40:dc:6a", "uplink_name": "ECS Aggregation South" }, { "type": "uap", "name": "E7 | Eng Lab (Right Front)", "model": "UAPA697", "serial": "8CEDE100CB25", "mac": "8c:ed:e1:00:cb:25", "ip": "10.10.10.103", "version": "8.6.11.18870", "uptime": 1095892, "state": 1, "adopted": true, "num_sta": 1, "satisfaction": 96, "radios": [ { "band": "ng", "channel": 5, "ht": 20 }, { "band": "na", "channel": 108, "ht": "20" }, { "band": "6e", "channel": 9, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:39:cb", "uplink_name": "ECS 48 PoE | 9" }, { "type": "uap", "name": "E7 Audience | SW Plant Room", "model": "UAPA6AB", "serial": "847848A09809", "mac": "84:78:48:a0:98:09", "ip": "10.10.10.231", "version": "8.6.11.18870", "uptime": 605588, "state": 1, "adopted": true, "num_sta": 0, "satisfaction": -1, "radios": [ { "band": "na", "channel": 124, "ht": "20" }, { "band": "6e", "channel": "auto", "ht": 20 }, { "band": "6e", "channel": 17, "ht": "20" } ], "uplink_mac": "84:78:48:5a:3c:61", "uplink_name": "ECS 24 PoE | SW | Access | 1" }, { "type": "uap", "name": "E7 | Build (Left Front)", "model": "UAPA697", "serial": "8CEDE100D691", "mac": "8c:ed:e1:00:d6:91", "ip": "10.10.10.198", "version": "8.6.11.18870", "uptime": 1174979, "state": 1, "adopted": true, "num_sta": 0, "satisfaction": -1, "radios": [ { "band": "ng", "channel": 9, "ht": 20 }, { "band": "na", "channel": 60, "ht": "20" }, { "band": "6e", "channel": 17, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:39:94", "uplink_name": "ECS 48 PoE | 8" }, { "type": "uap", "name": "U7 Pro XG Wall | TEC-7 Cab", "model": "UAPA6BA", "serial": "9041B2524260", "mac": "90:41:b2:52:42:60", "ip": "10.10.10.61", "version": "8.6.11.18870", "uptime": 484800, "state": 1, "adopted": true, "num_sta": 1, "satisfaction": 96, "radios": [ { "band": "ng", "channel": 3, "ht": 20 }, { "band": "na", "channel": 108, "ht": "20" }, { "band": "6e", "channel": 77, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:3e:f3", "uplink_name": "ECS 48 PoE | 7" }, { "type": "usw", "name": "UPS Tower", "model": "USWDA24", "serial": "1C0B8B3C145E", "mac": "1c:0b:8b:3c:14:5e", "ip": "192.168.1.150", "version": "1.5.0.378", "uptime": 0, "state": 0, "adopted": true, "num_port": 1, "uplink_mac": "8c:30:66:a2:5f:ee", "uplink_name": "USW Pro XG 8 PoE | Stores Rack" }, { "type": "uap", "name": "E7 | CEO (Left Door)", "model": "UAPA697", "serial": "1C0B8B009445", "mac": "1c:0b:8b:00:94:45", "ip": "10.10.10.24", "version": "8.6.11.18870", "uptime": 1035334, "state": 1, "adopted": true, "num_sta": 1, "satisfaction": 100, "radios": [ { "band": "ng", "channel": 13, "ht": 20 }, { "band": "na", "channel": 120, "ht": 20 }, { "band": "6e", "channel": 73, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:3c:5f", "uplink_name": "ECS 48 PoE | 11 Artum" }, { "type": "usw", "name": "ECS 48 PoE | 10", "model": "USWF069", "serial": "8CEDE1C23E4E", "mac": "8c:ed:e1:c2:3e:4e", "ip": "10.10.10.23", "version": "7.4.1.16850", "uptime": 2571555, "state": 1, "adopted": true, "num_port": 52, "poe_power": 2150, "uplink_mac": "94:2a:6f:40:e5:d2", "uplink_name": "ECS Aggregation North" }, { "type": "usw", "name": "ECS Aggregation South", "model": "USWF066", "serial": "942A6F40DC6A", "mac": "94:2a:6f:40:dc:6a", "ip": "10.10.10.208", "version": "3.0.8.852", "uptime": 615047, "state": 1, "adopted": true, "num_port": 54, "poe_power": 118, "uplink_mac": "a8:9c:6c:f4:d2:b8", "uplink_name": "1 Triton" }, { "type": "umbb", "name": "U5G Max Outdoor", "model": "UMBBE631", "serial": "8C3066D256DC", "mac": "8c:30:66:d2:56:dc", "ip": "10.10.10.181", "version": "7.4.1.19032", "uptime": 0, "state": 0, "adopted": true, "uplink_mac": "1c:6a:1b:68:4e:64", "uplink_name": "USW Pro Max 16 PoE | 18 (Desk)" }, { "type": "uap", "name": "U7 Pro XG Wall | TEC-13 Cab", "model": "UAPA6BA", "serial": "9041B25249C4", "mac": "90:41:b2:52:49:c4", "ip": "10.10.10.199", "version": "8.6.11.18870", "uptime": 1124253, "state": 1, "adopted": true, "num_sta": 2, "satisfaction": 98, "radios": [ { "band": "ng", "channel": 11, "ht": 20 }, { "band": "na", "channel": 112, "ht": "20" }, { "band": "6e", "channel": 1, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:3c:96", "uplink_name": "ECS 48 PoE | 13" }, { "type": "usw", "name": "USW Pro XG 8 PoE | 1 Replacement", "model": "USWED76", "serial": "1C0B8B08D7B3", "mac": "1c:0b:8b:08:d7:b3", "ip": "10.10.10.137", "version": "7.4.1.16850", "uptime": 221100, "state": 1, "adopted": true, "num_port": 10, "poe_power": 150, "uplink_mac": "94:2a:6f:40:dc:6a", "uplink_name": "ECS Aggregation South" }, { "type": "uap", "name": "U7 Pro XG Wall | SW SER (check)", "model": "UAPA6BA", "serial": "9041B2525D7C", "mac": "90:41:b2:52:5d:7c", "ip": "10.10.10.125", "version": "8.6.11.18870", "uptime": 0, "state": 0, "adopted": true, "num_sta": 0, "satisfaction": null, "radios": [ { "band": "na", "channel": "auto", "ht": "20" }, { "band": "6e", "channel": 77, "ht": "20" }, { "band": "ng", "channel": "auto", "ht": 20 } ], "uplink_mac": "90:41:b2:40:8c:6e", "uplink_name": "EAV XG 24 PoE | SW SER" }, { "type": "usw", "name": "ECS 24 PoE | 16 (Demo DC Lab) ", "model": "USWF067", "serial": "8478485C84E1", "mac": "84:78:48:5c:84:e1", "ip": "10.10.10.205", "version": "7.4.1.16850", "uptime": 918553, "state": 1, "adopted": true, "num_port": 26, "poe_power": 1050, "uplink_mac": "94:2a:6f:40:dc:6a", "uplink_name": "ECS Aggregation South" }, { "type": "uap", "name": "U7 Pro XG Wall | TEC-14 Cab", "model": "UAPA6BA", "serial": "9041B25249C0", "mac": "90:41:b2:52:49:c0", "ip": "10.10.10.195", "version": "8.6.11.18870", "uptime": 620422, "state": 1, "adopted": true, "num_sta": 2, "satisfaction": 100, "radios": [ { "band": "ng", "channel": 10, "ht": 20 }, { "band": "na", "channel": 124, "ht": "20" }, { "band": "6e", "channel": 77, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:37:6e", "uplink_name": "ECS 48 PoE | 14" }, { "type": "uap", "name": "E7 | Demo DC 2 (Right Middle)", "model": "UAPA697", "serial": "8CEDE100D749", "mac": "8c:ed:e1:00:d7:49", "ip": "10.10.10.239", "version": "8.6.11.18870", "uptime": 1183024, "state": 1, "adopted": true, "num_sta": 1, "satisfaction": 99, "radios": [ { "band": "ng", "channel": 6, "ht": 20 }, { "band": "na", "channel": 116, "ht": "20" }, { "band": "6e", "channel": 13, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:38:ef", "uplink_name": "ECS 48 PoE | 4" }, { "type": "uap", "name": "U7 Pro XG Wall | Meeting Room C", "model": "UAPA6BA", "serial": "9041B2524A28", "mac": "90:41:b2:52:4a:28", "ip": "10.10.10.154", "version": "8.6.11.18870", "uptime": 579952, "state": 1, "adopted": true, "num_sta": 0, "satisfaction": -1, "radios": [ { "band": "ng", "channel": 7, "ht": 20 }, { "band": "na", "channel": 64, "ht": "20" }, { "band": "6e", "channel": 61, "ht": "20" } ], "uplink_mac": "84:78:48:5a:3c:61", "uplink_name": "ECS 24 PoE | SW | Access | 1" }, { "type": "uap", "name": "E7 | Build 1 (Left Rear)", "model": "UAPA697", "serial": "8CEDE100D309", "mac": "8c:ed:e1:00:d3:09", "ip": "10.10.10.224", "version": "8.6.11.18870", "uptime": 1183813, "state": 1, "adopted": true, "num_sta": 0, "satisfaction": -1, "radios": [ { "band": "ng", "channel": 3, "ht": 20 }, { "band": "na", "channel": 112, "ht": "20" }, { "band": "6e", "channel": 81, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:3d:e0", "uplink_name": "ECS 48 PoE | 5" }, { "type": "usw", "name": "USW WAN | SW (Primary)", "model": "USWED74", "serial": "847848A2D420", "mac": "84:78:48:a2:d4:20", "ip": "10.10.10.119", "version": "7.3.109.16640", "uptime": 918936, "state": 1, "adopted": true, "num_port": 4, "uplink_mac": "84:78:48:5a:36:d4", "uplink_name": "ECS 24 PoE | SW | MGMT | 1" }, { "type": "usw", "name": "USW Pro XG 8 PoE | Stores Rack", "model": "USWED76", "serial": "8C3066A25FEE", "mac": "8c:30:66:a2:5f:ee", "ip": "10.10.10.142", "version": "7.4.1.16850", "uptime": 0, "state": 0, "adopted": true, "num_port": 10, "uplink_mac": "94:2a:6f:40:dc:6a", "uplink_name": "ECS Aggregation South" }, { "type": "uap", "name": "U7 Pro XG Wall | Meeting Room E", "model": "UAPA6BA", "serial": "9041B25244E4", "mac": "90:41:b2:52:44:e4", "ip": "10.10.10.169", "version": "8.6.11.18870", "uptime": 605689, "state": 1, "adopted": true, "num_sta": 0, "satisfaction": -1, "radios": [ { "band": "ng", "channel": 3, "ht": 20 }, { "band": "na", "channel": 124, "ht": "20" }, { "band": "6e", "channel": 69, "ht": "20" } ], "uplink_mac": "84:78:48:5a:3c:61", "uplink_name": "ECS 24 PoE | SW | Access | 1" }, { "type": "usw", "name": "ECS 48 PoE | 1", "model": "USWF069", "serial": "8C3066C27D78", "mac": "8c:30:66:c2:7d:78", "ip": "10.10.10.27", "version": "7.4.1.16850", "uptime": 0, "state": 0, "adopted": true, "num_port": 52, "uplink_mac": "94:2a:6f:40:dc:6a", "uplink_name": "ECS Aggregation South" }, { "type": "usw", "name": "EAV XG 24 PoE | NE SER ", "model": "USWF001", "serial": "9041B2408A0E", "mac": "90:41:b2:40:8a:0e", "ip": "10.10.10.19", "version": "7.5.2.16967", "uptime": 2579474, "state": 1, "adopted": true, "num_port": 29, "poe_power": 2150, "uplink_mac": "94:2a:6f:40:e5:d2", "uplink_name": "ECS Aggregation North" }, { "type": "uap", "name": "U7 Pro XG Wall | TEC-3 Cab", "model": "UAPA6BA", "serial": "9041B2524E70", "mac": "90:41:b2:52:4e:70", "ip": "10.10.10.176", "version": "8.6.11.18870", "uptime": 1190675, "state": 1, "adopted": true, "num_sta": 1, "satisfaction": 97, "radios": [ { "band": "ng", "channel": 8, "ht": 20 }, { "band": "na", "channel": 56, "ht": "20" }, { "band": "6e", "channel": 73, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:3d:a9", "uplink_name": "ECS 48 PoE | 3" }, { "type": "usw", "name": "USW WAN | SW (Secondary)", "model": "USWED74", "serial": "847848A2D4C0", "mac": "84:78:48:a2:d4:c0", "ip": "10.10.10.30", "version": "7.3.109.16640", "uptime": 2082041, "state": 1, "adopted": true, "num_port": 4, "uplink_mac": "84:78:48:5a:36:d4", "uplink_name": "ECS 24 PoE | SW | MGMT | 1" }, { "type": "uap", "name": "E7 | Cody Eng", "model": "UAPA697", "serial": "8CEDE100D675", "mac": "8c:ed:e1:00:d6:75", "ip": "10.10.10.170", "version": "8.6.11.18870", "uptime": 757205, "state": 1, "adopted": true, "num_sta": 4, "satisfaction": 100, "radios": [ { "band": "ng", "channel": 7, "ht": 20 }, { "band": "na", "channel": 116, "ht": "20" }, { "band": "6e", "channel": 81, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:3b:15", "uplink_name": "ECS 48 PoE | 15" }, { "type": "usw", "name": "ECS 48 PoE | 12", "model": "USWF069", "serial": "8CEDE1C23B83", "mac": "8c:ed:e1:c2:3b:83", "ip": "10.10.10.238", "version": "7.4.1.16850", "uptime": 2576529, "state": 1, "adopted": true, "num_port": 52, "poe_power": 2150, "uplink_mac": "94:2a:6f:40:e5:d2", "uplink_name": "ECS Aggregation North" }, { "type": "usw", "name": "ECS 48 PoE | 9", "model": "USWF069", "serial": "8CEDE1C239CB", "mac": "8c:ed:e1:c2:39:cb", "ip": "10.10.10.86", "version": "7.4.1.16850", "uptime": 2568713, "state": 1, "adopted": true, "num_port": 52, "poe_power": 2150, "uplink_mac": "94:2a:6f:40:e5:d2", "uplink_name": "ECS Aggregation North" }, { "type": "uap", "name": "E7 | Boris", "model": "UAPA697", "serial": "1C0B8B00BBAD", "mac": "1c:0b:8b:00:bb:ad", "ip": "10.10.10.13", "version": "8.6.11.18870", "uptime": 149877, "state": 1, "adopted": true, "num_sta": 1, "satisfaction": 99, "radios": [ { "band": "ng", "channel": 8, "ht": 20 }, { "band": "na", "channel": 64, "ht": "20" }, { "band": "6e", "channel": 9, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:37:6e", "uplink_name": "ECS 48 PoE | 14" }, { "type": "uap", "name": "E7 | Demo DC (Left Rear)", "model": "UAPA697", "serial": "8CEDE100D7CD", "mac": "8c:ed:e1:00:d7:cd", "ip": "10.10.10.122", "version": "8.6.11.18870", "uptime": 90257, "state": 1, "adopted": true, "num_sta": 0, "satisfaction": -1, "radios": [ { "band": "ng", "channel": "auto", "ht": 20 }, { "band": "na", "channel": 40, "ht": "20" }, { "band": "6e", "channel": 41, "ht": "20" } ], "uplink_mac": "8c:30:66:94:40:92", "uplink_name": "USW Pro XG 24 PoE | 2 | Replacement" }, { "type": "uap", "name": "E7 | Office ( Left Rear)", "model": "UAPA697", "serial": "1C0B8B00C1FD", "mac": "1c:0b:8b:00:c1:fd", "ip": "10.10.10.20", "version": "8.6.11.18870", "uptime": 485178, "state": 1, "adopted": true, "num_sta": 2, "satisfaction": 97, "radios": [ { "band": "ng", "channel": 9, "ht": 20 }, { "band": "na", "channel": 100, "ht": "20" }, { "band": "6e", "channel": 29, "ht": "20" } ], "uplink_mac": "8c:ed:e1:c2:3c:96", "uplink_name": "ECS 48 PoE | 13" }, { "type": "udb", "name": "UDB Switch | Stores", "model": "UDBA69F", "serial": "8CEDE11C2F18", "mac": "8c:ed:e1:1c:2f:18", "ip": "10.10.10.192", "version": "8.5.21.18681", "uptime": 0, "state": 0, "adopted": true, "uplink_mac": "8c:30:66:a2:5f:ee", "uplink_name": "USW Pro XG 8 PoE | Stores Rack" }, { "type": "usw", "name": "ECS Aggregation North", "model": "USWF066", "serial": "942A6F40E5D2", "mac": "94:2a:6f:40:e5:d2", "ip": "10.10.10.75", "version": "3.0.8.852", "uptime": 615247, "state": 1, "adopted": true, "num_port": 54, "poe_power": 117, "uplink_mac": "a8:9c:6c:f4:d2:b8", "uplink_name": "1 Triton" }, { "type": "usw", "name": "ECS 24 PoE | SW | Access | 1", "model": "USWF067", "serial": "8478485A3C61", "mac": "84:78:48:5a:3c:61", "ip": "10.10.10.113", "version": "7.4.1.16850", "uptime": 608290, "state": 1, "adopted": true, "num_port": 26, "poe_power": 1050, "uplink_mac": "94:2a:6f:40:dc:6a", "uplink_name": "ECS Aggregation South" }, { "type": "usw", "name": "USP RPS", "model": "USPRPS", "serial": "1C6A1BECAD6D", "mac": "1c:6a:1b:ec:ad:6d", "ip": "192.168.1.128", "version": "7.2.123.16565", "uptime": 672503, "state": 1, "adopted": true, "num_port": 1, "uplink_mac": "84:78:48:5a:36:d4", "uplink_name": "ECS 24 PoE | SW | MGMT | 1" }, { "type": "uap", "name": "E7 | SW SER", "model": "UAPA697", "serial": "8CEDE100D661", "mac": "8c:ed:e1:00:d6:61", "ip": "10.10.10.150", "version": "8.6.11.18870", "uptime": 602305, "state": 1, "adopted": true, "num_sta": 1, "satisfaction": 100, "radios": [ { "band": "ng", "channel": 4, "ht": 20 }, { "band": "na", "channel": 144, "ht": "20" }, { "band": "6e", "channel": 25, "ht": "20" } ], "uplink_mac": "84:78:48:5a:3c:61", "uplink_name": "ECS 24 PoE | SW | Access | 1" }, { "type": "usw", "name": "ECS 24 PoE", "model": "USWF067", "serial": "8478485A3DBD", "mac": "84:78:48:5a:3d:bd", "ip": "192.168.1.20", "version": "7.4.1.16850", "uptime": 262037, "state": 2, "adopted": false, "num_port": 0 }, { "type": "usw", "name": "ECS 24 PoE", "model": "USWF067", "serial": "8478485A398C", "mac": "84:78:48:5a:39:8c", "ip": "192.168.1.20", "version": "7.4.1.16850", "uptime": 262021, "state": 2, "adopted": false, "num_port": 0 } ], "inference_servers": [] }, "1-triton-efg": { "site_desc": null, "networks": [], "vpn_tunnels": [], "devices": [] }, "deploy-01": { "site_desc": null, "networks": [], "vpn_tunnels": [], "devices": [] }, "deploy-02": { "site_desc": null, "networks": [], "vpn_tunnels": [], "devices": [] }, "deploy-03": { "site_desc": null, "networks": [], "vpn_tunnels": [], "devices": [] }, "deploy-04": { "site_desc": null, "networks": [], "vpn_tunnels": [], "devices": [] }, "deploy-05": { "site_desc": null, "networks": [], "vpn_tunnels": [], "devices": [] }, "deploy-06": { "site_desc": null, "networks": [], "vpn_tunnels": [], "devices": [] }, "deploy-07": { "site_desc": null, "networks": [], "vpn_tunnels": [], "devices": [] }, "deploy-08": { "site_desc": null, "networks": [], "vpn_tunnels": [], "devices": [] } }, "wan_latency": null, "gw_version": null }, "inference": { "updated": "2026-07-25T08:26:56.421751+00:00", "clusters": [ { "name": "hmnd-ml-dev-eu-west-2", "region": "eu-west-2", "source": "aws+k8s", "cluster": { "status": "ACTIVE", "version": "1.33", "endpoint": "https://6F95294776BEAD181FAC5C4E01234F75.yl4.eu-west-2.eks.amazonaws.com", "platform_version": "eks.41" }, "nodegroups": [ { "name": "hmnd-ml-nodes-dev-eu-west-2", "status": "ACTIVE", "instance_types": [ "m5.xlarge", "m5.2xlarge", "c5.xlarge", "c5.2xlarge", "m5n.xlarge", "m5n.2xlarge", "m5.large", "c5.large" ], "desired": 3, "min": 2, "max": 20, "ami_type": "AL2023_x86_64_STANDARD", "capacity_type": "ON_DEMAND", "labels": {}, "health_issues": 0 }, { "name": "hmnd-ml-p5-dev-eu-west-2", "status": "ACTIVE", "instance_types": [ "p5.48xlarge" ], "desired": 6, "min": 6, "max": 6, "ami_type": "AL2023_x86_64_NVIDIA", "capacity_type": "ON_DEMAND", "labels": { "capacity-type": "reserved", "gpu-type": "h100", "az-placement": "eu-west-2b", "networking/efa": "true", "instance-family": "p5", "node-type": "gpu", "lustre-optimized": "true", "workload": "p5-reserved" }, "health_issues": 0 }, { "name": "hmnd-ml-p5-mig-dev-eu-west-2", "status": "ACTIVE", "instance_types": [ "p5.48xlarge" ], "desired": 2, "min": 2, "max": 2, "ami_type": "AL2023_x86_64_NVIDIA", "capacity_type": "ON_DEMAND", "labels": { "capacity-type": "reserved", "gpu-type": "h100", "az-placement": "eu-west-2b", "networking/efa": "true", "instance-family": "p5", "lustre-optimized": "true", "node-type": "gpu", "workload": "p5-reserved" }, "health_issues": 0 } ], "k8s": { "deployments": [ { "name": "bosch-tote-handling-09-03", "namespace": "inference", "replicas": 1, "ready": 1, "available": 1, "updated": 1, "images": [ "767828728064.dkr.ecr.eu-west-2.amazonaws.com/hmnd-training:006d95c" ], "created": "2026-03-13T18:00:29Z", "labels": { "app": "bosch-tote-handling-09-03", "app-type": "inference", "app.kubernetes.io/managed-by": "Helm", "model": "bosch-tote-handling-09-03", "release": "bosch-tote-handling-09-03" }, "resources": { "gpu_limit": "1", "cpu_request": "2", "memory_request": "8Gi" } }, { "name": "internal-demos-conveyor-18-11-map", "namespace": "inference", "replicas": 1, "ready": 1, "available": 1, "updated": 1, "images": [ "767828728064.dkr.ecr.eu-west-2.amazonaws.com/hmnd-training:4ce9e12" ], "created": "2026-03-26T12:24:19Z", "labels": { "app": "internal-demos-conveyor-18-11-map", "app-type": "inference", "app.kubernetes.io/managed-by": "Helm", "model": "internal-demos-conveyor-18-11-map", "release": "internal-demos-conveyor-18-11-map" }, "resources": { "gpu_limit": "1", "cpu_request": "2", "memory_request": "8Gi" } }, { "name": "main-2fd2-bosc-tote-09-03", "namespace": "inference", "replicas": 1, "ready": 1, "available": 1, "updated": 1, "images": [ "767828728064.dkr.ecr.eu-west-2.amazonaws.com/hmnd-training:2fd2c34" ], "created": "2026-04-17T09:40:10Z", "labels": { "app": "main-2fd2-bosc-tote-09-03", "app-type": "inference", "app.kubernetes.io/managed-by": "Helm", "model": "main-2fd2-bosc-tote-09-03", "release": "main-2fd2-bosch-tote-handling-09-03" }, "resources": { "gpu_limit": "1", "cpu_request": "2", "memory_request": "8Gi" } }, { "name": "main-bosc-tote-09-03", "namespace": "inference", "replicas": 1, "ready": 1, "available": 1, "updated": 1, "images": [ "767828728064.dkr.ecr.eu-west-2.amazonaws.com/hmnd-training:cb179df" ], "created": "2026-04-01T14:08:07Z", "labels": { "app": "main-bosc-tote-09-03", "app-type": "inference", "app.kubernetes.io/managed-by": "Helm", "model": "main-bosc-tote-09-03", "release": "main-bosch-tote-handling-09-03" }, "resources": { "gpu_limit": "1", "cpu_request": "2", "memory_request": "8Gi" } }, { "name": "main-pi0-robo-robostore", "namespace": "inference", "replicas": 1, "ready": 1, "available": 1, "updated": 1, "images": [ "767828728064.dkr.ecr.eu-west-2.amazonaws.com/hmnd-training:f5af6cc" ], "created": "2026-04-02T10:09:38Z", "labels": { "app": "main-pi0-robo-robostore", "app-type": "inference", "app.kubernetes.io/managed-by": "Helm", "model": "main-pi0-robo-robostore", "release": "main-robostore-2026-03-23" }, "resources": { "gpu_limit": "1", "cpu_request": "2", "memory_request": "8Gi" } }, { "name": "main-scha-mach-13-11-25-18k-map", "namespace": "inference", "replicas": 1, "ready": 1, "available": 1, "updated": 1, "images": [ "767828728064.dkr.ecr.eu-west-2.amazonaws.com/hmnd-training:f5af6cc" ], "created": "2026-04-01T13:59:50Z", "labels": { "app": "main-scha-mach-13-11-25-18k-map", "app-type": "inference", "app.kubernetes.io/managed-by": "Helm", "model": "main-scha-mach-13-11-25-18k-map", "release": "main-schaeffler-mf-13-11-25-18k-map" }, "resources": { "gpu_limit": "1", "cpu_request": "2", "memory_request": "8Gi" } }, { "name": "r20260302-pi0-symp-symphony-wa-with-termination", "namespace": "inference", "replicas": 1, "ready": 1, "available": 1, "updated": 1, "images": [ "767828728064.dkr.ecr.eu-west-2.amazonaws.com/hmnd-training:73290a8" ], "created": "2026-04-13T13:56:51Z", "labels": { "app": "r20260302-pi0-symp-symphony-wa-with-termination", "app-type": "inference", "app.kubernetes.io/managed-by": "Helm", "model": "r20260302-pi0-symp-symphony-wa-with-termination", "release": "r20260302-robostore-2026-03-23" }, "resources": { "gpu_limit": "1", "cpu_request": "2", "memory_request": "8Gi" } }, { "name": "r20260318-inte-conv-18-11-map", "namespace": "inference", "replicas": 1, "ready": 1, "available": 1, "updated": 1, "images": [ "767828728064.dkr.ecr.eu-west-2.amazonaws.com/hmnd-training:c991543" ], "created": "2026-04-02T11:25:11Z", "labels": { "app": "r20260318-inte-conv-18-11-map", "app-type": "inference", "app.kubernetes.io/managed-by": "Helm", "model": "r20260318-inte-conv-18-11-map", "release": "r20260318-internal-demos-conv-18-11-map" }, "resources": { "gpu_limit": "1", "cpu_request": "2", "memory_request": "8Gi" } }, { "name": "r20260318-scha-mach-13-11-25-18k-map", "namespace": "inference", "replicas": 1, "ready": 1, "available": 1, "updated": 1, "images": [ "767828728064.dkr.ecr.eu-west-2.amazonaws.com/hmnd-training:c991543" ], "created": "2026-04-02T11:25:17Z", "labels": { "app": "r20260318-scha-mach-13-11-25-18k-map", "app-type": "inference", "app.kubernetes.io/managed-by": "Helm", "model": "r20260318-scha-mach-13-11-25-18k-map", "release": "r20260318-schaeffler-mf-13-11-25-18k-map" }, "resources": { "gpu_limit": "1", "cpu_request": "2", "memory_request": "8Gi" } }, { "name": "r20260611-pi0-mach-schaeffler-alpha-rl", "namespace": "inference", "replicas": 1, "ready": 1, "available": 1, "updated": 1, "images": [ "767828728064.dkr.ecr.eu-west-2.amazonaws.com/hmnd-training:586bcb5" ], "created": "2026-06-12T13:25:31Z", "labels": { "app": "r20260611-pi0-mach-schaeffler-alpha-rl", "app-type": "inference", "app.kubernetes.io/managed-by": "Helm", "model": "r20260611-pi0-mach-schaeffler-alpha-rl", "release": "r20260611-schaeffler-mf-rl-27-may-6200" }, "resources": { "gpu_limit": "1", "cpu_request": "2", "memory_request": "8Gi" } }, { "name": "schaeffler-machine-feeding-13-11-25-18k-map", "namespace": "inference", "replicas": 1, "ready": 1, "available": 1, "updated": 1, "images": [ "767828728064.dkr.ecr.eu-west-2.amazonaws.com/hmnd-training:4ce9e12" ], "created": "2026-03-26T12:12:14Z", "labels": { "app": "schaeffler-machine-feeding-13-11-25-18k-map", "app-type": "inference", "app.kubernetes.io/managed-by": "Helm", "model": "schaeffler-machine-feeding-13-11-25-18k-map", "release": "schaeffler-machine-feeding-13-11-25-18k-map" }, "resources": { "gpu_limit": "1", "cpu_request": "2", "memory_request": "8Gi" } } ], "pods": [ { "name": "bosch-tote-handling-09-03-5f8fdfdd7d-ws9dh", "namespace": "inference", "phase": "Running", "node": "ip-10-3-95-87.eu-west-2.compute.internal", "restarts": 0, "ready": true, "images": [ "767828728064.dkr.ecr.eu-west-2.amazonaws.com/hmnd-training:006d95c" ], "started": "2026-06-02T12:08:51Z" }, { "name": "internal-demos-conveyor-18-11-map-7b587c7b96-9m5t6", "namespace": "inference", "phase": "Running", "node": "ip-10-3-95-87.eu-west-2.compute.internal", "restarts": 3, "ready": true, "images": [ "767828728064.dkr.ecr.eu-west-2.amazonaws.com/hmnd-training:4ce9e12" ], "started": "2026-05-22T12:37:02Z" }, { "name": "main-2fd2-bosc-tote-09-03-7dbfddf89f-s65dk", "namespace": "inference", "phase": "Failed", "node": "ip-10-3-92-36.eu-west-2.compute.internal", "restarts": 0, "ready": false, "images": [ "767828728064.dkr.ecr.eu-west-2.amazonaws.com/hmnd-training:2fd2c34" ], "started": "2026-07-13T06:36:23Z" }, { "name": "main-2fd2-bosc-tote-09-03-7dbfddf89f-zwrmk", "namespace": "inference", "phase": "Running", "node": "ip-10-3-92-36.eu-west-2.compute.internal", "restarts": 0, "ready": true, "images": [ "767828728064.dkr.ecr.eu-west-2.amazonaws.com/hmnd-training:2fd2c34" ], "started": "2026-07-13T07:43:58Z" }, { "name": "main-bosc-tote-09-03-6b8c6447db-9rsrx", "namespace": "inference", "phase": "Failed", "node": "ip-10-3-92-36.eu-west-2.compute.internal", "restarts": 0, "ready": false, "images": [], "started": "2026-07-13T06:36:24Z" }, { "name": "main-bosc-tote-09-03-6b8c6447db-rkfmt", "namespace": "inference", "phase": "Running", "node": "ip-10-3-92-36.eu-west-2.compute.internal", "restarts": 0, "ready": true, "images": [ "767828728064.dkr.ecr.eu-west-2.amazonaws.com/hmnd-training:cb179df" ], "started": "2026-07-13T07:44:06Z" }, { "name": "main-pi0-robo-robostore-69c8579f8d-xn77c", "namespace": "inference", "phase": "Running", "node": "ip-10-3-95-87.eu-west-2.compute.internal", "restarts": 4, "ready": true, "images": [ "767828728064.dkr.ecr.eu-west-2.amazonaws.com/hmnd-training:f5af6cc" ], "started": "2026-05-22T12:37:02Z" }, { "name": "main-scha-mach-13-11-25-18k-map-9b4fdcd46-62z46", "namespace": "inference", "phase": "Failed", "node": "ip-10-3-92-36.eu-west-2.compute.internal", "restarts": 0, "ready": false, "images": [], "started": "2026-07-13T06:36:23Z" }, { "name": "main-scha-mach-13-11-25-18k-map-9b4fdcd46-mn8zw", "namespace": "inference", "phase": "Running", "node": "ip-10-3-92-36.eu-west-2.compute.internal", "restarts": 0, "ready": true, "images": [ "767828728064.dkr.ecr.eu-west-2.amazonaws.com/hmnd-training:f5af6cc" ], "started": "2026-07-13T07:44:06Z" }, { "name": "r20260302-pi0-symp-symphony-wa-with-termination-6474ddf8bc2hbb2", "namespace": "inference", "phase": "Failed", "node": "ip-10-3-92-36.eu-west-2.compute.internal", "restarts": 0, "ready": false, "images": [], "started": "2026-07-13T06:36:23Z" }, { "name": "r20260302-pi0-symp-symphony-wa-with-termination-6474ddf8bcf4pkh", "namespace": "inference", "phase": "Running", "node": "ip-10-3-92-36.eu-west-2.compute.internal", "restarts": 0, "ready": true, "images": [ "767828728064.dkr.ecr.eu-west-2.amazonaws.com/hmnd-training:73290a8" ], "started": "2026-07-13T07:44:06Z" }, { "name": "r20260318-inte-conv-18-11-map-58c77f4f66-nbkb8", "namespace": "inference", "phase": "Running", "node": "ip-10-3-95-87.eu-west-2.compute.internal", "restarts": 0, "ready": true, "images": [ "767828728064.dkr.ecr.eu-west-2.amazonaws.com/hmnd-training:c991543" ], "started": "2026-07-13T06:36:23Z" }, { "name": "r20260318-scha-mach-13-11-25-18k-map-7695ffc97d-cpjn4", "namespace": "inference", "phase": "Running", "node": "ip-10-3-92-36.eu-west-2.compute.internal", "restarts": 0, "ready": true, "images": [ "767828728064.dkr.ecr.eu-west-2.amazonaws.com/hmnd-training:c991543" ], "started": "2026-07-13T07:43:58Z" }, { "name": "r20260318-scha-mach-13-11-25-18k-map-7695ffc97d-njnxd", "namespace": "inference", "phase": "Failed", "node": "ip-10-3-92-36.eu-west-2.compute.internal", "restarts": 0, "ready": false, "images": [ "767828728064.dkr.ecr.eu-west-2.amazonaws.com/hmnd-training:c991543" ], "started": "2026-07-13T06:36:23Z" }, { "name": "r20260611-pi0-mach-schaeffler-alpha-rl-648bc8576c-fq8x4", "namespace": "inference", "phase": "Running", "node": "ip-10-3-95-87.eu-west-2.compute.internal", "restarts": 0, "ready": true, "images": [ "767828728064.dkr.ecr.eu-west-2.amazonaws.com/hmnd-training:586bcb5" ], "started": "2026-06-12T13:25:31Z" }, { "name": "schaeffler-machine-feeding-13-11-25-18k-map-664cf66f68-vwv7p", "namespace": "inference", "phase": "Running", "node": "ip-10-3-95-87.eu-west-2.compute.internal", "restarts": 4, "ready": true, "images": [ "767828728064.dkr.ecr.eu-west-2.amazonaws.com/hmnd-training:4ce9e12" ], "started": "2026-05-22T12:37:01Z" } ], "ingresses": [ { "hostname": "bosch.tote-handling.09-03.eks-euw2.lan.thehumanoid.ai", "client": "bosch", "capability": "tote-handling", "version": "09-03", "cluster_alias": "eks-euw2", "name": "bosch-tote-handling-09-03", "paths": [ { "path": "/", "service": "bosch-tote-handling-09-03", "port": 9005 } ], "created": "2026-03-13T18:00:29Z", "annotations": { "kubernetes.io/ingress.class": "nginx", "nginx.ingress.kubernetes.io/backend-protocol": "GRPC", "nginx.ingress.kubernetes.io/grpc-backend": "true" }, "probe": { "latency_ms": 31, "status": "SERVING", "method": "grpc" } }, { "hostname": "internal-demos.conveyor.18-11-map.eks-euw2.lan.thehumanoid.ai", "client": "internal-demos", "capability": "conveyor", "version": "18-11-map", "cluster_alias": "eks-euw2", "name": "internal-demos-conveyor-18-11-map", "paths": [ { "path": "/", "service": "internal-demos-conveyor-18-11-map", "port": 9005 } ], "created": "2026-03-26T12:24:19Z", "annotations": { "kubernetes.io/ingress.class": "nginx", "nginx.ingress.kubernetes.io/backend-protocol": "GRPC", "nginx.ingress.kubernetes.io/grpc-backend": "true" }, "probe": { "latency_ms": 131, "status": "OK", "method": "https", "http_code": "200" } }, { "hostname": "main-2fd2.bosc.tote.09-03.eks-euw2.lan.thehumanoid.ai", "client": "main-2fd2", "capability": "bosc", "version": "tote", "cluster_alias": "09-03", "name": "main-2fd2-bosc-tote-09-03", "paths": [ { "path": "/", "service": "main-2fd2-bosc-tote-09-03", "port": 9005 } ], "created": "2026-04-17T09:40:10Z", "annotations": { "kubernetes.io/ingress.class": "nginx", "nginx.ingress.kubernetes.io/backend-protocol": "GRPC", "nginx.ingress.kubernetes.io/grpc-backend": "true" }, "probe": { "latency_ms": 35, "status": "SERVING", "method": "grpc" } }, { "hostname": "main.bosc.tote.09-03.eks-euw2.lan.thehumanoid.ai", "client": "main", "capability": "bosc", "version": "tote", "cluster_alias": "09-03", "name": "main-bosc-tote-09-03", "paths": [ { "path": "/", "service": "main-bosc-tote-09-03", "port": 9005 } ], "created": "2026-04-01T14:08:07Z", "annotations": { "kubernetes.io/ingress.class": "nginx", "nginx.ingress.kubernetes.io/backend-protocol": "GRPC", "nginx.ingress.kubernetes.io/grpc-backend": "true" }, "probe": { "latency_ms": 32, "status": "SERVING", "method": "grpc" } }, { "hostname": "main.pi0.robo.robostore.eks-euw2.lan.thehumanoid.ai", "client": "main", "capability": "pi0", "version": "robo", "cluster_alias": "robostore", "name": "main-pi0-robo-robostore", "paths": [ { "path": "/", "service": "main-pi0-robo-robostore", "port": 9005 } ], "created": "2026-04-02T10:09:38Z", "annotations": { "kubernetes.io/ingress.class": "nginx", "nginx.ingress.kubernetes.io/backend-protocol": "GRPC", "nginx.ingress.kubernetes.io/grpc-backend": "true" }, "probe": { "latency_ms": 232, "status": "OK", "method": "https", "http_code": "200" } }, { "hostname": "main.scha.mach.13-11-25-18k-map.eks-euw2.lan.thehumanoid.ai", "client": "main", "capability": "scha", "version": "mach", "cluster_alias": "13-11-25-18k-map", "name": "main-scha-mach-13-11-25-18k-map", "paths": [ { "path": "/", "service": "main-scha-mach-13-11-25-18k-map", "port": 9005 } ], "created": "2026-04-01T13:59:50Z", "annotations": { "kubernetes.io/ingress.class": "nginx", "nginx.ingress.kubernetes.io/backend-protocol": "GRPC", "nginx.ingress.kubernetes.io/grpc-backend": "true" }, "probe": { "latency_ms": 127, "status": "OK", "method": "https", "http_code": "200" } }, { "hostname": "r20260302.pi0.symp.symphony-wa-with-termination.eks-euw2.lan.thehumanoid.ai", "client": "r20260302", "capability": "pi0", "version": "symp", "cluster_alias": "symphony-wa-with-termination", "name": "r20260302-pi0-symp-symphony-wa-with-termination", "paths": [ { "path": "/", "service": "r20260302-pi0-symp-symphony-wa-with-termination", "port": 9005 } ], "created": "2026-04-13T13:56:51Z", "annotations": { "kubernetes.io/ingress.class": "nginx", "nginx.ingress.kubernetes.io/backend-protocol": "GRPC", "nginx.ingress.kubernetes.io/grpc-backend": "true" }, "probe": { "latency_ms": 207, "status": "OK", "method": "https", "http_code": "200" } }, { "hostname": "r20260318.inte.conv.18-11-map.eks-euw2.lan.thehumanoid.ai", "client": "r20260318", "capability": "inte", "version": "conv", "cluster_alias": "18-11-map", "name": "r20260318-inte-conv-18-11-map", "paths": [ { "path": "/", "service": "r20260318-inte-conv-18-11-map", "port": 9005 } ], "created": "2026-04-02T11:25:11Z", "annotations": { "kubernetes.io/ingress.class": "nginx", "nginx.ingress.kubernetes.io/backend-protocol": "GRPC", "nginx.ingress.kubernetes.io/grpc-backend": "true" }, "probe": { "latency_ms": 133, "status": "OK", "method": "https", "http_code": "200" } }, { "hostname": "r20260318.scha.mach.13-11-25-18k-map.eks-euw2.lan.thehumanoid.ai", "client": "r20260318", "capability": "scha", "version": "mach", "cluster_alias": "13-11-25-18k-map", "name": "r20260318-scha-mach-13-11-25-18k-map", "paths": [ { "path": "/", "service": "r20260318-scha-mach-13-11-25-18k-map", "port": 9005 } ], "created": "2026-04-02T11:25:17Z", "annotations": { "kubernetes.io/ingress.class": "nginx", "nginx.ingress.kubernetes.io/backend-protocol": "GRPC", "nginx.ingress.kubernetes.io/grpc-backend": "true" }, "probe": { "latency_ms": 157, "status": "OK", "method": "https", "http_code": "200" } }, { "hostname": "r20260611.pi0.mach.schaeffler-alpha-rl.eks-euw2.lan.thehumanoid.ai", "client": "r20260611", "capability": "pi0", "version": "mach", "cluster_alias": "schaeffler-alpha-rl", "name": "r20260611-pi0-mach-schaeffler-alpha-rl", "paths": [ { "path": "/", "service": "r20260611-pi0-mach-schaeffler-alpha-rl", "port": 9005 } ], "created": "2026-06-12T13:25:31Z", "annotations": { "kubernetes.io/ingress.class": "nginx", "nginx.ingress.kubernetes.io/backend-protocol": "GRPC", "nginx.ingress.kubernetes.io/grpc-backend": "true" }, "probe": { "latency_ms": 29, "status": "SERVING", "method": "grpc" } }, { "hostname": "schaeffler.machine-feeding.13-11-25-18k-map.eks-euw2.lan.thehumanoid.ai", "client": "schaeffler", "capability": "machine-feeding", "version": "13-11-25-18k-map", "cluster_alias": "eks-euw2", "name": "schaeffler-machine-feeding-13-11-25-18k-map", "paths": [ { "path": "/", "service": "schaeffler-machine-feeding-13-11-25-18k-map", "port": 9005 } ], "created": "2026-03-26T12:12:14Z", "annotations": { "kubernetes.io/ingress.class": "nginx", "nginx.ingress.kubernetes.io/backend-protocol": "GRPC", "nginx.ingress.kubernetes.io/grpc-backend": "true" }, "probe": { "latency_ms": 292, "status": "OK", "method": "https", "http_code": "200" } } ], "services": [ { "name": "bosch-tote-handling-09-03", "type": "ClusterIP", "cluster_ip": "172.20.214.97", "ports": [ { "port": 9005, "target": "grpc", "protocol": "TCP" } ] }, { "name": "internal-demos-conveyor-18-11-map", "type": "ClusterIP", "cluster_ip": "172.20.161.52", "ports": [ { "port": 9005, "target": "grpc", "protocol": "TCP" } ] }, { "name": "main-2fd2-bosc-tote-09-03", "type": "ClusterIP", "cluster_ip": "172.20.243.179", "ports": [ { "port": 9005, "target": "grpc", "protocol": "TCP" } ] }, { "name": "main-bosc-tote-09-03", "type": "ClusterIP", "cluster_ip": "172.20.47.168", "ports": [ { "port": 9005, "target": "grpc", "protocol": "TCP" } ] }, { "name": "main-pi0-robo-robostore", "type": "ClusterIP", "cluster_ip": "172.20.253.104", "ports": [ { "port": 9005, "target": "grpc", "protocol": "TCP" } ] }, { "name": "main-scha-mach-13-11-25-18k-map", "type": "ClusterIP", "cluster_ip": "172.20.189.3", "ports": [ { "port": 9005, "target": "grpc", "protocol": "TCP" } ] }, { "name": "r20260302-pi0-symp-symphony-wa-with-termination", "type": "ClusterIP", "cluster_ip": "172.20.116.243", "ports": [ { "port": 9005, "target": "grpc", "protocol": "TCP" } ] }, { "name": "r20260318-inte-conv-18-11-map", "type": "ClusterIP", "cluster_ip": "172.20.228.6", "ports": [ { "port": 9005, "target": "grpc", "protocol": "TCP" } ] }, { "name": "r20260318-scha-mach-13-11-25-18k-map", "type": "ClusterIP", "cluster_ip": "172.20.189.27", "ports": [ { "port": 9005, "target": "grpc", "protocol": "TCP" } ] }, { "name": "r20260611-pi0-mach-schaeffler-alpha-rl", "type": "ClusterIP", "cluster_ip": "172.20.32.119", "ports": [ { "port": 9005, "target": "grpc", "protocol": "TCP" }, { "port": 9213, "target": "metrics", "protocol": "TCP" } ] }, { "name": "schaeffler-machine-feeding-13-11-25-18k-map", "type": "ClusterIP", "cluster_ip": "172.20.92.103", "ports": [ { "port": 9005, "target": "grpc", "protocol": "TCP" } ] } ], "nodes": [ { "name": "ip-10-3-102-208.eu-west-2.compute.internal", "instance_type": "r7i.8xlarge", "zone": "eu-west-2c", "ready": true, "cpu": "32", "memory": "259756456Ki", "gpu": "0", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "31850m", "allocatable_memory": "256757160Ki", "os": "linux", "arch": "amd64", "node_group": "" }, { "name": "ip-10-3-102-31.eu-west-2.compute.internal", "instance_type": "r7i.12xlarge", "zone": "eu-west-2c", "ready": true, "cpu": "48", "memory": "389808280Ki", "gpu": "0", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "47810m", "allocatable_memory": "386808984Ki", "os": "linux", "arch": "amd64", "node_group": "" }, { "name": "ip-10-3-103-146.eu-west-2.compute.internal", "instance_type": "g6.8xlarge", "zone": "eu-west-2c", "ready": true, "cpu": "32", "memory": "127044216Ki", "gpu": "1", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "31850m", "allocatable_memory": "124044920Ki", "os": "linux", "arch": "amd64", "node_group": "" }, { "name": "ip-10-3-103-75.eu-west-2.compute.internal", "instance_type": "m6g.4xlarge", "zone": "eu-west-2c", "ready": true, "cpu": "16", "memory": "64391308Ki", "gpu": "0", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "15890m", "allocatable_memory": "61392012Ki", "os": "linux", "arch": "arm64", "node_group": "" }, { "name": "ip-10-3-105-201.eu-west-2.compute.internal", "instance_type": "m5.xlarge", "zone": "eu-west-2c", "ready": true, "cpu": "4", "memory": "15992748Ki", "gpu": "0", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "3800m", "allocatable_memory": "13690796Ki", "os": "linux", "arch": "amd64", "node_group": "hmnd-ml-nodes-dev-eu-west-2" }, { "name": "ip-10-3-106-21.eu-west-2.compute.internal", "instance_type": "r7i.8xlarge", "zone": "eu-west-2c", "ready": true, "cpu": "32", "memory": "259756464Ki", "gpu": "0", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "31850m", "allocatable_memory": "256757168Ki", "os": "linux", "arch": "amd64", "node_group": "" }, { "name": "ip-10-3-106-97.eu-west-2.compute.internal", "instance_type": "g6.8xlarge", "zone": "eu-west-2c", "ready": true, "cpu": "32", "memory": "127044224Ki", "gpu": "1", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "31850m", "allocatable_memory": "124044928Ki", "os": "linux", "arch": "amd64", "node_group": "" }, { "name": "ip-10-3-107-175.eu-west-2.compute.internal", "instance_type": "m6g.4xlarge", "zone": "eu-west-2c", "ready": true, "cpu": "16", "memory": "64391308Ki", "gpu": "0", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "15890m", "allocatable_memory": "61392012Ki", "os": "linux", "arch": "arm64", "node_group": "" }, { "name": "ip-10-3-64-123.eu-west-2.compute.internal", "instance_type": "g7e.2xlarge", "zone": "eu-west-2a", "ready": true, "cpu": "8", "memory": "65293116Ki", "gpu": "1", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "7910m", "allocatable_memory": "62068540Ki", "os": "linux", "arch": "amd64", "node_group": "" }, { "name": "ip-10-3-64-143.eu-west-2.compute.internal", "instance_type": "g7e.2xlarge", "zone": "eu-west-2a", "ready": true, "cpu": "8", "memory": "65293120Ki", "gpu": "1", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "7910m", "allocatable_memory": "62068544Ki", "os": "linux", "arch": "amd64", "node_group": "" }, { "name": "ip-10-3-64-97.eu-west-2.compute.internal", "instance_type": "g6.8xlarge", "zone": "eu-west-2a", "ready": true, "cpu": "32", "memory": "127044220Ki", "gpu": "1", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "31850m", "allocatable_memory": "124044924Ki", "os": "linux", "arch": "amd64", "node_group": "" }, { "name": "ip-10-3-65-146.eu-west-2.compute.internal", "instance_type": "g7e.2xlarge", "zone": "eu-west-2a", "ready": true, "cpu": "8", "memory": "65293124Ki", "gpu": "1", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "7910m", "allocatable_memory": "62068548Ki", "os": "linux", "arch": "amd64", "node_group": "" }, { "name": "ip-10-3-66-140.eu-west-2.compute.internal", "instance_type": "g6.8xlarge", "zone": "eu-west-2a", "ready": true, "cpu": "32", "memory": "127044220Ki", "gpu": "1", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "31850m", "allocatable_memory": "124044924Ki", "os": "linux", "arch": "amd64", "node_group": "" }, { "name": "ip-10-3-70-69.eu-west-2.compute.internal", "instance_type": "m5.xlarge", "zone": "eu-west-2a", "ready": true, "cpu": "4", "memory": "16164776Ki", "gpu": "0", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "3800m", "allocatable_memory": "13862824Ki", "os": "linux", "arch": "amd64", "node_group": "hmnd-ml-nodes-dev-eu-west-2" }, { "name": "ip-10-3-72-29.eu-west-2.compute.internal", "instance_type": "r7i.16xlarge", "zone": "eu-west-2a", "ready": true, "cpu": "64", "memory": "519860100Ki", "gpu": "0", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "63770m", "allocatable_memory": "511195012Ki", "os": "linux", "arch": "amd64", "node_group": "" }, { "name": "ip-10-3-73-175.eu-west-2.compute.internal", "instance_type": "g7e.2xlarge", "zone": "eu-west-2a", "ready": true, "cpu": "8", "memory": "65293156Ki", "gpu": "1", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "7910m", "allocatable_memory": "62068580Ki", "os": "linux", "arch": "amd64", "node_group": "" }, { "name": "ip-10-3-74-199.eu-west-2.compute.internal", "instance_type": "m6g.4xlarge", "zone": "eu-west-2a", "ready": true, "cpu": "16", "memory": "64391308Ki", "gpu": "0", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "15890m", "allocatable_memory": "61392012Ki", "os": "linux", "arch": "arm64", "node_group": "" }, { "name": "ip-10-3-80-127.eu-west-2.compute.internal", "instance_type": "r7i.8xlarge", "zone": "eu-west-2b", "ready": true, "cpu": "32", "memory": "259756460Ki", "gpu": "0", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "31850m", "allocatable_memory": "256757164Ki", "os": "linux", "arch": "amd64", "node_group": "" }, { "name": "ip-10-3-80-194.eu-west-2.compute.internal", "instance_type": "r7i.8xlarge", "zone": "eu-west-2b", "ready": true, "cpu": "32", "memory": "259756572Ki", "gpu": "0", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "31850m", "allocatable_memory": "256757276Ki", "os": "linux", "arch": "amd64", "node_group": "" }, { "name": "ip-10-3-80-81.eu-west-2.compute.internal", "instance_type": "m6g.4xlarge", "zone": "eu-west-2b", "ready": true, "cpu": "16", "memory": "64391308Ki", "gpu": "0", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "15890m", "allocatable_memory": "61392012Ki", "os": "linux", "arch": "arm64", "node_group": "" }, { "name": "ip-10-3-81-145.eu-west-2.compute.internal", "instance_type": "p5.48xlarge", "zone": "eu-west-2b", "ready": true, "cpu": "192", "memory": "2097112408Ki", "gpu": "8", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "188", "allocatable_memory": "2082227544Ki", "os": "linux", "arch": "amd64", "node_group": "hmnd-ml-p5-dev-eu-west-2" }, { "name": "ip-10-3-83-140.eu-west-2.compute.internal", "instance_type": "r7i.8xlarge", "zone": "eu-west-2b", "ready": true, "cpu": "32", "memory": "259756580Ki", "gpu": "0", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "31850m", "allocatable_memory": "256757284Ki", "os": "linux", "arch": "amd64", "node_group": "" }, { "name": "ip-10-3-83-17.eu-west-2.compute.internal", "instance_type": "p5.48xlarge", "zone": "eu-west-2b", "ready": true, "cpu": "192", "memory": "2097112420Ki", "gpu": "8", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "188", "allocatable_memory": "2082227556Ki", "os": "linux", "arch": "amd64", "node_group": "hmnd-ml-p5-dev-eu-west-2" }, { "name": "ip-10-3-83-49.eu-west-2.compute.internal", "instance_type": "p5.48xlarge", "zone": "eu-west-2b", "ready": true, "cpu": "192", "memory": "2097112360Ki", "gpu": "6", "gpu_mig": "6", "gpu_product": "", "allocatable_cpu": "188", "allocatable_memory": "2090616104Ki", "os": "linux", "arch": "amd64", "node_group": "hmnd-ml-p5-dev-eu-west-2" }, { "name": "ip-10-3-84-128.eu-west-2.compute.internal", "instance_type": "g6.8xlarge", "zone": "eu-west-2b", "ready": true, "cpu": "32", "memory": "127044216Ki", "gpu": "1", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "31850m", "allocatable_memory": "124044920Ki", "os": "linux", "arch": "amd64", "node_group": "" }, { "name": "ip-10-3-84-253.eu-west-2.compute.internal", "instance_type": "g6.8xlarge", "zone": "eu-west-2b", "ready": true, "cpu": "32", "memory": "127044088Ki", "gpu": "1", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "31850m", "allocatable_memory": "124044792Ki", "os": "linux", "arch": "amd64", "node_group": "" }, { "name": "ip-10-3-86-106.eu-west-2.compute.internal", "instance_type": "g6.8xlarge", "zone": "eu-west-2b", "ready": true, "cpu": "32", "memory": "127044088Ki", "gpu": "1", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "31850m", "allocatable_memory": "124044792Ki", "os": "linux", "arch": "amd64", "node_group": "" }, { "name": "ip-10-3-87-48.eu-west-2.compute.internal", "instance_type": "p5.48xlarge", "zone": "eu-west-2b", "ready": true, "cpu": "192", "memory": "2097112392Ki", "gpu": "8", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "188", "allocatable_memory": "2082227528Ki", "os": "linux", "arch": "amd64", "node_group": "hmnd-ml-p5-dev-eu-west-2" }, { "name": "ip-10-3-89-197.eu-west-2.compute.internal", "instance_type": "p5.48xlarge", "zone": "eu-west-2b", "ready": true, "cpu": "192", "memory": "2097112436Ki", "gpu": "8", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "188", "allocatable_memory": "2082227572Ki", "os": "linux", "arch": "amd64", "node_group": "hmnd-ml-p5-dev-eu-west-2" }, { "name": "ip-10-3-90-48.eu-west-2.compute.internal", "instance_type": "g6.8xlarge", "zone": "eu-west-2b", "ready": true, "cpu": "32", "memory": "127044088Ki", "gpu": "1", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "31850m", "allocatable_memory": "124044792Ki", "os": "linux", "arch": "amd64", "node_group": "" }, { "name": "ip-10-3-91-97.eu-west-2.compute.internal", "instance_type": "p5.48xlarge", "zone": "eu-west-2b", "ready": true, "cpu": "192", "memory": "2097112408Ki", "gpu": "8", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "188", "allocatable_memory": "2082227544Ki", "os": "linux", "arch": "amd64", "node_group": "hmnd-ml-p5-dev-eu-west-2" }, { "name": "ip-10-3-92-103.eu-west-2.compute.internal", "instance_type": "m5.xlarge", "zone": "eu-west-2b", "ready": true, "cpu": "4", "memory": "15992744Ki", "gpu": "0", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "3800m", "allocatable_memory": "13690792Ki", "os": "linux", "arch": "amd64", "node_group": "hmnd-ml-nodes-dev-eu-west-2" }, { "name": "ip-10-3-92-124.eu-west-2.compute.internal", "instance_type": "r7i.8xlarge", "zone": "eu-west-2b", "ready": true, "cpu": "32", "memory": "259756580Ki", "gpu": "0", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "31850m", "allocatable_memory": "256757284Ki", "os": "linux", "arch": "amd64", "node_group": "" }, { "name": "ip-10-3-92-36.eu-west-2.compute.internal", "instance_type": "p5.48xlarge", "zone": "eu-west-2b", "ready": true, "cpu": "192", "memory": "2097112352Ki", "gpu": "4", "gpu_mig": "6", "gpu_product": "", "allocatable_cpu": "188", "allocatable_memory": "2090616096Ki", "os": "linux", "arch": "amd64", "node_group": "hmnd-ml-p5-mig-dev-eu-west-2" }, { "name": "ip-10-3-95-87.eu-west-2.compute.internal", "instance_type": "p5.48xlarge", "zone": "eu-west-2b", "ready": true, "cpu": "192", "memory": "2097112348Ki", "gpu": "4", "gpu_mig": "6", "gpu_product": "", "allocatable_cpu": "188", "allocatable_memory": "2082227484Ki", "os": "linux", "arch": "amd64", "node_group": "hmnd-ml-p5-mig-dev-eu-west-2" }, { "name": "ip-10-3-97-208.eu-west-2.compute.internal", "instance_type": "c6i.xlarge", "zone": "eu-west-2c", "ready": true, "cpu": "4", "memory": "7964568Ki", "gpu": "0", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "3920m", "allocatable_memory": "6947736Ki", "os": "linux", "arch": "amd64", "node_group": "" } ], "namespaces": [ "agent-system", "agent-system-staging", "agent-system-test", "arc-runners", "arc-system", "bazel-remote", "cronjobs", "default", "dev", "efa-testing", "hmnd-rl", "inference", "kamino-dev", "kamino-dev-system", "kamino-system", "karpenter", "kube-node-lease", "kube-public", "kube-system", "kyverno", "logging", "longhorn-system", "monitoring", "prefect", "prefect-ml", "prod", "ray", "ray-system", "shared-storage", "simulation", "stray-dog", "vlm-proxy-dev", "volcano", "volcano-system" ] } }, { "name": "hmnd-eks-cluster", "region": "eu-west-2", "source": "aws+k8s", "cluster": { "status": "ACTIVE", "version": "1.33", "endpoint": "https://A4DC8392E035C92809FE94069A7D7999.gr7.eu-west-2.eks.amazonaws.com", "platform_version": "eks.41" }, "nodegroups": [ { "name": "hmnd-eks-cluster-node-group-a", "status": "ACTIVE", "instance_types": [ "t3.xlarge", "t3.2xlarge", "m5.xlarge" ], "desired": 0, "min": 0, "max": 1, "ami_type": "AL2023_x86_64_STANDARD", "capacity_type": "ON_DEMAND", "labels": {}, "health_issues": 0 }, { "name": "hmnd-eks-cluster-node-group-b", "status": "ACTIVE", "instance_types": [ "t3.xlarge", "t3.2xlarge", "m5.xlarge" ], "desired": 0, "min": 0, "max": 1, "ami_type": "AL2023_x86_64_STANDARD", "capacity_type": "ON_DEMAND", "labels": {}, "health_issues": 0 }, { "name": "hmnd-eks-cluster-node-group-c", "status": "ACTIVE", "instance_types": [ "t3.xlarge", "t3.2xlarge", "m5.xlarge" ], "desired": 0, "min": 0, "max": 1, "ami_type": "AL2023_x86_64_STANDARD", "capacity_type": "ON_DEMAND", "labels": {}, "health_issues": 0 } ], "k8s": { "deployments": [], "pods": [], "ingresses": [], "services": [], "nodes": [ { "name": "ip-10-253-31-169.eu-west-2.compute.internal", "instance_type": "i7ie.3xlarge", "zone": "eu-west-2b", "ready": false, "cpu": "12", "memory": "97283312Ki", "gpu": "0", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "11900m", "allocatable_memory": "96266480Ki", "os": "linux", "arch": "amd64", "node_group": "" }, { "name": "ip-10-253-38-213.eu-west-2.compute.internal", "instance_type": "t2.small", "zone": "eu-west-2c", "ready": false, "cpu": "1", "memory": "2002832Ki", "gpu": "0", "gpu_mig": "0", "gpu_product": "", "allocatable_cpu": "940m", "allocatable_memory": "1515408Ki", "os": "linux", "arch": "amd64", "node_group": "" } ], "namespaces": [ "default", "humanoid-console", "karpenter", "kube-node-lease", "kube-public", "kube-system", "monitoring", "ops", "perception", "reasoning", "shared-between-teams", "simulation", "skypilot", "skypilot-system", "vla" ] } } ], "onprem_clusters": [ { "name": "h100-01", "fqdn": "h100-01.lan.thehumanoid.ai", "ip": "10.44.11.101", "arch": "amd64", "gpu_type": "H100", "location": "lab", "source": "onprem" }, { "name": "h100-02", "fqdn": "h100-02.lan.thehumanoid.ai", "ip": "10.44.11.201", "arch": "amd64", "gpu_type": "H100", "location": "lab", "source": "onprem" }, { "name": "spark-01", "fqdn": "spark-01.lan.thehumanoid.ai", "ip": "10.44.11.12", "arch": "arm64", "gpu_type": "Spark", "location": "lab", "source": "onprem" }, { "name": "saudi-01", "fqdn": "saudi-01.lan.thehumanoid.ai", "ip": "158.252.119.42", "arch": "amd64", "gpu_type": "A10G", "location": "me-central-1", "source": "onprem+k8s", "api_latency_ms": 769, "k8s": { "deployments": [ { "name": "internal-demos-conveyor-18-11-map", "namespace": "inference", "replicas": 1, "ready": 1, "available": 1, "updated": 1, "images": [ "767828728064.dkr.ecr.eu-west-2.amazonaws.com/hmnd-training:latest" ], "created": "2026-01-23T14:12:04Z", "labels": { "app": "internal-demos-conveyor-18-11-map", "app-type": "inference", "app.kubernetes.io/managed-by": "Helm", "model": "internal-demos-conveyor-18-11-map", "release": "internal-demos-conveyor-18-11-map" }, "resources": { "gpu_limit": "1", "cpu_request": "4", "memory_request": "32Gi" } }, { "name": "r20260219-inte-conv-18-11-map", "namespace": "inference", "replicas": 1, "ready": 1, "available": 1, "updated": 1, "images": [ "767828728064.dkr.ecr.eu-west-2.amazonaws.com/hmnd-training:4ce9e12" ], "created": "2026-04-21T10:59:51Z", "labels": { "app": "r20260219-inte-conv-18-11-map", "app-type": "inference", "app.kubernetes.io/managed-by": "Helm", "model": "r20260219-inte-conv-18-11-map", "release": "r20260219-internal-demos-conv-18-11-map" }, "resources": { "gpu_limit": "1", "cpu_request": "4", "memory_request": "32Gi" } } ], "pods": [ { "name": "internal-demos-conveyor-18-11-map-5575469d6b-flwxl", "namespace": "inference", "phase": "Running", "node": "ip-172-31-33-154", "restarts": 0, "ready": true, "images": [ "767828728064.dkr.ecr.eu-west-2.amazonaws.com/hmnd-training:latest" ], "started": "2026-02-09T18:52:45Z" }, { "name": "r20260219-inte-conv-18-11-map-76d59c7c57-lgtv2", "namespace": "inference", "phase": "Running", "node": "ip-172-31-38-130", "restarts": 0, "ready": true, "images": [ "767828728064.dkr.ecr.eu-west-2.amazonaws.com/hmnd-training:4ce9e12" ], "started": "2026-04-21T10:59:51Z" } ], "ingresses": [ { "hostname": "ford.tote-handling.pi0-25-09-29-map.saudi-01.lan.thehumanoid.ai", "client": "ford", "capability": "tote-handling", "version": "pi0-25-09-29-map", "cluster_alias": "saudi-01", "name": "ford-tote-handling-pi0-25-09-29-map", "paths": [ { "path": "/", "service": "ford-tote-handling-pi0-25-09-29-map", "port": 9005 } ], "created": "2026-01-23T09:58:09Z", "annotations": { "kubernetes.io/ingress.class": "nginx", "nginx.ingress.kubernetes.io/backend-protocol": "GRPC", "nginx.ingress.kubernetes.io/grpc-backend": "true" }, "probe": { "latency_ms": 370, "status": "OK", "method": "https", "http_code": "503" } }, { "hostname": "internal-demos.conveyor.18-11-map.saudi-01.lan.thehumanoid.ai", "client": "internal-demos", "capability": "conveyor", "version": "18-11-map", "cluster_alias": "saudi-01", "name": "internal-demos-conveyor-18-11-map", "paths": [ { "path": "/", "service": "internal-demos-conveyor-18-11-map", "port": 9005 } ], "created": "2026-01-23T14:12:04Z", "annotations": { "kubernetes.io/ingress.class": "nginx", "nginx.ingress.kubernetes.io/backend-protocol": "GRPC", "nginx.ingress.kubernetes.io/grpc-backend": "true" }, "probe": { "latency_ms": 480, "status": "OK", "method": "https", "http_code": "200" } }, { "hostname": "r20260219.inte.conv.18-11-map.saudi-01.lan.thehumanoid.ai", "client": "r20260219", "capability": "inte", "version": "conv", "cluster_alias": "18-11-map", "name": "r20260219-inte-conv-18-11-map", "paths": [ { "path": "/", "service": "r20260219-inte-conv-18-11-map", "port": 9000 } ], "created": "2026-04-21T10:59:51Z", "annotations": { "kubernetes.io/ingress.class": "nginx", "nginx.ingress.kubernetes.io/backend-protocol": "GRPC", "nginx.ingress.kubernetes.io/grpc-backend": "true" }, "probe": { "latency_ms": 870, "status": "OK", "method": "https", "http_code": "200" } } ], "services": [ { "name": "ford-tote-handling-pi0-25-09-29-map", "type": "ClusterIP", "cluster_ip": "10.106.81.18", "ports": [ { "port": 9005, "target": "grpc", "protocol": "TCP" } ] }, { "name": "internal-demos-conveyor-18-11-map", "type": "ClusterIP", "cluster_ip": "10.102.3.23", "ports": [ { "port": 9005, "target": "grpc", "protocol": "TCP" } ] }, { "name": "r20260219-inte-conv-18-11-map", "type": "LoadBalancer", "cluster_ip": "10.98.118.189", "ports": [ { "port": 9000, "target": "grpc", "protocol": "TCP" } ] } ], "nodes": [ { "name": "ip-172-31-33-154", "instance_type": "", "zone": "", "ready": true, "cpu": "16", "memory": "65118216Ki", "gpu": "1", "gpu_mig": "0", "gpu_product": "NVIDIA-A10G", "allocatable_cpu": "16", "allocatable_memory": "56627208Ki", "os": "linux", "arch": "amd64", "node_group": "" }, { "name": "ip-172-31-38-130", "instance_type": "", "zone": "", "ready": true, "cpu": "16", "memory": "65118212Ki", "gpu": "1", "gpu_mig": "0", "gpu_product": "NVIDIA-A10G", "allocatable_cpu": "16", "allocatable_memory": "56627204Ki", "os": "linux", "arch": "amd64", "node_group": "" }, { "name": "ip-172-31-46-251", "instance_type": "", "zone": "", "ready": true, "cpu": "16", "memory": "65118216Ki", "gpu": "1", "gpu_mig": "0", "gpu_product": "NVIDIA-A10G", "allocatable_cpu": "16", "allocatable_memory": "56627208Ki", "os": "linux", "arch": "amd64", "node_group": "" } ], "namespaces": [ "default", "inference", "kube-flannel", "kube-node-lease", "kube-public", "kube-system", "logging", "longhorn-system", "monitoring" ] } }, { "name": "mumbai-01", "fqdn": "mumbai-01.lan.thehumanoid.ai", "ip": "3.110.31.221", "arch": "amd64", "gpu_type": "UNKNOWN", "location": "ap-south-1", "source": "onprem" } ] }, "netbox": { "updated": "2026-07-25T08:26:57.347389", "devices": [ { "name": "hmnd-ab-01", "role": "robot", "site": "hmnd-lon-lab", "location": "London Lab 5th Floor", "status": "active", "device_type": "Alpha Biped", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/68/" }, { "name": "hmnd-ab-02", "role": "robot", "site": "hmnd-lon-lab", "location": "London Lab 5th Floor", "status": "active", "device_type": "Alpha Biped", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/69/" }, { "name": "hmnd-aw-01", "role": "robot", "site": "hmnd-bos-lab", "location": "Robot Womb Boston", "status": "active", "device_type": "Alpha Wheeled", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/58/" }, { "name": "hmnd-aw-02", "role": "robot", "site": "hmnd-deploy-04", "location": "Bosch PoC", "status": "active", "device_type": "Alpha Wheeled", "primary_ip": "10.44.98.12", "dns_name": "aw2.hmnd.lan", "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/54/" }, { "name": "hmnd-aw-03", "role": "robot", "site": "hmnd-deploy-04", "location": "Bosch PoC", "status": "active", "device_type": "Alpha Wheeled", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/59/" }, { "name": "hmnd-aw-04", "role": "robot", "site": "hmnd-deploy-02", "location": "London Lab 6th Floor", "status": "active", "device_type": "Alpha Wheeled", "primary_ip": "10.44.98.14", "dns_name": "aw4.hmnd.lan", "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/55/" }, { "name": "hmnd-aw-05", "role": "robot", "site": "hmnd-deploy-03", "location": "KSA Showroom", "status": "active", "device_type": "Alpha Wheeled", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/60/" }, { "name": "hmnd-aw-06", "role": "robot", "site": "hmnd-deploy-02", "location": "London Lab 6th Floor", "status": "active", "device_type": "Alpha Wheeled", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/61/" }, { "name": "hmnd-aw-07", "role": "robot", "site": "hmnd-lon-lab", "location": "London Lab 5th Floor", "status": "active", "device_type": "Alpha Wheeled", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/62/" }, { "name": "hmnd-aw-08", "role": "robot", "site": "hmnd-deploy-02", "location": "London Lab 6th Floor", "status": "active", "device_type": "Alpha Wheeled", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/63/" }, { "name": "hmnd-aw-09", "role": "robot", "site": "hmnd-lon-lab", "location": "London Lab 5th Floor", "status": "active", "device_type": "Alpha Wheeled", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/64/" }, { "name": "hmnd-aw-10", "role": "robot", "site": "hmnd-deploy-02", "location": "London Lab 6th Floor", "status": "active", "device_type": "Alpha Wheeled", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/65/" }, { "name": "hmnd-aw-11", "role": "robot", "site": "hmnd-lon-lab", "location": "London Lab 5th Floor", "status": "active", "device_type": "Alpha Wheeled", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/66/" }, { "name": "hmnd-aw-12", "role": "robot", "site": "hmnd-deploy-02", "location": "London Lab 6th Floor", "status": "active", "device_type": "Alpha Wheeled", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/67/" }, { "name": "hmnd-deploy-01-gw-01", "role": "security-appliance", "site": "hmnd-deploy-01", "location": null, "status": "active", "device_type": "UniFi Cloud Gateway Fiber", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/10/" }, { "name": "hmnd-deploy-01-pi5-01", "role": "compute", "site": "hmnd-deploy-01", "location": null, "status": "active", "device_type": "Raspberry Pi 5", "primary_ip": "10.44.97.101", "dns_name": "pi1.hmnd.lan", "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/12/" }, { "name": "hmnd-deploy-01-screen-01", "role": "screen", "site": "hmnd-deploy-01", "location": null, "status": "active", "device_type": "GeekPi Screen", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/13/" }, { "name": "hmnd-deploy-01-sw-01", "role": "access-switch", "site": "hmnd-deploy-01", "location": null, "status": "active", "device_type": "Unifi Switch Pro XG 8 PoE", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/11/" }, { "name": "hmnd-deploy-02-ap-01", "role": "access-point", "site": "hmnd-deploy-02", "location": "CES Booth", "status": "active", "device_type": "E7", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/38/" }, { "name": "hmnd-deploy-02-ap-02", "role": "access-point", "site": "hmnd-deploy-02", "location": "CES Booth", "status": "active", "device_type": "U7 Pro Wall", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/41/" }, { "name": "hmnd-deploy-02-ap-03", "role": "access-point", "site": "hmnd-deploy-02", "location": "CES Booth", "status": "active", "device_type": "E7", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/42/" }, { "name": "hmnd-deploy-02-gw-01", "role": "security-appliance", "site": "hmnd-deploy-02", "location": "CES Booth", "status": "active", "device_type": "UniFi Cloud Gateway Fiber", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/34/" }, { "name": "hmnd-deploy-02-inference-01", "role": "compute", "site": "hmnd-deploy-02", "location": "CES Booth", "status": "active", "device_type": "H100", "primary_ip": "10.44.66.10", "dns_name": "h100-1.hmnd.lan", "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/43/" }, { "name": "hmnd-deploy-02-pi5-01", "role": "compute", "site": "hmnd-deploy-02", "location": "CES Booth", "status": "active", "device_type": "Raspberry Pi 5", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/37/" }, { "name": "hmnd-deploy-02-screen-01", "role": "screen", "site": "hmnd-deploy-02", "location": "CES Booth", "status": "active", "device_type": "GeekPi Screen", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/36/" }, { "name": "hmnd-deploy-02-sw-01", "role": "access-switch", "site": "hmnd-deploy-02", "location": "CES Booth", "status": "active", "device_type": "Unifi Switch Pro XG 8 PoE", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/35/" }, { "name": "hmnd-deploy-04-ap-01", "role": "access-point", "site": "hmnd-deploy-04", "location": "Martur Fompak", "status": "staged", "device_type": "E7", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/50/" }, { "name": "hmnd-deploy-04-ap-02", "role": "access-point", "site": "hmnd-deploy-04", "location": "Martur Fompak", "status": "staged", "device_type": "E7", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/51/" }, { "name": "hmnd-deploy-04-ap-03", "role": "access-point", "site": "hmnd-deploy-04", "location": "Martur Fompak", "status": "staged", "device_type": "U7 Pro Wall", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/52/" }, { "name": "hmnd-deploy-04-cam-01", "role": "camera", "site": "hmnd-deploy-04", "location": "Martur Fompak", "status": "active", "device_type": "Unifi G6 Bullet", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/39/" }, { "name": "hmnd-deploy-04-cam-02", "role": "camera", "site": "hmnd-deploy-04", "location": "Martur Fompak", "status": "active", "device_type": "Unifi G6 Bullet", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/40/" }, { "name": "hmnd-deploy-04-gw-01", "role": "security-appliance", "site": "hmnd-deploy-04", "location": "Martur Fompak", "status": "staged", "device_type": "UniFi Cloud Gateway Fiber", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/45/" }, { "name": "hmnd-deploy-04-inference-01", "role": "compute", "site": "hmnd-deploy-04", "location": "Martur Fompak", "status": "staged", "device_type": "RTX5090", "primary_ip": "10.44.98.59", "dns_name": "5090.hmnd.lan", "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/53/" }, { "name": "hmnd-deploy-04-pi5-01", "role": "compute", "site": "hmnd-deploy-04", "location": "Martur Fompak", "status": "staged", "device_type": "Raspberry Pi 5", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/47/" }, { "name": "hmnd-deploy-04-pi5-02", "role": "compute", "site": "hmnd-deploy-04", "location": "Martur Fompak", "status": "staged", "device_type": "Raspberry Pi 5", "primary_ip": "10.44.97.102", "dns_name": "pi2.hmnd.lan", "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/48/" }, { "name": "hmnd-deploy-04-pi5-03", "role": "compute", "site": "hmnd-deploy-04", "location": "Martur Fompak", "status": "staged", "device_type": "Raspberry Pi 5", "primary_ip": "10.44.97.103", "dns_name": "pi3.hmnd.lan", "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/49/" }, { "name": "hmnd-deploy-04-screen-01", "role": "screen", "site": "hmnd-deploy-04", "location": "Martur Fompak", "status": "staged", "device_type": "GeekPi Screen", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/44/" }, { "name": "hmnd-deploy-04-sw-01", "role": "access-switch", "site": "hmnd-deploy-04", "location": "Martur Fompak", "status": "staged", "device_type": "Unifi Switch Pro XG 8 PoE", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/46/" }, { "name": "hmnd-deploy-04-wan-01", "role": "wan-uplink", "site": "hmnd-deploy-04", "location": "Martur Fompak", "status": "active", "device_type": "Customer WAN Uplink", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/57/" }, { "name": "hmnd-deploy-04-wan-02", "role": "wan-uplink", "site": "hmnd-deploy-04", "location": "Martur Fompak", "status": "active", "device_type": "RUTM30", "primary_ip": null, "dns_name": null, "serial": "600585677", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/56/" }, { "name": "hmnd-lab-cam-01", "role": "camera", "site": "hmnd-lon-lab", "location": "London Lab 5th Floor", "status": "active", "device_type": "Unifi G6 Bullet", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/9/" }, { "name": "hmnd-lab-gw-01", "role": "security-appliance", "site": "hmnd-lon-lab", "location": "London Lab 5th Floor", "status": "active", "device_type": "Enterprise Fortress Gateway", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/1/" }, { "name": "hmnd-lab-gw-02", "role": "security-appliance", "site": "hmnd-lon-lab", "location": "London Lab 5th Floor", "status": "active", "device_type": "Enterprise Fortress Gateway", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/4/" }, { "name": "hmnd-lab-nas-01", "role": "network-attached-storage", "site": "hmnd-lon-lab", "location": "London Lab 5th Floor", "status": "active", "device_type": "UNAS Pro", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/5/" }, { "name": "hmnd-lab-nvr-01", "role": "network-video-recorder", "site": "hmnd-lon-lab", "location": "London Lab 5th Floor", "status": "active", "device_type": "UniFi Protect Network Video Recorder Pro", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/6/" }, { "name": "hmnd-lab-sw-01", "role": "access-switch", "site": "hmnd-lon-lab", "location": "London Lab 5th Floor", "status": "active", "device_type": "UniFi Switch Enterprise 48 PoE", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/2/" }, { "name": "hmnd-lab-sw-02", "role": "access-switch", "site": "hmnd-lon-lab", "location": "London Lab 5th Floor", "status": "active", "device_type": "Unifi Switch Pro XG 8 PoE", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/3/" }, { "name": "hmnd-lab-wire-01", "role": "patch-panel", "site": "hmnd-lon-lab", "location": "London Lab 5th Floor", "status": "active", "device_type": "24-Port Blank Keystone Patch Panel", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/7/" }, { "name": "hmnd-lab-wire-02", "role": "patch-panel", "site": "hmnd-lon-lab", "location": "London Lab 5th Floor", "status": "active", "device_type": "24-Port Blank Keystone Patch Panel", "primary_ip": null, "dns_name": null, "serial": "", "url": "https://bwgd4658.cloud.netboxapp.com/api/dcim/devices/8/" } ], "sites": [ { "name": "hmnd-bos-lab", "slug": "hmnd-bos-lab", "description": "hmnd boston hq", "region": "Boston", "facility": "WeWork 1 Broadway, Cambridge, MA 02142" }, { "name": "hmnd-deploy-01", "slug": "hmnd-deploy-01", "description": "Deployment Network in a Box | GeekPi 10inch 4U", "region": "Europe", "facility": "Symphony" }, { "name": "hmnd-deploy-02", "slug": "hmnd-deploy-02", "description": "Deployment Network in a Box | GeekPi 10inch 4U", "region": "Vegas", "facility": "WeWork 5 Merchant Square" }, { "name": "hmnd-deploy-03", "slug": "hmnd-deploy-03", "description": "Deployment Network in a Box | GeekPi 10inch 4U", "region": "Europe", "facility": "KSA" }, { "name": "hmnd-deploy-04", "slug": "hmnd-deploy-04", "description": "Deployment Network in a Box | GeekPi 10inch 8U", "region": "Europe", "facility": "Martur Fompak SAP" }, { "name": "hmnd-lon-lab", "slug": "hmnd-lab-lon", "description": "hmnd london hq", "region": "London", "facility": "WeWork 5 Merchant Square" } ], "ip_addresses": [ { "address": "10.44.10.17", "dns_name": "aw7.hmnd.lan", "status": "active", "assigned_device": "hmnd-deploy-02-ap-01" }, { "address": "10.44.10.18", "dns_name": "aw8.hmnd.lan", "status": "active", "assigned_device": null }, { "address": "10.44.50.101", "dns_name": "h100-01.hmnd.lan", "status": "active", "assigned_device": null }, { "address": "10.44.50.102", "dns_name": "h100-02.hmnd.lan", "status": "active", "assigned_device": null }, { "address": "10.44.66.10", "dns_name": "h100-1.hmnd.lan", "status": "active", "assigned_device": "hmnd-deploy-02-inference-01" }, { "address": "10.44.66.101", "dns_name": "h100-01.hmnd.lan", "status": "active", "assigned_device": null }, { "address": "10.44.66.102", "dns_name": "h100-02.hmnd.lan", "status": "active", "assigned_device": null }, { "address": "10.44.82.101", "dns_name": "h100-01.hmnd.lan", "status": "active", "assigned_device": null }, { "address": "10.44.82.102", "dns_name": "h100-02.hmnd.lan", "status": "active", "assigned_device": null }, { "address": "10.44.97.101", "dns_name": "pi1.hmnd.lan", "status": "active", "assigned_device": "hmnd-deploy-01-pi5-01" }, { "address": "10.44.97.102", "dns_name": "pi2.hmnd.lan", "status": "active", "assigned_device": "hmnd-deploy-04-pi5-02" }, { "address": "10.44.97.103", "dns_name": "pi3.hmnd.lan", "status": "active", "assigned_device": "hmnd-deploy-04-pi5-03" }, { "address": "10.44.98.12", "dns_name": "aw2.hmnd.lan", "status": "active", "assigned_device": "hmnd-aw-02" }, { "address": "10.44.98.14", "dns_name": "aw4.hmnd.lan", "status": "active", "assigned_device": "hmnd-aw-04" }, { "address": "10.44.98.59", "dns_name": "5090.hmnd.lan", "status": "active", "assigned_device": "hmnd-deploy-04-inference-01" }, { "address": "10.44.98.101", "dns_name": "h100-01.hmnd.lan", "status": "active", "assigned_device": null }, { "address": "10.44.98.102", "dns_name": "h100-02.hmnd.lan", "status": "active", "assigned_device": null } ], "prefixes": [ { "prefix": "10.44.0.0/20", "site": null, "vlan": null, "vlan_name": null, "status": "active", "description": "london lab prefix" }, { "prefix": "10.44.16.0/20", "site": null, "vlan": null, "vlan_name": null, "status": "reserved", "description": "" }, { "prefix": "10.44.32.0/20", "site": null, "vlan": null, "vlan_name": null, "status": "reserved", "description": "" }, { "prefix": "10.44.48.0/20", "site": null, "vlan": null, "vlan_name": null, "status": "active", "description": "" }, { "prefix": "10.44.50.0/24", "site": null, "vlan": 3, "vlan_name": "hmnd-inference", "status": "active", "description": "Inference GPU Compute" }, { "prefix": "10.44.64.0/20", "site": null, "vlan": null, "vlan_name": null, "status": "active", "description": "" }, { "prefix": "10.44.64.0/24", "site": null, "vlan": 1, "vlan_name": "hmnd-mgmt", "status": "active", "description": "Net Infra MGMT" }, { "prefix": "10.44.65.0/24", "site": null, "vlan": 2, "vlan_name": "hmnd-random", "status": "active", "description": "Emergent use case - Adapt to context" }, { "prefix": "10.44.66.0/24", "site": null, "vlan": 3, "vlan_name": "hmnd-inference", "status": "active", "description": "Inference GPU Compute" }, { "prefix": "10.44.67.0/24", "site": null, "vlan": 4, "vlan_name": "hmnd-human", "status": "active", "description": "Human Users" }, { "prefix": "10.44.68.0/24", "site": null, "vlan": 5, "vlan_name": "hmnd-robot", "status": "active", "description": "Robot Devices" }, { "prefix": "10.44.80.0/20", "site": null, "vlan": null, "vlan_name": null, "status": "active", "description": "" }, { "prefix": "10.44.82.0/24", "site": null, "vlan": 3, "vlan_name": "hmnd-inference", "status": "active", "description": "Inference GPU Compute" }, { "prefix": "10.44.96.0/20", "site": null, "vlan": null, "vlan_name": null, "status": "active", "description": "" }, { "prefix": "10.44.96.0/24", "site": null, "vlan": 1, "vlan_name": "hmnd-mgmt", "status": "active", "description": "Net Infra MGMT" }, { "prefix": "10.44.97.0/24", "site": null, "vlan": 2, "vlan_name": "hmnd-random", "status": "active", "description": "Emergent use case - Adapt to context" }, { "prefix": "10.44.98.0/24", "site": null, "vlan": 3, "vlan_name": "hmnd-inference", "status": "active", "description": "Inference GPU Compute" }, { "prefix": "10.44.99.0/24", "site": null, "vlan": 4, "vlan_name": "hmnd-human", "status": "active", "description": "Human Users" }, { "prefix": "10.44.100.0/24", "site": null, "vlan": 5, "vlan_name": "hmnd-robot", "status": "active", "description": "Robot Devices" }, { "prefix": "10.44.111.0/24", "site": null, "vlan": 16, "vlan_name": "hmnd-ui", "status": "active", "description": "Unifi One Click Wifi Users" } ], "vlans": [ { "vid": 1, "name": "hmnd-mgmt", "site": null, "status": "active" }, { "vid": 2, "name": "hmnd-random", "site": null, "status": "active" }, { "vid": 3, "name": "hmnd-inference", "site": null, "status": "active" }, { "vid": 4, "name": "hmnd-human", "site": null, "status": "active" }, { "vid": 5, "name": "hmnd-robot", "site": null, "status": "active" }, { "vid": 16, "name": "hmnd-ui", "site": null, "status": "active" } ], "drift": { "gaps": [ { "type": "missing_in_unifi", "severity": "critical", "device": "hmnd-deploy-01-gw-01", "robot": null, "netbox_value": "active", "actual_value": null, "site": "hmnd-deploy-01", "message": "hmnd-deploy-01-gw-01 (active) in Netbox but not found in Unifi" }, { "type": "missing_in_unifi", "severity": "critical", "device": "hmnd-deploy-01-pi5-01", "robot": null, "netbox_value": "active", "actual_value": null, "site": "hmnd-deploy-01", "message": "hmnd-deploy-01-pi5-01 (active) in Netbox but not found in Unifi" }, { "type": "missing_in_unifi", "severity": "critical", "device": "hmnd-deploy-01-sw-01", "robot": null, "netbox_value": "active", "actual_value": null, "site": "hmnd-deploy-01", "message": "hmnd-deploy-01-sw-01 (active) in Netbox but not found in Unifi" }, { "type": "missing_in_unifi", "severity": "critical", "device": "hmnd-deploy-02-ap-01", "robot": null, "netbox_value": "active", "actual_value": null, "site": "hmnd-deploy-02", "message": "hmnd-deploy-02-ap-01 (active) in Netbox but not found in Unifi" }, { "type": "missing_in_unifi", "severity": "critical", "device": "hmnd-deploy-02-ap-02", "robot": null, "netbox_value": "active", "actual_value": null, "site": "hmnd-deploy-02", "message": "hmnd-deploy-02-ap-02 (active) in Netbox but not found in Unifi" }, { "type": "missing_in_unifi", "severity": "critical", "device": "hmnd-deploy-02-ap-03", "robot": null, "netbox_value": "active", "actual_value": null, "site": "hmnd-deploy-02", "message": "hmnd-deploy-02-ap-03 (active) in Netbox but not found in Unifi" }, { "type": "missing_in_unifi", "severity": "critical", "device": "hmnd-deploy-02-gw-01", "robot": null, "netbox_value": "active", "actual_value": null, "site": "hmnd-deploy-02", "message": "hmnd-deploy-02-gw-01 (active) in Netbox but not found in Unifi" }, { "type": "missing_in_unifi", "severity": "critical", "device": "hmnd-deploy-02-inference-01", "robot": null, "netbox_value": "active", "actual_value": null, "site": "hmnd-deploy-02", "message": "hmnd-deploy-02-inference-01 (active) in Netbox but not found in Unifi" }, { "type": "missing_in_unifi", "severity": "critical", "device": "hmnd-deploy-02-pi5-01", "robot": null, "netbox_value": "active", "actual_value": null, "site": "hmnd-deploy-02", "message": "hmnd-deploy-02-pi5-01 (active) in Netbox but not found in Unifi" }, { "type": "missing_in_unifi", "severity": "critical", "device": "hmnd-deploy-02-sw-01", "robot": null, "netbox_value": "active", "actual_value": null, "site": "hmnd-deploy-02", "message": "hmnd-deploy-02-sw-01 (active) in Netbox but not found in Unifi" }, { "type": "missing_in_unifi", "severity": "info", "device": "hmnd-deploy-04-ap-01", "robot": null, "netbox_value": "staged", "actual_value": null, "site": "hmnd-deploy-04", "message": "hmnd-deploy-04-ap-01 (staged) in Netbox but not found in Unifi" }, { "type": "missing_in_unifi", "severity": "info", "device": "hmnd-deploy-04-ap-02", "robot": null, "netbox_value": "staged", "actual_value": null, "site": "hmnd-deploy-04", "message": "hmnd-deploy-04-ap-02 (staged) in Netbox but not found in Unifi" }, { "type": "missing_in_unifi", "severity": "info", "device": "hmnd-deploy-04-ap-03", "robot": null, "netbox_value": "staged", "actual_value": null, "site": "hmnd-deploy-04", "message": "hmnd-deploy-04-ap-03 (staged) in Netbox but not found in Unifi" }, { "type": "missing_in_unifi", "severity": "info", "device": "hmnd-deploy-04-gw-01", "robot": null, "netbox_value": "staged", "actual_value": null, "site": "hmnd-deploy-04", "message": "hmnd-deploy-04-gw-01 (staged) in Netbox but not found in Unifi" }, { "type": "missing_in_unifi", "severity": "info", "device": "hmnd-deploy-04-inference-01", "robot": null, "netbox_value": "staged", "actual_value": null, "site": "hmnd-deploy-04", "message": "hmnd-deploy-04-inference-01 (staged) in Netbox but not found in Unifi" }, { "type": "missing_in_unifi", "severity": "info", "device": "hmnd-deploy-04-pi5-01", "robot": null, "netbox_value": "staged", "actual_value": null, "site": "hmnd-deploy-04", "message": "hmnd-deploy-04-pi5-01 (staged) in Netbox but not found in Unifi" }, { "type": "missing_in_unifi", "severity": "info", "device": "hmnd-deploy-04-pi5-02", "robot": null, "netbox_value": "staged", "actual_value": null, "site": "hmnd-deploy-04", "message": "hmnd-deploy-04-pi5-02 (staged) in Netbox but not found in Unifi" }, { "type": "missing_in_unifi", "severity": "info", "device": "hmnd-deploy-04-pi5-03", "robot": null, "netbox_value": "staged", "actual_value": null, "site": "hmnd-deploy-04", "message": "hmnd-deploy-04-pi5-03 (staged) in Netbox but not found in Unifi" }, { "type": "missing_in_unifi", "severity": "info", "device": "hmnd-deploy-04-sw-01", "robot": null, "netbox_value": "staged", "actual_value": null, "site": "hmnd-deploy-04", "message": "hmnd-deploy-04-sw-01 (staged) in Netbox but not found in Unifi" }, { "type": "missing_in_unifi", "severity": "critical", "device": "hmnd-lab-gw-01", "robot": null, "netbox_value": "active", "actual_value": null, "site": "hmnd-lon-lab", "message": "hmnd-lab-gw-01 (active) in Netbox but not found in Unifi" }, { "type": "missing_in_unifi", "severity": "critical", "device": "hmnd-lab-gw-02", "robot": null, "netbox_value": "active", "actual_value": null, "site": "hmnd-lon-lab", "message": "hmnd-lab-gw-02 (active) in Netbox but not found in Unifi" }, { "type": "missing_in_unifi", "severity": "critical", "device": "hmnd-lab-nas-01", "robot": null, "netbox_value": "active", "actual_value": null, "site": "hmnd-lon-lab", "message": "hmnd-lab-nas-01 (active) in Netbox but not found in Unifi" }, { "type": "missing_in_unifi", "severity": "critical", "device": "hmnd-lab-nvr-01", "robot": null, "netbox_value": "active", "actual_value": null, "site": "hmnd-lon-lab", "message": "hmnd-lab-nvr-01 (active) in Netbox but not found in Unifi" }, { "type": "missing_in_unifi", "severity": "critical", "device": "hmnd-lab-sw-01", "robot": null, "netbox_value": "active", "actual_value": null, "site": "hmnd-lon-lab", "message": "hmnd-lab-sw-01 (active) in Netbox but not found in Unifi" }, { "type": "missing_in_unifi", "severity": "critical", "device": "hmnd-lab-sw-02", "robot": null, "netbox_value": "active", "actual_value": null, "site": "hmnd-lon-lab", "message": "hmnd-lab-sw-02 (active) in Netbox but not found in Unifi" }, { "type": "missing_in_netbox", "severity": "info", "device": "ECS 48 PoE | 9", "robot": null, "netbox_value": null, "actual_value": "usw", "site": "1-triton", "message": "ECS 48 PoE | 9 found in Unifi (1-triton) but not documented in Netbox" }, { "type": "missing_in_netbox", "severity": "info", "device": "USW Pro Max 16 PoE | 18 (Desk)", "robot": null, "netbox_value": null, "actual_value": "usw", "site": "1-triton", "message": "USW Pro Max 16 PoE | 18 (Desk) found in Unifi (1-triton) but not documented in Netbox" }, { "type": "missing_in_netbox", "severity": "info", "device": "E7 | SW SER", "robot": null, "netbox_value": null, "actual_value": "uap", "site": "1-triton", "message": "E7 | SW SER found in Unifi (1-triton) but not documented in Netbox" }, { "type": "missing_in_netbox", "severity": "info", "device": "UPS Tower", "robot": null, "netbox_value": null, "actual_value": "usw", "site": "1-triton", "message": "UPS Tower found in Unifi (1-triton) but not documented in Netbox" }, { "type": "missing_in_netbox", "severity": "info", "device": "1 Triton", "robot": null, "netbox_value": null, "actual_value": "udm", "site": "1-triton", "message": "1 Triton found in Unifi (1-triton) but not documented in Netbox" }, { "type": "missing_in_netbox", "severity": "info", "device": "U7 Pro XG Wall | TEC-3 Cab", "robot": null, "netbox_value": null, "actual_value": "uap", "site": "1-triton", "message": "U7 Pro XG Wall | TEC-3 Cab found in Unifi (1-triton) but not documented in Netbox" }, { "type": "missing_in_netbox", "severity": "info", "device": "ECS 24 PoE", "robot": null, "netbox_value": null, "actual_value": "usw", "site": "1-triton", "message": "ECS 24 PoE found in Unifi (1-triton) but not documented in Netbox" }, { "type": "missing_in_netbox", "severity": "info", "device": "E7 Campus | Float", "robot": null, "netbox_value": null, "actual_value": "uap", "site": "1-triton", "message": "E7 Campus | Float found in Unifi (1-triton) but not documented in Netbox" }, { "type": "missing_in_netbox", "severity": "info", "device": "AirWire", "robot": null, "netbox_value": null, "actual_value": "uacc", "site": "1-triton", "message": "AirWire found in Unifi (1-triton) but not documented in Netbox" }, { "type": "missing_in_netbox", "severity": "info", "device": "EAV XG 24 PoE | NE SER ", "robot": null, "netbox_value": null, "actual_value": "usw", "site": "1-triton", "message": "EAV XG 24 PoE | NE SER found in Unifi (1-triton) but not documented in Netbox" }, { "type": "missing_in_netbox", "severity": "info", "device": "E7 Lift Lobby ", "robot": null, "netbox_value": null, "actual_value": "uap", "site": "1-triton", "message": "E7 Lift Lobby found in Unifi (1-triton) but not documented in Netbox" }, { "type": "missing_in_netbox", "severity": "info", "device": "USW Flex 2.5G 8 PoE", "robot": null, "netbox_value": null, "actual_value": "usw", "site": "1-triton", "message": "USW Flex 2.5G 8 PoE found in Unifi (1-triton) but not documented in Netbox" }, { "type": "missing_in_netbox", "severity": "info", "device": "USW Pro XG 24 PoE | 2 | Replacement", "robot": null, "netbox_value": null, "actual_value": "usw", "site": "1-triton", "message": "USW Pro XG 24 PoE | 2 | Replacement found in Unifi (1-triton) but not documented in Netbox" }, { "type": "missing_in_netbox", "severity": "info", "device": "UDB Switch | Stores", "robot": null, "netbox_value": null, "actual_value": "udb", "site": "1-triton", "message": "UDB Switch | Stores found in Unifi (1-triton) but not documented in Netbox" }, { "type": "missing_in_netbox", "severity": "info", "device": "E7 Audience | SW Plant Room", "robot": null, "netbox_value": null, "actual_value": "uap", "site": "1-triton", "message": "E7 Audience | SW Plant Room found in Unifi (1-triton) but not documented in Netbox" }, { "type": "missing_in_netbox", "severity": "info", "device": "ECS Aggregation South", "robot": null, "netbox_value": null, "actual_value": "usw", "site": "1-triton", "message": "ECS Aggregation South found in Unifi (1-triton) but not documented in Netbox" }, { "type": "missing_in_netbox", "severity": "info", "device": "U5G Max Outdoor", "robot": null, "netbox_value": null, "actual_value": "umbb", "site": "1-triton", "message": "U5G Max Outdoor found in Unifi (1-triton) but not documented in Netbox" }, { "type": "missing_in_netbox", "severity": "info", "device": "USW Pro XG 8 PoE | Stores Rack", "robot": null, "netbox_value": null, "actual_value": "usw", "site": "1-triton", "message": "USW Pro XG 8 PoE | Stores Rack found in Unifi (1-triton) but not documented in Netbox" }, { "type": "missing_in_netbox", "severity": "info", "device": "USW WAN | SW (Secondary)", "robot": null, "netbox_value": null, "actual_value": "usw", "site": "1-triton", "message": "USW WAN | SW (Secondary) found in Unifi (1-triton) but not documented in Netbox" }, { "type": "missing_in_netbox", "severity": "info", "device": "ECS Aggregation North", "robot": null, "netbox_value": null, "actual_value": "usw", "site": "1-triton", "message": "ECS Aggregation North found in Unifi (1-triton) but not documented in Netbox" }, { "type": "missing_in_netbox", "severity": "info", "device": "USP RPS", "robot": null, "netbox_value": null, "actual_value": "usw", "site": "1-triton", "message": "USP RPS found in Unifi (1-triton) but not documented in Netbox" } ], "summary": { "total_gaps": 46, "critical": 16, "warnings": 0, "info": 30, "netbox_devices": 49, "unifi_devices": 21, "documented_pct": 119 } } }, "fleet_dns": { "updated": "2026-07-25T08:26:58.143589+00:00", "status": "ok", "zone_id": "Z0843494QQHALA69EIET", "records_upserted": 11, "records": [ { "name": "aw3.robots.hmnd.ai", "ip": "10.10.50.103", "site": "1-triton (last seen)" }, { "name": "ab3.robots.hmnd.ai", "ip": "10.10.51.103", "site": "1-triton (last seen)" }, { "name": "aw12.robots.hmnd.ai", "ip": "10.10.50.112", "site": "1-triton (last seen)" }, { "name": "aw2.robots.hmnd.ai", "ip": "10.10.50.102", "site": "1-triton (last seen)" }, { "name": "aw7.robots.hmnd.ai", "ip": "10.10.50.107", "site": "1-triton (last seen)" }, { "name": "aw6.robots.hmnd.ai", "ip": "10.10.50.106", "site": "1-triton (last seen)" }, { "name": "aw5.robots.hmnd.ai", "ip": "10.10.50.105", "site": "1-triton (last seen)" }, { "name": "aw10.robots.hmnd.ai", "ip": "10.10.50.110", "site": "1-triton (last seen)" }, { "name": "aw9.robots.hmnd.ai", "ip": "10.10.50.109", "site": "1-triton (last seen)" }, { "name": "aw11.robots.hmnd.ai", "ip": "10.10.50.111", "site": "1-triton (last seen)" }, { "name": "ab2.robots.hmnd.ai", "ip": "10.10.51.102", "site": "1-triton (last seen)" } ] }, "system_data": { "updated": "2026-07-25T08:27:09.849741+00:00", "robots": { "ab3": { "orin": { "kernel": "5.15.148-tegra", "os": "Ubuntu 22.04", "architecture": "aarch64", "hostname": "alphabiped03-orin", "uptime_seconds": "226", "memory_total_mb": "54793", "memory_free_mb": "51790", "release": null, "collected_at": "2026-03-27T14:31:12Z", "dns": { "servers": { "global": "8.8.8.8 1.1.1.1" }, "search_domains": {}, "dropins": "[Resolve]\nFallbackDNS=8.8.8.8", "zones": { "local": false, "route53": false, "private": false, "public": true } }, "docker_running": false, "can_interfaces_up": 0, "model": "Newton + Orin AGXi\u0000", "utilization": { "boots": [], "daily_seconds": {}, "total_boots_7d": 0, "total_on_seconds_7d": 0, "utilization_7d_pct": 0.0 }, "_source_mtime": "2026-03-27T14:32:14+00:00" }, "rtpc": { "kernel": "5.15.197-rt91-hmnd-0", "os": "Ubuntu 22.04", "architecture": "x86_64", "hostname": "alphabiped03-x86", "uptime_seconds": "404", "memory_total_mb": "47921", "memory_free_mb": "46687", "release": null, "collected_at": "2026-03-27T15:30:22Z", "dns": { "servers": { "wifi-0": "10.44.10.1" }, "search_domains": { "wifi-0": "hmnd.lan" }, "dropins": "no drop-ins", "zones": { "local": true, "route53": true, "private": true, "public": true } }, "wifi": [ { "name": "wifi-0", "driver": "iwlwifi", "firmware": "89.123cf747.0 ty-a0-gf-a0-89.uc", "mac": "c4:03:a8:32:e9:b6", "ssid": "Alpha", "channel": 36, "configured_ssid": "Alpha" } ], "interfaces": [ { "name": "eth-shore", "operstate": "DOWN", "ip": null, "mac": "00:13:95:6a:22:0a" }, { "name": "wifi-0", "operstate": "UP", "ip": "10.44.10.113", "mac": "c4:03:a8:32:e9:b6" } ], "nm_profiles": [ { "name": "Alpha", "type": "wifi", "interface": "wifi-0", "ssid": "Alpha", "autoconnect": true, "priority": 900, "key_mgmt": "wpa-psk", "zone": "external", "state": "activated" }, { "name": "hmnd0", "type": "wireguard", "interface": "hmnd0", "ssid": null, "autoconnect": false, "priority": 0, "key_mgmt": null, "zone": null, "state": "activated" }, { "name": "eth-shore", "type": "ethernet", "interface": "", "ssid": null, "autoconnect": true, "priority": 0, "key_mgmt": null, "zone": null, "state": "" }, { "name": "roadkill", "type": "wifi", "interface": "", "ssid": null, "autoconnect": true, "priority": 0, "key_mgmt": null, "zone": null, "state": "" } ], "dispatcher_scripts": [ "10-hmnd-disable-offloads", "70-hmnd-qos" ], "utilization": { "boots": [], "daily_seconds": {}, "total_boots_7d": 0, "total_on_seconds_7d": 0, "utilization_7d_pct": 0.0 }, "_source_mtime": "2026-03-27T15:32:10+00:00" } }, "aw10": { "orin": { "kernel": "5.15.148-tegra", "os": "Ubuntu 22.04", "architecture": "aarch64", "hostname": "orinalpha10", "uptime_seconds": "2878", "memory_total_mb": "54793", "memory_free_mb": "45991", "release": null, "collected_at": "2026-04-09T16:01:21Z", "dns": { "servers": { "global": "10.44.10.1" }, "search_domains": {}, "dropins": "[Resolve]\nDNS=10.44.10.1\nFallbackDNS=8.8.8.8", "zones": { "local": true, "route53": true, "private": true, "public": true } }, "docker_running": false, "can_interfaces_up": 0, "model": "Newton + Orin AGXi\u0000", "utilization": { "boots": [], "daily_seconds": {}, "total_boots_7d": 0, "total_on_seconds_7d": 0, "utilization_7d_pct": 0.0 }, "_source_mtime": "2026-04-09T16:02:09+00:00" }, "rtpc": { "kernel": "5.15.197-rt91-hmnd-0", "os": "Ubuntu 22.04", "architecture": "x86_64", "hostname": "alpha10", "uptime_seconds": "2874", "memory_total_mb": "47921", "memory_free_mb": "44601", "release": null, "collected_at": "2026-04-09T16:00:26Z", "dns": { "servers": { "wifi-0": "10.44.10.1" }, "search_domains": { "wifi-0": "hmnd.lan" }, "dropins": "no drop-ins", "zones": { "local": true, "route53": true, "private": true, "public": true } }, "wifi": [ { "name": "wifi-0", "driver": "iwlwifi", "firmware": "89.123cf747.0 ty-a0-gf-a0-89.uc", "mac": "5c:b2:6d:46:6d:88", "ssid": "Alpha", "channel": 104, "configured_ssid": "Alpha", "station": { "rx_packets": 321333, "tx_packets": 714183, "tx_retries": 23215, "tx_failed": 0, "beacon_loss": 0, "rx_drop_misc": 3807, "signal_dbm": -35, "signal_per_antenna": "[-44, -35]", "tx_bitrate": "243.7 MBit/s HE-MCS 10 HE-NSS 2 HE-GI 1 HE-DCM 0", "rx_bitrate": "258.0 MBit/s HE-MCS 10 HE-NSS 2 HE-GI 0 HE-DCM 0", "connected_seconds": 2893 } }, { "name": "wifi-1", "driver": "iwlwifi", "firmware": "89.123cf747.0 ty-a0-gf-a0-89.uc", "mac": "58:6d:67:f7:d1:9f", "ssid": null, "channel": null, "configured_ssid": null } ], "interfaces": [ { "name": "eth-shore", "operstate": "DOWN", "ip": null, "mac": "00:13:95:6a:22:10" }, { "name": "wifi-1", "operstate": "DOWN", "ip": null, "mac": "58:6d:67:f7:d1:9f" }, { "name": "wifi-0", "operstate": "UP", "ip": "10.44.10.20", "mac": "5c:b2:6d:46:6d:88" } ], "nm_profiles": [ { "name": "Alpha", "type": "wifi", "interface": "wifi-0", "ssid": "Alpha", "autoconnect": true, "priority": 900, "key_mgmt": "wpa-psk", "zone": "external", "state": "activated" }, { "name": "sw-internal", "type": "ethernet", "interface": "sw-internal", "ssid": null, "autoconnect": true, "priority": 0, "key_mgmt": null, "zone": "internal", "state": "activated" }, { "name": "hmnd0", "type": "wireguard", "interface": "hmnd0", "ssid": null, "autoconnect": false, "priority": 0, "key_mgmt": null, "zone": null, "state": "activated" }, { "name": "eth-shore", "type": "ethernet", "interface": "", "ssid": null, "autoconnect": true, "priority": 0, "key_mgmt": null, "zone": null, "state": "" } ], "dispatcher_scripts": [ "10-hmnd-disable-offloads", "70-hmnd-qos" ], "utilization": { "boots": [], "daily_seconds": {}, "total_boots_7d": 0, "total_on_seconds_7d": 0, "utilization_7d_pct": 0.0 }, "_source_mtime": "2026-04-09T16:02:08+00:00" } }, "ab1": { "rtpc": { "kernel": "5.15.197-rt91-hmnd-0", "os": "Ubuntu 22.04", "architecture": "x86_64", "hostname": "alphabiped01-x86", "uptime_seconds": "85924", "memory_total_mb": "47921", "memory_free_mb": "45538", "release": null, "collected_at": "2026-04-02T12:30:19Z", "dns": { "servers": { "wifi-0": "10.44.10.1" }, "search_domains": { "wifi-0": "hmnd.lan" }, "dropins": "no drop-ins", "zones": { "local": true, "route53": true, "private": true, "public": true } }, "wifi": [ { "name": "wifi-1", "driver": "iwlwifi", "firmware": "89.123cf747.0 ty-a0-gf-a0-89.uc", "mac": "8c:f8:c5:b5:f3:c0", "ssid": null, "channel": null, "configured_ssid": null }, { "name": "wifi-0", "driver": "iwlwifi", "firmware": "89.123cf747.0 ty-a0-gf-a0-89.uc", "mac": "8c:f8:c5:b5:f3:bf", "ssid": "Alpha", "channel": 85, "configured_ssid": "Alpha", "station": { "rx_packets": 2673194, "tx_packets": 2082579, "tx_retries": 36245, "tx_failed": 0, "beacon_loss": 0, "rx_drop_misc": 655565, "signal_dbm": -47, "signal_per_antenna": "[-47, -53]", "tx_bitrate": "286.7 MBit/s HE-MCS 11 HE-NSS 2 HE-GI 0 HE-DCM 0", "rx_bitrate": "286.7 MBit/s HE-MCS 11 HE-NSS 2 HE-GI 0 HE-DCM 0", "connected_seconds": 85425 } } ], "interfaces": [ { "name": "eth-shore", "operstate": "DOWN", "ip": null, "mac": "00:13:95:65:f1:5c" }, { "name": "wifi-0", "operstate": "UP", "ip": "10.44.10.99", "mac": "8c:f8:c5:b5:f3:bf" }, { "name": "wifi-1", "operstate": "DOWN", "ip": null, "mac": "8c:f8:c5:b5:f2:f2" } ], "nm_profiles": [ { "name": "Alpha", "type": "wifi", "interface": "wifi-0", "ssid": "Alpha", "autoconnect": true, "priority": 980, "key_mgmt": "wpa-psk", "zone": "external", "state": "activated" }, { "name": "sw-internal", "type": "ethernet", "interface": "sw-internal", "ssid": null, "autoconnect": false, "priority": 0, "key_mgmt": null, "zone": null, "state": "activated" }, { "name": "hmnd0", "type": "wireguard", "interface": "hmnd0", "ssid": null, "autoconnect": false, "priority": 0, "key_mgmt": null, "zone": null, "state": "activated" }, { "name": "eth-shore", "type": "ethernet", "interface": "", "ssid": null, "autoconnect": true, "priority": 0, "key_mgmt": null, "zone": null, "state": "" }, { "name": "hmnd-6ghz", "type": "wifi", "interface": "", "ssid": null, "autoconnect": true, "priority": 900, "key_mgmt": null, "zone": null, "state": "" }, { "name": "wifi", "type": "wifi", "interface": "", "ssid": null, "autoconnect": true, "priority": 0, "key_mgmt": null, "zone": null, "state": "" } ], "dispatcher_scripts": [ "10-hmnd-disable-offloads", "70-hmnd-qos" ], "utilization": { "boots": [], "daily_seconds": {}, "total_boots_7d": 0, "total_on_seconds_7d": 0, "utilization_7d_pct": 0.0 }, "_source_mtime": "2026-04-02T14:32:49+00:00" }, "orin": { "kernel": "5.15.148-tegra", "os": "Ubuntu 22.04", "architecture": "aarch64", "hostname": "alphabiped01-orin", "uptime_seconds": "84128", "memory_total_mb": "54793", "memory_free_mb": "50865", "release": null, "collected_at": "2026-04-02T12:01:13Z", "dns": { "servers": {}, "search_domains": {}, "dropins": "[Resolve]\nFallbackDNS=8.8.8.8", "zones": { "local": false, "route53": false, "private": false, "public": true } }, "docker_running": false, "can_interfaces_up": 0, "model": "Newton + Orin AGXi\u0000", "utilization": { "boots": [], "daily_seconds": {}, "total_boots_7d": 0, "total_on_seconds_7d": 0, "utilization_7d_pct": 0.0 }, "_source_mtime": "2026-04-02T12:02:19+00:00" } }, "aw11": { "rtpc": { "kernel": "5.15.197-rt91-hmnd-0", "os": "Ubuntu 22.04", "architecture": "x86_64", "hostname": "alpha11", "uptime_seconds": 21997, "memory_total_mb": 47924, "memory_free_mb": 42169, "release": null, "collected_at": "2026-04-10T13:15:40Z", "dns": { "servers": { "eth-shore": "10.44.10.1", "wifi-0": "10.44.10.1" }, "search_domains": { "eth-shore": "hmnd.lan", "wifi-0": "hmnd.lan" }, "dropins": "no drop-ins", "zones": { "local": true, "route53": true, "private": true, "public": true } }, "wifi": [ { "name": "wifi-1", "driver": "iwlwifi", "firmware": "89.123cf747.0 ty-a0-gf-a0-89.uc", "mac": "58:6d:67:f7:d1:37", "ssid": null, "channel": null, "configured_ssid": null }, { "name": "wifi-0", "driver": "iwlwifi", "firmware": "89.123cf747.0 ty-a0-gf-a0-89.uc", "mac": "58:6d:67:f7:d1:36", "ssid": "Alpha", "channel": 104, "configured_ssid": "Alpha", "station": { "rx_packets": 2409003, "tx_packets": 31997, "tx_retries": 28, "tx_failed": 0, "beacon_loss": 0, "rx_drop_misc": 23572, "signal_dbm": -39, "signal_per_antenna": "[-39, -46]", "tx_bitrate": "286.7 MBit/s HE-MCS 11 HE-NSS 2 HE-GI 0 HE-DCM 0", "rx_bitrate": "286.7 MBit/s HE-MCS 11 HE-NSS 2 HE-GI 0 HE-DCM 0", "connected_seconds": 21987 } } ], "interfaces": [ { "name": "eth-shore", "operstate": "UP", "ip": "10.44.10.14", "mac": "00:13:95:68:c3:93" }, { "name": "wifi-0", "operstate": "UP", "ip": "10.44.10.111", "mac": "58:6d:67:f7:d1:36" }, { "name": "wifi-1", "operstate": "DOWN", "ip": null, "mac": "58:6d:67:f7:d0:ff" } ], "nm_profiles": [ { "name": "eth-shore", "type": "ethernet", "interface": "eth-shore", "ssid": null, "autoconnect": true, "priority": 0, "key_mgmt": null, "zone": "external", "state": "activated" }, { "name": "sw-internal", "type": "ethernet", "interface": "sw-internal", "ssid": null, "autoconnect": false, "priority": 0, "key_mgmt": null, "zone": null, "state": "activated" }, { "name": "Alpha", "type": "wifi", "interface": "wifi-0", "ssid": "Alpha", "autoconnect": true, "priority": 900, "key_mgmt": "wpa-psk", "zone": "external", "state": "activated" }, { "name": "hmnd0", "type": "wireguard", "interface": "hmnd0", "ssid": null, "autoconnect": false, "priority": 0, "key_mgmt": null, "zone": null, "state": "activated" }, { "name": "hmnd-6ghz", "type": "wifi", "interface": "", "ssid": null, "autoconnect": true, "priority": 999, "key_mgmt": null, "zone": null, "state": "" } ], "dispatcher_scripts": [ "10-hmnd-disable-offloads", "70-hmnd-qos" ], "utilization": { "boots": [ { "index": -17, "start": "2026-03-30T23:20:14+00:00", "end": "2026-03-31T20:49:59+00:00", "duration_seconds": 77385, "running": false }, { "index": -16, "start": "2026-03-31T20:55:02+00:00", "end": "2026-04-01T13:38:39+00:00", "duration_seconds": 60217, "running": false }, { "index": -15, "start": "2026-04-01T13:44:26+00:00", "end": "2026-04-01T15:16:33+00:00", "duration_seconds": 5527, "running": false }, { "index": -14, "start": "2026-04-01T15:25:29+00:00", "end": "2026-04-01T15:43:00+00:00", "duration_seconds": 1051, "running": false }, { "index": -13, "start": "2026-04-01T15:45:45+00:00", "end": "2026-04-01T17:57:11+00:00", "duration_seconds": 7886, "running": false }, { "index": -12, "start": "2026-04-01T17:58:01+00:00", "end": "2026-04-01T19:28:44+00:00", "duration_seconds": 5443, "running": false }, { "index": -11, "start": "2026-04-01T19:32:03+00:00", "end": "2026-04-03T02:00:38+00:00", "duration_seconds": 109715, "running": false }, { "index": -10, "start": "2026-04-07T05:43:04+00:00", "end": "2026-04-07T07:56:51+00:00", "duration_seconds": 8027, "running": false }, { "index": -9, "start": "2026-04-07T07:57:39+00:00", "end": "2026-04-09T08:12:22+00:00", "duration_seconds": 173683, "running": false }, { "index": -8, "start": "2026-04-09T08:13:53+00:00", "end": "2026-04-09T12:34:54+00:00", "duration_seconds": 15661, "running": false }, { "index": -7, "start": "2026-04-09T12:36:53+00:00", "end": "2026-04-09T12:37:44+00:00", "duration_seconds": 51, "running": false }, { "index": -6, "start": "2026-04-09T12:40:38+00:00", "end": "2026-04-09T13:41:15+00:00", "duration_seconds": 3637, "running": false }, { "index": -5, "start": "2026-04-09T13:42:07+00:00", "end": "2026-04-09T13:50:46+00:00", "duration_seconds": 519, "running": false }, { "index": -4, "start": "2026-04-09T13:51:36+00:00", "end": "2026-04-09T21:05:44+00:00", "duration_seconds": 26048, "running": false }, { "index": -3, "start": "2026-04-09T21:08:49+00:00", "end": "2026-04-09T21:23:43+00:00", "duration_seconds": 894, "running": false }, { "index": -2, "start": "2026-04-09T21:25:02+00:00", "end": "2026-04-09T21:35:38+00:00", "duration_seconds": 636, "running": false }, { "index": -1, "start": "2026-04-09T21:36:57+00:00", "end": "2026-04-10T07:08:08+00:00", "duration_seconds": 34271, "running": false }, { "index": 0, "start": "2026-04-10T07:09:11+00:00", "end": "2026-04-10T13:15:55+00:00", "duration_seconds": 22004, "running": false } ], "daily_seconds": { "2026-03-30": 2386.0, "2026-03-31": 86097.0, "2026-04-01": 85103.0, "2026-04-02": 86400.0, "2026-04-03": 7238.0, "2026-04-07": 65768.0, "2026-04-08": 86400.0, "2026-04-09": 85571.0, "2026-04-10": 47692.0 }, "total_boots_7d": 0, "total_on_seconds_7d": 0, "utilization_7d_pct": 0.0 }, "_source_mtime": "2026-04-10T13:16:49+00:00" }, "orin": { "kernel": "5.15.148-tegra", "os": "Ubuntu 22.04", "architecture": "aarch64", "hostname": "orinalpha11", "uptime_seconds": 4554, "memory_total_mb": 54792, "memory_free_mb": 47130, "release": null, "collected_at": "2026-04-10T13:15:58Z", "dns": { "servers": { "global": "8.8.8.8" }, "search_domains": {}, "dropins": "[Resolve]\nFallbackDNS=8.8.8.8", "zones": { "local": false, "route53": false, "private": false, "public": true } }, "docker_running": false, "can_interfaces_up": 0, "model": "Newton + Orin AGXi\u0000", "utilization": { "boots": [ { "index": 0, "start": "2025-08-26T15:23:06+00:00", "end": "2026-04-10T13:16:17+00:00", "duration_seconds": 19605191, "running": false } ], "daily_seconds": { "2025-08-26": 31014.0, "2025-08-27": 86400.0, "2025-08-28": 86400.0, "2025-08-29": 86400.0, "2025-08-30": 86400.0, "2025-08-31": 86400.0, "2025-09-01": 86400.0, "2025-09-02": 86400.0, "2025-09-03": 86400.0, "2025-09-04": 86400.0, "2025-09-05": 86400.0, "2025-09-06": 86400.0, "2025-09-07": 86400.0, "2025-09-08": 86400.0, "2025-09-09": 86400.0, "2025-09-10": 86400.0, "2025-09-11": 86400.0, "2025-09-12": 86400.0, "2025-09-13": 86400.0, "2025-09-14": 86400.0, "2025-09-15": 86400.0, "2025-09-16": 86400.0, "2025-09-17": 86400.0, "2025-09-18": 86400.0, "2025-09-19": 86400.0, "2025-09-20": 86400.0, "2025-09-21": 86400.0, "2025-09-22": 86400.0, "2025-09-23": 86400.0, "2025-09-24": 86400.0, "2025-09-25": 86400.0, "2025-09-26": 86400.0, "2025-09-27": 86400.0, "2025-09-28": 86400.0, "2025-09-29": 86400.0, "2025-09-30": 86400.0, "2025-10-01": 86400.0, "2025-10-02": 86400.0, "2025-10-03": 86400.0, "2025-10-04": 86400.0, "2025-10-05": 86400.0, "2025-10-06": 86400.0, "2025-10-07": 86400.0, "2025-10-08": 86400.0, "2025-10-09": 86400.0, "2025-10-10": 86400.0, "2025-10-11": 86400.0, "2025-10-12": 86400.0, "2025-10-13": 86400.0, "2025-10-14": 86400.0, "2025-10-15": 86400.0, "2025-10-16": 86400.0, "2025-10-17": 86400.0, "2025-10-18": 86400.0, "2025-10-19": 86400.0, "2025-10-20": 86400.0, "2025-10-21": 86400.0, "2025-10-22": 86400.0, "2025-10-23": 86400.0, "2025-10-24": 86400.0, "2025-10-25": 86400.0, "2025-10-26": 86400.0, "2025-10-27": 86400.0, "2025-10-28": 86400.0, "2025-10-29": 86400.0, "2025-10-30": 86400.0, "2025-10-31": 86400.0, "2025-11-01": 86400.0, "2025-11-02": 86400.0, "2025-11-03": 86400.0, "2025-11-04": 86400.0, "2025-11-05": 86400.0, "2025-11-06": 86400.0, "2025-11-07": 86400.0, "2025-11-08": 86400.0, "2025-11-09": 86400.0, "2025-11-10": 86400.0, "2025-11-11": 86400.0, "2025-11-12": 86400.0, "2025-11-13": 86400.0, "2025-11-14": 86400.0, "2025-11-15": 86400.0, "2025-11-16": 86400.0, "2025-11-17": 86400.0, "2025-11-18": 86400.0, "2025-11-19": 86400.0, "2025-11-20": 86400.0, "2025-11-21": 86400.0, "2025-11-22": 86400.0, "2025-11-23": 86400.0, "2025-11-24": 86400.0, "2025-11-25": 86400.0, "2025-11-26": 86400.0, "2025-11-27": 86400.0, "2025-11-28": 86400.0, "2025-11-29": 86400.0, "2025-11-30": 86400.0, "2025-12-01": 86400.0, "2025-12-02": 86400.0, "2025-12-03": 86400.0, "2025-12-04": 86400.0, "2025-12-05": 86400.0, "2025-12-06": 86400.0, "2025-12-07": 86400.0, "2025-12-08": 86400.0, "2025-12-09": 86400.0, "2025-12-10": 86400.0, "2025-12-11": 86400.0, "2025-12-12": 86400.0, "2025-12-13": 86400.0, "2025-12-14": 86400.0, "2025-12-15": 86400.0, "2025-12-16": 86400.0, "2025-12-17": 86400.0, "2025-12-18": 86400.0, "2025-12-19": 86400.0, "2025-12-20": 86400.0, "2025-12-21": 86400.0, "2025-12-22": 86400.0, "2025-12-23": 86400.0, "2025-12-24": 86400.0, "2025-12-25": 86400.0, "2025-12-26": 86400.0, "2025-12-27": 86400.0, "2025-12-28": 86400.0, "2025-12-29": 86400.0, "2025-12-30": 86400.0, "2025-12-31": 86400.0, "2026-01-01": 86400.0, "2026-01-02": 86400.0, "2026-01-03": 86400.0, "2026-01-04": 86400.0, "2026-01-05": 86400.0, "2026-01-06": 86400.0, "2026-01-07": 86400.0, "2026-01-08": 86400.0, "2026-01-09": 86400.0, "2026-01-10": 86400.0, "2026-01-11": 86400.0, "2026-01-12": 86400.0, "2026-01-13": 86400.0, "2026-01-14": 86400.0, "2026-01-15": 86400.0, "2026-01-16": 86400.0, "2026-01-17": 86400.0, "2026-01-18": 86400.0, "2026-01-19": 86400.0, "2026-01-20": 86400.0, "2026-01-21": 86400.0, "2026-01-22": 86400.0, "2026-01-23": 86400.0, "2026-01-24": 86400.0, "2026-01-25": 86400.0, "2026-01-26": 86400.0, "2026-01-27": 86400.0, "2026-01-28": 86400.0, "2026-01-29": 86400.0, "2026-01-30": 86400.0, "2026-01-31": 86400.0, "2026-02-01": 86400.0, "2026-02-02": 86400.0, "2026-02-03": 86400.0, "2026-02-04": 86400.0, "2026-02-05": 86400.0, "2026-02-06": 86400.0, "2026-02-07": 86400.0, "2026-02-08": 86400.0, "2026-02-09": 86400.0, "2026-02-10": 86400.0, "2026-02-11": 86400.0, "2026-02-12": 86400.0, "2026-02-13": 86400.0, "2026-02-14": 86400.0, "2026-02-15": 86400.0, "2026-02-16": 86400.0, "2026-02-17": 86400.0, "2026-02-18": 86400.0, "2026-02-19": 86400.0, "2026-02-20": 86400.0, "2026-02-21": 86400.0, "2026-02-22": 86400.0, "2026-02-23": 86400.0, "2026-02-24": 86400.0, "2026-02-25": 86400.0, "2026-02-26": 86400.0, "2026-02-27": 86400.0, "2026-02-28": 86400.0, "2026-03-01": 86400.0, "2026-03-02": 86400.0, "2026-03-03": 86400.0, "2026-03-04": 86400.0, "2026-03-05": 86400.0, "2026-03-06": 86400.0, "2026-03-07": 86400.0, "2026-03-08": 86400.0, "2026-03-09": 86400.0, "2026-03-10": 86400.0, "2026-03-11": 86400.0, "2026-03-12": 86400.0, "2026-03-13": 86400.0, "2026-03-14": 86400.0, "2026-03-15": 86400.0, "2026-03-16": 86400.0, "2026-03-17": 86400.0, "2026-03-18": 86400.0, "2026-03-19": 86400.0, "2026-03-20": 86400.0, "2026-03-21": 86400.0, "2026-03-22": 86400.0, "2026-03-23": 86400.0, "2026-03-24": 86400.0, "2026-03-25": 86400.0, "2026-03-26": 86400.0, "2026-03-27": 86400.0, "2026-03-28": 86400.0, "2026-03-29": 86400.0, "2026-03-30": 86400.0, "2026-03-31": 86400.0, "2026-04-01": 86400.0, "2026-04-02": 86400.0, "2026-04-03": 86400.0, "2026-04-04": 86400.0, "2026-04-05": 86400.0, "2026-04-06": 86400.0, "2026-04-07": 86400.0, "2026-04-08": 86400.0, "2026-04-09": 86400.0, "2026-04-10": 47777.0 }, "total_boots_7d": 0, "total_on_seconds_7d": 0, "utilization_7d_pct": 0.0 }, "_source_mtime": "2026-04-10T13:16:49+00:00" } }, "aw2": { "orin": { "kernel": "5.15.148-tegra", "os": "Ubuntu 22.04", "architecture": "aarch64", "hostname": "orinalpha2", "uptime_seconds": "10364", "memory_total_mb": "54793", "memory_free_mb": "47974", "release": null, "collected_at": "2026-04-09T15:31:24Z", "dns": { "servers": { "global": "192.168.1.1" }, "search_domains": {}, "dropins": "[Resolve]\nDNS=192.168.1.1\nFallbackDNS=8.8.8.8\n[Resolve]\nFallbackDNS=8.8.8.8", "zones": { "local": false, "route53": false, "private": false, "public": true } }, "docker_running": false, "can_interfaces_up": 0, "model": "Newton + Orin AGXi\u0000", "utilization": { "boots": [], "daily_seconds": {}, "total_boots_7d": 0, "total_on_seconds_7d": 0, "utilization_7d_pct": 0.0 }, "_source_mtime": "2026-04-09T15:32:26+00:00" }, "rtpc": { "kernel": "5.15.197-rt91-hmnd-0", "os": "Ubuntu 22.04", "architecture": "x86_64", "hostname": "alpha2", "uptime_seconds": "10360", "memory_total_mb": "31844", "memory_free_mb": "28975", "release": null, "collected_at": "2026-04-09T15:30:23Z", "dns": { "servers": { "wifi-0": "172.31.0.2 8.8.8.8 1.1.1.1 10.44.10.1" }, "search_domains": { "wifi-0": "hmnd.lan" }, "dropins": "[Resolve]\nDNSStubListenerExtra=192.168.1.1", "zones": { "local": false, "route53": true, "private": true, "public": true } }, "wifi": [ { "name": "wifi-1", "driver": "iwlwifi", "firmware": "89.123cf747.0 ty-a0-gf-a0-89.uc", "mac": "8c:f8:c5:96:2e:cd", "ssid": null, "channel": null, "configured_ssid": null }, { "name": "wifi-0", "driver": "iwlwifi", "firmware": "89.123cf747.0 ty-a0-gf-a0-89.uc", "mac": "8c:f8:c5:96:2e:cc", "ssid": "Alpha", "channel": 104, "configured_ssid": "Alpha", "station": { "rx_packets": 162955, "tx_packets": 39453, "tx_retries": 13702, "tx_failed": 1, "beacon_loss": 0, "rx_drop_misc": 12707, "signal_dbm": -46, "signal_per_antenna": "[-46, -55]", "tx_bitrate": "270.8 MBit/s HE-MCS 11 HE-NSS 2 HE-GI 1 HE-DCM 0", "rx_bitrate": "286.7 MBit/s HE-MCS 11 HE-NSS 2 HE-GI 0 HE-DCM 0", "connected_seconds": 6435 } } ], "interfaces": [ { "name": "eth-shore", "operstate": "DOWN", "ip": null, "mac": "00:13:95:61:a7:cd" }, { "name": "wifi-0", "operstate": "UP", "ip": "10.44.10.10", "mac": "8c:f8:c5:96:2e:cc" }, { "name": "wifi-1", "operstate": "DOWN", "ip": null, "mac": "8c:f8:c5:b3:81:c5" } ], "nm_profiles": [ { "name": "Alpha", "type": "wifi", "interface": "wifi-0", "ssid": "Alpha", "autoconnect": true, "priority": 999, "key_mgmt": "wpa-psk", "zone": "external", "state": "activated" }, { "name": "sw-internal", "type": "ethernet", "interface": "sw-internal", "ssid": null, "autoconnect": false, "priority": 0, "key_mgmt": null, "zone": null, "state": "activated" }, { "name": "hmnd0", "type": "wireguard", "interface": "hmnd0", "ssid": null, "autoconnect": false, "priority": 0, "key_mgmt": null, "zone": null, "state": "activated" }, { "name": "eth-shore", "type": "ethernet", "interface": "", "ssid": null, "autoconnect": true, "priority": 0, "key_mgmt": null, "zone": null, "state": "" }, { "name": "hmnd-6ghz", "type": "wifi", "interface": "", "ssid": null, "autoconnect": true, "priority": 999, "key_mgmt": null, "zone": null, "state": "" }, { "name": "hmnd-test", "type": "wifi", "interface": "", "ssid": null, "autoconnect": true, "priority": 999, "key_mgmt": null, "zone": null, "state": "" }, { "name": "Roadkill", "type": "wifi", "interface": "", "ssid": null, "autoconnect": true, "priority": 0, "key_mgmt": null, "zone": null, "state": "" } ], "dispatcher_scripts": [ "10-hmnd-disable-offloads", "70-hmnd-qos" ], "utilization": { "boots": [], "daily_seconds": {}, "total_boots_7d": 0, "total_on_seconds_7d": 0, "utilization_7d_pct": 0.0 }, "_source_mtime": "2026-04-09T15:32:25+00:00" } }, "aw12": { "orin": { "kernel": "5.15.148-tegra", "os": "Ubuntu 22.04", "architecture": "aarch64", "hostname": "orinalpha12", "uptime_seconds": 77723, "memory_total_mb": 54792, "memory_free_mb": 39547, "release": null, "collected_at": "2026-04-10T12:20:25Z", "dns": { "servers": { "global": "10.44.10.1 192.168.1.1" }, "search_domains": { "global": "hmnd.lan lan.thehumanoid.ai robots.hmnd.ai" }, "dropins": "# Managed by Ansible (iac-robots playbook 04)\n# Points DNS at the RTPC stub resolver on the Moxa bridge.\n# 192.168.1.1 is site-independent \u2014 works at any location.\n# The RTPC forwards to whatever upstream DNS it learned from DHCP.\n[Resolve]\nDNS=192.168.1.1\nDomains=hmnd.lan lan.thehumanoid.ai robots.hmnd.ai\nFallbackDNS=8.8.8.8\n[Resolve]\nDNS=192.168.1.1\nFallbackDNS=8.8.8.8", "zones": { "local": true, "route53": true, "private": true, "public": true } }, "docker_running": false, "can_interfaces_up": 0, "model": "Newton + Orin AGXi\u0000", "utilization": { "boots": [ { "index": 0, "start": "2025-08-26T15:23:06+00:00", "end": "2026-04-10T12:20:53+00:00", "duration_seconds": 19601867, "running": false } ], "daily_seconds": { "2025-08-26": 31014.0, "2025-08-27": 86400.0, "2025-08-28": 86400.0, "2025-08-29": 86400.0, "2025-08-30": 86400.0, "2025-08-31": 86400.0, "2025-09-01": 86400.0, "2025-09-02": 86400.0, "2025-09-03": 86400.0, "2025-09-04": 86400.0, "2025-09-05": 86400.0, "2025-09-06": 86400.0, "2025-09-07": 86400.0, "2025-09-08": 86400.0, "2025-09-09": 86400.0, "2025-09-10": 86400.0, "2025-09-11": 86400.0, "2025-09-12": 86400.0, "2025-09-13": 86400.0, "2025-09-14": 86400.0, "2025-09-15": 86400.0, "2025-09-16": 86400.0, "2025-09-17": 86400.0, "2025-09-18": 86400.0, "2025-09-19": 86400.0, "2025-09-20": 86400.0, "2025-09-21": 86400.0, "2025-09-22": 86400.0, "2025-09-23": 86400.0, "2025-09-24": 86400.0, "2025-09-25": 86400.0, "2025-09-26": 86400.0, "2025-09-27": 86400.0, "2025-09-28": 86400.0, "2025-09-29": 86400.0, "2025-09-30": 86400.0, "2025-10-01": 86400.0, "2025-10-02": 86400.0, "2025-10-03": 86400.0, "2025-10-04": 86400.0, "2025-10-05": 86400.0, "2025-10-06": 86400.0, "2025-10-07": 86400.0, "2025-10-08": 86400.0, "2025-10-09": 86400.0, "2025-10-10": 86400.0, "2025-10-11": 86400.0, "2025-10-12": 86400.0, "2025-10-13": 86400.0, "2025-10-14": 86400.0, "2025-10-15": 86400.0, "2025-10-16": 86400.0, "2025-10-17": 86400.0, "2025-10-18": 86400.0, "2025-10-19": 86400.0, "2025-10-20": 86400.0, "2025-10-21": 86400.0, "2025-10-22": 86400.0, "2025-10-23": 86400.0, "2025-10-24": 86400.0, "2025-10-25": 86400.0, "2025-10-26": 86400.0, "2025-10-27": 86400.0, "2025-10-28": 86400.0, "2025-10-29": 86400.0, "2025-10-30": 86400.0, "2025-10-31": 86400.0, "2025-11-01": 86400.0, "2025-11-02": 86400.0, "2025-11-03": 86400.0, "2025-11-04": 86400.0, "2025-11-05": 86400.0, "2025-11-06": 86400.0, "2025-11-07": 86400.0, "2025-11-08": 86400.0, "2025-11-09": 86400.0, "2025-11-10": 86400.0, "2025-11-11": 86400.0, "2025-11-12": 86400.0, "2025-11-13": 86400.0, "2025-11-14": 86400.0, "2025-11-15": 86400.0, "2025-11-16": 86400.0, "2025-11-17": 86400.0, "2025-11-18": 86400.0, "2025-11-19": 86400.0, "2025-11-20": 86400.0, "2025-11-21": 86400.0, "2025-11-22": 86400.0, "2025-11-23": 86400.0, "2025-11-24": 86400.0, "2025-11-25": 86400.0, "2025-11-26": 86400.0, "2025-11-27": 86400.0, "2025-11-28": 86400.0, "2025-11-29": 86400.0, "2025-11-30": 86400.0, "2025-12-01": 86400.0, "2025-12-02": 86400.0, "2025-12-03": 86400.0, "2025-12-04": 86400.0, "2025-12-05": 86400.0, "2025-12-06": 86400.0, "2025-12-07": 86400.0, "2025-12-08": 86400.0, "2025-12-09": 86400.0, "2025-12-10": 86400.0, "2025-12-11": 86400.0, "2025-12-12": 86400.0, "2025-12-13": 86400.0, "2025-12-14": 86400.0, "2025-12-15": 86400.0, "2025-12-16": 86400.0, "2025-12-17": 86400.0, "2025-12-18": 86400.0, "2025-12-19": 86400.0, "2025-12-20": 86400.0, "2025-12-21": 86400.0, "2025-12-22": 86400.0, "2025-12-23": 86400.0, "2025-12-24": 86400.0, "2025-12-25": 86400.0, "2025-12-26": 86400.0, "2025-12-27": 86400.0, "2025-12-28": 86400.0, "2025-12-29": 86400.0, "2025-12-30": 86400.0, "2025-12-31": 86400.0, "2026-01-01": 86400.0, "2026-01-02": 86400.0, "2026-01-03": 86400.0, "2026-01-04": 86400.0, "2026-01-05": 86400.0, "2026-01-06": 86400.0, "2026-01-07": 86400.0, "2026-01-08": 86400.0, "2026-01-09": 86400.0, "2026-01-10": 86400.0, "2026-01-11": 86400.0, "2026-01-12": 86400.0, "2026-01-13": 86400.0, "2026-01-14": 86400.0, "2026-01-15": 86400.0, "2026-01-16": 86400.0, "2026-01-17": 86400.0, "2026-01-18": 86400.0, "2026-01-19": 86400.0, "2026-01-20": 86400.0, "2026-01-21": 86400.0, "2026-01-22": 86400.0, "2026-01-23": 86400.0, "2026-01-24": 86400.0, "2026-01-25": 86400.0, "2026-01-26": 86400.0, "2026-01-27": 86400.0, "2026-01-28": 86400.0, "2026-01-29": 86400.0, "2026-01-30": 86400.0, "2026-01-31": 86400.0, "2026-02-01": 86400.0, "2026-02-02": 86400.0, "2026-02-03": 86400.0, "2026-02-04": 86400.0, "2026-02-05": 86400.0, "2026-02-06": 86400.0, "2026-02-07": 86400.0, "2026-02-08": 86400.0, "2026-02-09": 86400.0, "2026-02-10": 86400.0, "2026-02-11": 86400.0, "2026-02-12": 86400.0, "2026-02-13": 86400.0, "2026-02-14": 86400.0, "2026-02-15": 86400.0, "2026-02-16": 86400.0, "2026-02-17": 86400.0, "2026-02-18": 86400.0, "2026-02-19": 86400.0, "2026-02-20": 86400.0, "2026-02-21": 86400.0, "2026-02-22": 86400.0, "2026-02-23": 86400.0, "2026-02-24": 86400.0, "2026-02-25": 86400.0, "2026-02-26": 86400.0, "2026-02-27": 86400.0, "2026-02-28": 86400.0, "2026-03-01": 86400.0, "2026-03-02": 86400.0, "2026-03-03": 86400.0, "2026-03-04": 86400.0, "2026-03-05": 86400.0, "2026-03-06": 86400.0, "2026-03-07": 86400.0, "2026-03-08": 86400.0, "2026-03-09": 86400.0, "2026-03-10": 86400.0, "2026-03-11": 86400.0, "2026-03-12": 86400.0, "2026-03-13": 86400.0, "2026-03-14": 86400.0, "2026-03-15": 86400.0, "2026-03-16": 86400.0, "2026-03-17": 86400.0, "2026-03-18": 86400.0, "2026-03-19": 86400.0, "2026-03-20": 86400.0, "2026-03-21": 86400.0, "2026-03-22": 86400.0, "2026-03-23": 86400.0, "2026-03-24": 86400.0, "2026-03-25": 86400.0, "2026-03-26": 86400.0, "2026-03-27": 86400.0, "2026-03-28": 86400.0, "2026-03-29": 86400.0, "2026-03-30": 86400.0, "2026-03-31": 86400.0, "2026-04-01": 86400.0, "2026-04-02": 86400.0, "2026-04-03": 86400.0, "2026-04-04": 86400.0, "2026-04-05": 86400.0, "2026-04-06": 86400.0, "2026-04-07": 86400.0, "2026-04-08": 86400.0, "2026-04-09": 86400.0, "2026-04-10": 44453.0 }, "total_boots_7d": 0, "total_on_seconds_7d": 0, "utilization_7d_pct": 0.0 }, "_source_mtime": "2026-04-10T12:21:23+00:00" }, "rtpc": { "kernel": "5.15.197-rt91-hmnd-0", "os": "Ubuntu 22.04", "architecture": "x86_64", "hostname": "alpha12", "uptime_seconds": 77769, "memory_total_mb": 47921, "memory_free_mb": 34708, "release": null, "collected_at": "2026-04-10T12:19:40Z", "dns": { "servers": { "wifi-0": "10.44.10.1" }, "search_domains": { "wifi-0": "hmnd.lan" }, "dropins": "# Managed by Ansible (iac-robots playbook 04)\n# Exposes the DNS stub resolver on sw-internal (192.168.1.1)\n# so the Orin and other Moxa devices can use the RTPC as a DNS forwarder.\n# The stub resolver automatically uses whatever upstream DNS the RTPC\n# learned from DHCP \u2014 works at any site without reconfiguration.\n[Resolve]\nDNSStubListenerExtra=192.168.1.1", "zones": { "local": true, "route53": true, "private": true, "public": true } }, "wifi": [ { "name": "wifi-1", "driver": "iwlwifi", "firmware": "89.123cf747.0 ty-a0-gf-a0-89.uc", "mac": "c4:03:a8:32:ee:44", "ssid": null, "channel": null, "configured_ssid": null }, { "name": "wifi-0", "driver": "iwlwifi", "firmware": "89.123cf747.0 ty-a0-gf-a0-89.uc", "mac": "c4:03:a8:32:ee:43", "ssid": "Alpha", "channel": 157, "configured_ssid": "Alpha", "station": { "rx_packets": 29712955, "tx_packets": 60993054, "tx_retries": 8651, "tx_failed": 0, "beacon_loss": 0, "rx_drop_misc": 87401, "signal_dbm": -44, "signal_per_antenna": "[-52, -44]", "tx_bitrate": "286.7 MBit/s HE-MCS 11 HE-NSS 2 HE-GI 0 HE-DCM 0", "rx_bitrate": "286.7 MBit/s HE-MCS 11 HE-NSS 2 HE-GI 0 HE-DCM 0", "connected_seconds": 77770 } } ], "interfaces": [ { "name": "eth-shore", "operstate": "DOWN", "ip": null, "mac": "00:13:95:68:c2:fb" }, { "name": "wifi-0", "operstate": "UP", "ip": "10.44.10.22", "mac": "c4:03:a8:32:ee:43" }, { "name": "wifi-1", "operstate": "DOWN", "ip": null, "mac": "c4:03:a8:32:e6:cd" } ], "nm_profiles": [ { "name": "Alpha", "type": "wifi", "interface": "wifi-0", "ssid": "Alpha", "autoconnect": true, "priority": 999, "key_mgmt": "wpa-psk", "zone": "external", "state": "activated" }, { "name": "sw-internal", "type": "ethernet", "interface": "sw-internal", "ssid": null, "autoconnect": false, "priority": 0, "key_mgmt": null, "zone": null, "state": "activated" }, { "name": "hmnd0", "type": "wireguard", "interface": "hmnd0", "ssid": null, "autoconnect": false, "priority": 0, "key_mgmt": null, "zone": null, "state": "activated" }, { "name": "eth-shore", "type": "ethernet", "interface": "", "ssid": null, "autoconnect": true, "priority": 0, "key_mgmt": null, "zone": null, "state": "" } ], "dispatcher_scripts": [ "10-hmnd-disable-offloads", "70-hmnd-qos" ], "utilization": { "boots": [ { "index": -50, "start": "2026-03-24T11:03:18+00:00", "end": "2026-03-24T16:15:48+00:00", "duration_seconds": 18750, "running": false }, { "index": -49, "start": "2026-03-24T16:16:40+00:00", "end": "2026-03-24T17:25:56+00:00", "duration_seconds": 4156, "running": false }, { "index": -48, "start": "2026-03-24T19:40:44+00:00", "end": "2026-03-24T19:43:50+00:00", "duration_seconds": 186, "running": false }, { "index": -47, "start": "2026-03-24T20:52:51+00:00", "end": "2026-03-24T20:58:31+00:00", "duration_seconds": 340, "running": false }, { "index": -46, "start": "2026-03-24T21:08:33+00:00", "end": "2026-03-24T21:10:26+00:00", "duration_seconds": 113, "running": false }, { "index": -45, "start": "2026-03-25T08:41:37+00:00", "end": "2026-03-25T08:48:07+00:00", "duration_seconds": 390, "running": false }, { "index": -44, "start": "2026-03-25T09:16:13+00:00", "end": "2026-03-25T09:25:10+00:00", "duration_seconds": 537, "running": false }, { "index": -43, "start": "2026-03-25T09:36:45+00:00", "end": "2026-03-25T09:38:59+00:00", "duration_seconds": 134, "running": false }, { "index": -42, "start": "2026-03-25T09:40:27+00:00", "end": "2026-03-25T09:41:48+00:00", "duration_seconds": 81, "running": false }, { "index": -41, "start": "2026-03-25T10:07:31+00:00", "end": "2026-03-25T10:08:58+00:00", "duration_seconds": 87, "running": false }, { "index": -40, "start": "2026-03-25T10:25:04+00:00", "end": "2026-03-25T10:46:15+00:00", "duration_seconds": 1271, "running": false }, { "index": -39, "start": "2026-03-25T12:22:30+00:00", "end": "2026-03-25T12:24:18+00:00", "duration_seconds": 108, "running": false }, { "index": -38, "start": "2026-03-25T12:26:24+00:00", "end": "2026-03-25T12:27:35+00:00", "duration_seconds": 71, "running": false }, { "index": -37, "start": "2026-03-25T12:33:28+00:00", "end": "2026-03-25T12:41:07+00:00", "duration_seconds": 459, "running": false }, { "index": -36, "start": "2026-03-25T12:43:01+00:00", "end": "2026-03-25T16:29:56+00:00", "duration_seconds": 13615, "running": false }, { "index": -35, "start": "2026-03-25T17:07:14+00:00", "end": "2026-03-25T17:27:21+00:00", "duration_seconds": 1207, "running": false }, { "index": -34, "start": "2026-03-25T17:28:55+00:00", "end": "2026-03-25T19:41:53+00:00", "duration_seconds": 7978, "running": false }, { "index": -33, "start": "2026-03-25T20:45:59+00:00", "end": "2026-03-25T23:59:34+00:00", "duration_seconds": 11615, "running": false }, { "index": -32, "start": "2026-03-26T00:01:36+00:00", "end": "2026-03-26T02:37:59+00:00", "duration_seconds": 9383, "running": false }, { "index": -31, "start": "2026-03-26T04:15:45+00:00", "end": "2026-03-26T04:42:49+00:00", "duration_seconds": 1624, "running": false }, { "index": -30, "start": "2026-03-26T07:01:38+00:00", "end": "2026-03-26T13:53:09+00:00", "duration_seconds": 24691, "running": false }, { "index": -29, "start": "2026-03-26T13:53:58+00:00", "end": "2026-03-26T14:24:25+00:00", "duration_seconds": 1827, "running": false }, { "index": -28, "start": "2026-03-26T14:25:15+00:00", "end": "2026-03-26T17:43:56+00:00", "duration_seconds": 11921, "running": false }, { "index": -27, "start": "2026-03-26T18:02:47+00:00", "end": "2026-03-26T18:16:22+00:00", "duration_seconds": 815, "running": false }, { "index": -26, "start": "2026-03-27T20:56:59+00:00", "end": "2026-03-27T21:08:53+00:00", "duration_seconds": 714, "running": false }, { "index": -25, "start": "2026-03-27T21:09:30+00:00", "end": "2026-03-27T21:09:30+00:00", "duration_seconds": 0, "running": false }, { "index": -24, "start": "2026-03-27T22:51:56+00:00", "end": "2026-03-27T22:58:47+00:00", "duration_seconds": 411, "running": false }, { "index": -23, "start": "2026-03-28T17:22:53+00:00", "end": "2026-03-28T17:49:11+00:00", "duration_seconds": 1578, "running": false }, { "index": -22, "start": "2026-03-30T05:45:40+00:00", "end": "2026-03-30T07:21:38+00:00", "duration_seconds": 5758, "running": false }, { "index": -21, "start": "2026-03-30T09:17:10+00:00", "end": "2026-03-30T09:18:46+00:00", "duration_seconds": 96, "running": false }, { "index": -20, "start": "2026-03-30T09:21:23+00:00", "end": "2026-03-30T09:22:25+00:00", "duration_seconds": 62, "running": false }, { "index": -19, "start": "2026-03-30T09:39:31+00:00", "end": "2026-03-30T09:42:10+00:00", "duration_seconds": 159, "running": false }, { "index": -18, "start": "2026-03-30T14:31:39+00:00", "end": "2026-03-30T14:33:14+00:00", "duration_seconds": 95, "running": false }, { "index": -17, "start": "2026-03-30T14:34:15+00:00", "end": "2026-03-30T14:36:51+00:00", "duration_seconds": 156, "running": false }, { "index": -16, "start": "2026-03-30T14:39:27+00:00", "end": "2026-03-30T14:46:32+00:00", "duration_seconds": 425, "running": false }, { "index": -15, "start": "2026-03-30T14:49:52+00:00", "end": "2026-03-30T16:39:49+00:00", "duration_seconds": 6597, "running": false }, { "index": -14, "start": "2026-03-31T12:49:50+00:00", "end": "2026-03-31T17:41:17+00:00", "duration_seconds": 17487, "running": false }, { "index": -13, "start": "2026-04-01T11:27:35+00:00", "end": "2026-04-01T19:29:48+00:00", "duration_seconds": 28933, "running": false }, { "index": -12, "start": "2026-04-02T08:49:52+00:00", "end": "2026-04-02T09:28:56+00:00", "duration_seconds": 2344, "running": false }, { "index": -11, "start": "2026-04-02T09:29:40+00:00", "end": "2026-04-02T09:31:23+00:00", "duration_seconds": 103, "running": false }, { "index": -10, "start": "2026-04-02T09:32:09+00:00", "end": "2026-04-02T16:44:48+00:00", "duration_seconds": 25959, "running": false }, { "index": -9, "start": "2026-04-02T17:49:20+00:00", "end": "2026-04-02T17:50:55+00:00", "duration_seconds": 95, "running": false }, { "index": -8, "start": "2026-04-02T17:57:54+00:00", "end": "2026-04-02T18:01:15+00:00", "duration_seconds": 201, "running": false }, { "index": -7, "start": "2026-04-02T18:03:53+00:00", "end": "2026-04-02T18:09:52+00:00", "duration_seconds": 359, "running": false }, { "index": -6, "start": "2026-04-02T18:13:38+00:00", "end": "2026-04-02T18:16:28+00:00", "duration_seconds": 170, "running": false }, { "index": -5, "start": "2026-04-07T09:44:28+00:00", "end": "2026-04-07T13:33:02+00:00", "duration_seconds": 13714, "running": false }, { "index": -4, "start": "2026-04-07T14:19:17+00:00", "end": "2026-04-07T16:43:00+00:00", "duration_seconds": 8623, "running": false }, { "index": -3, "start": "2026-04-08T07:51:08+00:00", "end": "2026-04-08T17:01:57+00:00", "duration_seconds": 33049, "running": false }, { "index": -2, "start": "2026-04-09T08:36:44+00:00", "end": "2026-04-09T08:48:15+00:00", "duration_seconds": 691, "running": false }, { "index": -1, "start": "2026-04-09T09:11:29+00:00", "end": "2026-04-09T14:27:20+00:00", "duration_seconds": 18951, "running": false }, { "index": 0, "start": "2026-04-09T14:43:39+00:00", "end": "2026-04-10T12:20:06+00:00", "duration_seconds": 77787, "running": false } ], "daily_seconds": { "2026-03-24": 23545.0, "2026-03-25": 37553.0, "2026-03-26": 50261.0, "2026-03-27": 1125.0, "2026-03-28": 1578.0, "2026-03-30": 13348.0, "2026-03-31": 17487.0, "2026-04-01": 28933.0, "2026-04-02": 29231.0, "2026-04-07": 22337.0, "2026-04-08": 33049.0, "2026-04-09": 53023.0, "2026-04-10": 44406.0 }, "total_boots_7d": 0, "total_on_seconds_7d": 0, "utilization_7d_pct": 0.0 }, "_source_mtime": "2026-04-10T12:21:22+00:00" } }, "aw3": { "orin": { "kernel": "5.15.148-tegra", "os": "Ubuntu 22.04", "architecture": "aarch64", "hostname": "orinalpha3", "uptime_seconds": "675", "memory_total_mb": "54793", "memory_free_mb": "45612", "release": null, "collected_at": "2026-04-07T14:31:25Z", "dns": { "servers": { "global": "192.168.1.1" }, "search_domains": { "global": "hmnd.lan lan.thehumanoid.ai robots.hmnd.ai" }, "dropins": "# Managed by Ansible (iac-robots playbook 04)\n# Points DNS at the RTPC stub resolver on the Moxa bridge.\n# 192.168.1.1 is site-independent \u2014 works at any location.\n# The RTPC forwards to whatever upstream DNS it learned from DHCP.\n[Resolve]\nDNS=192.168.1.1\nDomains=hmnd.lan lan.thehumanoid.ai robots.hmnd.ai\nFallbackDNS=8.8.8.8\n[Resolve]\nFallbackDNS=8.8.8.8", "zones": { "local": true, "route53": true, "private": true, "public": true } }, "docker_running": false, "can_interfaces_up": 0, "model": "Newton + Orin AGXi\u0000", "utilization": { "boots": [], "daily_seconds": {}, "total_boots_7d": 0, "total_on_seconds_7d": 0, "utilization_7d_pct": 0.0 }, "_source_mtime": "2026-04-07T14:32:41+00:00" }, "rtpc": { "kernel": "5.15.197-rt91-hmnd-0", "os": "Ubuntu 22.04", "architecture": "x86_64", "hostname": "alpha3", "uptime_seconds": "699", "memory_total_mb": "47930", "memory_free_mb": "41604", "release": null, "collected_at": "2026-04-07T14:30:21Z", "dns": { "servers": { "wifi-0": "10.44.10.1" }, "search_domains": { "wifi-0": "hmnd.lan" }, "dropins": "[Resolve]\nDNSStubListenerExtra=192.168.2.1\n\n\n\n\n\n\n# Managed by Ansible (iac-robots playbook 04)\n# Exposes the DNS stub resolver on sw-internal (192.168.1.1)\n# so the Orin and other Moxa devices can use the RTPC as a DNS forwarder.\n# The stub resolver automatically uses whatever upstream DNS the RTPC\n# learned from DHCP \u2014 works at any site without reconfiguration.\n[Resolve]\nDNSStubListenerExtra=192.168.1.1", "zones": { "local": true, "route53": true, "private": true, "public": true } }, "wifi": [ { "name": "wifi-1", "driver": "iwlwifi", "firmware": "89.123cf747.0 ty-a0-gf-a0-89.uc", "mac": "8c:f8:c5:b3:81:12", "ssid": null, "channel": null, "configured_ssid": null }, { "name": "wifi-0", "driver": "iwlwifi", "firmware": "89.123cf747.0 ty-a0-gf-a0-89.uc", "mac": "8c:f8:c5:b3:81:11", "ssid": "Alpha", "channel": 64, "configured_ssid": "Alpha", "station": { "rx_packets": 139751, "tx_packets": 492072, "tx_retries": 1881, "tx_failed": 0, "beacon_loss": 0, "rx_drop_misc": 2886, "signal_dbm": -41, "signal_per_antenna": "[-45, -41]", "tx_bitrate": "286.7 MBit/s HE-MCS 11 HE-NSS 2 HE-GI 0 HE-DCM 0", "rx_bitrate": "229.4 MBit/s HE-MCS 9 HE-NSS 2 HE-GI 0 HE-DCM 0", "connected_seconds": 722 } } ], "interfaces": [ { "name": "eth-shore", "operstate": "DOWN", "ip": null, "mac": "00:13:95:65:05:ca" }, { "name": "wifi-0", "operstate": "UP", "ip": "10.44.10.13", "mac": "8c:f8:c5:b3:81:11" }, { "name": "wifi-1", "operstate": "DOWN", "ip": null, "mac": "8c:f8:c5:96:2d:cd" } ], "nm_profiles": [ { "name": "Alpha", "type": "wifi", "interface": "wifi-0", "ssid": "Alpha", "autoconnect": true, "priority": 500, "key_mgmt": "wpa-psk", "zone": "external", "state": "activated" }, { "name": "sw-internal", "type": "ethernet", "interface": "sw-internal", "ssid": null, "autoconnect": false, "priority": 0, "key_mgmt": null, "zone": null, "state": "activated" }, { "name": "hmnd0", "type": "wireguard", "interface": "hmnd0", "ssid": null, "autoconnect": false, "priority": 0, "key_mgmt": null, "zone": null, "state": "activated" }, { "name": "eth-shore", "type": "ethernet", "interface": "", "ssid": null, "autoconnect": true, "priority": 0, "key_mgmt": null, "zone": null, "state": "" }, { "name": "hmnd-infra", "type": "wifi", "interface": "", "ssid": null, "autoconnect": false, "priority": 100, "key_mgmt": null, "zone": null, "state": "" }, { "name": "hmnd-robot", "type": "wifi", "interface": "", "ssid": null, "autoconnect": true, "priority": 999, "key_mgmt": null, "zone": null, "state": "" } ], "dispatcher_scripts": [ "10-hmnd-disable-offloads", "70-hmnd-qos" ], "utilization": { "boots": [], "daily_seconds": {}, "total_boots_7d": 0, "total_on_seconds_7d": 0, "utilization_7d_pct": 0.0 }, "_source_mtime": "2026-04-07T14:32:40+00:00" } }, "aw4": { "orin": { "kernel": "5.15.148-tegra", "os": "Ubuntu 22.04", "architecture": "aarch64", "hostname": "orinalpha4", "uptime_seconds": 164642, "memory_total_mb": 54793, "memory_free_mb": 40376, "release": null, "collected_at": "2026-04-10T12:20:20Z", "dns": { "servers": { "global": "10.44.10.1 192.168.1.1" }, "search_domains": { "global": "hmnd.lan lan.thehumanoid.ai robots.hmnd.ai" }, "dropins": "# Managed by Ansible (iac-robots playbook 04)\n# Points DNS at the RTPC stub resolver on the Moxa bridge.\n# 192.168.1.1 is site-independent \u2014 works at any location.\n# The RTPC forwards to whatever upstream DNS it learned from DHCP.\n[Resolve]\nDNS=192.168.1.1\nDomains=hmnd.lan lan.thehumanoid.ai robots.hmnd.ai\nFallbackDNS=8.8.8.8\n[Resolve]\nFallbackDNS=8.8.8.8", "zones": { "local": true, "route53": true, "private": true, "public": true } }, "docker_running": false, "can_interfaces_up": 0, "model": "Newton + Orin AGXi\u0000", "utilization": { "boots": [ { "index": -8, "start": "2026-04-02T16:54:04+00:00", "end": "2026-04-03T04:01:18+00:00", "duration_seconds": 40034, "running": false }, { "index": -7, "start": "2026-04-04T13:22:57+00:00", "end": "2026-04-04T17:31:18+00:00", "duration_seconds": 14901, "running": false }, { "index": -6, "start": "2026-04-07T05:41:37+00:00", "end": "2026-04-07T06:38:05+00:00", "duration_seconds": 3388, "running": false }, { "index": -5, "start": "2026-04-07T06:42:39+00:00", "end": "2026-04-07T08:46:46+00:00", "duration_seconds": 7447, "running": false }, { "index": -4, "start": "2026-04-07T08:56:49+00:00", "end": "2026-04-07T08:57:07+00:00", "duration_seconds": 18, "running": false }, { "index": -3, "start": "2026-04-07T09:03:14+00:00", "end": "2026-04-07T20:35:25+00:00", "duration_seconds": 41531, "running": false }, { "index": -2, "start": "2026-04-07T20:44:25+00:00", "end": "2026-04-07T20:45:01+00:00", "duration_seconds": 36, "running": false }, { "index": -1, "start": "2026-04-07T20:48:44+00:00", "end": "2026-04-08T14:32:29+00:00", "duration_seconds": 63825, "running": false }, { "index": 0, "start": "2026-04-08T14:36:30+00:00", "end": "2026-04-10T12:20:55+00:00", "duration_seconds": 164665, "running": false } ], "daily_seconds": { "2026-04-02": 25556.0, "2026-04-03": 14478.0, "2026-04-04": 14901.0, "2026-04-07": 63896.0, "2026-04-08": 86159.0, "2026-04-09": 86400.0, "2026-04-10": 44455.0 }, "total_boots_7d": 0, "total_on_seconds_7d": 0, "utilization_7d_pct": 0.0 }, "_source_mtime": "2026-04-10T12:21:20+00:00" }, "rtpc": { "kernel": "5.15.183-rt85", "os": "Ubuntu 22.04", "architecture": "x86_64", "hostname": "alpha4", "uptime_seconds": 164678, "memory_total_mb": 47919, "memory_free_mb": 40041, "release": null, "collected_at": "2026-04-10T12:19:28Z", "dns": { "servers": { "wifi-0": "10.44.10.1" }, "search_domains": { "wifi-0": "hmnd.lan" }, "dropins": "# Managed by Ansible (iac-robots playbook 04)\n# Exposes the DNS stub resolver on sw-internal (192.168.1.1)\n# so the Orin and other Moxa devices can use the RTPC as a DNS forwarder.\n# The stub resolver automatically uses whatever upstream DNS the RTPC\n# learned from DHCP \u2014 works at any site without reconfiguration.\n[Resolve]\nDNSStubListenerExtra=192.168.1.1", "zones": { "local": true, "route53": true, "private": true, "public": true } }, "wifi": [ { "name": "wifi-1", "driver": "iwlwifi", "firmware": "89.123cf747.0 ty-a0-gf-a0-89.uc", "mac": "90:10:57:98:ab:b7", "ssid": null, "channel": null, "configured_ssid": null }, { "name": "wifi-0", "driver": "iwlwifi", "firmware": "89.123cf747.0 ty-a0-gf-a0-89.uc", "mac": "90:10:57:98:ab:b6", "ssid": "Alpha", "channel": 104, "configured_ssid": "Alpha", "station": { "rx_packets": 331134685, "tx_packets": 173634287, "tx_retries": 684013, "tx_failed": 132, "beacon_loss": 0, "rx_drop_misc": 158094, "signal_dbm": -35, "signal_per_antenna": "[-35, -38]", "tx_bitrate": "286.7 MBit/s HE-MCS 11 HE-NSS 2 HE-GI 0 HE-DCM 0", "rx_bitrate": "286.7 MBit/s HE-MCS 11 HE-NSS 2 HE-GI 0 HE-DCM 0", "connected_seconds": 164690 } } ], "interfaces": [ { "name": "eth-shore", "operstate": "DOWN", "ip": null, "mac": "00:13:95:68:c2:f3" }, { "name": "wifi-0", "operstate": "UP", "ip": "10.44.10.104", "mac": "90:10:57:98:ab:b6" }, { "name": "wifi-1", "operstate": "DOWN", "ip": null, "mac": "90:10:57:97:83:58" } ], "nm_profiles": [ { "name": "Alpha", "type": "wifi", "interface": "wifi-0", "ssid": "Alpha", "autoconnect": true, "priority": 900, "key_mgmt": "wpa-psk", "zone": "external", "state": "activated" }, { "name": "sw-internal", "type": "ethernet", "interface": "sw-internal", "ssid": null, "autoconnect": false, "priority": 0, "key_mgmt": null, "zone": null, "state": "activated" }, { "name": "hmnd0", "type": "wireguard", "interface": "hmnd0", "ssid": null, "autoconnect": false, "priority": 0, "key_mgmt": null, "zone": null, "state": "activated" }, { "name": "eth-shore", "type": "ethernet", "interface": "", "ssid": null, "autoconnect": true, "priority": 0, "key_mgmt": null, "zone": null, "state": "" }, { "name": "hmnd-6ghz", "type": "wifi", "interface": "", "ssid": null, "autoconnect": true, "priority": 999, "key_mgmt": null, "zone": null, "state": "" } ], "dispatcher_scripts": [ "10-hmnd-disable-offloads", "70-hmnd-qos" ], "utilization": { "boots": [ { "index": -42, "start": "2026-03-27T06:01:30+00:00", "end": "2026-03-27T11:00:41+00:00", "duration_seconds": 17951, "running": false }, { "index": -41, "start": "2026-03-27T11:03:02+00:00", "end": "2026-03-27T11:03:59+00:00", "duration_seconds": 57, "running": false }, { "index": -40, "start": "2026-03-27T11:04:51+00:00", "end": "2026-03-27T18:49:38+00:00", "duration_seconds": 27887, "running": false }, { "index": -39, "start": "2026-03-27T19:11:17+00:00", "end": "2026-03-27T19:37:40+00:00", "duration_seconds": 1583, "running": false }, { "index": -38, "start": "2026-03-27T19:45:09+00:00", "end": "2026-03-27T19:53:50+00:00", "duration_seconds": 521, "running": false }, { "index": -37, "start": "2026-03-28T17:23:08+00:00", "end": "2026-03-28T17:49:34+00:00", "duration_seconds": 1586, "running": false }, { "index": -36, "start": "2026-03-30T15:07:54+00:00", "end": "2026-03-30T15:40:32+00:00", "duration_seconds": 1958, "running": false }, { "index": -35, "start": "2026-03-30T15:41:21+00:00", "end": "2026-03-30T16:09:13+00:00", "duration_seconds": 1672, "running": false }, { "index": -34, "start": "2026-03-30T16:10:08+00:00", "end": "2026-03-30T16:11:32+00:00", "duration_seconds": 84, "running": false }, { "index": -33, "start": "2026-03-30T16:12:33+00:00", "end": "2026-03-30T16:16:22+00:00", "duration_seconds": 229, "running": false }, { "index": -32, "start": "2026-03-30T16:17:26+00:00", "end": "2026-03-30T17:14:53+00:00", "duration_seconds": 3447, "running": false }, { "index": -31, "start": "2026-03-30T17:34:26+00:00", "end": "2026-03-30T18:27:27+00:00", "duration_seconds": 3181, "running": false }, { "index": -30, "start": "2026-03-30T18:32:13+00:00", "end": "2026-03-30T18:33:36+00:00", "duration_seconds": 83, "running": false }, { "index": -29, "start": "2026-03-30T18:34:42+00:00", "end": "2026-03-31T12:33:54+00:00", "duration_seconds": 64752, "running": false }, { "index": -28, "start": "2026-03-31T12:40:37+00:00", "end": "2026-03-31T12:49:09+00:00", "duration_seconds": 512, "running": false }, { "index": -27, "start": "2026-03-31T12:51:31+00:00", "end": "2026-03-31T12:57:24+00:00", "duration_seconds": 353, "running": false }, { "index": -26, "start": "2026-03-31T12:58:23+00:00", "end": "2026-03-31T15:33:26+00:00", "duration_seconds": 9303, "running": false }, { "index": -25, "start": "2026-03-31T15:41:26+00:00", "end": "2026-03-31T15:50:46+00:00", "duration_seconds": 560, "running": false }, { "index": -24, "start": "2026-03-31T15:52:31+00:00", "end": "2026-03-31T17:45:23+00:00", "duration_seconds": 6772, "running": false }, { "index": -23, "start": "2026-03-31T17:48:03+00:00", "end": "2026-03-31T17:48:47+00:00", "duration_seconds": 44, "running": false }, { "index": -22, "start": "2026-03-31T17:52:45+00:00", "end": "2026-03-31T17:53:34+00:00", "duration_seconds": 49, "running": false }, { "index": -21, "start": "2026-03-31T17:58:13+00:00", "end": "2026-03-31T17:58:57+00:00", "duration_seconds": 44, "running": false }, { "index": -20, "start": "2026-03-31T18:09:56+00:00", "end": "2026-03-31T19:28:05+00:00", "duration_seconds": 4689, "running": false }, { "index": -19, "start": "2026-03-31T20:01:15+00:00", "end": "2026-04-01T01:18:37+00:00", "duration_seconds": 19042, "running": false }, { "index": -18, "start": "2026-04-01T01:39:43+00:00", "end": "2026-04-01T01:41:26+00:00", "duration_seconds": 103, "running": false }, { "index": -17, "start": "2026-04-01T01:42:49+00:00", "end": "2026-04-01T01:44:24+00:00", "duration_seconds": 95, "running": false }, { "index": -16, "start": "2026-04-01T01:48:53+00:00", "end": "2026-04-01T04:46:16+00:00", "duration_seconds": 10643, "running": false }, { "index": -15, "start": "2026-04-01T04:47:40+00:00", "end": "2026-04-01T04:55:05+00:00", "duration_seconds": 445, "running": false }, { "index": -14, "start": "2026-04-01T04:58:12+00:00", "end": "2026-04-01T05:04:16+00:00", "duration_seconds": 364, "running": false }, { "index": -13, "start": "2026-04-01T05:06:43+00:00", "end": "2026-04-01T05:06:56+00:00", "duration_seconds": 13, "running": false }, { "index": -12, "start": "2026-04-01T05:11:40+00:00", "end": "2026-04-01T05:11:58+00:00", "duration_seconds": 18, "running": false }, { "index": -11, "start": "2026-04-01T05:14:38+00:00", "end": "2026-04-01T17:15:34+00:00", "duration_seconds": 43256, "running": false }, { "index": -10, "start": "2026-04-01T17:16:44+00:00", "end": "2026-04-01T17:17:30+00:00", "duration_seconds": 46, "running": false }, { "index": -9, "start": "2026-04-01T17:18:41+00:00", "end": "2026-04-03T04:08:57+00:00", "duration_seconds": 125416, "running": false }, { "index": -8, "start": "2026-04-04T13:21:57+00:00", "end": "2026-04-04T17:38:11+00:00", "duration_seconds": 15374, "running": false }, { "index": -7, "start": "2026-04-07T05:40:37+00:00", "end": "2026-04-07T06:38:04+00:00", "duration_seconds": 3447, "running": false }, { "index": -6, "start": "2026-04-07T06:40:58+00:00", "end": "2026-04-07T08:46:47+00:00", "duration_seconds": 7549, "running": false }, { "index": -5, "start": "2026-04-07T08:55:17+00:00", "end": "2026-04-07T09:00:30+00:00", "duration_seconds": 313, "running": false }, { "index": -4, "start": "2026-04-07T09:01:42+00:00", "end": "2026-04-07T20:35:26+00:00", "duration_seconds": 41624, "running": false }, { "index": -3, "start": "2026-04-07T20:42:54+00:00", "end": "2026-04-07T20:46:02+00:00", "duration_seconds": 188, "running": false }, { "index": -2, "start": "2026-04-07T20:47:14+00:00", "end": "2026-04-08T14:32:27+00:00", "duration_seconds": 63913, "running": false }, { "index": -1, "start": "2026-04-08T14:33:19+00:00", "end": "2026-04-08T14:33:32+00:00", "duration_seconds": 13, "running": false }, { "index": 0, "start": "2026-04-08T14:34:59+00:00", "end": "2026-04-10T12:20:07+00:00", "duration_seconds": 164708, "running": false } ], "daily_seconds": { "2026-03-27": 47999.0, "2026-03-28": 1586.0, "2026-03-30": 30172.0, "2026-03-31": 81885.0, "2026-04-01": 83779.0, "2026-04-02": 86400.0, "2026-04-03": 14937.0, "2026-04-04": 15374.0, "2026-04-07": 64687.0, "2026-04-08": 86261.0, "2026-04-09": 86400.0, "2026-04-10": 44407.0 }, "total_boots_7d": 0, "total_on_seconds_7d": 0, "utilization_7d_pct": 0.0 }, "_source_mtime": "2026-04-10T12:21:20+00:00" } }, "aw6": { "orin": { "kernel": "5.15.148-tegra", "os": "Ubuntu 22.04", "architecture": "aarch64", "hostname": "orinalpha6", "uptime_seconds": "1637", "memory_total_mb": "54793", "memory_free_mb": "52668", "release": null, "collected_at": "2026-04-09T14:31:53Z", "dns": { "servers": {}, "search_domains": {}, "dropins": "[Resolve]\nFallbackDNS=8.8.8.8", "zones": { "local": false, "route53": false, "private": false, "public": true } }, "docker_running": false, "can_interfaces_up": 0, "model": "Newton + Orin AGXi\u0000", "utilization": { "boots": [], "daily_seconds": {}, "total_boots_7d": 0, "total_on_seconds_7d": 0, "utilization_7d_pct": 0.0 }, "_source_mtime": "2026-04-09T14:34:45+00:00" }, "rtpc": { "kernel": "5.15.197-rt91-hmnd-0", "os": "Ubuntu 22.04", "architecture": "x86_64", "hostname": "alpha6", "uptime_seconds": "1638", "memory_total_mb": "47921", "memory_free_mb": "44490", "release": null, "collected_at": "2026-04-09T14:30:24Z", "dns": { "servers": { "wifi-0": "10.44.10.1" }, "search_domains": { "wifi-0": "hmnd.lan" }, "dropins": "no drop-ins", "zones": { "local": true, "route53": true, "private": true, "public": true } }, "wifi": [ { "name": "wifi-1", "driver": "iwlwifi", "firmware": "89.123cf747.0 ty-a0-gf-a0-89.uc", "mac": "90:10:57:97:84:26", "ssid": null, "channel": null, "configured_ssid": null }, { "name": "wifi-0", "driver": "iwlwifi", "firmware": "89.123cf747.0 ty-a0-gf-a0-89.uc", "mac": "90:10:57:97:84:25", "ssid": "Alpha", "channel": 157, "configured_ssid": "Alpha", "station": { "rx_packets": 57270, "tx_packets": 60396, "tx_retries": 62, "tx_failed": 0, "beacon_loss": 0, "rx_drop_misc": 6457, "signal_dbm": -45, "signal_per_antenna": "[-45, -49]", "tx_bitrate": "286.7 MBit/s HE-MCS 11 HE-NSS 2 HE-GI 0 HE-DCM 0", "rx_bitrate": "286.7 MBit/s HE-MCS 11 HE-NSS 2 HE-GI 0 HE-DCM 0", "connected_seconds": 1673 } } ], "interfaces": [ { "name": "eth-shore", "operstate": "DOWN", "ip": null, "mac": "00:13:95:65:f1:60" }, { "name": "wifi-0", "operstate": "UP", "ip": "10.44.10.16", "mac": "90:10:57:97:84:25" }, { "name": "wifi-1", "operstate": "DOWN", "ip": null, "mac": "90:10:57:97:81:e6" } ], "nm_profiles": [ { "name": "Alpha", "type": "wifi", "interface": "wifi-0", "ssid": "Alpha", "autoconnect": true, "priority": 900, "key_mgmt": "wpa-psk", "zone": "external", "state": "activated" }, { "name": "sw-internal", "type": "ethernet", "interface": "sw-internal", "ssid": null, "autoconnect": false, "priority": 0, "key_mgmt": null, "zone": null, "state": "activated" }, { "name": "hmnd0", "type": "wireguard", "interface": "hmnd0", "ssid": null, "autoconnect": false, "priority": 0, "key_mgmt": null, "zone": null, "state": "activated" }, { "name": "eth-shore", "type": "ethernet", "interface": "", "ssid": null, "autoconnect": true, "priority": 0, "key_mgmt": null, "zone": null, "state": "" }, { "name": "hmnd-robot-6ghz", "type": "wifi", "interface": "", "ssid": null, "autoconnect": true, "priority": 999, "key_mgmt": null, "zone": null, "state": "" } ], "dispatcher_scripts": [ "10-hmnd-disable-offloads", "70-hmnd-qos" ], "utilization": { "boots": [], "daily_seconds": {}, "total_boots_7d": 0, "total_on_seconds_7d": 0, "utilization_7d_pct": 0.0 }, "_source_mtime": "2026-04-09T14:34:45+00:00" } }, "aw7": { "rtpc": { "kernel": "5.15.197-rt91-hmnd-0", "os": "Ubuntu 22.04", "architecture": "x86_64", "hostname": "alpha7", "uptime_seconds": 6009, "memory_total_mb": 47921, "memory_free_mb": 31145, "release": null, "collected_at": "2026-04-10T12:19:38Z", "dns": { "servers": { "wifi-0": "10.44.10.1" }, "search_domains": { "wifi-0": "hmnd.lan" }, "dropins": "no drop-ins", "zones": { "local": true, "route53": true, "private": true, "public": true } }, "wifi": [ { "name": "wifi-1", "driver": "iwlwifi", "firmware": "89.123cf747.0 ty-a0-gf-a0-89.uc", "mac": "90:10:57:98:aa:27", "ssid": null, "channel": null, "configured_ssid": null }, { "name": "wifi-0", "driver": "iwlwifi", "firmware": "89.123cf747.0 ty-a0-gf-a0-89.uc", "mac": "90:10:57:98:aa:26", "ssid": "Alpha", "channel": 37, "configured_ssid": "Alpha", "station": { "rx_packets": 1333129, "tx_packets": 1907737, "tx_retries": 1834, "tx_failed": 0, "beacon_loss": 0, "rx_drop_misc": 19194, "signal_dbm": -45, "signal_per_antenna": "[-45, -48]", "tx_bitrate": "286.7 MBit/s HE-MCS 11 HE-NSS 2 HE-GI 0 HE-DCM 0", "rx_bitrate": "286.7 MBit/s HE-MCS 11 HE-NSS 2 HE-GI 0 HE-DCM 0", "connected_seconds": 5805 } } ], "interfaces": [ { "name": "eth-shore", "operstate": "DOWN", "ip": null, "mac": "00:13:95:69:b6:36" }, { "name": "wifi-0", "operstate": "UP", "ip": "10.44.10.17", "mac": "90:10:57:98:aa:26" }, { "name": "wifi-1", "operstate": "DOWN", "ip": null, "mac": "90:10:57:98:ab:f2" } ], "nm_profiles": [ { "name": "Alpha", "type": "wifi", "interface": "wifi-0", "ssid": "Alpha", "autoconnect": true, "priority": 901, "key_mgmt": "wpa-psk", "zone": "external", "state": "activated" }, { "name": "sw-internal", "type": "ethernet", "interface": "sw-internal", "ssid": null, "autoconnect": false, "priority": 0, "key_mgmt": null, "zone": null, "state": "activated" }, { "name": "hmnd0", "type": "wireguard", "interface": "hmnd0", "ssid": null, "autoconnect": false, "priority": 0, "key_mgmt": null, "zone": null, "state": "activated" }, { "name": "eth-shore", "type": "ethernet", "interface": "", "ssid": null, "autoconnect": true, "priority": 0, "key_mgmt": null, "zone": null, "state": "" }, { "name": "hmnd-6ghz", "type": "wifi", "interface": "", "ssid": null, "autoconnect": true, "priority": 999, "key_mgmt": null, "zone": null, "state": "" }, { "name": "hmnd-robot-6ghz", "type": "wifi", "interface": "", "ssid": null, "autoconnect": true, "priority": 999, "key_mgmt": null, "zone": null, "state": "" } ], "dispatcher_scripts": [ "10-hmnd-disable-offloads", "70-hmnd-qos" ], "utilization": { "boots": [ { "index": -49, "start": "2026-03-25T09:54:28+00:00", "end": "2026-03-25T11:47:38+00:00", "duration_seconds": 6790, "running": false }, { "index": -48, "start": "2026-03-25T11:50:49+00:00", "end": "2026-03-25T11:57:31+00:00", "duration_seconds": 402, "running": false }, { "index": -47, "start": "2026-03-25T12:00:47+00:00", "end": "2026-03-25T12:38:16+00:00", "duration_seconds": 2249, "running": false }, { "index": -46, "start": "2026-03-25T13:37:02+00:00", "end": "2026-03-25T14:41:31+00:00", "duration_seconds": 3869, "running": false }, { "index": -45, "start": "2026-03-25T17:16:34+00:00", "end": "2026-03-25T17:49:52+00:00", "duration_seconds": 1998, "running": false }, { "index": -44, "start": "2026-03-25T17:51:42+00:00", "end": "2026-03-25T17:57:17+00:00", "duration_seconds": 335, "running": false }, { "index": -43, "start": "2026-03-25T17:57:59+00:00", "end": "2026-03-25T18:45:00+00:00", "duration_seconds": 2821, "running": false }, { "index": -42, "start": "2026-03-25T18:45:37+00:00", "end": "2026-03-25T18:47:44+00:00", "duration_seconds": 127, "running": false }, { "index": -41, "start": "2026-03-25T18:48:32+00:00", "end": "2026-03-25T19:14:55+00:00", "duration_seconds": 1583, "running": false }, { "index": -40, "start": "2026-03-25T19:15:48+00:00", "end": "2026-03-25T19:40:54+00:00", "duration_seconds": 1506, "running": false }, { "index": -39, "start": "2026-03-25T19:48:16+00:00", "end": "2026-03-25T19:56:18+00:00", "duration_seconds": 482, "running": false }, { "index": -38, "start": "2026-03-26T08:32:57+00:00", "end": "2026-03-26T10:02:09+00:00", "duration_seconds": 5352, "running": false }, { "index": -37, "start": "2026-03-26T10:05:52+00:00", "end": "2026-03-26T10:34:30+00:00", "duration_seconds": 1718, "running": false }, { "index": -36, "start": "2026-03-26T10:38:42+00:00", "end": "2026-03-26T10:42:11+00:00", "duration_seconds": 209, "running": false }, { "index": -35, "start": "2026-03-26T10:43:18+00:00", "end": "2026-03-26T10:44:41+00:00", "duration_seconds": 83, "running": false }, { "index": -34, "start": "2026-03-26T10:45:24+00:00", "end": "2026-03-26T11:05:15+00:00", "duration_seconds": 1191, "running": false }, { "index": -33, "start": "2026-03-26T12:05:46+00:00", "end": "2026-03-26T14:05:51+00:00", "duration_seconds": 7205, "running": false }, { "index": -32, "start": "2026-03-27T09:52:43+00:00", "end": "2026-03-27T13:34:01+00:00", "duration_seconds": 13278, "running": false }, { "index": -31, "start": "2026-03-27T13:42:53+00:00", "end": "2026-03-27T14:55:41+00:00", "duration_seconds": 4368, "running": false }, { "index": -30, "start": "2026-03-27T15:05:22+00:00", "end": "2026-03-27T15:09:42+00:00", "duration_seconds": 260, "running": false }, { "index": -29, "start": "2026-03-27T15:10:29+00:00", "end": "2026-03-27T15:15:43+00:00", "duration_seconds": 314, "running": false }, { "index": -28, "start": "2026-03-27T15:16:56+00:00", "end": "2026-03-27T15:19:36+00:00", "duration_seconds": 160, "running": false }, { "index": -27, "start": "2026-03-27T15:20:46+00:00", "end": "2026-03-27T15:25:19+00:00", "duration_seconds": 273, "running": false }, { "index": -26, "start": "2026-03-28T09:30:23+00:00", "end": "2026-03-28T13:55:51+00:00", "duration_seconds": 15928, "running": false }, { "index": -25, "start": "2026-03-28T14:40:43+00:00", "end": "2026-03-28T17:47:18+00:00", "duration_seconds": 11195, "running": false }, { "index": -24, "start": "2026-03-30T07:05:15+00:00", "end": "2026-03-31T20:38:51+00:00", "duration_seconds": 135216, "running": false }, { "index": -23, "start": "2026-04-01T08:33:19+00:00", "end": "2026-04-01T13:32:04+00:00", "duration_seconds": 17925, "running": false }, { "index": -22, "start": "2026-04-01T13:33:33+00:00", "end": "2026-04-01T13:36:53+00:00", "duration_seconds": 200, "running": false }, { "index": -21, "start": "2026-04-01T13:38:44+00:00", "end": "2026-04-01T16:31:50+00:00", "duration_seconds": 10386, "running": false }, { "index": -20, "start": "2026-04-02T08:00:44+00:00", "end": "2026-04-02T09:44:03+00:00", "duration_seconds": 6199, "running": false }, { "index": -19, "start": "2026-04-02T09:55:31+00:00", "end": "2026-04-02T10:03:38+00:00", "duration_seconds": 487, "running": false }, { "index": -18, "start": "2026-04-02T10:04:40+00:00", "end": "2026-04-02T10:09:02+00:00", "duration_seconds": 262, "running": false }, { "index": -17, "start": "2026-04-02T10:09:43+00:00", "end": "2026-04-02T10:10:11+00:00", "duration_seconds": 28, "running": false }, { "index": -16, "start": "2026-04-02T10:11:00+00:00", "end": "2026-04-02T10:19:00+00:00", "duration_seconds": 480, "running": false }, { "index": -15, "start": "2026-04-02T12:16:53+00:00", "end": "2026-04-02T19:07:58+00:00", "duration_seconds": 24665, "running": false }, { "index": -14, "start": "2026-04-02T19:08:55+00:00", "end": "2026-04-02T19:19:31+00:00", "duration_seconds": 636, "running": false }, { "index": -13, "start": "2026-04-02T19:20:23+00:00", "end": "2026-04-02T19:53:53+00:00", "duration_seconds": 2010, "running": false }, { "index": -12, "start": "2026-04-02T20:00:20+00:00", "end": "2026-04-02T20:05:50+00:00", "duration_seconds": 330, "running": false }, { "index": -11, "start": "2026-04-02T20:06:53+00:00", "end": "2026-04-02T20:06:53+00:00", "duration_seconds": 0, "running": false }, { "index": -10, "start": "2026-04-02T20:08:05+00:00", "end": "2026-04-02T20:12:54+00:00", "duration_seconds": 289, "running": false }, { "index": -9, "start": "2026-04-02T20:13:53+00:00", "end": "2026-04-02T20:16:28+00:00", "duration_seconds": 155, "running": false }, { "index": -8, "start": "2026-04-02T20:17:18+00:00", "end": "2026-04-02T20:35:43+00:00", "duration_seconds": 1105, "running": false }, { "index": -7, "start": "2026-04-02T20:50:48+00:00", "end": "2026-04-02T20:55:23+00:00", "duration_seconds": 275, "running": false }, { "index": -6, "start": "2026-04-08T08:33:01+00:00", "end": "2026-04-08T09:26:49+00:00", "duration_seconds": 3228, "running": false }, { "index": -5, "start": "2026-04-08T09:27:52+00:00", "end": "2026-04-08T15:56:37+00:00", "duration_seconds": 23325, "running": false }, { "index": -4, "start": "2026-04-09T08:32:17+00:00", "end": "2026-04-09T12:04:33+00:00", "duration_seconds": 12736, "running": false }, { "index": -3, "start": "2026-04-10T08:13:25+00:00", "end": "2026-04-10T08:43:08+00:00", "duration_seconds": 1783, "running": false }, { "index": -2, "start": "2026-04-10T08:44:02+00:00", "end": "2026-04-10T09:15:24+00:00", "duration_seconds": 1882, "running": false }, { "index": -1, "start": "2026-04-10T10:08:08+00:00", "end": "2026-04-10T10:38:54+00:00", "duration_seconds": 1846, "running": false }, { "index": 0, "start": "2026-04-10T10:39:38+00:00", "end": "2026-04-10T12:20:06+00:00", "duration_seconds": 6028, "running": false } ], "daily_seconds": { "2026-03-25": 22162.0, "2026-03-26": 15758.0, "2026-03-27": 18653.0, "2026-03-28": 27123.0, "2026-03-30": 60885.0, "2026-03-31": 74331.0, "2026-04-01": 28511.0, "2026-04-02": 36921.0, "2026-04-08": 26553.0, "2026-04-09": 12736.0, "2026-04-10": 11539.0 }, "total_boots_7d": 0, "total_on_seconds_7d": 0, "utilization_7d_pct": 0.0 }, "_source_mtime": "2026-04-10T12:21:20+00:00" }, "orin": { "kernel": "5.15.148-tegra", "os": "Ubuntu 22.04", "architecture": "aarch64", "hostname": "orinalpha7", "uptime_seconds": 5999, "memory_total_mb": 54793, "memory_free_mb": 36542, "release": null, "collected_at": "2026-04-10T12:20:20Z", "dns": { "servers": { "global": "10.44.10.1" }, "search_domains": {}, "dropins": "[Resolve]\nDNS=10.44.10.1\nFallbackDNS=8.8.8.8", "zones": { "local": true, "route53": true, "private": true, "public": true } }, "docker_running": false, "can_interfaces_up": 0, "model": "Newton + Orin AGXi\u0000", "utilization": { "boots": [ { "index": 0, "start": "2025-08-26T15:23:06+00:00", "end": "2026-04-10T12:20:53+00:00", "duration_seconds": 19601867, "running": false } ], "daily_seconds": { "2025-08-26": 31014.0, "2025-08-27": 86400.0, "2025-08-28": 86400.0, "2025-08-29": 86400.0, "2025-08-30": 86400.0, "2025-08-31": 86400.0, "2025-09-01": 86400.0, "2025-09-02": 86400.0, "2025-09-03": 86400.0, "2025-09-04": 86400.0, "2025-09-05": 86400.0, "2025-09-06": 86400.0, "2025-09-07": 86400.0, "2025-09-08": 86400.0, "2025-09-09": 86400.0, "2025-09-10": 86400.0, "2025-09-11": 86400.0, "2025-09-12": 86400.0, "2025-09-13": 86400.0, "2025-09-14": 86400.0, "2025-09-15": 86400.0, "2025-09-16": 86400.0, "2025-09-17": 86400.0, "2025-09-18": 86400.0, "2025-09-19": 86400.0, "2025-09-20": 86400.0, "2025-09-21": 86400.0, "2025-09-22": 86400.0, "2025-09-23": 86400.0, "2025-09-24": 86400.0, "2025-09-25": 86400.0, "2025-09-26": 86400.0, "2025-09-27": 86400.0, "2025-09-28": 86400.0, "2025-09-29": 86400.0, "2025-09-30": 86400.0, "2025-10-01": 86400.0, "2025-10-02": 86400.0, "2025-10-03": 86400.0, "2025-10-04": 86400.0, "2025-10-05": 86400.0, "2025-10-06": 86400.0, "2025-10-07": 86400.0, "2025-10-08": 86400.0, "2025-10-09": 86400.0, "2025-10-10": 86400.0, "2025-10-11": 86400.0, "2025-10-12": 86400.0, "2025-10-13": 86400.0, "2025-10-14": 86400.0, "2025-10-15": 86400.0, "2025-10-16": 86400.0, "2025-10-17": 86400.0, "2025-10-18": 86400.0, "2025-10-19": 86400.0, "2025-10-20": 86400.0, "2025-10-21": 86400.0, "2025-10-22": 86400.0, "2025-10-23": 86400.0, "2025-10-24": 86400.0, "2025-10-25": 86400.0, "2025-10-26": 86400.0, "2025-10-27": 86400.0, "2025-10-28": 86400.0, "2025-10-29": 86400.0, "2025-10-30": 86400.0, "2025-10-31": 86400.0, "2025-11-01": 86400.0, "2025-11-02": 86400.0, "2025-11-03": 86400.0, "2025-11-04": 86400.0, "2025-11-05": 86400.0, "2025-11-06": 86400.0, "2025-11-07": 86400.0, "2025-11-08": 86400.0, "2025-11-09": 86400.0, "2025-11-10": 86400.0, "2025-11-11": 86400.0, "2025-11-12": 86400.0, "2025-11-13": 86400.0, "2025-11-14": 86400.0, "2025-11-15": 86400.0, "2025-11-16": 86400.0, "2025-11-17": 86400.0, "2025-11-18": 86400.0, "2025-11-19": 86400.0, "2025-11-20": 86400.0, "2025-11-21": 86400.0, "2025-11-22": 86400.0, "2025-11-23": 86400.0, "2025-11-24": 86400.0, "2025-11-25": 86400.0, "2025-11-26": 86400.0, "2025-11-27": 86400.0, "2025-11-28": 86400.0, "2025-11-29": 86400.0, "2025-11-30": 86400.0, "2025-12-01": 86400.0, "2025-12-02": 86400.0, "2025-12-03": 86400.0, "2025-12-04": 86400.0, "2025-12-05": 86400.0, "2025-12-06": 86400.0, "2025-12-07": 86400.0, "2025-12-08": 86400.0, "2025-12-09": 86400.0, "2025-12-10": 86400.0, "2025-12-11": 86400.0, "2025-12-12": 86400.0, "2025-12-13": 86400.0, "2025-12-14": 86400.0, "2025-12-15": 86400.0, "2025-12-16": 86400.0, "2025-12-17": 86400.0, "2025-12-18": 86400.0, "2025-12-19": 86400.0, "2025-12-20": 86400.0, "2025-12-21": 86400.0, "2025-12-22": 86400.0, "2025-12-23": 86400.0, "2025-12-24": 86400.0, "2025-12-25": 86400.0, "2025-12-26": 86400.0, "2025-12-27": 86400.0, "2025-12-28": 86400.0, "2025-12-29": 86400.0, "2025-12-30": 86400.0, "2025-12-31": 86400.0, "2026-01-01": 86400.0, "2026-01-02": 86400.0, "2026-01-03": 86400.0, "2026-01-04": 86400.0, "2026-01-05": 86400.0, "2026-01-06": 86400.0, "2026-01-07": 86400.0, "2026-01-08": 86400.0, "2026-01-09": 86400.0, "2026-01-10": 86400.0, "2026-01-11": 86400.0, "2026-01-12": 86400.0, "2026-01-13": 86400.0, "2026-01-14": 86400.0, "2026-01-15": 86400.0, "2026-01-16": 86400.0, "2026-01-17": 86400.0, "2026-01-18": 86400.0, "2026-01-19": 86400.0, "2026-01-20": 86400.0, "2026-01-21": 86400.0, "2026-01-22": 86400.0, "2026-01-23": 86400.0, "2026-01-24": 86400.0, "2026-01-25": 86400.0, "2026-01-26": 86400.0, "2026-01-27": 86400.0, "2026-01-28": 86400.0, "2026-01-29": 86400.0, "2026-01-30": 86400.0, "2026-01-31": 86400.0, "2026-02-01": 86400.0, "2026-02-02": 86400.0, "2026-02-03": 86400.0, "2026-02-04": 86400.0, "2026-02-05": 86400.0, "2026-02-06": 86400.0, "2026-02-07": 86400.0, "2026-02-08": 86400.0, "2026-02-09": 86400.0, "2026-02-10": 86400.0, "2026-02-11": 86400.0, "2026-02-12": 86400.0, "2026-02-13": 86400.0, "2026-02-14": 86400.0, "2026-02-15": 86400.0, "2026-02-16": 86400.0, "2026-02-17": 86400.0, "2026-02-18": 86400.0, "2026-02-19": 86400.0, "2026-02-20": 86400.0, "2026-02-21": 86400.0, "2026-02-22": 86400.0, "2026-02-23": 86400.0, "2026-02-24": 86400.0, "2026-02-25": 86400.0, "2026-02-26": 86400.0, "2026-02-27": 86400.0, "2026-02-28": 86400.0, "2026-03-01": 86400.0, "2026-03-02": 86400.0, "2026-03-03": 86400.0, "2026-03-04": 86400.0, "2026-03-05": 86400.0, "2026-03-06": 86400.0, "2026-03-07": 86400.0, "2026-03-08": 86400.0, "2026-03-09": 86400.0, "2026-03-10": 86400.0, "2026-03-11": 86400.0, "2026-03-12": 86400.0, "2026-03-13": 86400.0, "2026-03-14": 86400.0, "2026-03-15": 86400.0, "2026-03-16": 86400.0, "2026-03-17": 86400.0, "2026-03-18": 86400.0, "2026-03-19": 86400.0, "2026-03-20": 86400.0, "2026-03-21": 86400.0, "2026-03-22": 86400.0, "2026-03-23": 86400.0, "2026-03-24": 86400.0, "2026-03-25": 86400.0, "2026-03-26": 86400.0, "2026-03-27": 86400.0, "2026-03-28": 86400.0, "2026-03-29": 86400.0, "2026-03-30": 86400.0, "2026-03-31": 86400.0, "2026-04-01": 86400.0, "2026-04-02": 86400.0, "2026-04-03": 86400.0, "2026-04-04": 86400.0, "2026-04-05": 86400.0, "2026-04-06": 86400.0, "2026-04-07": 86400.0, "2026-04-08": 86400.0, "2026-04-09": 86400.0, "2026-04-10": 44453.0 }, "total_boots_7d": 0, "total_on_seconds_7d": 0, "utilization_7d_pct": 0.0 }, "_source_mtime": "2026-04-10T12:21:21+00:00" } }, "aw8": { "orin": { "kernel": "5.15.148-tegra", "os": "Ubuntu 22.04", "architecture": "aarch64", "hostname": "orinalpha8", "uptime_seconds": "13667", "memory_total_mb": "54793", "memory_free_mb": "46648", "release": null, "collected_at": "2026-03-24T21:02:10Z", "dns": { "servers": {}, "search_domains": {}, "dropins": "[Resolve]\nFallbackDNS=8.8.8.8", "zones": { "local": false, "route53": false, "private": false, "public": true } }, "docker_running": false, "can_interfaces_up": 0, "model": "Newton + Orin AGXi\u0000", "utilization": { "boots": [], "daily_seconds": {}, "total_boots_7d": 0, "total_on_seconds_7d": 0, "utilization_7d_pct": 0.0 }, "_source_mtime": "2026-03-24T21:03:51+00:00" }, "rtpc": { "kernel": "5.15.197-rt91-hmnd-0", "os": "Ubuntu 22.04", "architecture": "x86_64", "hostname": "alpha8", "uptime_seconds": "13654", "memory_total_mb": "47921", "memory_free_mb": "42279", "release": null, "collected_at": "2026-03-24T21:00:28Z", "dns": { "servers": { "wifi-0": "10.44.52.1" }, "search_domains": { "wifi-0": "hmnd.lan" }, "dropins": "no drop-ins", "zones": { "local": true, "route53": true, "private": true, "public": true } }, "wifi": [ { "name": "wifi-1", "driver": "iwlwifi", "firmware": "89.123cf747.0 ty-a0-gf-a0-89.uc", "mac": "5c:b2:6d:46:6d:57", "ssid": null, "channel": null, "configured_ssid": null }, { "name": "wifi-0", "driver": "iwlwifi", "firmware": "89.123cf747.0 ty-a0-gf-a0-89.uc", "mac": "5c:b2:6d:46:6d:56", "ssid": "hmnd-robot-6ghz", "channel": 213, "configured_ssid": "hmnd-robot-6ghz" } ], "interfaces": [ { "name": "wifi-0", "operstate": "UP", "ip": "10.44.52.108", "mac": "5c:b2:6d:46:6d:56" }, { "name": "wifi-1", "operstate": "DOWN", "ip": null, "mac": "58:6d:67:f7:cf:d8" } ], "nm_profiles": [ { "name": "hmnd-robot-6ghz", "type": "wifi", "interface": "wifi-0", "ssid": "hmnd-robot-6ghz", "autoconnect": true, "priority": 999, "key_mgmt": "sae", "zone": "external", "state": "activated" }, { "name": "sw-internal", "type": "ethernet", "interface": "sw-internal", "ssid": null, "autoconnect": false, "priority": 0, "key_mgmt": null, "zone": null, "state": "activated" }, { "name": "hmnd0", "type": "wireguard", "interface": "hmnd0", "ssid": null, "autoconnect": false, "priority": 0, "key_mgmt": null, "zone": null, "state": "activated" }, { "name": "eth-shore", "type": "ethernet", "interface": "", "ssid": null, "autoconnect": true, "priority": 0, "key_mgmt": null, "zone": null, "state": "" } ], "dispatcher_scripts": [ "10-hmnd-disable-offloads", "70-hmnd-qos" ], "utilization": { "boots": [], "daily_seconds": {}, "total_boots_7d": 0, "total_on_seconds_7d": 0, "utilization_7d_pct": 0.0 }, "_source_mtime": "2026-03-24T21:03:50+00:00" } }, "aw9": { "orin": { "kernel": "5.15.148-tegra", "os": "Ubuntu 22.04", "architecture": "aarch64", "hostname": "orinalpha9", "uptime_seconds": 17006, "memory_total_mb": 54793, "memory_free_mb": 40292, "release": null, "collected_at": "2026-04-10T12:20:21Z", "dns": { "servers": {}, "search_domains": {}, "dropins": "[Resolve]\nFallbackDNS=8.8.8.8", "zones": { "local": false, "route53": false, "private": false, "public": true } }, "docker_running": false, "can_interfaces_up": 0, "model": "Newton + Orin AGXi\u0000", "utilization": { "boots": [ { "index": 0, "start": "2025-08-26T15:23:06+00:00", "end": "2026-04-10T12:20:53+00:00", "duration_seconds": 19601867, "running": false } ], "daily_seconds": { "2025-08-26": 31014.0, "2025-08-27": 86400.0, "2025-08-28": 86400.0, "2025-08-29": 86400.0, "2025-08-30": 86400.0, "2025-08-31": 86400.0, "2025-09-01": 86400.0, "2025-09-02": 86400.0, "2025-09-03": 86400.0, "2025-09-04": 86400.0, "2025-09-05": 86400.0, "2025-09-06": 86400.0, "2025-09-07": 86400.0, "2025-09-08": 86400.0, "2025-09-09": 86400.0, "2025-09-10": 86400.0, "2025-09-11": 86400.0, "2025-09-12": 86400.0, "2025-09-13": 86400.0, "2025-09-14": 86400.0, "2025-09-15": 86400.0, "2025-09-16": 86400.0, "2025-09-17": 86400.0, "2025-09-18": 86400.0, "2025-09-19": 86400.0, "2025-09-20": 86400.0, "2025-09-21": 86400.0, "2025-09-22": 86400.0, "2025-09-23": 86400.0, "2025-09-24": 86400.0, "2025-09-25": 86400.0, "2025-09-26": 86400.0, "2025-09-27": 86400.0, "2025-09-28": 86400.0, "2025-09-29": 86400.0, "2025-09-30": 86400.0, "2025-10-01": 86400.0, "2025-10-02": 86400.0, "2025-10-03": 86400.0, "2025-10-04": 86400.0, "2025-10-05": 86400.0, "2025-10-06": 86400.0, "2025-10-07": 86400.0, "2025-10-08": 86400.0, "2025-10-09": 86400.0, "2025-10-10": 86400.0, "2025-10-11": 86400.0, "2025-10-12": 86400.0, "2025-10-13": 86400.0, "2025-10-14": 86400.0, "2025-10-15": 86400.0, "2025-10-16": 86400.0, "2025-10-17": 86400.0, "2025-10-18": 86400.0, "2025-10-19": 86400.0, "2025-10-20": 86400.0, "2025-10-21": 86400.0, "2025-10-22": 86400.0, "2025-10-23": 86400.0, "2025-10-24": 86400.0, "2025-10-25": 86400.0, "2025-10-26": 86400.0, "2025-10-27": 86400.0, "2025-10-28": 86400.0, "2025-10-29": 86400.0, "2025-10-30": 86400.0, "2025-10-31": 86400.0, "2025-11-01": 86400.0, "2025-11-02": 86400.0, "2025-11-03": 86400.0, "2025-11-04": 86400.0, "2025-11-05": 86400.0, "2025-11-06": 86400.0, "2025-11-07": 86400.0, "2025-11-08": 86400.0, "2025-11-09": 86400.0, "2025-11-10": 86400.0, "2025-11-11": 86400.0, "2025-11-12": 86400.0, "2025-11-13": 86400.0, "2025-11-14": 86400.0, "2025-11-15": 86400.0, "2025-11-16": 86400.0, "2025-11-17": 86400.0, "2025-11-18": 86400.0, "2025-11-19": 86400.0, "2025-11-20": 86400.0, "2025-11-21": 86400.0, "2025-11-22": 86400.0, "2025-11-23": 86400.0, "2025-11-24": 86400.0, "2025-11-25": 86400.0, "2025-11-26": 86400.0, "2025-11-27": 86400.0, "2025-11-28": 86400.0, "2025-11-29": 86400.0, "2025-11-30": 86400.0, "2025-12-01": 86400.0, "2025-12-02": 86400.0, "2025-12-03": 86400.0, "2025-12-04": 86400.0, "2025-12-05": 86400.0, "2025-12-06": 86400.0, "2025-12-07": 86400.0, "2025-12-08": 86400.0, "2025-12-09": 86400.0, "2025-12-10": 86400.0, "2025-12-11": 86400.0, "2025-12-12": 86400.0, "2025-12-13": 86400.0, "2025-12-14": 86400.0, "2025-12-15": 86400.0, "2025-12-16": 86400.0, "2025-12-17": 86400.0, "2025-12-18": 86400.0, "2025-12-19": 86400.0, "2025-12-20": 86400.0, "2025-12-21": 86400.0, "2025-12-22": 86400.0, "2025-12-23": 86400.0, "2025-12-24": 86400.0, "2025-12-25": 86400.0, "2025-12-26": 86400.0, "2025-12-27": 86400.0, "2025-12-28": 86400.0, "2025-12-29": 86400.0, "2025-12-30": 86400.0, "2025-12-31": 86400.0, "2026-01-01": 86400.0, "2026-01-02": 86400.0, "2026-01-03": 86400.0, "2026-01-04": 86400.0, "2026-01-05": 86400.0, "2026-01-06": 86400.0, "2026-01-07": 86400.0, "2026-01-08": 86400.0, "2026-01-09": 86400.0, "2026-01-10": 86400.0, "2026-01-11": 86400.0, "2026-01-12": 86400.0, "2026-01-13": 86400.0, "2026-01-14": 86400.0, "2026-01-15": 86400.0, "2026-01-16": 86400.0, "2026-01-17": 86400.0, "2026-01-18": 86400.0, "2026-01-19": 86400.0, "2026-01-20": 86400.0, "2026-01-21": 86400.0, "2026-01-22": 86400.0, "2026-01-23": 86400.0, "2026-01-24": 86400.0, "2026-01-25": 86400.0, "2026-01-26": 86400.0, "2026-01-27": 86400.0, "2026-01-28": 86400.0, "2026-01-29": 86400.0, "2026-01-30": 86400.0, "2026-01-31": 86400.0, "2026-02-01": 86400.0, "2026-02-02": 86400.0, "2026-02-03": 86400.0, "2026-02-04": 86400.0, "2026-02-05": 86400.0, "2026-02-06": 86400.0, "2026-02-07": 86400.0, "2026-02-08": 86400.0, "2026-02-09": 86400.0, "2026-02-10": 86400.0, "2026-02-11": 86400.0, "2026-02-12": 86400.0, "2026-02-13": 86400.0, "2026-02-14": 86400.0, "2026-02-15": 86400.0, "2026-02-16": 86400.0, "2026-02-17": 86400.0, "2026-02-18": 86400.0, "2026-02-19": 86400.0, "2026-02-20": 86400.0, "2026-02-21": 86400.0, "2026-02-22": 86400.0, "2026-02-23": 86400.0, "2026-02-24": 86400.0, "2026-02-25": 86400.0, "2026-02-26": 86400.0, "2026-02-27": 86400.0, "2026-02-28": 86400.0, "2026-03-01": 86400.0, "2026-03-02": 86400.0, "2026-03-03": 86400.0, "2026-03-04": 86400.0, "2026-03-05": 86400.0, "2026-03-06": 86400.0, "2026-03-07": 86400.0, "2026-03-08": 86400.0, "2026-03-09": 86400.0, "2026-03-10": 86400.0, "2026-03-11": 86400.0, "2026-03-12": 86400.0, "2026-03-13": 86400.0, "2026-03-14": 86400.0, "2026-03-15": 86400.0, "2026-03-16": 86400.0, "2026-03-17": 86400.0, "2026-03-18": 86400.0, "2026-03-19": 86400.0, "2026-03-20": 86400.0, "2026-03-21": 86400.0, "2026-03-22": 86400.0, "2026-03-23": 86400.0, "2026-03-24": 86400.0, "2026-03-25": 86400.0, "2026-03-26": 86400.0, "2026-03-27": 86400.0, "2026-03-28": 86400.0, "2026-03-29": 86400.0, "2026-03-30": 86400.0, "2026-03-31": 86400.0, "2026-04-01": 86400.0, "2026-04-02": 86400.0, "2026-04-03": 86400.0, "2026-04-04": 86400.0, "2026-04-05": 86400.0, "2026-04-06": 86400.0, "2026-04-07": 86400.0, "2026-04-08": 86400.0, "2026-04-09": 86400.0, "2026-04-10": 44453.0 }, "total_boots_7d": 0, "total_on_seconds_7d": 0, "utilization_7d_pct": 0.0 }, "_source_mtime": "2026-04-10T12:21:21+00:00" }, "rtpc": { "kernel": "5.15.197-rt91-hmnd-0", "os": "Ubuntu 22.04", "architecture": "x86_64", "hostname": "alpha9", "uptime_seconds": 17017, "memory_total_mb": 47921, "memory_free_mb": 34538, "release": null, "collected_at": "2026-04-10T12:19:38Z", "dns": { "servers": { "eth-shore": "10.44.10.1", "wifi-0": "10.44.10.1" }, "search_domains": { "eth-shore": "hmnd.lan", "wifi-0": "hmnd.lan" }, "dropins": "no drop-ins", "zones": { "local": true, "route53": true, "private": true, "public": true } }, "wifi": [ { "name": "wifi-1", "driver": "iwlwifi", "firmware": "89.123cf747.0 ty-a0-gf-a0-89.uc", "mac": "5c:b2:6d:46:6d:0c", "ssid": null, "channel": null, "configured_ssid": null }, { "name": "wifi-0", "driver": "iwlwifi", "firmware": "89.123cf747.0 ty-a0-gf-a0-89.uc", "mac": "5c:b2:6d:46:6d:0b", "ssid": "Alpha", "channel": 64, "configured_ssid": "Alpha", "station": { "rx_packets": 569143, "tx_packets": 2329, "tx_retries": 8, "tx_failed": 0, "beacon_loss": 0, "rx_drop_misc": 69023, "signal_dbm": -44, "signal_per_antenna": "[-46, -44]", "tx_bitrate": "286.7 MBit/s HE-MCS 11 HE-NSS 2 HE-GI 0 HE-DCM 0", "rx_bitrate": "286.7 MBit/s HE-MCS 11 HE-NSS 2 HE-GI 0 HE-DCM 0", "connected_seconds": 17010 } } ], "interfaces": [ { "name": "eth-shore", "operstate": "UP", "ip": "10.44.10.190", "mac": "00:13:95:69:b6:3a" }, { "name": "wifi-0", "operstate": "UP", "ip": "10.44.10.19", "mac": "5c:b2:6d:46:6d:0b" }, { "name": "wifi-1", "operstate": "DOWN", "ip": null, "mac": "5c:b2:6d:46:6e:6e" } ], "nm_profiles": [ { "name": "eth-shore", "type": "ethernet", "interface": "eth-shore", "ssid": null, "autoconnect": true, "priority": 0, "key_mgmt": null, "zone": "external", "state": "activated" }, { "name": "sw-internal", "type": "ethernet", "interface": "sw-internal", "ssid": null, "autoconnect": false, "priority": 0, "key_mgmt": null, "zone": null, "state": "activated" }, { "name": "Alpha", "type": "wifi", "interface": "wifi-0", "ssid": "Alpha", "autoconnect": true, "priority": 0, "key_mgmt": "wpa-psk", "zone": "external", "state": "activated" }, { "name": "hmnd0", "type": "wireguard", "interface": "hmnd0", "ssid": null, "autoconnect": false, "priority": 0, "key_mgmt": null, "zone": null, "state": "activated" }, { "name": "hmnd-6ghz", "type": "wifi", "interface": "", "ssid": null, "autoconnect": true, "priority": 999, "key_mgmt": null, "zone": null, "state": "" } ], "dispatcher_scripts": [ "10-hmnd-disable-offloads", "70-hmnd-qos" ], "utilization": { "boots": [ { "index": -31, "start": "2026-03-20T10:11:23+00:00", "end": "2026-03-20T14:12:35+00:00", "duration_seconds": 14472, "running": false }, { "index": -30, "start": "2026-03-20T14:38:54+00:00", "end": "2026-03-20T14:40:00+00:00", "duration_seconds": 66, "running": false }, { "index": -29, "start": "2026-03-20T14:41:50+00:00", "end": "2026-03-20T15:53:54+00:00", "duration_seconds": 4324, "running": false }, { "index": -28, "start": "2026-03-20T15:55:09+00:00", "end": "2026-03-20T17:54:59+00:00", "duration_seconds": 7190, "running": false }, { "index": -27, "start": "2026-03-23T10:33:33+00:00", "end": "2026-03-23T10:35:46+00:00", "duration_seconds": 133, "running": false }, { "index": -26, "start": "2026-03-23T10:37:36+00:00", "end": "2026-03-23T18:26:52+00:00", "duration_seconds": 28156, "running": false }, { "index": -25, "start": "2026-03-24T11:41:10+00:00", "end": "2026-03-24T19:24:43+00:00", "duration_seconds": 27813, "running": false }, { "index": -24, "start": "2026-03-25T09:42:38+00:00", "end": "2026-03-25T17:53:07+00:00", "duration_seconds": 29429, "running": false }, { "index": -23, "start": "2026-03-25T17:54:31+00:00", "end": "2026-03-25T17:57:07+00:00", "duration_seconds": 156, "running": false }, { "index": -22, "start": "2026-03-25T17:57:56+00:00", "end": "2026-03-25T18:00:20+00:00", "duration_seconds": 144, "running": false }, { "index": -21, "start": "2026-03-25T18:01:11+00:00", "end": "2026-03-25T18:02:22+00:00", "duration_seconds": 71, "running": false }, { "index": -20, "start": "2026-03-25T18:03:18+00:00", "end": "2026-03-25T19:08:51+00:00", "duration_seconds": 3933, "running": false }, { "index": -19, "start": "2026-03-26T10:37:52+00:00", "end": "2026-03-26T10:39:03+00:00", "duration_seconds": 71, "running": false }, { "index": -18, "start": "2026-03-26T10:48:34+00:00", "end": "2026-03-26T10:49:36+00:00", "duration_seconds": 62, "running": false }, { "index": -17, "start": "2026-03-26T10:50:17+00:00", "end": "2026-03-26T10:50:35+00:00", "duration_seconds": 18, "running": false }, { "index": -16, "start": "2026-03-26T10:51:19+00:00", "end": "2026-03-26T18:27:29+00:00", "duration_seconds": 27370, "running": false }, { "index": -15, "start": "2026-03-27T10:19:40+00:00", "end": "2026-03-27T18:00:56+00:00", "duration_seconds": 27676, "running": false }, { "index": -14, "start": "2026-03-28T17:19:37+00:00", "end": "2026-03-28T17:45:41+00:00", "duration_seconds": 1564, "running": false }, { "index": -13, "start": "2026-03-30T09:54:40+00:00", "end": "2026-03-30T17:49:28+00:00", "duration_seconds": 28488, "running": false }, { "index": -12, "start": "2026-03-31T09:09:52+00:00", "end": "2026-03-31T10:32:08+00:00", "duration_seconds": 4936, "running": false }, { "index": -11, "start": "2026-03-31T10:49:31+00:00", "end": "2026-03-31T21:03:13+00:00", "duration_seconds": 36822, "running": false }, { "index": -10, "start": "2026-04-01T07:26:37+00:00", "end": "2026-04-01T18:40:29+00:00", "duration_seconds": 40432, "running": false }, { "index": -9, "start": "2026-04-02T07:57:00+00:00", "end": "2026-04-02T18:08:22+00:00", "duration_seconds": 36682, "running": false }, { "index": -8, "start": "2026-04-07T06:41:22+00:00", "end": "2026-04-07T19:26:33+00:00", "duration_seconds": 45911, "running": false }, { "index": -7, "start": "2026-04-08T07:56:33+00:00", "end": "2026-04-08T19:25:19+00:00", "duration_seconds": 41326, "running": false }, { "index": -6, "start": "2026-04-09T05:52:45+00:00", "end": "2026-04-09T06:06:36+00:00", "duration_seconds": 831, "running": false }, { "index": -5, "start": "2026-04-09T06:09:26+00:00", "end": "2026-04-09T09:33:56+00:00", "duration_seconds": 12270, "running": false }, { "index": -4, "start": "2026-04-09T09:34:56+00:00", "end": "2026-04-09T09:35:52+00:00", "duration_seconds": 56, "running": false }, { "index": -3, "start": "2026-04-09T09:38:12+00:00", "end": "2026-04-09T13:25:48+00:00", "duration_seconds": 13656, "running": false }, { "index": -2, "start": "2026-04-09T13:26:34+00:00", "end": "2026-04-09T14:47:10+00:00", "duration_seconds": 4836, "running": false }, { "index": -1, "start": "2026-04-09T15:09:09+00:00", "end": "2026-04-10T06:24:44+00:00", "duration_seconds": 54935, "running": false }, { "index": 0, "start": "2026-04-10T07:36:10+00:00", "end": "2026-04-10T12:20:07+00:00", "duration_seconds": 17037, "running": false } ], "daily_seconds": { "2026-03-20": 26052.0, "2026-03-23": 28289.0, "2026-03-24": 27813.0, "2026-03-25": 33733.0, "2026-03-26": 27521.0, "2026-03-27": 27676.0, "2026-03-28": 1564.0, "2026-03-30": 28488.0, "2026-03-31": 41758.0, "2026-04-01": 40432.0, "2026-04-02": 36682.0, "2026-04-07": 45911.0, "2026-04-08": 41326.0, "2026-04-09": 63500.0, "2026-04-10": 40121.0 }, "total_boots_7d": 0, "total_on_seconds_7d": 0, "utilization_7d_pct": 0.0 }, "_source_mtime": "2026-04-10T12:21:21+00:00" } } }, "source_mtime": "2026-04-10T13:16:49+00:00" }, "console": { "updated": "2026-07-25T08:27:34Z", "available": true, "statusCode": 200, "robots": [ { "id": 41, "name": "UMI 2", "shortName": "umi-2", "status": "active", "isSimulation": false, "model": "UMI", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 39, "name": "Egocentric 2", "shortName": "egocentric2", "status": "active", "isSimulation": false, "model": "Egocentric", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 42, "name": "Biped 3", "shortName": "biped3", "status": "active", "isSimulation": false, "model": "Alpha (Biped)", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 33, "name": "Quest 0", "shortName": "quest0", "status": "active", "isSimulation": false, "model": "Egocentric", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 43, "name": "Quest 1", "shortName": "quest1", "status": "active", "isSimulation": false, "model": "Quest", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 44, "name": "Quest 2", "shortName": "quest2", "status": "active", "isSimulation": false, "model": "Quest", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 45, "name": "Quest 3", "shortName": "quest3", "status": "active", "isSimulation": false, "model": "Quest", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 46, "name": "Quest 4", "shortName": "quest4", "status": "active", "isSimulation": false, "model": "Quest", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 47, "name": "Quest 5", "shortName": "quest5", "status": "active", "isSimulation": false, "model": "Quest", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 48, "name": "Quest 6", "shortName": "quest6", "status": "active", "isSimulation": false, "model": "Quest", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 1, "name": "H1", "shortName": "h1", "status": "active", "isSimulation": false, "model": null, "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 2, "name": "Adam", "shortName": "adam", "status": "inactive", "isSimulation": false, "model": null, "location": null, "deactivationDate": "2025-09-04T13:13:49.315757", "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 49, "name": "Quest 7", "shortName": "quest7", "status": "active", "isSimulation": false, "model": "Quest", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 50, "name": "Quest 8", "shortName": "quest8", "status": "active", "isSimulation": false, "model": "Quest", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 51, "name": "Quest 9", "shortName": "quest9", "status": "active", "isSimulation": false, "model": "Quest", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 52, "name": "Quest 10", "shortName": "quest10", "status": "active", "isSimulation": false, "model": "Quest", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 53, "name": "Quest 11", "shortName": "quest11", "status": "active", "isSimulation": false, "model": "Quest", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 54, "name": "Quest 12", "shortName": "quest12", "status": "active", "isSimulation": false, "model": "Quest", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 55, "name": "Quest 13", "shortName": "quest13", "status": "active", "isSimulation": false, "model": "Quest", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 56, "name": "Quest 14", "shortName": "quest14", "status": "active", "isSimulation": false, "model": "Quest", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 57, "name": "Quest 15", "shortName": "quest15", "status": "active", "isSimulation": false, "model": "Quest", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 58, "name": "Quest 16", "shortName": "quest16", "status": "active", "isSimulation": false, "model": "Quest", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 7, "name": "Dev", "shortName": "dev", "status": "inactive", "isSimulation": false, "model": null, "location": null, "deactivationDate": "2025-09-23T09:44:11.495060", "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 4, "name": "H1 (Sim)", "shortName": "h1-isaac", "status": "active", "isSimulation": true, "model": null, "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 18, "name": "Alpha 1 (Sim)", "shortName": "alpha-1-sim", "status": "active", "isSimulation": true, "model": "Alpha (Wheelbase)", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 59, "name": "Quest 17", "shortName": "quest17", "status": "active", "isSimulation": false, "model": "Quest", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 21, "name": "Alpha 2", "shortName": "alpha2", "status": "active", "isSimulation": false, "model": "Alpha (Wheelbase)", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 22, "name": "Alpha 5", "shortName": "alpha5", "status": "active", "isSimulation": false, "model": "Alpha (Wheelbase)", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 60, "name": "Quest 18", "shortName": "quest18", "status": "active", "isSimulation": false, "model": "Quest", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 61, "name": "Quest 19", "shortName": "quest19", "status": "active", "isSimulation": false, "model": "Quest", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 20, "name": "Test 1", "shortName": "test-1", "status": "active", "isSimulation": false, "model": "Alpha (Wheelbase)", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 62, "name": "Alpha Biped 4", "shortName": "alpha_biped4", "status": "active", "isSimulation": false, "model": null, "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 24, "name": "Alpha 7", "shortName": "alpha7", "status": "active", "isSimulation": false, "model": "Alpha (Wheelbase)", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 63, "name": "Alpha Biped 2", "shortName": "alpha_biped2", "status": "active", "isSimulation": false, "model": null, "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 25, "name": "Alpha 6", "shortName": "alpha6", "status": "active", "isSimulation": false, "model": "Alpha (Wheelbase)", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 19, "name": "Alpha 3", "shortName": "alpha3", "status": "active", "isSimulation": false, "model": "Alpha (Wheelbase)", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 17, "name": "Alpha 1", "shortName": "alpha1", "status": "active", "isSimulation": false, "model": "Alpha (Wheelbase)", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 26, "name": "Alpha 8", "shortName": "alpha8", "status": "active", "isSimulation": false, "model": "Alpha (Wheelbase)", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 27, "name": "Alpha Biped 1", "shortName": "alphabiped01-orin", "status": "active", "isSimulation": false, "model": "Alpha (Biped)", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 28, "name": "XDOF UMI 1", "shortName": "xdof-umi-1", "status": "active", "isSimulation": false, "model": "UMI", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 5, "name": "rainbow-r2", "shortName": "rby2", "status": "active", "isSimulation": false, "model": "(Rainbow) RB-Y1", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 32, "name": "Alpha 11", "shortName": "alpha11", "status": "active", "isSimulation": false, "model": "Alpha (Wheelbase)", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 34, "name": "Alpha 12", "shortName": "alpha12", "status": "active", "isSimulation": false, "model": "Alpha (Wheelbase)", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 38, "name": "Alpha 9", "shortName": "alpha9", "status": "active", "isSimulation": false, "model": "Alpha (Wheelbase)", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 31, "name": "Alpha 10", "shortName": "alpha10", "status": "active", "isSimulation": false, "model": "Alpha (Wheelbase)", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 23, "name": "Alpha 4", "shortName": "alpha4", "status": "active", "isSimulation": false, "model": "Alpha (Wheelbase)", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 30, "name": "UMI 1", "shortName": "umi-1", "status": "active", "isSimulation": false, "model": "UMI", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 29, "name": "xdof-umi-20250523", "shortName": "xdof-umi-20250523", "status": "active", "isSimulation": false, "model": "UMI", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 8, "name": "rainbow-r3", "shortName": "rby3", "status": "active", "isSimulation": false, "model": "(Rainbow) RB-Y1", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 9, "name": "rainbow-r1", "shortName": "rby1", "status": "active", "isSimulation": false, "model": "(Rainbow) RB-Y1", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 12, "name": "rainbow-r4", "shortName": "rby4", "status": "active", "isSimulation": false, "model": "(Rainbow) RB-Y1", "location": "Lab", "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 13, "name": "rainbow-r5", "shortName": "rby5", "status": "active", "isSimulation": false, "model": "(Rainbow) RB-Y1", "location": "Lab", "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 11, "name": "rainbow-r6", "shortName": "rby6", "status": "active", "isSimulation": false, "model": "(Rainbow) RB-Y1", "location": "Lab", "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 14, "name": "rainbow-r7", "shortName": "rby7", "status": "active", "isSimulation": false, "model": "(Rainbow) RB-Y1", "location": "Lab", "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 16, "name": "Rainbow (Sim)", "shortName": "rby1-isaac", "status": "active", "isSimulation": true, "model": "(Rainbow) RB-Y1", "location": "isaac-sim", "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} }, { "id": 37, "name": "Egocentric 1", "shortName": "egocentric1", "status": "active", "isSimulation": false, "model": "Egocentric", "location": null, "deactivationDate": null, "recordingCount": 0, "episodeCount": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingGood": 0, "operators": {} } ], "recentRecordings": [], "recentEpisodes": [ { "id": 913439, "recordingId": 9011, "robotName": "Alpha 4", "taskTitle": "QGF distillation", "operator": "Sergei Fedotov", "s3Url": "s3://hm-rainbow-mount/alpha8/record-pkg-id=9011/2026_07_25-07_49_40/", "createdAt": "2026-07-25T08:23:32.429378", "uploadStatus": "Converted", "pickQuality": null, "controlMode": null, "pedalAction": "not_found", "project": null, "item": null }, { "id": 913438, "recordingId": 9011, "robotName": "Alpha 4", "taskTitle": "QGF distillation", "operator": "Sergei Fedotov", "s3Url": "s3://hm-rainbow-mount/alpha8/record-pkg-id=9011/2026_07_25-07_49_19/", "createdAt": "2026-07-25T08:23:32.077049", "uploadStatus": null, "pickQuality": null, "controlMode": null, "pedalAction": null, "project": null, "item": null }, { "id": 913437, "recordingId": 9011, "robotName": "Alpha 4", "taskTitle": "QGF distillation", "operator": "Sergei Fedotov", "s3Url": "s3://hm-rainbow-mount/alpha8/record-pkg-id=9011/2026_07_25-07_48_48/", "createdAt": "2026-07-25T08:23:31.781863", "uploadStatus": null, "pickQuality": null, "controlMode": null, "pedalAction": null, "project": null, "item": null }, { "id": 913436, "recordingId": 9011, "robotName": "Alpha 4", "taskTitle": "QGF distillation", "operator": "Sergei Fedotov", "s3Url": "s3://hm-rainbow-mount/alpha8/record-pkg-id=9011/2026_07_25-07_48_17/", "createdAt": "2026-07-25T08:23:31.442947", "uploadStatus": null, "pickQuality": null, "controlMode": null, "pedalAction": null, "project": null, "item": null }, { "id": 913435, "recordingId": 9011, "robotName": "Alpha 4", "taskTitle": "QGF distillation", "operator": "Sergei Fedotov", "s3Url": "s3://hm-rainbow-mount/alpha8/record-pkg-id=9011/2026_07_25-07_47_47/", "createdAt": "2026-07-25T08:23:31.092453", "uploadStatus": "Converted", "pickQuality": null, "controlMode": null, "pedalAction": "not_found", "project": null, "item": null }, { "id": 913434, "recordingId": 9011, "robotName": "Alpha 4", "taskTitle": "QGF distillation", "operator": "Sergei Fedotov", "s3Url": "s3://hm-rainbow-mount/alpha8/record-pkg-id=9011/2026_07_25-07_47_22/", "createdAt": "2026-07-25T08:23:30.795991", "uploadStatus": null, "pickQuality": null, "controlMode": null, "pedalAction": null, "project": null, "item": null }, { "id": 913433, "recordingId": 9011, "robotName": "Alpha 4", "taskTitle": "QGF distillation", "operator": "Sergei Fedotov", "s3Url": "s3://hm-rainbow-mount/alpha8/record-pkg-id=9011/2026_07_25-07_46_56/", "createdAt": "2026-07-25T08:23:30.443619", "uploadStatus": null, "pickQuality": null, "controlMode": null, "pedalAction": null, "project": null, "item": null }, { "id": 913432, "recordingId": 9011, "robotName": "Alpha 4", "taskTitle": "QGF distillation", "operator": "Sergei Fedotov", "s3Url": "s3://hm-rainbow-mount/alpha8/record-pkg-id=9011/2026_07_25-07_46_33/", "createdAt": "2026-07-25T08:23:30.094819", "uploadStatus": null, "pickQuality": null, "controlMode": null, "pedalAction": null, "project": null, "item": null }, { "id": 913431, "recordingId": 9011, "robotName": "Alpha 4", "taskTitle": "QGF distillation", "operator": "Sergei Fedotov", "s3Url": "s3://hm-rainbow-mount/alpha8/record-pkg-id=9011/2026_07_25-07_46_00/", "createdAt": "2026-07-25T08:23:29.793361", "uploadStatus": null, "pickQuality": null, "controlMode": null, "pedalAction": null, "project": null, "item": null }, { "id": 913430, "recordingId": 9011, "robotName": "Alpha 4", "taskTitle": "QGF distillation", "operator": "Sergei Fedotov", "s3Url": "s3://hm-rainbow-mount/alpha8/record-pkg-id=9011/2026_07_25-07_45_30/", "createdAt": "2026-07-25T08:23:29.439415", "uploadStatus": null, "pickQuality": null, "controlMode": null, "pedalAction": null, "project": null, "item": null }, { "id": 913429, "recordingId": 9011, "robotName": "Alpha 4", "taskTitle": "QGF distillation", "operator": "Sergei Fedotov", "s3Url": "s3://hm-rainbow-mount/alpha8/record-pkg-id=9011/2026_07_25-07_44_54/", "createdAt": "2026-07-25T08:23:29.087731", "uploadStatus": null, "pickQuality": null, "controlMode": null, "pedalAction": null, "project": null, "item": null }, { "id": 913428, "recordingId": 9011, "robotName": "Alpha 4", "taskTitle": "QGF distillation", "operator": "Sergei Fedotov", "s3Url": "s3://hm-rainbow-mount/alpha8/record-pkg-id=9011/2026_07_25-07_44_49/", "createdAt": "2026-07-25T08:23:28.794198", "uploadStatus": null, "pickQuality": null, "controlMode": null, "pedalAction": null, "project": null, "item": null }, { "id": 913427, "recordingId": 9011, "robotName": "Alpha 4", "taskTitle": "QGF distillation", "operator": "Sergei Fedotov", "s3Url": "s3://hm-rainbow-mount/alpha8/record-pkg-id=9011/2026_07_25-07_44_16/", "createdAt": "2026-07-25T08:23:28.451951", "uploadStatus": "Converted", "pickQuality": null, "controlMode": null, "pedalAction": "not_found", "project": null, "item": null }, { "id": 913426, "recordingId": 9011, "robotName": "Alpha 4", "taskTitle": "QGF distillation", "operator": "Sergei Fedotov", "s3Url": "s3://hm-rainbow-mount/alpha8/record-pkg-id=9011/2026_07_25-07_43_45/", "createdAt": "2026-07-25T08:23:28.101429", "uploadStatus": "Converted", "pickQuality": null, "controlMode": null, "pedalAction": "not_found", "project": null, "item": null }, { "id": 913425, "recordingId": 9011, "robotName": "Alpha 4", "taskTitle": "QGF distillation", "operator": "Sergei Fedotov", "s3Url": "s3://hm-rainbow-mount/alpha8/record-pkg-id=9011/2026_07_25-07_43_19/", "createdAt": "2026-07-25T08:23:27.805067", "uploadStatus": null, "pickQuality": null, "controlMode": null, "pedalAction": null, "project": null, "item": null }, { "id": 913424, "recordingId": 9011, "robotName": "Alpha 4", "taskTitle": "QGF distillation", "operator": "Sergei Fedotov", "s3Url": "s3://hm-rainbow-mount/alpha8/record-pkg-id=9011/2026_07_25-07_42_42/", "createdAt": "2026-07-25T08:23:27.458173", "uploadStatus": null, "pickQuality": null, "controlMode": null, "pedalAction": null, "project": null, "item": null }, { "id": 913423, "recordingId": 9011, "robotName": "Alpha 4", "taskTitle": "QGF distillation", "operator": "Sergei Fedotov", "s3Url": "s3://hm-rainbow-mount/alpha8/record-pkg-id=9011/2026_07_25-07_42_14/", "createdAt": "2026-07-25T08:23:27.106892", "uploadStatus": null, "pickQuality": null, "controlMode": null, "pedalAction": null, "project": null, "item": null }, { "id": 913422, "recordingId": 9011, "robotName": "Alpha 4", "taskTitle": "QGF distillation", "operator": "Sergei Fedotov", "s3Url": "s3://hm-rainbow-mount/alpha8/record-pkg-id=9011/2026_07_25-07_41_32/", "createdAt": "2026-07-25T08:23:26.808115", "uploadStatus": null, "pickQuality": null, "controlMode": null, "pedalAction": null, "project": null, "item": null }, { "id": 913421, "recordingId": 9011, "robotName": "Alpha 4", "taskTitle": "QGF distillation", "operator": "Sergei Fedotov", "s3Url": "s3://hm-rainbow-mount/alpha4/record-pkg-id=9011/2026_07_24-09_46_53/", "createdAt": "2026-07-25T08:23:26.457835", "uploadStatus": null, "pickQuality": null, "controlMode": null, "pedalAction": null, "project": null, "item": null }, { "id": 913420, "recordingId": 9011, "robotName": "Alpha 4", "taskTitle": "QGF distillation", "operator": "Sergei Fedotov", "s3Url": "s3://hm-rainbow-mount/alpha4/record-pkg-id=9011/2026_07_24-09_46_27/", "createdAt": "2026-07-25T08:23:26.105258", "uploadStatus": null, "pickQuality": null, "controlMode": null, "pedalAction": null, "project": null, "item": null } ], "evaluationSessions": [ { "id": 19, "title": "Machine feeding joint pi0.5 model", "description": "", "robotName": "rainbow-r5", "modelId": 13, "modelName": "", "evaluator": "Diogo Almeida", "numTrials": 12, "createdAt": "2025-10-01T10:48:35.950648", "comment": "", "trials": [ { "id": 147, "videoLink": "s3://hm-rainbow-mount/rby5/session-id=19/2025_10_01-11_25_40/20251001_112540.mp4", "comment": "", "createdAt": "2025-10-01T11:26:12.323244" }, { "id": 148, "videoLink": "s3://hm-rainbow-mount/rby5/session-id=19/2025_10_01-11_39_56/20251001_113956.mp4", "comment": "", "createdAt": "2025-10-01T11:41:14.429190" }, { "id": 149, "videoLink": "s3://hm-rainbow-mount/rby5/session-id=19/2025_10_01-11_42_49/20251001_114249.mp4", "comment": "", "createdAt": "2025-10-01T11:46:13.940381" }, { "id": 150, "videoLink": "s3://hm-rainbow-mount/rby5/session-id=19/2025_10_01-12_05_03/20251001_120503.mp4", "comment": "", "createdAt": "2025-10-01T12:06:18.034990" }, { "id": 151, "videoLink": "s3://hm-rainbow-mount/rby5/session-id=19/2025_10_01-12_06_30/20251001_120630.mp4", "comment": "", "createdAt": "2025-10-01T12:11:18.566627" }, { "id": 152, "videoLink": "s3://hm-rainbow-mount/rby5/session-id=19/2025_10_01-12_16_15/20251001_121615.mp4", "comment": "", "createdAt": "2025-10-01T12:16:19.867108" }, { "id": 153, "videoLink": "s3://hm-rainbow-mount/rby5/session-id=19/2025_10_01-12_19_00/20251001_121900.mp4", "comment": "", "createdAt": "2025-10-01T12:21:19.655204" }, { "id": 154, "videoLink": "s3://hm-rainbow-mount/rby5/session-id=19/2025_10_01-12_17_03/20251001_121703.mp4", "comment": "", "createdAt": "2025-10-01T12:21:20.884304" }, { "id": 155, "videoLink": "s3://hm-rainbow-mount/rby5/session-id=19/2025_10_01-12_31_45/20251001_123145.mp4", "comment": "", "createdAt": "2025-10-01T17:55:40.556708" }, { "id": 156, "videoLink": "s3://hm-rainbow-mount/rby5/session-id=19/2025_10_01-18_00_24/20251001_180024.mp4", "comment": "", "createdAt": "2025-10-01T18:00:45.260729" }, { "id": 157, "videoLink": "s3://hm-rainbow-mount/rby5/session-id=19/2025_10_01-18_02_12/20251001_180212.mp4", "comment": "", "createdAt": "2025-10-01T18:05:44.921730" }, { "id": 158, "videoLink": "s3://hm-rainbow-mount/rby5/session-id=19/2025_10_01-18_12_31/20251001_181231.mp4", "comment": "", "createdAt": "2025-10-01T18:15:48.874933" } ] }, { "id": 18, "title": "test", "description": "test", "robotName": "Adam", "modelId": 1, "modelName": "", "evaluator": "Anastacia Koncz", "numTrials": 0, "createdAt": "2025-08-15T11:15:39.513476", "comment": "test", "trials": [] }, { "id": 17, "title": "Calibration check ", "description": "Checking if calibration has worked on R4", "robotName": "rainbow-r4", "modelId": 9, "modelName": "", "evaluator": "Diogo Almeida", "numTrials": 0, "createdAt": "2025-06-17T15:49:45.588009", "comment": "", "trials": [] }, { "id": 16, "title": "After 8 test for Calibration", "description": "Testing using Alfa 8 base positions", "robotName": "rainbow-r3", "modelId": 11, "modelName": "", "evaluator": "Diogo Almeida", "numTrials": 11, "createdAt": "2025-06-17T14:53:16.591907", "comment": "", "trials": [ { "id": 136, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=16/2025_06_17-14_54_00/20250617_145400.mp4", "comment": "", "createdAt": "2025-06-17T14:54:49.905366" }, { "id": 137, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=16/2025_06_17-14_55_44/20250617_145544.mp4", "comment": "", "createdAt": "2025-06-17T14:59:52.580251" }, { "id": 138, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=16/2025_06_17-14_58_57/20250617_145857.mp4", "comment": "", "createdAt": "2025-06-17T14:59:54.042936" }, { "id": 139, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=16/2025_06_17-14_57_00/20250617_145700.mp4", "comment": "", "createdAt": "2025-06-17T14:59:54.078920" }, { "id": 140, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=16/2025_06_17-15_04_29/20250617_150429.mp4", "comment": "", "createdAt": "2025-06-17T15:04:51.766335" }, { "id": 141, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=16/2025_06_17-14_59_53/20250617_145953.mp4", "comment": "", "createdAt": "2025-06-17T15:04:55.177510" }, { "id": 142, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=16/2025_06_17-15_02_12/20250617_150212.mp4", "comment": "", "createdAt": "2025-06-17T15:04:55.741057" }, { "id": 143, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=16/2025_06_17-15_00_59/20250617_150059.mp4", "comment": "", "createdAt": "2025-06-17T15:05:00.990219" }, { "id": 144, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=16/2025_06_17-15_03_20/20250617_150320.mp4", "comment": "", "createdAt": "2025-06-17T15:05:01.769933" }, { "id": 145, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=16/2025_06_17-15_06_29/20250617_150629.mp4", "comment": "", "createdAt": "2025-06-17T15:09:52.458076" }, { "id": 146, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=16/2025_06_17-15_05_37/20250617_150537.mp4", "comment": "", "createdAt": "2025-06-17T15:09:54.944052" } ] }, { "id": 15, "title": "Quavers test for Calibration", "description": "10 Picks for testing ", "robotName": "rainbow-r3", "modelId": 10, "modelName": "", "evaluator": "Diogo Almeida", "numTrials": 10, "createdAt": "2025-06-17T14:39:20.845237", "comment": "", "trials": [ { "id": 126, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=15/2025_06_17-14_39_42/20250617_143942.mp4", "comment": "", "createdAt": "2025-06-17T14:40:00.677642" }, { "id": 127, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=15/2025_06_17-14_40_25/20250617_144025.mp4", "comment": "", "createdAt": "2025-06-17T14:44:49.904151" }, { "id": 128, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=15/2025_06_17-14_41_01/20250617_144101.mp4", "comment": "", "createdAt": "2025-06-17T14:44:54.148725" }, { "id": 129, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=15/2025_06_17-14_42_23/20250617_144223.mp4", "comment": "", "createdAt": "2025-06-17T14:44:54.917811" }, { "id": 130, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=15/2025_06_17-14_41_54/20250617_144154.mp4", "comment": "", "createdAt": "2025-06-17T14:44:54.957773" }, { "id": 131, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=15/2025_06_17-14_43_06/20250617_144306.mp4", "comment": "", "createdAt": "2025-06-17T14:45:02.645140" }, { "id": 132, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=15/2025_06_17-14_43_35/20250617_144335.mp4", "comment": "", "createdAt": "2025-06-17T14:45:03.537990" }, { "id": 133, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=15/2025_06_17-14_44_16/20250617_144416.mp4", "comment": "", "createdAt": "2025-06-17T14:45:06.076231" }, { "id": 134, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=15/2025_06_17-14_44_46/20250617_144446.mp4", "comment": "", "createdAt": "2025-06-17T14:45:06.879725" }, { "id": 135, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=15/2025_06_17-14_45_17/20250617_144517.mp4", "comment": "", "createdAt": "2025-06-17T14:49:50.661709" } ] }, { "id": 14, "title": "Crumpets test for Calibration", "description": "", "robotName": "rainbow-r3", "modelId": 9, "modelName": "", "evaluator": "Diogo Almeida", "numTrials": 14, "createdAt": "2025-06-17T14:14:02.495402", "comment": "Actually Jack Running ", "trials": [ { "id": 112, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=14/2025_06_17-14_16_55/20250617_141655.mp4", "comment": "", "createdAt": "2025-06-17T14:19:44.750612" }, { "id": 113, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=14/2025_06_17-14_15_53/20250617_141553.mp4", "comment": "", "createdAt": "2025-06-17T14:19:44.770938" }, { "id": 114, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=14/2025_06_17-14_24_08/20250617_142408.mp4", "comment": "", "createdAt": "2025-06-17T14:24:46.008096" }, { "id": 115, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=14/2025_06_17-14_20_27/20250617_142027.mp4", "comment": "", "createdAt": "2025-06-17T14:24:46.924700" }, { "id": 116, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=14/2025_06_17-14_22_16/20250617_142216.mp4", "comment": "", "createdAt": "2025-06-17T14:24:47.682270" }, { "id": 117, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=14/2025_06_17-14_28_52/20250617_142852.mp4", "comment": "", "createdAt": "2025-06-17T14:29:50.396566" }, { "id": 118, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=14/2025_06_17-14_27_09/20250617_142709.mp4", "comment": "", "createdAt": "2025-06-17T14:29:50.564045" }, { "id": 119, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=14/2025_06_17-14_27_40/20250617_142740.mp4", "comment": "", "createdAt": "2025-06-17T14:29:53.784205" }, { "id": 120, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=14/2025_06_17-14_29_30/20250617_142930.mp4", "comment": "", "createdAt": "2025-06-17T14:29:54.010849" }, { "id": 121, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=14/2025_06_17-14_28_22/20250617_142822.mp4", "comment": "", "createdAt": "2025-06-17T14:29:56.615276" }, { "id": 122, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=14/2025_06_17-14_26_28/20250617_142628.mp4", "comment": "", "createdAt": "2025-06-17T14:29:59.953662" }, { "id": 123, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=14/2025_06_17-14_24_59/20250617_142459.mp4", "comment": "", "createdAt": "2025-06-17T14:30:02.386922" }, { "id": 124, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=14/2025_06_17-14_25_44/20250617_142544.mp4", "comment": "", "createdAt": "2025-06-17T14:30:02.589025" }, { "id": 125, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=14/2025_06_17-14_30_03/20250617_143003.mp4", "comment": "", "createdAt": "2025-06-17T14:34:47.171728" } ] }, { "id": 13, "title": "Testing quavers", "description": "", "robotName": "rainbow-r2", "modelId": 8, "modelName": "", "evaluator": "Diogo Almeida", "numTrials": 0, "createdAt": "2025-05-19T15:27:00.915273", "comment": "", "trials": [] }, { "id": 12, "title": "Quavers", "description": "", "robotName": "rainbow-r2", "modelId": 8, "modelName": "", "evaluator": "Diogo Almeida", "numTrials": 0, "createdAt": "2025-05-19T15:25:33.153170", "comment": "", "trials": [] }, { "id": 11, "title": "Quavers on R2", "description": "", "robotName": "rainbow-r2", "modelId": 8, "modelName": "", "evaluator": "Diogo Almeida", "numTrials": 0, "createdAt": "2025-05-19T15:25:02.803864", "comment": "", "trials": [] }, { "id": 10, "title": "Crumpets with recoveries", "description": "Evaluate whether adding recoveries to the crumpet dataset leads to better behavior", "robotName": "rainbow-r3", "modelId": 5, "modelName": "", "evaluator": "Diogo Almeida", "numTrials": 21, "createdAt": "2025-05-12T17:38:16.761466", "comment": "", "trials": [ { "id": 26, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=10/20250512_174055.mp4", "comment": "", "createdAt": "2025-05-13T16:49:06.013737" }, { "id": 27, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=10/20250512_174138.mp4", "comment": "", "createdAt": "2025-05-13T16:49:06.207897" }, { "id": 28, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=10/20250512_174219.mp4", "comment": "", "createdAt": "2025-05-13T16:49:06.399085" }, { "id": 29, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=10/20250512_174311.mp4", "comment": "", "createdAt": "2025-05-13T16:49:06.581359" }, { "id": 30, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=10/20250512_174342.mp4", "comment": "", "createdAt": "2025-05-13T16:49:06.767381" }, { "id": 64, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=10/2025_05_12-17_40_28/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:17.399279" }, { "id": 65, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=10/2025_05_12-17_39_26/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:17.435117" }, { "id": 66, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=10/2025_05_12-17_41_38/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:17.765644" }, { "id": 67, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=10/2025_05_12-17_40_55/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:17.776915" }, { "id": 68, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=10/2025_05_12-17_42_19/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:18.358176" }, { "id": 69, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=10/2025_05_12-17_43_11/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:18.382103" }, { "id": 25, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=10/20250512_174028.mp4", "comment": "The video was deleted and moved. ", "createdAt": "2025-05-13T16:49:05.824099" }, { "id": 74, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=10/2025_05_12-17_43_42/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:19.105587" }, { "id": 104, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=10/2025_05_12-17_40_55/evaluation_head.mp4", "comment": "Added manually via portal", "createdAt": "2025-05-13T19:38:25.095050" }, { "id": 105, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=10/2025_05_12-17_40_55/evaluation_head.mp4", "comment": "added manually", "createdAt": "2025-05-13T19:39:36.108102" }, { "id": 106, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=10/2025_05_12-17_40_55/evaluation_head.mp4", "comment": "added manually", "createdAt": "2025-05-13T19:41:42.970321" }, { "id": 107, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=10/2025_05_12-17_40_55/evaluation_head.mp4", "comment": "manual", "createdAt": "2025-05-13T19:43:38.169637" }, { "id": 108, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=10/2025_05_12-17_40_55/evaluation_head.mp4", "comment": "rest api created", "createdAt": "2025-05-13T19:48:15.012413" }, { "id": 109, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=10/2025_05_12-17_40_55/evaluation_head.mp4", "comment": "rest api created", "createdAt": "2025-05-13T19:49:00.072077" }, { "id": 110, "videoLink": "s3://hm-rainbow-mount/rby3/evaluation_session_id=2/20250507_090447.mp4", "comment": "created manually for testing. The video is not from this eval. ", "createdAt": "2025-05-13T20:05:24.131848" }, { "id": 8, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=10/20250512_173926.mp4", "comment": "The video was deleted and moved. ", "createdAt": "2025-05-12T17:55:57.755970" } ] }, { "id": 9, "title": "Quality street million model", "description": "", "robotName": "rainbow-r3", "modelId": 7, "modelName": "", "evaluator": "Diogo Almeida", "numTrials": 16, "createdAt": "2025-05-12T17:28:57.735596", "comment": "", "trials": [ { "id": 56, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=9/20250512_172942.mp4", "comment": "", "createdAt": "2025-05-13T16:49:11.615968" }, { "id": 57, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=9/20250512_173021.mp4", "comment": "", "createdAt": "2025-05-13T16:49:11.808316" }, { "id": 58, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=9/20250512_173048.mp4", "comment": "", "createdAt": "2025-05-13T16:49:11.996690" }, { "id": 59, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=9/20250512_173154.mp4", "comment": "", "createdAt": "2025-05-13T16:49:12.192215" }, { "id": 60, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=9/20250512_173235.mp4", "comment": "", "createdAt": "2025-05-13T16:49:12.392561" }, { "id": 61, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=9/20250512_173320.mp4", "comment": "", "createdAt": "2025-05-13T16:49:12.591853" }, { "id": 62, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=9/20250512_173426.mp4", "comment": "", "createdAt": "2025-05-13T16:49:12.773149" }, { "id": 63, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=9/20250512_173502.mp4", "comment": "", "createdAt": "2025-05-13T16:49:12.960146" }, { "id": 95, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=9/2025_05_12-17_30_21/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:23.785364" }, { "id": 96, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=9/2025_05_12-17_29_42/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:23.802959" }, { "id": 98, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=9/2025_05_12-17_31_54/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:24.754857" }, { "id": 99, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=9/2025_05_12-17_30_48/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:24.779063" }, { "id": 100, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=9/2025_05_12-17_32_35/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:24.801720" }, { "id": 101, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=9/2025_05_12-17_33_20/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:25.161835" }, { "id": 102, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=9/2025_05_12-17_34_26/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:25.375613" }, { "id": 103, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=9/2025_05_12-17_35_02/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:25.399112" } ] }, { "id": 8, "title": "After eight slow", "description": "", "robotName": "rainbow-r3", "modelId": 3, "modelName": "", "evaluator": "Diogo Almeida", "numTrials": 34, "createdAt": "2025-05-12T17:06:46.848120", "comment": "", "trials": [ { "id": 39, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/20250512_170856.mp4", "comment": "", "createdAt": "2025-05-13T16:49:08.439245" }, { "id": 40, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/20250512_171032.mp4", "comment": "", "createdAt": "2025-05-13T16:49:08.617615" }, { "id": 41, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/20250512_171125.mp4", "comment": "", "createdAt": "2025-05-13T16:49:08.808420" }, { "id": 42, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/20250512_171215.mp4", "comment": "", "createdAt": "2025-05-13T16:49:09.005489" }, { "id": 43, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/20250512_171335.mp4", "comment": "", "createdAt": "2025-05-13T16:49:09.190138" }, { "id": 44, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/20250512_171413.mp4", "comment": "", "createdAt": "2025-05-13T16:49:09.379944" }, { "id": 45, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/20250512_171528.mp4", "comment": "", "createdAt": "2025-05-13T16:49:09.568643" }, { "id": 46, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/20250512_171655.mp4", "comment": "", "createdAt": "2025-05-13T16:49:09.765192" }, { "id": 47, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/20250512_171733.mp4", "comment": "", "createdAt": "2025-05-13T16:49:09.946138" }, { "id": 48, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/20250512_171900.mp4", "comment": "", "createdAt": "2025-05-13T16:49:10.132396" }, { "id": 49, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/20250512_171942.mp4", "comment": "", "createdAt": "2025-05-13T16:49:10.312037" }, { "id": 50, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/20250512_172026.mp4", "comment": "", "createdAt": "2025-05-13T16:49:10.500841" }, { "id": 51, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/20250512_172153.mp4", "comment": "", "createdAt": "2025-05-13T16:49:10.691083" }, { "id": 52, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/20250512_172230.mp4", "comment": "", "createdAt": "2025-05-13T16:49:10.874803" }, { "id": 53, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/20250512_172329.mp4", "comment": "", "createdAt": "2025-05-13T16:49:11.060869" }, { "id": 54, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/20250512_172409.mp4", "comment": "", "createdAt": "2025-05-13T16:49:11.245338" }, { "id": 55, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/20250512_172744.mp4", "comment": "", "createdAt": "2025-05-13T16:49:11.432216" }, { "id": 75, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/2025_05_12-17_10_32/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:19.571631" }, { "id": 82, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/2025_05_12-17_12_15/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:20.376919" }, { "id": 84, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/2025_05_12-17_15_28/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:21.274083" }, { "id": 85, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/2025_05_12-17_14_13/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:21.286263" }, { "id": 88, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/2025_05_12-17_19_42/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:21.770645" }, { "id": 91, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/2025_05_12-17_22_30/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:22.886795" }, { "id": 93, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/2025_05_12-17_21_53/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:22.907431" }, { "id": 94, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/2025_05_12-17_27_44/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:23.776187" }, { "id": 80, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/2025_05_12-17_08_56/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:20.157685" }, { "id": 81, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/2025_05_12-17_11_25/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:20.179864" }, { "id": 83, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/2025_05_12-17_13_35/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:20.387708" }, { "id": 86, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/2025_05_12-17_16_55/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:21.295107" }, { "id": 87, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/2025_05_12-17_17_33/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:21.305199" }, { "id": 89, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/2025_05_12-17_19_00/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:21.781884" }, { "id": 90, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/2025_05_12-17_20_26/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:21.976005" }, { "id": 92, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/2025_05_12-17_23_29/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:22.897769" }, { "id": 97, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=8/2025_05_12-17_24_09/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:23.813617" } ] }, { "id": 7, "title": "After eight fast", "description": "Fast policy; image transforms; admittance controller fixes", "robotName": "rainbow-r3", "modelId": 1, "modelName": "", "evaluator": "Artem Shutak", "numTrials": 16, "createdAt": "2025-05-12T13:06:10.148062", "comment": "", "trials": [ { "id": 31, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=7/20250512_131323.mp4", "comment": "", "createdAt": "2025-05-13T16:49:06.946121" }, { "id": 32, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=7/20250512_131620.mp4", "comment": "", "createdAt": "2025-05-13T16:49:07.125978" }, { "id": 33, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=7/20250512_131747.mp4", "comment": "", "createdAt": "2025-05-13T16:49:07.309428" }, { "id": 34, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=7/20250512_131839.mp4", "comment": "", "createdAt": "2025-05-13T16:49:07.493933" }, { "id": 35, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=7/20250512_131929.mp4", "comment": "", "createdAt": "2025-05-13T16:49:07.684980" }, { "id": 36, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=7/20250512_132030.mp4", "comment": "", "createdAt": "2025-05-13T16:49:07.873477" }, { "id": 37, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=7/20250512_132203.mp4", "comment": "", "createdAt": "2025-05-13T16:49:08.056300" }, { "id": 38, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=7/20250512_132303.mp4", "comment": "", "createdAt": "2025-05-13T16:49:08.240949" }, { "id": 70, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=7/2025_05_12-13_18_39/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:18.573073" }, { "id": 71, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=7/2025_05_12-13_16_20/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:18.585408" }, { "id": 72, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=7/2025_05_12-13_17_47/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:18.623436" }, { "id": 73, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=7/2025_05_12-13_13_23/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:18.632346" }, { "id": 78, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=7/2025_05_12-13_22_03/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:19.604058" }, { "id": 76, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=7/2025_05_12-13_19_29/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:19.581390" }, { "id": 77, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=7/2025_05_12-13_20_30/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:19.595161" }, { "id": 79, "videoLink": "s3://hm-rainbow-mount/rby3/session-id=7/2025_05_12-13_23_03/evaluation_head.mp4", "comment": "", "createdAt": "2025-05-13T19:20:19.634749" } ] }, { "id": 6, "title": "Test evaluatoin", "description": "sdfsdf", "robotName": "Dev", "modelId": 2, "modelName": "", "evaluator": "Artem Shutak", "numTrials": 1, "createdAt": "2025-05-09T15:01:34.678375", "comment": "testy test", "trials": [ { "id": 111, "videoLink": "s3://hm-rainbow-mount/rby3/evaluation_session_id=2/20250507_090447.mp4", "comment": "pupupuuu", "createdAt": "2025-05-13T20:06:02.348484" } ] }, { "id": 4, "title": "Test Session", "description": "", "robotName": "H1", "modelId": 1, "modelName": "", "evaluator": "Artem Shutak", "numTrials": 0, "createdAt": "2025-05-07T19:58:29.001508", "comment": "", "trials": [] }, { "id": 3, "title": "Test Session", "description": "", "robotName": "H1", "modelId": 1, "modelName": "", "evaluator": "Artem Shutak", "numTrials": 0, "createdAt": "2025-05-07T19:58:22.039435", "comment": "", "trials": [] }, { "id": 1, "title": "Test evaluation", "description": "Some description of the test", "robotName": "Dev", "modelId": 1, "modelName": "", "evaluator": "Artem Shutak", "numTrials": 3, "createdAt": "2025-05-01T22:15:15.623386", "comment": "Tratatatata", "trials": [ { "id": 6, "videoLink": "s3://hm-rainbow-mount/rby3/evaluation_session_id=2/20250507_090707.mp4", "comment": "new one\nanother one", "createdAt": "2025-05-12T08:52:23.639843" }, { "id": 7, "videoLink": "s3://hm-rainbow-mount/rby3/evaluation_session_id=2/2025_05_07-09_07_07/test_vid.mp4", "comment": "", "createdAt": "2025-05-12T17:05:14.311387" }, { "id": 9, "videoLink": "s3://hm-rainbow-mount/rby3/evaluation_session_id=2/20250507_090707.mp4", "comment": "", "createdAt": "2025-05-13T16:15:39.082521" } ] } ], "mlModels": [], "users": [ { "id": 3, "name": "Sergei Fedotov", "email": "sfed@skl.vc" }, { "id": 1, "name": "Artem Shutak", "email": "ashu@skl.vc" }, { "id": 4, "name": "Diogo Almeida", "email": "dalm@skl.vc" }, { "id": 5, "name": "Arjun Liddar", "email": "alid@skl.vc" }, { "id": 6, "name": "Amadeea Sara Filip", "email": "asfi@skl.vc" }, { "id": 7, "name": "Mhara Koutsoudi", "email": "mkou@skl.vc" }, { "id": 8, "name": "Anastacia Koncz", "email": "anak@skl.vc" }, { "id": 9, "name": "Casey Atienza", "email": "kcva@skl.vc" }, { "id": 10, "name": "Daniel Ponsford", "email": "danp@skl.vc" }, { "id": 11, "name": "Warsame Sheikhdon", "email": "wash@skl.vc" }, { "id": 14, "name": "Viktoria Glypti", "email": "veig@skl.vc" }, { "id": 16, "name": "Grace Johnson", "email": "graj@thehumanoid.ai" }, { "id": 17, "name": "Jamie Uy", "email": "jamu@thehumanoid.ai" }, { "id": 18, "name": "Jaimin Pithwa", "email": "jaip@thehumanoid.ai" }, { "id": 19, "name": "Claudio Coppola", "email": "ccop@thehumanoid.ai" }, { "id": 20, "name": "Brian Delhaisse", "email": "bdel@skl.vc" }, { "id": 21, "name": "Ezeddin Al Hakim", "email": "ezah@skl.vc" }, { "id": 22, "name": "Jake Longley", "email": "jakl@thehumanoid.ai" }, { "id": 23, "name": "Team Red", "email": "alid@skl.vc" }, { "id": 24, "name": "Team Green", "email": "alid@skl.vc" }, { "id": 25, "name": "Team Blue", "email": "alid@skl.vc" }, { "id": 26, "name": "Polina Shorenko", "email": "pols@skl.vc" }, { "id": 27, "name": "Krova Farik", "email": "kfar@skl.vc" }, { "id": 28, "name": "XDOF", "email": "xdof-operator@thehumanoid.ai" }, { "id": 29, "name": "Mohamed Hisham", "email": "mhis@thehumanoid.ai" }, { "id": 30, "name": "Dilip Naidu Govind", "email": "dina@thehumanoid.ai" }, { "id": 31, "name": "Hamza Shekhdon", "email": "hash@thehumanoid.ai" }, { "id": 32, "name": "Alexandra Vorrea", "email": "alvo@thehumanoid.ai" }, { "id": 33, "name": "Saleh Zaiba", "email": "saza@thehumanoid.ai" }, { "id": 35, "name": "Tobias Jacob", "email": "tjac@thehumanoid.ai" }, { "id": 36, "name": "Bao Tran", "email": "batr@thehumanoid.ai" }, { "id": 34, "name": "Barbara Charles", "email": "bach@thehumanoid.ai" }, { "id": 37, "name": "Rodrigo Zenha", "email": "rzen@skl.vc" }, { "id": 38, "name": "Dmitrii Ovsiannikov", "email": "dovs@skl.vc" }, { "id": 39, "name": "Yoo-Jin Jung", "email": "yjj@skl.vc" }, { "id": 40, "name": "Federica Leo", "email": "federica.leo@thehumanoid.ai" }, { "id": 41, "name": "Shihabuddeen Waqar", "email": "shihab.waqar@thehumanoid.ai" }, { "id": 42, "name": "Rajat Sharma", "email": "rash@thehumanoid.ai" }, { "id": 43, "name": "Noor K Mahdy", "email": "noor.mahdy@thehumanoid.ai" }, { "id": 44, "name": "Karim Shaban", "email": "ksha@skl.vc" }, { "id": 45, "name": "Aiden Pocha", "email": "aiden.pocha@hmnd.ai" }, { "id": 15, "name": "Dan Mason", "email": "dmas@thehumanoid.ai" }, { "id": 46, "name": "Richard Weight", "email": "riwe@thehumanoid.ai" }, { "id": 47, "name": "Nia Oloyede", "email": "nolo@thehumanoid.ai" }, { "id": 48, "name": "Saurabh Kumar", "email": "saurabh.kumar@thehumanoid.ai" }, { "id": 49, "name": "Matteo Presutto", "email": "mapr@thehumanoid.ai" }, { "id": 50, "name": "Jacob Moss", "email": "jamo@thehumanoid.ai" }, { "id": 51, "name": "Edward Holder", "email": "edward.holder@thehumanoid.ai" }, { "id": 52, "name": "Zayd Liloul", "email": "zali@thehumanoid.ai" }, { "id": 53, "name": "Malorie Iovino", "email": "malorie.iovino@thehumanoid.ai" }, { "id": 54, "name": "Saeid Samadi", "email": "ssam@skl.vc" }, { "id": 55, "name": "Omkar Ninawe", "email": "onin@skl.vc" }, { "id": 56, "name": "Joe Jeffcock", "email": "joe.jeffcock@thehumanoid.ai" }, { "id": 57, "name": "Fanqi Xu", "email": "fanqi.xu@thehumanoid.ai" }, { "id": 58, "name": "Fanqi", "email": "fanqi.xu@thehumanoid.ai" }, { "id": 59, "name": "Aisha Marenah", "email": "aisha.marenah@thehumanoid.ai" }, { "id": 60, "name": "Adrian Petrescu", "email": "adpe@thehumanoid.ai" }, { "id": 61, "name": "Jin Rhee", "email": "jin.rhee@thehumanoid.ai" }, { "id": 62, "name": "Raphael Shuaibu", "email": "raphael.shuaibu@thehumanoid.ai" } ], "projects": [ { "id": 6, "name": "Alpha Ring Picking", "description": "Metadata Template for Ring picking with alpha", "recordings": 0, "totalEpisodes": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingNotRequired": 0, "labelingGood": 0, "metadataKeys": [ "Item", "prompt", "Pick Quality", "Comment", "Technical Issue", "Task Stage", "Gripper Type" ], "requiredKeys": [ "Item", "prompt", "Pick Quality", "Task Stage", "Gripper Type" ] }, { "id": 27, "name": "Amazon", "description": "", "recordings": 0, "totalEpisodes": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingNotRequired": 0, "labelingGood": 0, "metadataKeys": [ "Control Mode", "is_onsite", "Project", "Box type", "Pick Quality", "Comment", "Technical Issue", "Task Stage", "Amazon Recovery Type", "Certified" ], "requiredKeys": [ "Control Mode", "is_onsite", "Project", "Pick Quality", "Task Stage", "Amazon Recovery Type", "Certified" ] }, { "id": 29, "name": "Beta feasibility study", "description": "", "recordings": 0, "totalEpisodes": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingNotRequired": 0, "labelingGood": 0, "metadataKeys": [ "Project", "Tote", "Task Stage", "Tote pick type" ], "requiredKeys": [ "Project", "Tote", "Task Stage", "Tote pick type" ] }, { "id": 13, "name": "Bosch", "description": "PoC", "recordings": 0, "totalEpisodes": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingNotRequired": 0, "labelingGood": 0, "metadataKeys": [ "Item", "is_onsite", "Pick Quality", "Comment", "Re-reviewed", "Bosch_Stage", "Technical Issue" ], "requiredKeys": [ "Item", "Pick Quality", "Bosch_Stage" ] }, { "id": 5, "name": "Bussing", "description": "Template with the new bussing task annotations", "recordings": 0, "totalEpisodes": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingNotRequired": 0, "labelingGood": 0, "metadataKeys": [ "Item", "Arm", "prompt", "Pick Quality", "Comment", "Technical Issue", "Item 2" ], "requiredKeys": [ "Item", "Arm", "prompt", "Pick Quality" ] }, { "id": 31, "name": "Capability Factory - Ring on stick", "description": "", "recordings": 0, "totalEpisodes": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingNotRequired": 0, "labelingGood": 0, "metadataKeys": [ "Item", "Project", "Pick Quality", "Task Stage" ], "requiredKeys": [ "Item", "Project", "Pick Quality", "Task Stage" ] }, { "id": 21, "name": "Ceva", "description": "", "recordings": 0, "totalEpisodes": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingNotRequired": 0, "labelingGood": 0, "metadataKeys": [ "is_onsite", "Project", "Ceva Task Stage", "Payload", "Pick Quality", "Technical Issue", "Gripper Type" ], "requiredKeys": [ "Project", "Ceva Task Stage", "Payload", "Pick Quality", "Gripper Type" ] }, { "id": 10, "name": "Common", "description": "The template where we have one required field: pick_quality. We can use it where the whole package is homogeneous and doesn't need episode-level labelling", "recordings": 0, "totalEpisodes": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingNotRequired": 0, "labelingGood": 0, "metadataKeys": [ "Pick Quality", "Comment", "Technical Issue" ], "requiredKeys": [ "Pick Quality" ] }, { "id": 16, "name": "FT Tests", "description": "", "recordings": 0, "totalEpisodes": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingNotRequired": 0, "labelingGood": 0, "metadataKeys": [ "Item", "Project", "Control Mode", "is_onsite", "FT Tests Task Stage", "Item 2", "Pick Quality", "Comment" ], "requiredKeys": [ "Item", "Project", "Control Mode", "is_onsite", "FT Tests Task Stage", "Item 2", "Pick Quality" ] }, { "id": 23, "name": "Household Biped", "description": "", "recordings": 0, "totalEpisodes": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingNotRequired": 0, "labelingGood": 0, "metadataKeys": [ "Recovery Type", "Household Biped Project", "Household Items", "Pick Quality", "Comment", "Re-reviewed", "Technical Issue", "Arm", "Household totes", "Terminal", "Certified" ], "requiredKeys": [ "Household Biped Project", "Household Items", "Pick Quality", "Arm", "Household totes", "Terminal", "Certified" ] }, { "id": 20, "name": "Household Egocentric", "description": "", "recordings": 0, "totalEpisodes": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingNotRequired": 0, "labelingGood": 0, "metadataKeys": [ "Item", "Arm", "Household Egocentric Stage", "Household Items", "Robot height", "Pick Quality", "Comment", "Technical Issue", "Terminal" ], "requiredKeys": [ "Item", "Arm", "Household Egocentric Stage", "Robot height", "Pick Quality", "Terminal" ] }, { "id": 12, "name": "Martur Fompak", "description": "", "recordings": 0, "totalEpisodes": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingNotRequired": 0, "labelingGood": 0, "metadataKeys": [ "Item", "Technical Issue", "Action Type", "Martur Task Stage", "Pick Quality", "Comment", "Re-reviewed", "prompt" ], "requiredKeys": [ "Item", "Martur Task Stage", "Pick Quality", "prompt" ] }, { "id": 7, "name": "Picking and Placing tote", "description": "Picking and Placing tote", "recordings": 0, "totalEpisodes": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingNotRequired": 0, "labelingGood": 0, "metadataKeys": [ "Re-reviewed", "prompt", "Item", "Pick Quality", "Arm", "Comment", "Technical Issue", "Task Stage with Tote", "Action Type" ], "requiredKeys": [ "prompt", "Item", "Pick Quality", "Arm", "Task Stage with Tote" ] }, { "id": 28, "name": "Ring Picking 2026", "description": "", "recordings": 0, "totalEpisodes": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingNotRequired": 0, "labelingGood": 0, "metadataKeys": [ "Bin size", "is_onsite", "Project", "Pick Quality", "Comment", "Technical Issue", "Task Stage", "Ring Picking Pick Recovery Type", "Ring Type", "Gripper Type", "Bin depth", "Torso turn direction", "Certified", "Robot Rotation" ], "requiredKeys": [ "Bin size", "is_onsite", "Project", "Pick Quality", "Task Stage", "Ring Picking Pick Recovery Type", "Ring Type", "Gripper Type", "Bin depth", "Certified" ] }, { "id": 14, "name": "Robostore", "description": "", "recordings": 0, "totalEpisodes": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingNotRequired": 0, "labelingGood": 0, "metadataKeys": [ "Project", "Control Mode", "is_onsite", "Robostore Task Stage", "Robostore Right Item", "Robostore Left Item", "Pick Quality", "Comment", "Technical Issue", "is_robostore_wait_handover" ], "requiredKeys": [ "Project", "Control Mode", "is_onsite", "Robostore Task Stage", "Robostore Right Item", "Robostore Left Item", "Pick Quality" ] }, { "id": 17, "name": "Robostore Bussing", "description": "", "recordings": 0, "totalEpisodes": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingNotRequired": 0, "labelingGood": 0, "metadataKeys": [ "Project", "Control Mode", "is_onsite", "prompt", "Pick Quality", "Comment", "Technical Issue", "Robostore Bussing Task Stage", "Action Type" ], "requiredKeys": [ "Project", "Control Mode", "is_onsite", "prompt", "Pick Quality", "Robostore Bussing Task Stage" ] }, { "id": 9, "name": "Robot Skills", "description": "", "recordings": 0, "totalEpisodes": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingNotRequired": 0, "labelingGood": 0, "metadataKeys": [ "Pre-training tasks", "prompt", "Item", "Pick Quality", "Arm", "Comment", "Technical Issue" ], "requiredKeys": [ "Pre-training tasks", "prompt", "Item", "Pick Quality", "Arm" ] }, { "id": 30, "name": "Schaeffler Pilot", "description": "", "recordings": 0, "totalEpisodes": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingNotRequired": 0, "labelingGood": 0, "metadataKeys": [ "Tote", "Pick Quality", "Comment", "Task Stage" ], "requiredKeys": [ "Tote", "Pick Quality", "Task Stage" ] }, { "id": 11, "name": "Symphony Biped", "description": "", "recordings": 0, "totalEpisodes": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingNotRequired": 0, "labelingGood": 0, "metadataKeys": [ "Symphony Biped Task Stage", "Pick Quality", "Comment", "Technical Issue", "Action Type" ], "requiredKeys": [ "Symphony Biped Task Stage", "Pick Quality" ] }, { "id": 8, "name": "Symphony Bussing", "description": "", "recordings": 0, "totalEpisodes": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingNotRequired": 0, "labelingGood": 0, "metadataKeys": [ "Item 2", "prompt", "Item", "Pick Quality", "Arm", "Comment", "Technical Issue", "Bussing_Symphony", "Action Type" ], "requiredKeys": [ "prompt", "Item", "Pick Quality", "Arm", "Bussing_Symphony" ] }, { "id": 18, "name": "Symphony II", "description": "", "recordings": 0, "totalEpisodes": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingNotRequired": 0, "labelingGood": 0, "metadataKeys": [ "Symphony II Task Stage", "Project", "Tote", "Symphony II Item", "Pick Quality", "Comment", "Technical Issue", "With new UMI mount" ], "requiredKeys": [ "Symphony II Task Stage", "Project", "Tote", "Symphony II Item", "Pick Quality" ] }, { "id": 24, "name": "Symphony II Objects", "description": "", "recordings": 0, "totalEpisodes": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingNotRequired": 0, "labelingGood": 0, "metadataKeys": [ "Recovery Type", "Environment", "Symphony II Task Stage", "Project", "Tote", "Pick Quality", "Comment", "Technical Issue", "Symphony II right item", "Symphony II left item", "dev_branch", "Terminal", "Gripper Type" ], "requiredKeys": [ "Environment", "Symphony II Task Stage", "Project", "Tote", "Pick Quality", "Symphony II right item", "Symphony II left item", "Gripper Type" ] }, { "id": 19, "name": "System 1.5 [Memory]", "description": "A memory layer for the VLA", "recordings": 0, "totalEpisodes": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingNotRequired": 0, "labelingGood": 0, "metadataKeys": [ "Project", "Pick Quality" ], "requiredKeys": [ "Project", "Pick Quality" ] }, { "id": 26, "name": "Test", "description": "", "recordings": 0, "totalEpisodes": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingNotRequired": 0, "labelingGood": 0, "metadataKeys": [ "Task Stage" ], "requiredKeys": [ "Task Stage" ] }, { "id": 15, "name": "UMI", "description": "", "recordings": 0, "totalEpisodes": 0, "labelingComplete": 0, "labelingRequired": 0, "labelingNotRequired": 0, "labelingGood": 0, "metadataKeys": [ "Item", "Item 2", "prompt", "Project", "UMI task stages", "Pick Quality", "Comment", "Technical Issue", "Arm", "Gripper Type" ], "requiredKeys": [ "Item", "UMI task stages", "Pick Quality", "Gripper Type" ] } ], "episodeStats": { "total": 375577, "uploadStatus": { "Converted": 1969, "MissingMetadataFile": 2, "MissingMetadata": 2, "NoVideoTopics": 1 }, "pickQuality": { "Good quality": 215, "Mistake": 194, "Bad quality": 67 }, "controlMode": { "teleop": 313 }, "pedalAction": { "saved": 840, "not_found": 647, "rejected": 487 }, "sampled": 2000 }, "stats": { "totalRecordings": 9052, "totalEpisodes": 0, "byRobot": {}, "byTemplate": {}, "byOperator": {}, "byOperatorDetail": {}, "episodesByRobot": {}, "labeling": { "complete": 0, "required": 0, "notRequired": 0, "good": 0 } } }, "changes": { "compared_to": "2026-07-25T08:22:22.990558+00:00", "computed_at": "2026-07-25T08:27:34.029096+00:00", "summary": { "added": 0, "removed": 0, "changed": 35 }, "sections": { "console": { "robots": { "added": [], "removed": [], "changed": [ { "name": "Quest 7", "field": "recordingCount", "from": 1, "to": 0 }, { "name": "Quest 7", "field": "episodeCount", "from": 49, "to": 0 }, { "name": "Alpha 12", "field": "recordingCount", "from": 57, "to": 0 }, { "name": "Alpha 12", "field": "episodeCount", "from": 1851, "to": 0 }, { "name": "Alpha 10", "field": "recordingCount", "from": 11, "to": 0 }, { "name": "Alpha 10", "field": "episodeCount", "from": 22, "to": 0 }, { "name": "UMI 1", "field": "recordingCount", "from": 37, "to": 0 }, { "name": "UMI 1", "field": "episodeCount", "from": 1323, "to": 0 }, { "name": "UMI 2", "field": "recordingCount", "from": 4, "to": 0 }, { "name": "UMI 2", "field": "episodeCount", "from": 149, "to": 0 }, { "name": "Biped 3", "field": "recordingCount", "from": 102, "to": 0 }, { "name": "Biped 3", "field": "episodeCount", "from": 2592, "to": 0 }, { "name": "Alpha 4", "field": "recordingCount", "from": 1, "to": 0 }, { "name": "Alpha 4", "field": "episodeCount", "from": 662, "to": 0 }, { "name": "Alpha 2", "field": "recordingCount", "from": 18, "to": 0 }, { "name": "Alpha 2", "field": "episodeCount", "from": 62, "to": 0 }, { "name": "Egocentric 1", "field": "recordingCount", "from": 2, "to": 0 }, { "name": "Egocentric 1", "field": "episodeCount", "from": 11, "to": 0 }, { "name": "Alpha 7", "field": "recordingCount", "from": 13, "to": 0 }, { "name": "Alpha 7", "field": "episodeCount", "from": 492, "to": 0 }, { "name": "Quest 5", "field": "recordingCount", "from": 1, "to": 0 }, { "name": "Alpha 1 (Sim)", "field": "recordingCount", "from": 4, "to": 0 }, { "name": "Alpha 1 (Sim)", "field": "episodeCount", "from": 234, "to": 0 }, { "name": "Alpha 11", "field": "recordingCount", "from": 40, "to": 0 }, { "name": "Alpha 11", "field": "episodeCount", "from": 2753, "to": 0 }, { "name": "Alpha 3", "field": "recordingCount", "from": 53, "to": 0 }, { "name": "Alpha 3", "field": "episodeCount", "from": 1531, "to": 0 }, { "name": "Alpha 5", "field": "recordingCount", "from": 91, "to": 0 }, { "name": "Alpha 5", "field": "episodeCount", "from": 3826, "to": 0 }, { "name": "Alpha 6", "field": "recordingCount", "from": 56, "to": 0 }, { "name": "Alpha 6", "field": "episodeCount", "from": 1617, "to": 0 }, { "name": "Alpha 9", "field": "recordingCount", "from": 1, "to": 0 }, { "name": "Alpha 9", "field": "episodeCount", "from": 1, "to": 0 }, { "name": "Egocentric 2", "field": "recordingCount", "from": 8, "to": 0 }, { "name": "Egocentric 2", "field": "episodeCount", "from": 110, "to": 0 } ] } } } } }