We recently fixed the timing of the arm and torso commands on a wheeled dual-arm humanoid, on site with a robotics startup in San Francisco. The robot's two 7-axis arms and 4-axis torso share one CAN bus. The startup's AI application publishes joint targets on ROS 2 topics, and the control stack that ships with the robot turns them into commands for the motor controllers. The arms and the torso expect a command every 4 ms (250 Hz).
The first thing we checked on site was this:
"Are the commands really going out every 4 ms? If not, how often do they slip?"
Short answer: with the stock stack, no. Measured at the actual transmit time on the CAN bus, the p99 deviation of the command interval from 4 ms (jitter) was 4.0–21.9 ms: at least 1 command in 100 was off by more than a full period (4 ms). Every 2-minute case also had 415–3,653 intervals longer than two periods. Applying the same CPU isolation and real-time priority as ours to the stock stack did not bring the p99 of its worst stream below 5.6 ms. Replacing only the arm and torso command path with the WIM real-time stack on the Jetson AGX Orin gave a p99 of 0.13–0.41 ms under the same conditions, a steady 250 Hz, and 0–1 gaps. Over a 30-minute comparison run the gaps were 18,422 versus 0. The stock stack kept the grippers, the chassis and the sensors, and the application did not change a line.
This post follows Time is a control input. That post covered the real-time layer itself. Here we put that layer on a humanoid with a shared bus and measured the difference fairly, with the same conditions applied to the stock stack.
What we changed
In the stock stack an arm command passes through two processes. A higher-level control process takes the target and interpolates it, hands it over a messaging middleware to a bus driver process, and the driver writes the CAN frame. We replaced only the arm and torso part of that path with a single real-time node.
Red: the path that is replaced. Green: the WIM real-time node. Grey: unchanged. Nothing changes on the application side.
The WIM real-time node follows the same principles as the previous post.
| Layer | Setting |
|---|---|
| Kernel | PREEMPT_RT |
| CPU | One of eight cores isolated for the control loop and the CAN interrupt only. Everything else runs on the other seven |
| Scheduling | Control loop at SCHED_FIFO 70, 4 ms period with absolute-time wakeups. The CAN interrupt thread stays at the kernel default (FIFO 50) |
| Memory | mlockall() and all buffers at start. No allocation, exceptions, blocking I/O, logging or mutexes inside the loop |
| ROS 2 | A thread outside the loop receives the topics. It exchanges data with the loop only through a Seqlock and an SPSC (single-producer, single-consumer) queue |
| Shared bus | Other devices on the same bus (grippers, chassis, sensors) are left alone. Before taking over the arm and torso transmit, the node checks on the bus that the previous sender has stopped |
How we measured
- Measurement point: the transmit timestamps the kernel records on the robot's CAN interface. Same point and same method for both stacks.
- Streams: left arm, right arm and torso commands. Each case reports the worst of the three.
- Jitter: how far the interval between two consecutive commands deviates from 4 ms, . We report p99 and max.
- Gap: an interval longer than 8.24 ms. When one command is missing, the interval becomes 8 ms; adding a 0.24 ms measurement margin, this counts the cases where at least one whole command was skipped. The threshold is our own definition.
- Motion: the application sends the same joint trajectory (a large motion of both arms and the torso). Each case runs 120 s. Jitter is computed after the first 30 s; gaps are counted over the whole time targets are arriving.
- Conditions: 2 stacks × CPU isolation × priority × 2 application settings (default and a lighter tuned setting), 10 cases in total. The application's target-forwarding threads ran at the same real-time priority (FIFO 20) in every case.
For a fair comparison we applied the same CPU isolation to the stock stack (cases 3–6): its control process, its bus driver process and the CAN interrupt all moved to the isolated core. In cases 5–6 every thread of its control process was also raised to SCHED_FIFO 70, the same as our loop.
Result 1: two seconds are enough to see it

Left-arm command intervals over two seconds under the same condition (CPU isolated, default application setting). One dot is one command. Even with isolation, the stock stack swings between 2 and 6 ms and has several 9–28 ms gaps. The WIM stack stays on the 4 ms line for the same two seconds.
Result 2: all 10 conditions

(a) Bars are jitter p99, ▼ is the max (log axis). The dotted line is 4 ms, one whole period. (b) Intervals longer than 8.24 ms per case. The left six are the stock stack, the right four the WIM stack.
| Metric (worst of three streams) | Stock stack, cases 1–6 | WIM stack, cases 7–10 |
|---|---|---|
| Jitter p99 | 5.6–21.9 ms | 0.34–0.41 ms |
| Jitter max | 47.8–111.4 ms | 0.6–22.0 ms (0.6 ms with isolation) |
| Actual command rate | 160–245 Hz | 250 Hz |
| Gaps over 8.24 ms (per case) | 415–3,653 | 0–1 |
| Missed ticks, send errors (node records) | n/a | 0 |
Across all streams the stock stack's p99 is 4.0–21.9 ms and the WIM stack's 0.13–0.41 ms. Comparing matched conditions (1 with 7, 2 with 8, 3 with 9, 4 with 10), the WIM stack's p99 is 17–62 times lower.
The measures applied to the stock stack did help. CPU isolation alone cut the gaps from 3,653 to 415 (default application setting). Even with raised priority, though, the p99 of its worst stream stayed at 5.6 ms and the torso commands never exceeded 197 Hz. The remaining delay did not shrink with more CPU. We look at why after the 30-minute results.
Result 3: 30-minute comparison
To complement the 2-minute cases, we also ran each stack once for 30 minutes with the same repeating trajectory (different from the 10 cases: a sine of ±0.10 rad on the arms and ±0.03 rad on the torso at 0.1–0.5 Hz). In this comparison the stock stack ran as delivered (no isolation) and the WIM stack in its operating configuration, isolation included.

| Metric | Stock stack (as delivered) | WIM stack |
|---|---|---|
| Jitter p99 (left arm / right arm / torso) | 6.7 / 6.9 / 7.7 ms | 0.34 / 0.66 / 0.66 ms |
| Jitter max | 50.2 / 50.2 / 55.5 ms | 0.61 / 1.05 / 1.08 ms |
| Actual command rate | 241 / 240 / 195 Hz | 250 / 250 / 250 Hz |
| Gaps over 8.24 ms | 18,422 (one every 0.096 s on average) | 0 (over 1,692 s) |
| Missed ticks (node records) | n/a | 0 of 442,497 ticks |
Why the stock stack could not hold 4 ms
In the stock stack a command passes three stages before it reaches the bus. The control process builds it periodically, the messaging middleware delivers it, and the bus driver process receives it and writes it to CAN. Each stage is a separate process or thread that wakes up on its own, so the wait at each stage varies from command to command. Even if the command is built on time, the time it reaches the bus wobbles by the sum of the three delays.

Schematic, not measured. Top: the stock stack. When a command is delayed at any stage, the interval to the previous command grows, and past 8.24 ms it counts as a gap. The late command can also leave almost together with the next one (bunching). Bottom: the WIM stack builds and sends each command in the same 4 ms loop, with no waiting stage in between.
The traces of this structure show up directly in the interval distribution measured on the bus.

Intervals of the commands that actually went out on the bus, under the same condition (CPU isolated, default application setting, cases 3 and 9). The dotted line is 4 ms; the red dashed line is the 8.24 ms gap threshold.
- Left arm: only 46% of the stock stack's commands fall within 4 ± 0.5 ms. 16% of the intervals are shorter than 3 ms and 15% longer than 5 ms, which matches commands bunching up behind a late one.
- Torso: 19% fall within 4 ± 0.5 ms, and 47% are longer than 5 ms. Most torso commands leave later than 4 ms, so the actual rate was only 196 Hz.
- WIM stack: 99.8% or more of both streams fall within 4 ± 0.5 ms.
CPU isolation and priority can make each stage faster, but they do not remove the number of stages or the waits between them. We believe this is why the stock stack's p99 did not go below 5.6 ms (inferred).
What we learned
More priority and more CPU were not enough. In these measurements, only the configuration that builds the command and writes it to the bus inside the same real-time loop held 4 ms.
The WIM stack needs isolation too. One WIM case without isolation (case 8) had a single 22 ms interval. The node recorded no missed tick at that moment, so the delay happened after the loop, on the way to the bus. We did not identify the cause. The two isolated cases had a max of 0.6 ms, and the operating configuration includes isolation.
Limitations:
- One robot, measured on one day. The 10 cases are 2 minutes each, and the 30-minute comparison was run once per stack.
- The 30-minute comparison ran the stock stack as delivered. The comparison with equal isolation is in the 2-minute cases.
- In cases 5–6 only the control process was raised; the bus driver process kept its priority. We did not measure the stock stack with both raised.
- The 8.24 ms gap threshold is our own definition.
- In the cases with the tuned application setting, the number of targets that arrived in 2 minutes differed somewhat between the stacks (about 29,000–33,000). This does not affect the interval measurement, but the inputs were not identical.
Summary
On the same robot, with the same motion and the same CPU isolation, the p99 jitter of the arm and torso commands (worst stream) was 5.6 ms or more with the stock stack and 0.41 ms or less with the WIM real-time stack. The only thing we changed was the path the arm and torso commands take to the bus. The application and the other devices on the bus were left as they were. If you have the same question about your robot, start by measuring when the commands actually leave on the bus. The period measured inside the software and the period seen on the bus can differ.