Skip to main content
Embedded/Real-time

Sending a humanoid's arm commands exactly every 4 ms: a stock control stack vs a real-time stack under 10 conditions

We moved the arm and torso commands of a wheeled dual-arm humanoid from its stock control stack to the WIM real-time stack on a Jetson AGX Orin, which writes them to CAN directly. Under 10 conditions with the same motion, the stock stack showed a p99 jitter (deviation of the command interval from 4 ms) of 4.0–21.9 ms even with the same CPU isolation and real-time priority applied, while the WIM stack held 0.13–0.41 ms. In a 30-minute comparison run, intervals longer than two periods occurred 18,422 times versus 0.

WRWIM Robotics Team
·
real-timepreempt-rtjetsoncan-bushumanoidbenchmark

We recently fixed the timing of the arm and torso commands on a wheeled dual-arm humanoid, on site with a robotics startup in San Francisco. The robot's two 7-axis arms and 4-axis torso share one CAN bus. The startup's AI application publishes joint targets on ROS 2 topics, and the control stack that ships with the robot turns them into commands for the motor controllers. The arms and the torso expect a command every 4 ms (250 Hz).

The first thing we checked on site was this:

"Are the commands really going out every 4 ms? If not, how often do they slip?"

Short answer: with the stock stack, no. Measured at the actual transmit time on the CAN bus, the p99 deviation of the command interval from 4 ms (jitter) was 4.0–21.9 ms: at least 1 command in 100 was off by more than a full period (4 ms). Every 2-minute case also had 415–3,653 intervals longer than two periods. Applying the same CPU isolation and real-time priority as ours to the stock stack did not bring the p99 of its worst stream below 5.6 ms. Replacing only the arm and torso command path with the WIM real-time stack on the Jetson AGX Orin gave a p99 of 0.13–0.41 ms under the same conditions, a steady 250 Hz, and 0–1 gaps. Over a 30-minute comparison run the gaps were 18,422 versus 0. The stock stack kept the grippers, the chassis and the sensors, and the application did not change a line.

This post follows Time is a control input. That post covered the real-time layer itself. Here we put that layer on a humanoid with a shared bus and measured the difference fairly, with the same conditions applied to the stock stack.

What we changed​

In the stock stack an arm command passes through two processes. A higher-level control process takes the target and interpolates it, hands it over a messaging middleware to a bus driver process, and the driver writes the CAN frame. We replaced only the arm and torso part of that path with a single real-time node.

Red: the path that is replaced. Green: the WIM real-time node. Grey: unchanged. Nothing changes on the application side.

The WIM real-time node follows the same principles as the previous post.

LayerSetting
KernelPREEMPT_RT
CPUOne of eight cores isolated for the control loop and the CAN interrupt only. Everything else runs on the other seven
SchedulingControl loop at SCHED_FIFO 70, 4 ms period with absolute-time wakeups. The CAN interrupt thread stays at the kernel default (FIFO 50)
Memorymlockall() and all buffers at start. No allocation, exceptions, blocking I/O, logging or mutexes inside the loop
ROS 2A thread outside the loop receives the topics. It exchanges data with the loop only through a Seqlock and an SPSC (single-producer, single-consumer) queue
Shared busOther devices on the same bus (grippers, chassis, sensors) are left alone. Before taking over the arm and torso transmit, the node checks on the bus that the previous sender has stopped

How we measured​

  • Measurement point: the transmit timestamps the kernel records on the robot's CAN interface. Same point and same method for both stacks.
  • Streams: left arm, right arm and torso commands. Each case reports the worst of the three.
  • Jitter: how far the interval between two consecutive commands deviates from 4 ms, ∣J∣=∣Δt−4 ms∣|J| = |\Delta t - 4\,\text{ms}|. We report p99 and max.
  • Gap: an interval longer than 8.24 ms. When one command is missing, the interval becomes 8 ms; adding a 0.24 ms measurement margin, this counts the cases where at least one whole command was skipped. The threshold is our own definition.
  • Motion: the application sends the same joint trajectory (a large motion of both arms and the torso). Each case runs 120 s. Jitter is computed after the first 30 s; gaps are counted over the whole time targets are arriving.
  • Conditions: 2 stacks × CPU isolation × priority × 2 application settings (default and a lighter tuned setting), 10 cases in total. The application's target-forwarding threads ran at the same real-time priority (FIFO 20) in every case.

For a fair comparison we applied the same CPU isolation to the stock stack (cases 3–6): its control process, its bus driver process and the CAN interrupt all moved to the isolated core. In cases 5–6 every thread of its control process was also raised to SCHED_FIFO 70, the same as our loop.

Result 1: two seconds are enough to see it​

Two seconds of left-arm command intervals under the same condition. The stock stack scatters around 4 ms with 9–28 ms gaps; the WIM stack stays on the 4 ms line

Left-arm command intervals over two seconds under the same condition (CPU isolated, default application setting). One dot is one command. Even with isolation, the stock stack swings between 2 and 6 ms and has several 9–28 ms gaps. The WIM stack stays on the 4 ms line for the same two seconds.

Result 2: all 10 conditions​

Jitter p99 and max, and gap counts for 10 cases. The six stock-stack cases show p99 5.6–21.9 ms and 415–3,653 gaps; the four WIM cases show p99 0.34–0.41 ms and 0–1 gaps

(a) Bars are jitter p99, ▼ is the max (log axis). The dotted line is 4 ms, one whole period. (b) Intervals longer than 8.24 ms per case. The left six are the stock stack, the right four the WIM stack.

Metric (worst of three streams)Stock stack, cases 1–6WIM stack, cases 7–10
Jitter p995.6–21.9 ms0.34–0.41 ms
Jitter max47.8–111.4 ms0.6–22.0 ms (0.6 ms with isolation)
Actual command rate160–245 Hz250 Hz
Gaps over 8.24 ms (per case)415–3,6530–1
Missed ticks, send errors (node records)n/a0

Across all streams the stock stack's p99 is 4.0–21.9 ms and the WIM stack's 0.13–0.41 ms. Comparing matched conditions (1 with 7, 2 with 8, 3 with 9, 4 with 10), the WIM stack's p99 is 17–62 times lower.

The measures applied to the stock stack did help. CPU isolation alone cut the gaps from 3,653 to 415 (default application setting). Even with raised priority, though, the p99 of its worst stream stayed at 5.6 ms and the torso commands never exceeded 197 Hz. The remaining delay did not shrink with more CPU. We look at why after the 30-minute results.

Result 3: 30-minute comparison​

To complement the 2-minute cases, we also ran each stack once for 30 minutes with the same repeating trajectory (different from the 10 cases: a sine of ±0.10 rad on the arms and ±0.03 rad on the torso at 0.1–0.5 Hz). In this comparison the stock stack ran as delivered (no isolation) and the WIM stack in its operating configuration, isolation included.

30-minute comparison run. Stock stack p99 6.7–7.7 ms, max 50–56 ms, torso at 195 Hz. WIM stack p99 0.34–0.66 ms, max 0.6–1.1 ms, all three streams at 250 Hz

MetricStock stack (as delivered)WIM stack
Jitter p99 (left arm / right arm / torso)6.7 / 6.9 / 7.7 ms0.34 / 0.66 / 0.66 ms
Jitter max50.2 / 50.2 / 55.5 ms0.61 / 1.05 / 1.08 ms
Actual command rate241 / 240 / 195 Hz250 / 250 / 250 Hz
Gaps over 8.24 ms18,422 (one every 0.096 s on average)0 (over 1,692 s)
Missed ticks (node records)n/a0 of 442,497 ticks

Why the stock stack could not hold 4 ms​

In the stock stack a command passes three stages before it reaches the bus. The control process builds it periodically, the messaging middleware delivers it, and the bus driver process receives it and writes it to CAN. Each stage is a separate process or thread that wakes up on its own, so the wait at each stage varies from command to command. Even if the command is built on time, the time it reaches the bus wobbles by the sum of the three delays.

Schematic. In the stock stack each command passes the control process, the messaging middleware and the bus driver process, and the varying delay at each stage opens gaps or bunches two commands on the bus. The WIM stack builds and sends each command in one loop, so the spacing stays even

Schematic, not measured. Top: the stock stack. When a command is delayed at any stage, the interval to the previous command grows, and past 8.24 ms it counts as a gap. The late command can also leave almost together with the next one (bunching). Bottom: the WIM stack builds and sends each command in the same 4 ms loop, with no waiting stage in between.

The traces of this structure show up directly in the interval distribution measured on the bus.

Distribution of command intervals on the bus under the same condition. Only 46% of the stock stack's left-arm commands and 19% of its torso commands fall within 4 ± 0.5 ms; for the WIM stack almost 100% of both streams sit at 4 ms

Intervals of the commands that actually went out on the bus, under the same condition (CPU isolated, default application setting, cases 3 and 9). The dotted line is 4 ms; the red dashed line is the 8.24 ms gap threshold.

  • Left arm: only 46% of the stock stack's commands fall within 4 ± 0.5 ms. 16% of the intervals are shorter than 3 ms and 15% longer than 5 ms, which matches commands bunching up behind a late one.
  • Torso: 19% fall within 4 ± 0.5 ms, and 47% are longer than 5 ms. Most torso commands leave later than 4 ms, so the actual rate was only 196 Hz.
  • WIM stack: 99.8% or more of both streams fall within 4 ± 0.5 ms.

CPU isolation and priority can make each stage faster, but they do not remove the number of stages or the waits between them. We believe this is why the stock stack's p99 did not go below 5.6 ms (inferred).

What we learned​

More priority and more CPU were not enough. In these measurements, only the configuration that builds the command and writes it to the bus inside the same real-time loop held 4 ms.

The WIM stack needs isolation too. One WIM case without isolation (case 8) had a single 22 ms interval. The node recorded no missed tick at that moment, so the delay happened after the loop, on the way to the bus. We did not identify the cause. The two isolated cases had a max of 0.6 ms, and the operating configuration includes isolation.

Limitations:

  • One robot, measured on one day. The 10 cases are 2 minutes each, and the 30-minute comparison was run once per stack.
  • The 30-minute comparison ran the stock stack as delivered. The comparison with equal isolation is in the 2-minute cases.
  • In cases 5–6 only the control process was raised; the bus driver process kept its priority. We did not measure the stock stack with both raised.
  • The 8.24 ms gap threshold is our own definition.
  • In the cases with the tuned application setting, the number of targets that arrived in 2 minutes differed somewhat between the stacks (about 29,000–33,000). This does not affect the interval measurement, but the inputs were not identical.

Summary​

On the same robot, with the same motion and the same CPU isolation, the p99 jitter of the arm and torso commands (worst stream) was 5.6 ms or more with the stock stack and 0.41 ms or less with the WIM real-time stack. The only thing we changed was the path the arm and torso commands take to the bus. The application and the other devices on the bus were left as they were. If you have the same question about your robot, start by measuring when the commands actually leave on the bus. The period measured inside the software and the period seen on the bus can differ.