When people talk about robot control, they talk about control laws: PID, ADRC, inverse dynamics, learned policies. In the field, what breaks first is usually something earlier and duller. The command did not go out on time.

A robot trading company in Japan came to us with a humanoid teleoperation setup built on UFACTORY xArm arms. To follow the operator's hand they push dense targets into each arm, and the arm stutters, chatters and jumps. They tried the box's other interfaces. They tried designing an inverse-dynamics feedforward, but swinging cables kept the model from matching. We had solved the same problem on the same arm a few weeks earlier, at a frontier lab in San Francisco that researches VLA models, without changing a single setting in the vendor box. What we changed was time.

"The box closes every position loop for us. So why does the arm shake when we push it faster?"

The short answer: with position streaming, the send timestamps are the trajectory. A late send becomes a jump. A constant delay becomes tracking error proportional to speed. A hard real-time host fixes the first by keeping every send within 5 µs, and that precision is what makes the second fixable: we measured the box's lag at 6.5 ms and sent every point that much early. Against the vendor box on its own, lateness went from 1.57 s to zero and tool-tip error at the same instant from 165 mm to 0.3 mm on a figure-8 (283 mm to 0.64 mm on a 3D weave), with a 1.57 kg bag swinging from the gripper and zero box errors in 30 runs. The path got about 140 times tighter. The error at any given instant got about 500 times smaller. The arm itself, its firmware and its controller were left exactly as shipped, and not one line of control code was written for it. Pushed to 1,000 mm/s with the load still swinging, the tool tip stayed within 1 mm RMS. Nothing in this stack is specific to this arm: it sits in front of any robot that takes streamed positions over TCP, and it is the same layer that drives EtherCAT servos at 1 kHz with 1.2 µs of jitter.

Most of this post is about the real-time layer itself: why timing becomes motion, what we configure, how we verify it, and how we spend the determinism we get. The field case is the evidence. Every figure is drawn from raw field logs.

Why timing becomes motion

There are three ways a timing problem turns into a motion problem. All three were visible on the arm.

1. A late cycle is a jump

When you stream positions to the vendor box, it receives "where to go" every 4 ms and never decides "how to go". It has no interpolator. It chases the newest point at up to 180°/s. So the sequence of targets, and the moments they arrive, is the velocity profile. If the sender stalls, the arm stops. When the sender resumes, the next target is far away and the arm lurches toward it. This is why cyclic position commands without real-time guarantees make a robot jump.

Blue: the J4 command stream actually sent at 125 °/s on the isolated core. Red: the same stream with a single 28 ms stall inserted (27,808 µs is the worst wake-up we measured once on a non-isolated run that day). During the stall the box is asked for 0 °/s. Right after, it is asked for 999 °/s in one cycle, 5.5 times its 180 °/s chase limit. The arm stops, then lurches. That is chatter.

2. A constant delay is error proportional to speed

Even with perfect sends, the arm arrives late. A device that is a pure delay L leaves a tracking error of L × speed. With the box measured below at L = 6.5 ms, a joint turning at 125 °/s is always 0.81° behind where it was told to be. That error can be removed, but only by sending early by exactly L, and "exactly" requires a send clock that does not wander.

3. Stale feedback cannot close a loop

The box reports state on its own schedule, about every 10 ms (99.6 Hz), so the measurement our loop sees is about 5 ms old on average and 11 ms in the worst case (feedback age p50 4,987 µs, p99 10,399 µs). Raising feedback gain on information that old corrects the past and shakes the arm. That is also why a model-based controller (inverse dynamics, payload models) cannot rescue a host with poor timing: every one of those terms has to arrive on time to mean anything.

The principle

Before tuning any control law, make time deterministic. Then measure the delays that remain, and spend that determinism on feedforward instead of on feedback.

The real-time stack, layer by layer

This is the configuration we run on Jetson AGX Orin. "Field" means it was set and measured on the arm in this post. "Blog" points to an earlier post where we measured that layer in isolation.

Layer	Setting	Why	Measured effect
Kernel	PREEMPT_RT (6.8.12-rt-tegra in the field; 5.15.148-rt-tegra in earlier posts), /sys/kernel/realtime = 1	Makes almost all kernel code preemptible so a high-priority thread wakes on time	Combined load: non-RT 318 µs → RT 56 µs max (Blog)
CPU isolation	isolcpus=managed_irq,domain,11 irqaffinity=0-10 kthread_cpus=0-10	Keeps the scheduler, managed IRQs and kernel threads off the control core	Isolated CPU11 max 5 µs vs CPU5 13 µs (Field). Under GPU load 159 → 15 µs, EtherCAT 113 → 6 µs, storage 145 → 7 µs (Blog)
Tick and RCU	nohz_full=8-11 rcu_nocbs=8-11 rcu_nocb_poll (needs a custom kernel; the OTA kernel ignores them)	Removes the scheduler tick and RCU callbacks from isolated cores	56 → 24 µs under stress (Blog)
Scheduling	Control thread pinned to the isolated core, SCHED_FIFO 80; rtprio 90, memlock unlimited; sched_rt_runtime_us=-1	Above default IRQ threads (50); no RT throttling	Wake-up lateness p99 0 µs in 40,953 production cycles (Field)
Memory	mlockall() before affinity and FIFO; prefault the stack; no allocation in the RT path	No page faults inside the loop	Memory-lock limit lifted in the field; measured in Blog
Power and clocks	nvpmodel MAXN, jetson_clocks, CPU governor performance (2.2016 GHz), cpuidle off on the control core	No frequency or idle-state exit latency	Part of the 5 µs baseline (Field)
Time base	CLOCK_MONOTONIC everywhere; nvpps ptp_tsc_sync_dis in the device tree; NTP kept clean	The Jetson default locks the host counter to an Ethernet PHC and changes its rate by ±140 ppm about every 110 s	EtherCAT DC deviation 328 → 3.5 µs (Blog)
IPC	Lock-free SPSC queues between RT and non-RT, fixed-size messages, release/acquire ordering, 64-byte aligned indices	A mutex hands priority to whoever holds it	Mutex 78 to 103 µs → lock-free 0.74 to 0.82 µs (Blog)
Monitoring	The RT thread only pushes samples into a queue; a non-RT thread publishes to ROS 2	Observability without touching the deadline	Push mean 0.3 µs, p99 0.7 µs (Blog)
Network	Dedicated wired NIC to the device, static IP, no gateway; no second client on the same box	Shared links and competing clients add queueing you cannot schedule	Round trip mean 721 µs, p99 841 µs (Field)

How we verify it

A configuration is a hypothesis until it is measured on the machine, under the load it will actually see. We check four things, in this order.

Baseline: cyclictest, round trip, report rate

(a) cyclictest, 60 s at 1 kHz: isolated CPU11 max 5 µs, non-isolated CPU5 max 13 µs. (b) Round trip to the box, 2,000 calls paced at 250 Hz: mean 721 µs, max 916 µs. (c) The box's own state reports: every 10.04 ms (99.6 Hz), split between about 9.67 and 10.75 ms, while we command every 4 ms.

Measure alone, then measure under load

The first cyclictest that day ran at the same time as two other measurements and hit a single 27,808 µs wake-up: seven 250 Hz cycles lost at once, the stall simulated in the first figure. The standalone re-run gave 5 µs. A baseline is only valid with every other job stopped. A production check is only valid with every real job running.

In production: the loop's own record

The control loop logs its own execution time, wake-up lateness and a deadline-miss flag every cycle. Across 36 runs and 381,255 cycles there were 2 deadline misses, each a single stall where one cycle took about 10.5 ms. Every other cycle finished within 1,951 µs of its 4,000 µs budget.

Four production runs, 40,953 cycles. (a) Execution per cycle, max 1,951 µs; the peak near 900 µs is the SDK round trip. (b) Wake-up lateness: p99 0 µs, max 46 µs.

The plant: how late is the device?

Only once the host is deterministic can you measure the device behind it. With feedforward and feedback off, we swept each joint with sines (0.2 to 4 Hz and 6 to 10 Hz, 14 sweeps) and fitted H(jω) = e^(−jωL) / (1 + jωτ).

Phase falls linearly with frequency. Equivalent delay sits at 6.1 to 7.0 ms at every frequency, and gain is flat to 10 Hz. L+τ over 14 fits: 6.17 to 6.96 ms, median 6.54 ms. The box is a pure delay.

The full budget, command to motor

Stage	Time	Status
Teleop or planner input (Cartesian target)	Device and camera delay, typically tens of ms	Out of scope here
TF, IK (7-DoF), joint trajectory and spline at 250 Hz	Inside the 4 ms cycle	Stage times not broken out
Real-time send loop	wake-up ≤ 5 µs, execution ≤ 1,951 µs	Measured
Network to the box (TCP)	round trip 0.72 ms mean, 0.84 ms p99	Measured
Box servo → joint → report	command computed → observed: 6.54 ms	Measured (internals not broken down)
After the lead	−0.04 / +0.09 ms behind the plan	Measured

Of the 6.54 ms, the network is 0.72 ms (11%) and host jitter is 0.08%. The rest belongs to the box, including about 2 ms of zero-order hold at 250 Hz.

Spending determinism: timing as a control lever

A host that keeps time to 5 µs can do something a jittery host cannot: send the future. Knowing the device is 6.5 ms late, we send the angle the arm should reach 6.5 ms from now.

u = q_ref + k_ff · q̇_ref · t_lead + k_p · e + k_i · ∫e

# servo gains, all joints
kff: 1.0
t_lead_s: 0.0065
kp: 0.0     # feedback is 5 to 11 ms stale: don't
ki: 0.0

If the send time wandered by milliseconds, the lead would be wrong by the same amount every cycle, and the error would come back as noise. At 5 µs the lead is accurate to 0.08% of itself.

3D weave, ±40 ms around J4's fastest moment (125 °/s). The command (orange) runs 6.5 ms, about 0.81°, ahead of the reference, and the measured points land on the reference line. There is one measured point every 10 ms because of the box's report rate.

The same determinism makes shaping possible. Every generator is acceleration-continuous (S-curve, jerk-limited PTP), the 250 Hz interpolator supplies q̇_ref, and a ZVD input shaper tuned to the 1.03 Hz swinging payload sits on the stop ramp. Streamed teleop targets get an online safety shaper in the driver. None of these work if the 4 ms grid itself is uneven.

What it bought: the xArm field case

Setup: UFACTORY xArm 7 (XS13, firmware v2.6.0) over TCP (4 ms command stream, state reports every 10 ms), Jetson AGX Orin with the stack above, UFACTORY F/T sensor and Gripper G2, a 1.573 kg bag hanging from the gripper, four days on site at a frontier lab in San Francisco that researches VLA models, October 2026. The vendor box was left exactly as shipped.

Vendor box vs. the real-time layer

We gave the vendor box every chance first. Its built-in planner with dense waypoints and blending ran 1.57 s behind the plan and wobbled 1.1 to 1.6 mm off the path. With a stop at every turn it stayed closer but took 2.26 times as long. Its Cartesian velocity interface slew-limited the tool to about 70 mm/s² and let joints drift up to 72°. The same box, fed by the WIM real-time layer, stayed on the path and on time.

Figure-8, 165 mm/s. Tool-tip paths from the logged joint angles through xArm 7 forward kinematics, distance from plan drawn ×10. The vendor box wobbles around the plan (1.11 mm RMS, 3.47 mm max): this is the customer's chatter. With the real-time layer the tool tip stays on the line (0.008 mm RMS).

3D weave, 590 mm/s, same method. The vendor box zigzags at the turns (1.56 mm RMS, 4.82 mm max). The real-time layer stays on the path (0.041 mm RMS; the kinks at the turnarounds are ≤ 0.79 mm before magnification).

The same story on a time axis: the vendor box draws the figure-8 about 1.57 s behind the plan. The real-time layer is on it.

The numbers

One run each, same arm, same bag, same box. (a) Behind the plan: 1,575 / 1,080 ms for the vendor box, zero for the real-time layer (within the 0.5 ms measurement grid). (b) Tool-tip error at the same instant, from forward kinematics: 165 → 0.305 mm on the figure-8 and 283 → 0.645 mm on the weave. (c) Distance from the path regardless of time: 1.11 → 0.008 mm and 1.56 → 0.041 mm.

Across all 15 real-time-layer runs per path, tool-tip error RMS averaged 0.267 mm (figure-8) and 0.576 mm (weave), lateness −0.04 ms and +0.09 ms, and bag swing after the stop 0.037 and 0.062 N·m.

Speed ladder with the same bag: tool-tip error RMS 0.50 mm at 646.5 mm/s, 0.55 at 775, 0.66 at 900 and 0.86 mm at 1,000 mm/s (max 3.80 mm), zero box errors. One run per speed.

With the bag attached, all 30 real-time-layer runs finished with zero box errors and a wrist F/T peak of 2.99 N·m (4 N·m rated).

TCP box vs. direct EtherCAT

The xArm is the hard case: the box only opens over TCP, so 6.5 ms of device delay stays, and the best a host can do is cancel it with a lead. When our controller drives servos directly over EtherCAT, the same real-time layer takes the whole loop down to microseconds: on the same Orin, 1 kHz loop jitter at or below 1.2 µs with zero missed cycles, and drive-to-drive DC deviation at or below 3.5 µs once the nvpps lock is off (nvpps post). The architecture changes. The order of work does not. Make time deterministic, measure what is left, then control.

Checklist for your own host

# 1) Kernel and isolation
uname -v | grep PREEMPT_RT
cat /sys/kernel/realtime            # 1
cat /proc/cmdline | tr ' ' '\n' | grep -E 'isolcpus|irqaffinity|kthread_cpus|nohz_full|rcu_nocbs'
cat /proc/sys/kernel/sched_rt_runtime_us   # -1

# 2) Baseline on the control core, everything else stopped
sudo cyclictest -m -p 90 -i 1000 -a 11 -t 1 -D 60 -q -h 2000

# 3) Same test with the real load running (GPU, camera, logging, ROS 2)

# 4) Jetson time base: is the host counter locked to an Ethernet PHC?
dmesg | grep "TSC config ptx"
ls /sys/firmware/devicetree/base/bus@0/nvpps/

# 5) Device: round trip, report rate, then per-joint sine sweeps -> delay -> lead

• Log execution time, wake-up lateness and a deadline-miss flag in the loop itself, every cycle, through a lock-free queue.
• Never let a second client talk to the same device during a test.
• Declare the payload before streaming starts. On the xArm, changing payload or TCP offset puts the box in a hold state until it is re-enabled.

Summary

• With position streaming, send timing is the trajectory: one 28 ms stall asks the arm for 999 °/s, 5.5 times what it can chase.
• PREEMPT_RT, core isolation, FIFO scheduling, locked memory, fixed clocks, a clean time base and lock-free IPC kept the send loop within 5 µs. In production, 2 deadline misses in 381,255 cycles.
• Determinism made the device measurable (a pure 6.5 ms delay) and that delay cancellable by sending early.
• On an xArm whose vendor box we never touched, that alone took the arm from 1.57 s behind the plan to on time, tool-tip error at the same instant from 165 mm to 0.3 mm, and path error from 1.11 mm RMS to 0.008 mm, with zero box errors in 30 runs under a swinging 1.57 kg load.
• A humanoid teleoperation setup that chattered at every push now follows the operator's hand at up to 1 m/s within 1 mm, and the only thing that changed is the computer that keeps time.

References

• WIM Tech, Is PREEMPT_RT Enough? Validating Real-Time Performance on Jetson Orin
• WIM Tech, Why Our EtherCAT Sync Fell Apart Every 110 Seconds
• WIM Tech, Lock-free vs Mutex: Robot Control System IPC Performance Benchmark
• WIM Tech, What is the Real Cost of std::memory_order on ARM64?
• WIM Tech, 1 kHz Real-Time Robot Control System Monitoring Architecture
• UFACTORY xArm Developer Manual and xArm-Python-SDK (position streaming, state report)
• N. C. Singer, W. P. Seering, "Preshaping Command Inputs to Reduce System Vibration," ASME J. Dyn. Sys., Meas., Control, 1990