We were running an IgH EtherCAT master on a single Jetson AGX Orin, driving two servo drives at 1 kHz. The test rig couples the two axes mechanically, so if the two drives do not execute the same command at the same instant, the axes push against each other. During a high-speed run at 23.5 rad/s, that is exactly what happened. Torque spiked.

The logs showed windows of a few seconds where the clock synchronization between drives, the EtherCAT DC (Distributed Clocks) deviation, jumped to tens and then hundreds of microseconds. Real-time loop jitter stayed at or below 1.2 µs, and not a single cycle was missed. The drive clocks drifted apart anyway, roughly every 110 seconds.

"The control loop is fine. Why do the drive clocks slip every two minutes?"

The short answer: it was not the motors, not the kernel, and not our control code. The PTP-TSC hardware lock that NVIDIA's nvpps driver enables at boot was changing the rate of the Jetson host clock by up to ±140 ppm roughly every 110 seconds. Disabling that lock with a single device tree property, ptp_tsc_sync_dis, cut the maximum DC deviation from 328 µs to 3.5 µs, and the share of samples above 10 µs from 7.4% to 0% (300 s with the lock on, 20 minutes with it off). In the high-speed run where the problem first appeared, the maximum went from 326 µs to 2.7 to 5.2 µs. Not a single line of control code changed.

This post walks through how we tracked it down. We took three wrong turns along the way, and one of them made the problem worse. At the end, we show how to check whether the same setting is active on your Jetson.

Test setup

Item	Details
Host	Jetson AGX Orin, JetPack 6.2.1 (Jetson Linux R36.4.4), kernel 5.15.148-rt-tegra (PREEMPT_RT)
EtherCAT master	IgH EtherCAT Master 1.6.13, dedicated Intel I210 NIC (ec_igb native driver)
Slaves	Two Panasonic MINAS A6B servo drives on an in-house test rig with the two axes mechanically coupled
Cycle	1 ms (1 kHz), DC synchronization enabled
Clocks recorded	Reference drive ESC clock, DC deviation of drive 2 (ESC register 0x092C, System Time Difference), hardware clock (PHC) of the EtherCAT NIC (I210), PHC of the on-board Ethernet eno1, host CLOCK_MONOTONIC and CLOCK_MONOTONIC_RAW

Background: what DC aligns, and where the host clock comes in

Every EtherCAT slave controller (ESC) has its own hardware clock. DC keeps these clocks aligned to the first DC slave, the reference clock. Each drive applies its command at a SYNC0 event defined on that shared timebase. That is why two drives move at the same instant even if the master's frame arrives a little late.

The remaining question is how the reference clock and the master (host) clock are kept in step. The industry uses two directions.

Approach	What it does	IgH API	Typical example
Align the bus to the host	Every cycle, pass the host time and send a frame that steers the reference clock toward it	ecrt_master_application_time() + ecrt_master_sync_reference_clock()	acontis EC-Master default (Bus Shift)
Align the host to the bus	The host reads the reference clock time and adjusts its own cycle	ecrt_master_reference_clock_time()	Beckhoff TwinCAT approach (the master also syncs to the reference clock)

PLEM, our control software, uses the first approach. That approach carries a premise: the host timer frequency stays constant. The acontis documentation states it plainly: "The timer frequency must never change after the application start." If the host clock wobbles, the wobble is carried onto the bus every cycle.

How much wobble matters? Multiply the rate error (ppm) by time to get the accumulated offset.

$$\Delta t = \frac{\Delta f}{f} \times T = 140 \times 10^{-6} \times 1\,\text{s} = 140\,\mu\text{s}$$

A host clock running 140 ppm fast or slow drifts 140 µs per second. Per 1 ms cycle, that is 0.14 µs. If a rate change like that lasts even a few seconds, it is large enough to produce deviations of hundreds of microseconds.

What remained after the first fix

We had already found one direct cause of the torque spikes. The SYNC0 shift (the time margin between the start of the cycle and the SYNC0 event, which buys time for the frame to reach the drives) was set to 0. We raised it to 300 µs.

The windows where DC deviation jumped to tens or hundreds of microseconds remained. When one hit, it briefly consumed almost the entire 300 µs margin, and in one high-speed run (a 326 µs transient) the torque spiked again inside that window. A bigger margin only covered the symptom. We still did not know why the clock was moving.

Wrong turn 1: the real-time loop and the drive settings

We suspected our own side first.

• Real-time loop: jitter at or below 1.2 µs, zero missed cycles. The loop was on time.
• Drive settings and motion: the same transients appeared with the motors stopped, and even with the drives never brought into the operational state while the bus was running. Motion and drive settings were not involved.

Wrong turn 2: NTP

Next suspect: NTP. This board synced to internet time servers over Wi-Fi, and its state was poor (offset −108 ms, jitter 38 ms). NTP adjusts the clock by changing its rate slightly (slewing), which made it a plausible suspect.

The verdict was "not the main cause."

• Transients still appeared with NTP fully disabled.
• The same rate changes showed up in CLOCK_MONOTONIC_RAW, a clock NTP cannot touch.

NTP only added small wobbles of its own. We also learned something else here. Linux CLOCK_REALTIME, CLOCK_MONOTONIC, and CLOCK_MONOTONIC_RAW all count the same hardware counter. If that counter itself wobbles, no choice of clock avoids it. (PLEM uses CLOCK_MONOTONIC throughout.)

Separating the amplifier from the trigger

We ran the same conditions while switching PLEM's DC approach for diagnosis. The motors were stopped in all of these runs.

Run	DC approach	Result
Default	Steer the reference clock to the host time every cycle	Transients of hundreds of µs around 103 s and 228 s
Diagnostic 1	Steering disabled	No transients over 300 s
Diagnostic 2	Host follows the reference clock	Stayed in sync for 333 s (deviation at or below 87 ns), absorbing four ±145 ppm host rate events in the meantime

The picture split into two layers. The steering approach was the amplifier, and the trigger was on the host clock side. Switching approaches would have avoided the symptom, but it meant reworking the control thread's cycle alignment and the alignment across multiple buses. We decided to find the trigger first.

The decisive experiment: two unrelated clocks see the same event

To tell whether the host clock is moving, you need a clock that has nothing to do with the host. Our rig had two, each running on its own oscillator (crystal):

• the ESC clock of the reference drive (the drive's oscillator)
• the hardware clock (PHC) of the Intel I210 NIC used for EtherCAT (/dev/ptp0, the NIC's oscillator)

Together with the PHC of the on-board Ethernet eno1 and the host clocks (CLOCK_MONOTONIC, CLOCK_MONOTONIC_RAW), we recorded them side by side for 630 seconds with NTP off. DC was set to measure phase only, neither steering nor following.

• Rate-change events appeared in "reference drive minus host" and "I210 minus host" at the same time and with the same magnitude: correlation 1.000 and slope 1.000 over 2-second windows.
• Comparing the two clocks with the host removed, the rate difference was constant at −25.91 ppm (standard deviation 0.60 ppm over 2-second windows).
• NTP was off, and MONOTONIC − RAW was smooth.

If two oscillators with no relationship to each other see the same event at the same magnitude, the one that moved is their common denominator: the host. And since RAW moved as well, the hardware counter the host counts was changing rate.

One more thing looked odd. When we compared the host counter with the PHC of eno1, an on-board Ethernet port with no cable plugged in, the two stayed locked near 0 ppm outside the events and slipped only during the events, at −141.8, −142.2, −145.8, and +135.4 ppm. In the first 350 seconds there were three large slips, accumulating about −46 ms of phase. The host clock was tied to an Ethernet port that had nothing to do with control.

With EtherCAT shut off entirely, comparing only the clocks for 10 minutes gave the same result. The two PHCs held a constant −26.9 ppm between them, while the host counter alone swung between 0 and ±140 ppm. This was a Jetson host phenomenon, independent of EtherCAT.

The host counter is locked to the free-running eno1 clock, and its wobble enters the reference clock every cycle, showing up as DC deviation on the drive side. The I210 PHC runs on an oscillator unrelated to the host, which makes it the reference for judging who moved.

Who tied the host clock to an Ethernet port: nvpps

The boot log had the clue. The nvpps module was loaded and left these lines:

primary-emac found ethernet@6800000
using ptp notifier method on emac ethernet@6800000
TSC config ptx 0x113

nvpps is NVIDIA's PPS (pulse per second) driver. NVIDIA's DRIVE OS documentation describes the PTP-TSC sync feature as a hardware assist that uses the Ethernet MAC's PPS to align the increment rate of the TSC (Time Stamp Counter) with that MAC's PTP hardware clock (PHC). Its purpose is to align camera fsync to PTP second boundaries and to remove drift from the correlation between TSC and PTP timestamps. The documentation also says the reference MAC should be a PTP client of an external grandmaster, directly or through a switch.

The public nvpps source for Jetson Linux R36.4.4 shows the same structure.

• The driver reads the ARM generic timer counter (cntvct, the Linux arch counter) as the "TSC".
• The reference MAC for the lock comes from primary-emac in the device tree. The default AGX Orin device tree sets it to the on-board MGBE0, which is eno1.
• A timer of about one second checks the lock state and re-arms the lock if it has dropped.
• If the device tree has the ptp_tsc_sync_dis property, the whole configuration is skipped.

The live device tree on our board had no ptp_tsc_sync_dis. The feature was on by default. But eno1 had no cable and no PTP grandmaster. The host counter was locked to a clock that ran with no reference at all. One thing that can trip you up when searching: the DRIVE OS 6.0.7 documentation names the disable property ptp_tsc_dis, while the DRIVE OS 7.0.3 documentation and the Jetson Linux R36.4.4 nvpps source use ptp_tsc_sync_dis.

Wrong turn 3: unbinding the driver made it worse

To see what the hardware was actually doing, we read the TSC lock status register defined in the nvpps source, read-only, every 0.1 seconds for 20 minutes.

• The one-second counter in the status register advanced for only 10 to 20 seconds every 109 to 124 seconds (median 110 s), and stood still the rest of the time.
• In each frozen interval, the host counter rate was pinned to one of a few values, such as 0, −20, or ±142 ppm.
• The lock bit (LOCKED) did not predict the rate.

Our first model was "when the lock drops, the clock slips at −141 ppm." The full data contradicted it: there were intervals at +142 ppm while locked and at 0 ppm while unlocked. The explanation that best fits the data now is that while the hardware stops updating for about 100 seconds, the last correction value stays in effect.

So was the driver's one-second re-alignment the culprit? We left the hardware configuration in place, unbound only the nvpps device from its driver, and measured for another 20 minutes.

The result was the opposite of what we expected. The slips did not go away; they got larger. The share of time with more than 100 ppm of rate error rose from 28% to 78%, and accumulated phase reached 45.7 ms. The ~110-second period stayed the same. The driver's software re-alignment was not the cause. The problem was the hardware lock configuration set at boot.

Turn it off, and it is gone

We added the valueless property ptp_tsc_sync_dis to the nvpps node in the device tree, rebooted, and measured for 20 minutes.

• All lock configuration registers at 0, the one-second counter stopped, zero slips.
• The rate difference between the host counter and the I210 held constant at −26.26 ppm. Only the native difference between the two oscillators remained.

In that state we powered the drives and measured DC for 20 minutes (1,170 s), with the motors stopped.

Drive 2 DC deviation	Lock on (default, 300 s)	Lock off (1,170 s)
p99	about 93 µs	0.18 µs
Max	328 µs	3.5 µs
Share above 10 µs	7.4% (every minute)	0% (all 20 minutes)

Two small bumps of a few microseconds remained. One (at 757 s, 3.8 µs) coincided with NTP changing its frequency correction. The other (at 1,063 s, −4.5 µs) we could not explain. Both are orders of magnitude below the transients of hundreds of microseconds.

Then we went back to the high-speed run where the problem first appeared, coupling the two axes and driving them at 23.5 rad/s with the same product build, a 300 µs SYNC0 shift, and about 33 seconds of motion per run.

Run	Lock	Max DC deviation during motion	Notes
Before fix	On	326 µs	Transient from 23 to 36 s, torque spike inside it
After fix 1	Off	2.7 µs	Zero protective stops, zero missed cycles, zero WKC mismatches during operation
After fix 2	Off	2.8 µs	Same as above
After fix 3	Off	5.2 µs	A single event a few ms long at constant speed, right after a 7.6 µs loop jitter. Different in shape from the old multi-second transients

A run at 14 rad/s also stayed at 2.3 µs maximum during motion.

(a) DC deviation of drive 2, p99 and maximum, with the 1 kHz bus running and the motors stopped. (b) Maximum during motion in runs with the two axes coupled at 23.5 rad/s. The vertical axis is logarithmic; the red dashed line marks 10 µs.

Is it safe to turn this lock off?

The feature exists for a reason, so before disabling it we checked whether anything depended on it.

• In our configuration, synchronization across multiple stereo cameras is handled by the deserializer hardware.
• The camera fsync generator was disabled.
• A camera check after reboot passed.

However, if a device connects eno1 to a real PTP grandmaster and keeps time over PTP, this lock is doing exactly what it was designed for, and it should stay on. The feature becomes a problem when the reference MAC runs with no reference.

In PLEM we settled it like this.

• The real-time setup step adds this property automatically on boards that have an nvpps node. Instead of attaching an overlay to each boot label, it writes the property into the base device tree itself, so the setting survives operations that create new boot labels, such as installing a camera driver.
• Devices that use eno1 for PTP have an option to keep the lock.
• The DC approach (steering the bus to the host time) stays as it is. With the trigger gone, there is no pressing reason to change it.

Does your Jetson do this? How to check

1. Check the device tree

ls /sys/firmware/devicetree/base/bus@0/nvpps/

If ptp_tsc_sync_dis is in the list, the lock is off. If it is not, the default applies, which means on.

2. Check the boot log

sudo dmesg | grep -i "TSC config ptx"

If this line appears, the lock was configured at boot.

3. Measure it yourself

Record the host counter side by side with the PHC of a NIC that has its own oscillator. The script below prints the rate difference between the two clocks in ppm every 2 seconds. It reads the PHC device, so run it as root.

# phc_vs_host.py: print the rate difference between the host counter (CLOCK_MONOTONIC_RAW) and a NIC hardware clock (PHC) every 2 s
import os, sys, time

dev = sys.argv[1] if len(sys.argv) > 1 else "/dev/ptp0"
fd = os.open(dev, os.O_RDONLY)
phc = ((~fd) << 3) | 3  # FD_TO_CLOCKID: clock id for reading the PHC with clock_gettime

def sample():
    r0 = time.clock_gettime_ns(time.CLOCK_MONOTONIC_RAW)
    p = time.clock_gettime_ns(phc)
    r1 = time.clock_gettime_ns(time.CLOCK_MONOTONIC_RAW)
    return (r0 + r1) // 2, p

prev = sample()
while True:
    time.sleep(2.0)
    cur = sample()
    d_host = cur[0] - prev[0]
    d_phc = cur[1] - prev[1]
    ppm = (d_host - d_phc) / d_phc * 1e6
    print(f"{time.strftime('%H:%M:%S')}  host - PHC = {ppm:+8.2f} ppm", flush=True)
    prev = cur

sudo python3 phc_vs_host.py /dev/ptp0

If the output stays constant somewhere between a few and a few tens of ppm, that is the native difference between the two oscillators. If it steps by more than 100 ppm roughly every 110 seconds, the host is moving. Ten minutes is enough to see it.

Do not use the PHC of the on-board eno1 as the comparison. Even with the lock off, it matches the host counter within 0.01 ppm, which suggests the two share the same SoC oscillator. The PHC of a separate NIC (in our case, the I210 used for EtherCAT) is the reference to judge by.

What we still do not know

• We have not explained where the ~110-second period itself comes from. There is no trace of it in the kernel log or in eno1 interrupts. It is hardware behavior, so it needs an explanation from NVIDIA. If you have seen the same thing, we would like to compare notes.
• The NVIDIA developer forum also has a report of CLOCK_MONOTONIC_RAW frequency jumps on AGX Orin with JetPack 6.0 (link). That thread did not identify a cause. We do not know whether the cause is the same, but the symptom looks similar.
• These measurements were taken on a Jetson AGX Orin with JetPack 6.2.1 (R36.4.4). We have not checked JetPack 7 yet.

Takeaways

• If DC collapses periodically while the control loop looks fine, suspect the host clock. Put two unrelated clocks side by side and it becomes obvious who is moving.
• On Jetson Orin (JetPack 6), if the device tree has an nvpps node and no ptp_tsc_sync_dis, the PTP-TSC lock is enabled at boot. With no PTP grandmaster on the reference Ethernet port, that lock can swing the host counter by up to ±140 ppm roughly every 110 seconds.
• One device tree property, ptp_tsc_sync_dis, cut the maximum DC deviation from 328 µs to 3.5 µs, and the maximum during high-speed motion from 326 µs to 2.7 to 5.2 µs. The control code did not change.
• Steering the bus to the host time is a standard industry choice, but it rests on the premise that the host clock is steady. The lesson here is that a platform default can break that premise.

References

• NVIDIA DRIVE OS 7.0.3, Time Sync: https://developer.nvidia.com/docs/drive/drive-os/7.0.3/public/drive-os-linux-sdk/embedded-software-components/DRIVE_AGX_SoC/Networking/time_sync_details.html
• NVIDIA, Jetson Linux R36.4.4 (JetPack 6.2.1): https://developer.nvidia.com/embedded/jetson-linux-r3644
• nvpps source (Jetson Linux R36.4.4, OE4T mirror): https://github.com/OE4T/linux-nv-oot/blob/patches-r36.4.4/drivers/nvpps/nvpps_main.c
• IgH EtherCAT Master, Application Interface: https://docs.etherlab.org/ethercat/1.6/doxygen/group__ApplicationInterface.html
• acontis EC-Master, DCM: https://public.acontis.com/manuals/EC-Master/3.1/html/ec-master-class-a/dcm_technical.html
• Beckhoff, Distributed Clocks: https://infosys.beckhoff.com/content/1033/tc3_io_intro/1446575627.html
• Beckhoff, EtherCAT ESC Datasheet Section II (registers 0x0910, 0x092C): https://download.beckhoff.com/download/Document/io/ethercat-development-products/ethercat_esc_datasheet_sec2_registers_v3.3.pdf