← ComputingSphere

Reading room · Case study · 21 October 2026

The substation GPS that overwrote the master: DNP3 events an hour in the future

The symptom reached the control centre as a data quality problem: events from one bay arrived with timestamps in the future, and the SCADA flagged them as suspect. The master's configuration looked right: it wrote the time to that outstation every 30 minutes, and every write was accepted. Yet the events kept coming in an hour ahead. The cause was in the substation: a GPS clock that kept writing its own, wrong, time into the same device.

Two clocks, one device

Many bay IEDs accept time from two sources: the SCADA master over DNP3 (a WRITE of the Time and Date object, group 50) and a local reference, a GPS receiver over IRIG-B, NTP or PTP. They are meant to agree. Here the substation's GPS applied a daylight-saving rule the site does not use, so its time was exactly one hour ahead, and the IED took it a couple of minutes after every DNP3 sync.

The sequence repeats every 30 minutes:

  1. The outstation raises IIN1.4 NEED_TIME; the master runs DELAY_MEASURE and writes the time. The outstation accepts it: no error, NEED_TIME clears.
  2. About two minutes later the GPS sets the clock one hour ahead. Nothing goes over DNP3: the outstation has no reason to ask for the time again, because from its point of view its clock is synced.
  3. Every event from then on (binary changes, analog events with time, group 2 and group 32) carries a timestamp roughly an hour after the moment the packet was captured.

What the capture shows

Outstations ranking: outstation 5 with the finding events ahead of the synced clock, outstation 6 without it
The ranking: only outstation 5 has the finding "events ahead of the synced clock". Outstation 6, polled the same way and synced by the same master, is clean.

Comparing every event's own timestamp with the time it was captured, against the clock the master wrote:

  • 885 events from outstation 5, 820 of them in the future, by up to 3,599,910 ms, one hour minus the network delay. The only ones that were on time are those of the two minutes after each sync.
  • 4 time syncs, all applied: the master's writes were accepted every time. This is not a master that fails to sync, nor an outstation that refuses it.
  • NEED_TIME only 4 times, once per sync cycle. The usual alarm for clock trouble stays quiet, because the device believes its time.
  • Outstation 6: 887 events, none in the future. Same master, same sync interval: the problem is local to the bay.
Analysis of outstation 5: 820 events stamped ahead of the clock the master synced
The analysis of outstation 5 leads with the finding: 820 events stamped ahead of the clock the master synced.

Why it matters

An hour is a large error, so the SCADA caught it and flagged the events as suspect, which was the right thing to do and also the trouble: every sequence of events from that bay became unusable. A smaller offset is worse. A GPS a few seconds off puts a breaker trip before the protection pickup that caused it, and nobody flags it. In both cases the fault-analysis tools downstream, the sequence-of-events report, post-mortem of a protection operation, and any scheme that orders events across devices, work on the wrong order.

Report executive summary listing 820 events stamped ahead of the clock the master synced
The same finding in the commissioning report, ready to attach to the ticket for the substation team.

How to fix it

  • Pick one time master per device. If the substation has a GPS, let it own the time and disable the SCADA's time write to that IED (or the other way around). Two sources fighting is the root cause, whatever their offset.
  • Check the GPS configuration: time zone, daylight-saving rule, and whether the IED expects UTC or local time on IRIG-B. An offset of exactly one hour almost always is one of these.
  • Verify from the traffic: after the change, events should land within milliseconds of their capture time, the way outstation 6's do.

This is the kind of problem that is invisible from either end alone. The master sees accepted time writes; the substation sees a GPS-synced clock. Only the traffic between them, with every event timestamp compared against the time the master wrote, shows that the two disagree.

Try it on your own capture

ComputingSphere DNP3 decodes a pcap once, splits it per channel and per outstation, and ranks the outstations worst first with the same numbers used here. It only reads what already went over the wire. Every outstation gets its event timing checked against the clock the master wrote, with the events ahead of it counted. This capture is among the examples of every free account, to reproduce every number in this post; or upload your own: the Free plan covers 1,000 messages a month.

More cases from the same capture