IEEE 1588 Based Synchronization Distribution System Implementation

Practical Implementation of a IEEE 1588 Based Synchronization Distribution System DA N I E L G A L L A N T I T S F N OV E M B E R 2 0 1 8

Agenda Practical Implementation of a T-BC ToD Synchronization Between Master and Slave ToD Synchronization within the Network Element Challenges of Meeting 5ns Time Error Possible Solutions for Meeting 5ns and Beyond 2

Implementation of a Telecom Boundary Clock (T-BC) G.8273.2 Implementation Block Diagram ITU-T G.8273.2 Functional Block Diagram Delay 1PPS Packet time asymmetry Timing Service Monitor TOD TOD Time stamps Freq ToD Adjust PHY Slave PHY Master Time input ToD Time selector (e.g., 1pps+TOD) Packet I/O Packet I/O Frequency output (e.g., 2MHz) 1PPS Time Stamper Time Stamper Time and frequency generator TOD CPU Time- stamped Packets PP PP Packet I/O Packet I/O (packet processing) (packet processing) MASTER SIDE Stack Servo SLAVE SIDE Frequency selector TCXO/ OCXO PEC DCO PLL Frequency OUT Physical layer clock (e.g., EEC/SEC) Physical layer Frequency IN Physical layer Frequency OUT Physical layer Frequency IN Physical layer Frequency OUT PLL PHYSICAL LAYER G.8273.2 Y.1368.2(14)_FA.1 Ext. frequency input (e.g., 2MHz) Main physical components of a T-BC: Master and Slave PHY (ToD/timestampers) Processor (1588 stack & servo) 1588 PLL (with DCO) SyncE PLL (independent from 1588 PLL) TCXO/OCXO (for frequency stability) Main functions of a T-BC: Packet processing (PTP stack, timestamping) PEC (uses timestamps to generate clocks) Physical layer clock (unaltered by PTP) 3

The Goal of Network Synchronization All ToD Counters Synchronized PRTC GNSS T-GM T-BC T-TSC 1PPS 1PPS TOD TOD TOD ToD Freq ToD Adjust Freq Freq PHY Master PHY Slave PHY Slave PHY Master ToD ToD ToD ToD 1PPS ToD Adjust Packet I/O Packet I/O Time Stamper Time Stamper Time Stamper Time Stamper CPU CPU Time- stamped Packets Time- stamped Packets Stack Servo Stack Servo TCXO/ OCXO DCO DCO TCXO/ OCXO PLL PLL PLL Clk OUT Clk OUT Clk OUT PLL PLL PLL Clk OUT 1PPS Physical layer Frequency Physical layer Frequency All ToD counters within the timing chain must be synchronized with the T-GM: They contain the same time value They turn over at the same time (with nano-second accuracy) 4

T-GM to T-BC ToD Synchronization Using 1588 to synchronize ToD between T-GM Master port and T-BC Slave port PRTC GNSS T-GM T-BC T-TSC 1PPS 1PPS TOD TOD TOD ToD Freq ToD Adjust Freq Freq PHY Master PHY Slave PHY Slave PHY Master ToD ToD ToD ToD 1PPS ToD Adjust Packet I/O Packet I/O Time Stamper Time Stamper Time Stamper Time Stamper 1588 Loop CPU CPU Time- stamped Packets Time- stamped Packets Stack Servo Stack Servo TCXO/ OCXO DCO DCO TCXO/ OCXO PLL PLL PLL Clk OUT Clk OUT Clk OUT PLL PLL PLL Clk OUT 1PPS Physical layer Frequency Physical layer Frequency Three stages of ToD synchronization: 1. Initial adjustment: T-BC extracts ToD from 1588 packet sent by T-GM 2. Lock acquisition: T-BC servo uses timestamp information to adjust ToD counter frequency minimizing time error 3. Locked: T-BC servo makes fine adjustments of ToD frequency to keep time error at zero 5

Option 1 Option 1: T-BC ToD Synchronization > Using a Physical Interface ToD counters must be synchronized between Slave and Master ports within T-BC PRTC GNSS T-GM T-BC T-TSC 1PPS 1PPS TOD TOD TOD ToD Sync I/F Freq ToD Adjust Freq PHY Master PHY Slave Freq PHY Slave PHY Master ToD ToD ToD ToD 1PPS ToD Adjust Packet I/O Packet I/O Time Stamper Time Stamper Time Stamper Time Stamper CPU CPU Time- stamped Packets Time- stamped Packets Stack Servo Stack Servo TCXO/ OCXO DCO DCO TCXO/ OCXO PLL PLL PLL Clk OUT Clk OUT Clk OUT PLL PLL PLL Clk OUT 1PPS Physical layer Frequency Physical layer Frequency If timestamping is performed in separate physical locations within the T-BC Synchronization of ToD at slave and master ports must be synchronized (or it will contribute to time error) Proprietary interfaces help ToD synchronization between PHYs 6

Option 2 Option 2: T-BC ToD Synchronization > Using a Central ToD/Timestamper Central ToD PRTC T-BC GNSS T-GM T-TSC 1PPS 1PPS TOD TOD TOD NPU ToD Freq ToD Adjust Freq PHY Master PHY Master Master PHY Slave PHY Slave PHY ToD ToD ToD M A C M A C MII MII 1PPS Packet I/O Packet I/O Time Stamper Time Stamper Time Stamper Time- stamped Packets ToD Adjust CPU CPU Time- stamped Packets Freq Stack Servo Stack Servo TCXO/ OCXO DCO DCO TCXO/ OCXO PLL PLL PLL Clk OUT Clk OUT Clk OUT PLL PLL PLL Clk OUT 1PPS Physical layer Frequency Physical layer Frequency ToD and Timestamping can be located at a central point in the system Synchronization of ToD at slave and master ports must be synchronized (or it will contribute to time error) Ortimestampers must reference a central point of ToD 7

Multi Circuit-Pack System Architecture ToD Most carrier grade equipment is broken down into multiple circuit packs connected together with a backplane MASTER LC LC LC LC SLAVE PHY PHY PHY PHY LC ToD ToD ToD ToD PHY Timestamping/ToD and PTP stack functions (Line Card) are separated from the central clock generation functions (Timing Card) TS ToD TS TS TS TS CPU CPU CPU CPU Stack Stack Stack Stack CPU Stack There s still a need to synchronize ToD from the slave line card to ToD of master line cards Backplane TC TC CPU CPU CPU CPU Servo Servo Servo Servo DCO DCO PLL PLL PLL PLL 8

Distribution of ToD across a Multi Circuit-Pack System Control plane ToD Distribution ToD from Slave line card is distributed to all Master line cards ToD is distributed over control plane Precision not important (1 second to get there) 1PPS provides ToD 1-second rollover precision ToD MASTER LC LC LC LC SLAVE PHY PHY PHY PHY LC ToD ToD ToD ToD PHY TS ToD TS TS 1PPS Distribution 1PPS is generated by PLL/DCO (controlled by 1588 Servo) Objective is to align all 1PPS signals across all line cards Any 1PPS phase misalignment looks like constant time error (cTE) TS TS CPU CPU CPU CPU Stack Stack Stack Stack CPU Stack 1PPS Backplane Challenges Backplane trace lengths must be matched or compensated Delay uncertainty of backplane drivers/receivers must be considered in timing budget I/O delay of line card PLL must also be considered TC TC CPU CPU CPU CPU Servo Servo Servo Servo DCO DCO PLL PLL PLL PLL 9

Solution 1 Solution 1: Manual Calibration of 1PPS Distribution Path Backplane Trace Delays 1PPS backplane trace delays are manually measured Variance in PCB trace length during manufacturing causes delay uncertainty Line Card Slave Slave Master Line Card Line Card ToD ToD ToD Line Card Slave TS ToD TS TS TS PLL Delay Adjust PLL t PLL PLL t PLL I/O Delay t Backplane Driver/Receiver Delays Driver + Receiver delay uncertainty over PVT : 2-3 ns typ Trace length differences Line Card PLL Keep I-O delay as low as possible by using ZDM All deterministic delays can be compensated for using adjustable delay at Line Card PLL. Driver/Receiver delay uncertainty Backplane Challenges: Delay uncertainty from PCB traces, drivers/receivers, and Line Card PLL adds to the time error budget Drivers/receiver delay uncertainty is the main contributor 1PPS DCO Timing Card Master PLL PVT - Process/Voltage/Temperature, ZDM Zero Delay Mode 10

Solution 2 Solution 2: On-chip 1PPS Delay Measurement & Adjust Line Card Slave Slave Master Measuring 1PPS path delays 1PPS path delays are measured at start-up Must be done for each Line Card (each 1PPS path) Delay measurement performed at central point in the system. In this case at the Timing Card. Could use existing SyncE backplane traces to provide return path Measured trace delay differences are adjusted at line cards 1PPS delay = Round Trip Delay minus Return Path delay Line Card Line Card ToD ToD ToD Line Card Slave TS ToD TS TS TS PLL Delay Adjust PLL t PLL PLL t t Backplane Advantage: Compensates for both backplane trace delay and driver/receiver delay uncertainty Challenges: Assumption that forward and return paths are symmetrical Return path must also be measured (using TxSyncE/RxSyncE pair?) 1PPS Delay DCO Measurement PLL I-O Timing Card Master 11

Solution 3 Solution 3 - ToD Distribution Without 1PPS Distribution Synchronizes ToDB with ToDA without a physical time alignment signal (i.e. no 1PPS required) 1588-like ToD distribution but with the advantage that Master and Slave are working from the same frequency (using existing backplane frequency distribution) No servo loop or acquisition process required 1588 messaging calculates One Way Delay (OWD) and Time Error Offset (e) No PDV Static asymmetry (can be calibrated out) Control Plane T-BC Switch PHY PHY Line Card - Master Line Card - Slave PHY NP PHY NP PHY PHY t1 ToDA ToDB Rx SyncE t2 t3 t4 PLL PLL t4 frequency Challenges: Control plane must be 1588 aware (e.g. Transparent clocking) ToD transfer accuracy depends on timestamp resolution Backplane DCO PLL Timing Card - Master 12

Conclusions Time error of a T-BC consist of PTP servo error + misalignment of ToD Distribution of ToD in a multi circuit-pack system takes careful design Meeting 5ns time error is a difficult challenge. New design practices and synchronization distribution architectures will be needed 13

Thank You