High-Performance Time Server Core for FPGA System-on-Chip

electronics Article High-Performance Time Server Core for FPGA System-on-Chip Julian Viejo ∗, Jorge Juan-Chico , Manuel J. Bellido, Paulino Ruiz-de-Clavijo, David Guerrero, Enrique Ostua and German Cano Electronics Technology Department, E.T.S. Ingeniería Informática, University of Seville, Avda. Reina Mercedes s/n, 41012 Seville, Spain; [email protected] (J.J.-C.); [email protected] (M.J.B.); [email protected] (P.R.-d.-C.); [email protected] (D.G.); [email protected] (E.O.); [email protected] (G.C.) *Correspondence: [email protected]; Tel.: +34-954-556160 Received: 11 April 2019; Accepted: 7 May 2019; Published: 11 May 2019   Abstract: This paper presents the complete design and implementation of a low-cost, low-footprint, network time protocol server core for field programmable gate arrays. The core uses a carefully designed modular architecture, which is fully implemented in hardware using digital circuits and systems. Most remarkable novelties introduced are a hardware-optimized timekeeping algorithm implementation, and a full-hardware protocol stack and automatic network configuration. As a result, the core is able to achieve similar accuracy and performance to typical high-performance network time protocol server equipment. The core uses a standard global positioning system receiver as time reference, has a small footprint and can easily fit in a low-range field-programmable chip, greatly scaling down from previous system-on-chip time synchronization systems. Accuracy and performance results show that the core can serve hundreds of thousands of network time clients with negligible accuracy degradation, in contrast to state-of-the-art high-performance time server equipment. Therefore, this core provides a valuable time server solution for a wide range of emerging embedded and distributed network applications such as the Internet of Things and the smart grid, at a fraction of the cost and footprint of current discrete and embedded solutions. Keywords: system-on-chip; digital integrated circuits; field programmable gate array; network time synchronization; network time protocol; hardware timestamping; Internet of Things 1. Introduction Network time synchronization allows the nodes in a distributed system to share a common time through exchanging time synchronization packets with one or more reference nodes (the time servers). Time synchronization is recognized as a key component in all types of distributed systems, such as worldwide Internet services, industrial data acquisition systems [ 1 ], power distribution [ 2 ] and Internet of Things (IoT) applications [ 3 ]. Synchronization accuracy requirements are very dependent on the application field and particular tasks. Industrial and smart grid applications commonly require 1 µ s to 100 ms accuracy, whereas mobile networks, audio-video transmission and measurement applications normally aim at sub-microsecond accuracy [4]. Several synchronization algorithms over communication networks have been proposed by the scientific community with different levels of complexity and precision. Nowadays, the two main network synchronization protocols are network time protocol (NTP) [ 5 ], and IEEE 1588 precision time protocol (PTP) [ 6 ]. Both define various accuracy categories among their servers, with stratum-1 servers (NTP) or master clocks (PTP) being the most accurate references in the network. These top-level time sources are usually high-performance discrete synchronization servers from several manufacturers [7–10] that use an external time reference, such as global positioning system (GPS) [ 11 ], as a time source. These Electronics 2019,8, 528; doi:10.3390/electronics8050528 www.mdpi.com/journal/electronics Electronics 2019,8, 528 2 of 28 synchronization servers commonly support both protocols and employ dedicated hardware to handle the critical task of timestamping synchronization packets. Synchronization servers also achieve an accuracy to the reference from 1 µ s to a few milliseconds, with a maximum request ratio from 1000 to 10,000 requests per second, according to manufacturers specifications [ 8 – 10 ]. This exactitude and performance amounts an average cost that ranges from $3000 to $10,000 as of 2019. A dedicated discrete synchronization server may be a good solution for large and centralized infrastructures as found in the telecommunications industry. However, cost, size and/or power consumption render these systems inadequate for most embedded and highly distributed applications. Since a massive adoption of highly distributed technologies, such as the IoT, is expected in the forthcoming years [ 12 ], several authors have proposed NTP and PTP servers and/or clients solutions from the embedded systems perspective, frequently implemented as a system-on-chip (SoC) on a field programmable gate array (FPGA) device. Software-only embedded implementations are cheaper and easier to develop. Nevertheless, these solutions can only provide limited accuracy (e.g., embedded NTP implementations in [ 13 , 14 ]) report an accuracy in the range of several milliseconds, whereas other approaches based on the advanced RISC machine (ARM) architecture [ 15 , 16 ] are close to 1 ms). Similarly, the software-only PTP system in [ 17 ] reports an accuracy from 10 µ s to 200 µ s, depending on the server load. To approach to limited-resource systems, such as numerous IoT applications, less accurate techniques [ 18 ] must be applied instead of complex protocols, such as NTP or PTP. In contrast, hardware-assisted embedded approaches are more difficult to develop and have the extra cost of the dedicated hardware, although the expansion of FPGAs and modern FPGA-based SoC platforms have greatly dropped the cost barrier in the last few years. Hardware-assisted solutions aim to sub-microsecond accuracy in order to cope with more demanding applications, such as audio-video data synchronization [19], smart grid control [20] or high-accuracy data acquisition [21]. This contribution goes one step forward by developing an NTP server core for FPGA completely elaborated in hardware that handles both the timekeeping functions and network communications, and can operate stand-alone or embedded in a SoC. The NTP core provides highly distributed applications with a complete time server synchronization solution at a lower size and cost than previously developed SoC-based time synchronization systems (see Figure 1). It means that the full functionality of an NTP server can be embedded as a module in any FPGA-based system, while offering an accuracy and performance comparable to that found in large and expensive high-performance NTP servers. Figure 1. Qualitative physical scale comparison of time server implementation approaches from discrete equipment to hardware core (this paper). Unit cost and power consumption roughly downscale with the physical size. Electronics 2019,8, 528 3 of 28 The rest of the paper is organized as follows. Section 2presents previous and related work, Section 3introduces the NTP protocol and Section 4offers an overview of the structure and operation of the core. Sections 5and 6detail design and implementation. Accuracy and performance results are presented and discussed in Section 7and the main conclusions are summarized in Section 8. 2. Previous and Related Work 2.1. Previous Work The authors have done research on accurate hardware-based time synchronization since 2007. A fully-hardware NTP client is first introduced in [ 22 , 23 ] with application to industrial remote terminal units (RTU). Further development in hardware-oriented clock discipline algorithms is presented in [ 24 ] allowing for better accuracy and performance. Later on, the authors started working in a more flexible and general purpose modular hardware architecture for NTP client and server development. An overview of the new architecture is introduced in the summary of a conference’s invited talk in [ 25 ], together with some preliminary performance estimations derived from early, functionally-incomplete prototypes. This paper is the evolution of the initial architecture applied to the development of a general-purpose fully-functional NTP server. Detailed description of the the architecture and its building blocks is offered here for the first time, together with up to date extensive performance results. 2.2. Related Work In recent years, several time synchronization systems have been proposed from the embedded systems perspective, using the flexibility of the FPGA platform. Some of these approaches are commented on below. Software-only NTP implementations in [ 13 , 14 ] generate a clock signal with an accuracy in the range of several milliseconds with respect to the time reference. This accuracy is far from the range of 1 µ s of typical NTP equipment. Other software implementations based on the ARM architecture can be found in [ 15 , 16 ]. The accuracy reported in [ 15 ] is in the range of the expected accuracy of 1 ms typical of software-only NTP servers, even though a voltage controlled temperature compensated crystal oscillator (VCTCCO) is used to improve the accuracy of the local clock. The authors of [ 16 ] basically did a port of the standard NTP server software to their ARM-based development platform and did not report any experimental results. Regarding the full hardware NTP client/server in [ 19 ], it is designed on a low-cost FPGA chip integrated with an oven controlled crystal oscillator (OCXO). The implementation is not described in detail. Results show a high degree of accuracy in the pulse-per-second (PPS) signal generated by the server and a synchronized NTP client (below 5 µ s). However, the accuracy of the NTP server’s PPS signal with respect to its time source (GPS receiver) is not reported. It is also not clear if the protocol used is the standard NTP or an NTP variant implemented on top of Ethernet without the internet protocol (IP) level. Moreira et al. [ 20 ] developed various PTP client and server implementations for the Xilinx Zynq-7000 platform. This platform includes both programmable logic and an ARM processor. The authors followed the hardware–software SoC approach by using dedicated synchronization hardware tied to the Ethernet controller together with PTP software running on the Linux operation system. Results obtained by connecting client and server through a single Ethernet link (no switch in the middle) report an offset error of 40 ns, while network tests are below 1 µs. A similar approach comes from the White Rabbit (WR) Project [ 26 ]. The WR Project aims at sub-nanosecond accuracy by using extensions to the PTP and Ethernet protocols. The system proposed in [ 21 ] uses a mixed software–hardware approach that combines FPGA-based specific hardware together with controlling software running in an ARM microprocessor. The whole system is implemented as a SoC using the Xilinx’s Zynq development platform. The result is a stand-alone Electronics 2019,8, 528 4 of 28 WR node that should be able to achieve nanosecond accuracy when used inside the very specialized WR ecosystem. Table 1offers a quick overview of the commented related work. Table 1. Comparison of related works. Implementation Scale Application Protocol Accuracy Hwang, S.Y. [15] Software Embedded Distributed systems Network time protocol (NTP) ≈1 ms Chao, C.C. [16] Software Embedded General purpose NTP ≈1 ms Kuwano, S. [19] Hardware Embedded Digitized radio over fiber (DROF) systems NTP ≈1µs Moreira, N. [20]Hardware/ Software System-on-chip (SoC) Smart grids and substation automation systems (SAS) Precision time protocol (PTP) ≈40 ns Jimenez, M. [21]Hardware/ Software SoC Scientific infrastructures PTP * <1 ns * with extensions on PTP and Ethernet protocols. When compared to other author’s related work, the approach used in this paper has the following advantages: • Scale: The system is a very small core compared to previous SoC approaches, and can be used stand-alone or integrated in any other SoC project. • Hardware-only: It does not require a software stack or even a processor and associated resources, allowing for very efficient operation in both performance and power consumption. • Standard logic design: It can be easily ported to any platform with a minimum effort, and does not need any additional external devices to operate. • The cores uses a modular design that can be extended and/or reused to tackle other applications: use of other protocols, client design, etc. In summary, this contribution tries to fill an important niche: producing a very low-footprint embedded time server solution that is as accurate as discrete commercial equipment, with minimum hardware requirements and synthesizable on standard digital programmable logic. Compared to previous SoC-scale approaches, the proposed design is more suited to fulfill the cost, performance and efficiency demands of the IoT field. 3. NTP Synchronization The NTP synchronization protocol is based on exchanging packets between clients and servers. This mechanism is called on-wire protocol and aims to determine the offset of the client’s local clock, with respect to the server’s clock, and the latency of the network connection. Figure 2describes the operation of the NTP on-wire protocol. The client sends a request to the server by issuing an user datagram protocol (UDP) data packet including the time at its local clock T1 (origin timestamp). A new time T2 is registered with the reception time as given by the server’s local clock (receive timestamp) when the request is received at the server. After processing the request, the server issues a reply including the time at which the reply leaves the server T3 (transmit timestamp). As soon as the client receives the reply, the arrival time T4 is also annotated (destination timestamp). This set of timestamps help the client calculate the round trip time ( δ ) and the time offset between the server’s and the client’s clocks ( θ ). Assuming a symmetric connection (equal delay in both directions), these times can be calculated as: δ= (T4−T1)−(T3−T2) θ=(T2−T1) + (T3−T4) 2 (1) Electronics 2019,8, 528 5 of 28 Figure 2. On-wire network time protocol (NTP) operation. The client can correct its local clock to match the server’s time as accurately as possible using the clock offset, the round trip delay and additional statistical measurements based on the calculated offset and delay. Software implementations of NTP clients normally achieve time synchronization accuracy within a millisecond with respect to NTP servers on the Internet [ 27 ]. There are two main sources of error that could limit the accuracy. The first one is asymmetry in the network communication when time spent by the client’s request to reach the server differs from time spent by the server’s response to reach the client. This asymmetry is due to various latencies found in network paths and in network equipments. While the PTP is able to compensate for systematic known latency asymmetry [ 28 ], NTP does not provide any standard compensation mechanism. The second main source of error comes from the variable latency between the instant in which the timestamp is registered in the datagram, and the real instant the datagram leaves or reaches the host. However, in common software implementation, these timestamps are registered by client/server software running as a user level application. Therefore, the timestamp error will depend on time spent processing the datagram, as it goes through the protocol stack and software layers. Hence, this error will largely depend on system load and detailed software implementation. The NTP implements a set of advanced algorithms in order to mitigate these errors and improve accuracy [ 5 ]. In a first stage, the NTP client sends requests to multiple servers and the algorithm calculates offset ( θ ), delay ( δ ), dispersion ( e ), and jitter ( ϕ ) for every server. Dispersion represents the maximum error due to the frequency tolerance and time spent since the last packet was sent. The jitter is defined as the root-mean-square (RMS) average of the most recent offset differences, representing the nominal error in estimating the offset. While jitter is rarely considered a major factor in ranking server quality, this parameter is a valuable indicator of timekeeping performance and network congestion state. In a second stage, these data are processed by the mitigation algorithms consisting of selection, clustering, combination and clock discipline. First, the selection algorithm scans all associations and casts off the falsetickers, which have demonstrably incorrect time, leaving the truechimers as a result. Second, in a series of rounds, the clustering algorithm discards the associations statistically furthest from the centroid, until a specified minimum number of survivors remain. Third, the combination algorithm produces the best and final statistics on a weighted average basis, and the best time offsets are obtained. Finally, the clock discipline algorithm is applied to calculate the needed time and frequency corrections in order to match the server’s time as accurately as possible and maintain a stable clock frequency. The precision of the NTP synchronization can be largely improved by executing the timestamping operation in lower layers [ 29 ]. However, to achieve the highest precision in the timestamping operation, Electronics 2019,8, 528 6 of 28 the Ethernet network interface card (NIC) must carry out this task as soon as the packets arrive or leave the interface. Therefore, accuracy in the client’s clock can be within one microsecond [ 29 ]. This way, the precision of an NTP client/server system can be similar to that of a PTP systems, where hardware-assisted timestamping is mandatory in a standards-compliant implementation. In addition to the NTP specification, there exists the simple network time protocol (SNTP) specification [ 30 ]. Both specifications share the same communication protocol and data format. However, NTP uses sophisticated algorithms described above that ensure a correct synchronization with multiple servers under highly variable latency data links, which is common in a worldwide network such as the Internet. By contrast, SNTP covers the synchronization with a single reference (as happens in Stratum 1 servers) and allows the peers to use simplified stateless algorithms in clients and servers, being better suited to be implemented in embedded systems. Regardless, any NTP or SNTP client will communicate seamlessly with either a SNTP or a full NTP server. 4. NTP Server Core Overview The NTP server core described in this paper synchronizes with a single reference, a GPS receiver, and operates as a primary (Stratum 1) server following the SNTP. The core can be used in the standard scenario depicted in Figure 3, where the NTP server provides NTP clients in local or remote networks with an NTP synchronization service. Moreover, a configuration server delivers networking configuration parameters (IP address, network mask and so on) to hosts in the network. The configuration server has been explicitly included in that figure since the NTP server core can obtain internal configuration parameters from the configuration server, in addition to the normal network parameters. Consequently, the complete configuration for any number of NTP servers can be maintained in a central place and can be easily updated, even at run-time, if necessary. In particular, the bootstrap protocol (BOOTP) [ 31 ] is used for configuration due to its simplicity compared to the dynamic host configuration protocol (DHCP). In fact, most DHCP services also implement the BOOTP, therefore, in most cases, the configuration requirements of the NTP core can be solved by utilizing existing infrastructure. Figure 3. NTP synchronization system overview. The NTP server has been designed as a pure digital core requiring a minimum set of external elements (see Figure 4): (i) A physical layer adapter (PHY) that interfaces to the media access control (MAC) controller in the server through the media-independent interface (MII); and (ii) A recommended standard 232 (RS-232) transceiver to convert the output voltage level of the GPS receiver into the appropriate level at the core, if necessary. Such core also uses the pulse per second (PPS) signal from the GPS receiver as well as an optional external digital clock signal (osc), used as the base frequency referencefor the local clock. The server also outputs its local clock time and internal PPS reference (signals local_time and PPSout, respectively) that can be used for testing as well as for delivering accurate time to nearby cores and systems. Electronics 2019,8, 528 7 of 28 Figure 4. NTP server core and external elements. Internally, the NTP core consists of five main modules connected as depicted in Figure 5. The MAC controller and the universal asynchronous receiver-transmitter (UART) are standard modules that handle the communication with the Ethernet local network and serial port, respectively. The modules in the NTP core can be grouped into two subsystems: (i) The timekeeping subsystem, formed by the UART, the time reception module and the synchronization module; And (ii) communications subsystem, formed by the comprising protocol and configuration interface (P&CI), and the MAC controller. In the former, the time reception module processes the data received from the GPS unit through the UART and sends the information to the synchronization module, which uses it to maintain the local clock synchronized with the GPS time. In the latter, the P&CI handles the initial configuration of the system and network communications, by means of various Internet protocols, using the MAC controller as the lower-level network interface. Figure 5. NTP server core block diagram. Electronics 2019,8, 528 8 of 28 The NTP server core architecture is based on the general NTP server architecture proposed in [ 25 ]. Both the synchronization module and the P&CI borrow some ideas and characteristics from some of the modules in the NTP client prototypes in [ 22 , 23 ]. However, the modules in the NTP server core constitute a new server-oriented implementation with greatly extended and improved functionality. In general, the system operates as follows: upon startup, the P&CI enters the configuration phase and sends a BOOTP request to the network through the MAC controller. The BOOTP response brings back the configuration of the core that the P&CI distributes to the corresponding modules. Thereafter, the system starts normal operation. One of the key aspects of the design is that, once initialized, the timekeeping subsystem works independently of the communication subsystem. On the one hand, the active edge of the PPS signal triggers the reception of the data received from the GPS in the time reception module, collected through the UART. This information is processed and the current time is extracted and handed to the synchronization module, which uses the GPS time and the active edge of the PPS signal to adjust and discipline the local clock. On the other hand, the P&CI waits for NTP requests from clients in the network and, upon reception of an NTP request, an NTP response is built using the local time from the synchronization module. Afterwards, the NTP packet is passed to the MAC controller for delivery. Splitting tasks between the two subsystems allows the accuracy of the local clock time to be conserved independently of the network load. At the same time, the timekeeping operations triggered by the PPS signal do not affect the NTP network operations carried out by P&CI. This fact contributes to maintaining a constant latency when processing NTP requests, which is vital to minimize errors in the synchronization of clients, as mentioned in Section 3. The NTP server core uses standard synchronous digital design techniques and it has been optimized to be implemented in low-density cost-effective FPGA chips, and to be able to achieve microsecond accuracy at a system clock frequency as low as 50 MHz. The NTP server core has been implemented in a Xilinx XC3S500E FPGA chip [ 32 ]. This low-range and slightly outdated chip has been intentionally selected to demonstrate the performance of the core, even in modest hardware. Most modules in the core implement a control unit described in a hardware description language (HDL) that controls a processing data path. Xilinx System Generator for digital signal processor (DSP) [ 33 ] has been utilized to implement some of the data processing blocks, so as to shorten the design time. However, the design can be easily ported to any other platform provided that basic arithmetic and system-level blocks are available. Additionally, the modularity of the design allows the future re-usability of the main blocks depicted in Figure 5in other synchronization applications. The next sections further describe the design and operation of the NTP server core’s building blocks. 5. Timekeeping Subsystem The synchronization module is the core of the timekeeping subsystem, in which the UART and the time reception modules are working as an interface to the GPS receiver. These modules are described in this section. 5.1. UART The UART is used by the NTP server to receive time data from the GPS receiver through the popular electronic industries alliance (EIA) RS-232. RS-232 has been selected for its low-cost and wide availability. UART’s are standard modules easily available in embedded systems design libraries. In the present paper, a Xilinx’s intellectual property (IP) UART core has been used [ 34 ]. It is connected to an external RS-232 transceiver and includes both transmitter and receiver modules, although only the latter is needed in this case. GPS receivers universally use the national marine electronics association specifications NMEA-0183 [ 35 ] to transmit data. The specification defines both the communication parameters and the format of the data. The UART uses a fixed data format of 8 data bits, no parity bit and one Electronics 2019,8, 528 9 of 28 stop bit (8-N-1), according to the NMEA-0183 specification. Although this specification establishes a communication speed of 4800 bps, many GPS receivers in the market support higher baud rates to be able to transfer several frames in a single second. For this reason, the UART baud rate can be configured through the uart_conf bus coming from the P&CI (see Figure 5). A 4-bit code determines the baud rate according to Table 2. Table 2. Universal asynchronous receiver-transmitter’s (UART’s) baud rate configuration parameter. Decimal Value Baud Rate (bps) 0 4800 1 9600 2 19,200 3 38,400 4 57,600 5 115,200 6–15 4800 As soon as the GPS receiver transmits new data to the UART, they are stored in its internal 16-byte first-in, first-out (FIFO) queue and made available to the time reception module, which controls the UART’s communications. The internal FIFOs are implemented using FPGAs logic blocks since their sizes are relatively small. 5.2. Time Reception Module The mission of the time reception module is to collect information from the GPS receiver and transform it into time data and control signals that can be used by the synchronization module to maintain an accurate time in the local clock. The time reception module has two main blocks depicted in Figure 6: (i) Processing unit; and (ii) time format converter. The module operates as follows: the GPS receiver signals the starting of a new second with an active edge of the PPS signal. Following the edge, the GPS receiver emits a set of NMEA-0183 messages encoded in plain ASCII text. The message that is relevant for time synchronization applications is the recommended minimum sentence C (RMC), which covers date and time corresponding to the active edge of the PPS signal. The time reception module operation is triggered by the active edge of the PPS signal. Furthermore, a 1-bit configuration parameter included within the trm_conf bus indicates whether the expected active edge is a rising (1) or a falling (0) edge. Next, the time reception module starts receiving data from the UART, identifies the RMC frame and extracts time, GPS status and date information that is encoded in frame fields 1, 2 and 7, respectively. Figure 7shows a sample RMC frame with decoded fields. After processing the RMC frame, date and time information is converted into NTP time format (seconds 00:00, 1 January 1900) and handed to the synchronization module through the 32-bit signal gps_time. Both the status of the GPS (active or not active) and the status of time conversion (correct or not correct) are also made available to the synchronization module through output signals gps_st and time_conv_st, respectively. Figure 6. Time reception module block diagram. Electronics 2019,8, 528 16 of 28 • If the incoming packet is an NTP request from a client then the MAC address and IP address of the client are both extracted from the packet together with the origin timestamp T1 . They are delivered to the packet builder block through the packet_data bus, so that the appropriate reply can be built later. • If the incoming packet is an ARP request then the MAC address and IP address of the sender are both extracted and placed on the packet_data bus, so that the packet builder can store it with the aim to build the corresponding reply. • If the incoming packet is a BOOTP reply then it is processed to extract the IP configuration of the server (IP address, network mask and default router) and the filename string field of the BOOTP reply. All these pieces are made available to the configuration block through the conf_data bus. The reception block has been implemented by an FSM coded in HDL. The FSM reads the bytes in the incoming packet and distributes the information in a set of registers which drive the data outputs of the block. Once the incoming packet is processed, the status of the reception block is updated and the P&CI control unit is signaled through the recept_st signal so that appropriate actions can be taken. 6.3. Configuration Block and Timers The configuration block centralizes and stores the various bits of information required by the P&CI itself and by the other modules in the system. The parameters administered by the configuration block are shown in Table 3together with the bus used to deliver each parameter to its destination. The parameters obtained by the BOOTP configuration process are called dynamic and are loaded at system startup. Nonetheless, they can be updated at run time by a new BOOTP request. The parameters that are hard-coded in the design’s programming file (bitstream) are called static and contain fixed values that cannot be altered after the FPGA chip has been programmed. The MAC address and the IP configuration parameters are made available to the packet builder in the pb_conf bus. Similarly, parameters for the UART, time reception and synchronization modules are transferred through their respective buses, as already described in previous sections. Table 3. Configuration parameters stored in the configuration block. Parameter Type Bus Server’s MAC address static pb_conf Internet protocol (IP) address dynamic pb_conf IP mask dynamic pb_conf IP gateway dynamic pb_conf Baud rate dynamic uart_conf Pulse-per-second (PPS) active edge dynamic trm_conf Prescaler factor static sync_conf Convergence factors static sync_conf Initial Dstatic sync_conf System clock frequency static t_conf Bootstrap protocol (BOOTP) timeout static t_conf The baud rate of the UART and the PPS active edge of the GPS receiver are dynamic parameters encoded in the text field filename of the BOOTP reply that are used by the reception block. The first character of the field represents the baud rate according to Table 2, whereas the second character is “0” for a falling PPS active edge and “1” for the opposite. This way of encoding some of the internal parameters of the NTP server makes it easy to alter their values by editing the BOOTP configuration file in the BOOTP server. Figure 14 displays a sample self-explicative configuration file for the internet systems consortium (ISC) DHCP/BOOTP server [ 45 ]. In the example, ntp-server-1 has been configured with an UART baud rate of 19, 200 bps and a rising PPS active edge, whereas ntp-server-2 uses a baud rate of 9600 bps and a falling PPS active edge. Electronics 2019,8, 528 17 of 28 Electronics 2019,xx, 5 17 of 28 subnet 192.168.4.0 netmask 255.255.255.0 { option routers 192.168.4.1; allow bootp; host ntp-server-1 { hardware ethernet 40:8d:1a:2b:3c:4d; fixed-address 192.168.4.10; filename "21"; // 19200bps (2), rising edge (1) } host ntp-server-2 { hardware ethernet 40:8d:1a:b2:c3:d4; fixed-address 192.168.4.20; filename "10"; // 9600bps (1), falling edge (0) } } Figure 14. Sample DHCP server configuration. ISC DHCP server format. The P&CI uses a timer to control the sending of BOOTP requests. The timers block is configured with the system’s clock frequency in Hertz and a timeout number of seconds given by the BOOTP timeout parameter. Timers’ configuration goes in the t_conf bus. The timers block can be extended to support additional timers, with timeout outputs bundled in the t_out signal connected to the P&CI control unit (Figure 13). The configuration block has been completely described in HDL by using a main FSM that manages the processing of data coming from the reception module and the distribution to the output registers. A secondary FSM is used to decode and check for possible errors in the filename field, as well as assign the resulting binary values to the trm_conf and uart_conf buses. Static parameters are internally stored in BRAM and can be directly modified in the bitstream file, without having to run the whole implementation process again, by using the Xilinx tool data2mem [ 46 ]. This tool is especially useful not only to change the server’s MAC address, which needs to be unique for every programmed chip, but also either for testing different configurations or for adapting the implemented design to a different master clock frequency. During the initialization of the block, the parameters stored in the BRAM are moved to conventional registers that drive the corresponding output signals. The timers block has been designed by using regular counters from the Xilinx System Generator tool library. 6.4. Packet Builder, Packet RAM and Transmission Block The packet builder is in charge of building the different types of outgoing packets the NTP server core needs to support: BOOTP requests, NTP replies, ARP requests and ARP replies. The packet builder is supported by a packet RAM that acts as a workspace for the packet builder. It holds a template of an Ethernet frame for every packet type needed. The templates are located at known addresses within the packet RAM, conforming to the memory map in Table 4. When the control unit requests building a particular type of packet through the pb_ctl signal, the packet builder gathers the needed information from its input signals or internal registers, and completes the packet fields in the appropriate template within the packet RAM. If necessary, checksums are also calculated and stored, like in the transmission of UDP packets. Once the needed packet is built, the control unit is informed through the status signals in the pb_st bus. Table 4. Packet RAM memory map. Packet Type Address Range (hex.) BOOTP request 00–55 NTP reply 56–6C ARP request 6D–7B ARP reply 7C–8A Figure 14. Sample dynamic host configuration protocol (DHCP) server configuration. Internet systems consortium (ISC) DHCP server format. The P&CI uses a timer to control the sending of BOOTP requests. The timers block is configured with the system’s clock frequency in Hertz and a timeout number of seconds given by the BOOTP timeout parameter. Timers’ configuration goes in the t_conf bus. The timers block can be extended to support additional timers, with timeout outputs bundled in the t_out signal connected to the P&CI control unit (Figure 13). The configuration block has been completely described in HDL by using a main FSM that manages the processing of data coming from the reception module and the distribution to the output registers. A secondary FSM is used to decode and check for possible errors in the filename field, as well as assign the resulting binary values to the trm_conf and uart_conf buses. Static parameters are internally stored in BRAM and can be directly modified in the bitstream file, without having to run the whole implementation process again, by using the Xilinx tool data2mem [ 46 ]. This tool is especially useful not only to change the server’s MAC address, which needs to be unique for every programmed chip, but also either for testing different configurations or for adapting the implemented design to a different master clock frequency. During the initialization of the block, the parameters stored in the BRAM are moved to conventional registers that drive the corresponding output signals. The timers block has been designed by using regular counters from the Xilinx System Generator tool library. 6.4. Packet Builder, Packet RAM and Transmission Block The packet builder is in charge of building the different types of outgoing packets the NTP server core needs to support: BOOTP requests, NTP replies, ARP requests and ARP replies. The packet builder is supported by a packet RAM that acts as a workspace for the packet builder. It holds a template of an Ethernet frame for every packet type needed. The templates are located at known addresses within the packet RAM, conforming to the memory map in Table 4. When the control unit requests building a particular type of packet through the pb_ctl signal, the packet builder gathers the needed information from its input signals or internal registers, and completes the packet fields in the appropriate template within the packet RAM. If necessary, checksums are also calculated and stored, such as in the transmission of UDP packets. Once the needed packet is built, the control unit is informed through the status signals in the pb_st bus. The packet builder is also responsible both for collecting the origin timestamp ( T1 ) of incoming NTP requests processed by the reception block and for issuing the corresponding receive timestamp (T2) by using the local_time signal from the synchronization module. These timestamps are stored in the packet builder internal registers to be used to build the NTP reply to the last NTP request. Electronics 2019,8, 528 18 of 28 Table 4. Packet RAM memory map. Packet Type Address Range (hex.) BOOTP request 00–55 NTP reply 56–6C Address resolution protocol (ARP) request 6D–7B ARP reply 7C–8A The packet building process is similar for all the types of packets. BOOTP request packets are directed to the local broadcast address carrying the MAC address of the server with it. NTP reply packets are built with the information of the corresponding request previously stored (origin and receive timestamps) together with the local time and local clock status coming from the synchronization module. The transmit timestamp ( T3 ) is obtained from the local_time signal. The stratum of the server is set in terms of the sync_st signal. If the signal value is 0, then the GPS receiver time is not valid or the synchronization with the GPS is not correct. In this case, the stratum of the server is set to 0, otherwise it is set to 1, since it is a primary server connected to a GPS reference. The server_st bus is used to fill in the precision field of the NTP reply. Other NTP fields are completed following the NTP specification [5]. In a similar way, ARP replies use the previously stored sender’s MAC and IP addresses and complete the data with the server’s MAC and IP addresses and other information required by the ARP protocol [ 47 ]. The packet builder and RAM also have provisions to build ARP request packets, but this functionality is not used in the current version of the server, since the only communication initiated at the server are BOOTP requests that are issued to the LAN’s broadcast address. Once an outgoing packet has been built, the control unit activates the transmission block and indicates the type of packet to be transmitted by means of the trans_ctl bus. The transmission block then reads the packet previously produced in the packet RAM from the address corresponding to the type of transmission and it passes the data to the output FIFO queue of the MAC controller. The transmission block also detects any condition of the MAC controller that would require a new packet to be built, e.g., a collision in the MAC level that would need to rebuild the NTP reply with a refreshed transmit timestamp. Both the packet building process and the data transmission to the MAC are controlled by FSMs described in HDL. The packet RAM is built out of BRAM memory available in the FPGA chip. BRAM has the advantage of being a synchronous memory, therefore contributing to have a fixed transmission delay time, which is important for the stability of the NTP time synchronization. 6.5. P&CI Control Unit and Operation The role of the control unit is to coordinate the operation of the rest of the blocks that form the P&CI module. This task is relatively simple because, as stated in the previous sections, all communication and configuration tasks, including NTP timestamping, are completely handled by the blocks in the module in a very independent way, only requiring some coordination. The time server service can be given as an example: when an NTP request packet arrives to the reception module, the packet is fully processed before the control unit is signaled. If the reception has been correct then the control unit only needs to activate the packet builder as well as the transmission block in turn, and the NTP reply is automatically crafted and sent out. The algorithm to control this service can be described by the pseudo-code in Figure 15, which can be easily implemented by an FSM. Analogous simple algorithms are used to coordinate the configuration service and the address resolution service that have to do with BOOTP packets in the first case and ARP packets in the second case. In practice, the main challenge for the P&CI control unit is to run the three services in parallel in a coordinated way, avoiding any conflicts among them. For instance, the configuration service may need to issue a BOOTP request, while the time server service is replying to an NTP request. To solve Electronics 2019,8, 528 19 of 28 this problem, the P&CI control unit uses separate FSMs to control each service. The access of these services to the underlying blocks is controlled by a service arbiter that only grants access to one service at a time. Figure 16 shows the overall architecture of the arbitration system. Each service block is connected to the status signals of the processing blocks in the module and can actuate on their control signals as well, but only when access has been granted by the service arbiter. Electronics 2019,xx, 5 19 of 28 wait recept_st if recept_st == INCOMING_NTP_REQUEST pb_ctl = BUILD_NTP_REPLY wait pb_st if pb_st == BUILD_OK trans_ctl = SEND_NTP_PACKET wait trans_st if trans_st != SEND_OK handle_transmission_error() else handle_pb_error() Figure 15. Time server service control algorithm. Error handling intentionally omitted for simplicity. In practice, the main challenge for the P&CI control unit is to run the three services in parallel in a coordinated way, avoiding any conflicts among them. For instance, the configuration service may need to issue a BOOTP request, while the time server service is replying to an NTP request. To solve this problem, the P&CI control unit uses separate FSMs to control each service. The access of these services to the underlying blocks is controlled by a service arbiter that only grants access to one service at a time. Figure 16 shows the overall architecture of the arbitration system. Each service block is connected to the status signals of the processing blocks in the module and can actuate on their control signals as well, but only when access has been granted by the service arbiter. Figure 16. Protocol and configuration interface control unit service architecture. The handshake mechanism between services and the arbiter is very simple: if Service i wants to access the processing blocks then it will assert the request signal rq_i and will wait for signal gt_i to be asserted by the arbiter granting the requested access. When the service block finishes its tasks, it de-asserts the request signal and the arbiter may grant access to another service. Collision of services (two or more services requesting access at the same time) during the normal operation of the system is very unlikely to happen, since NTP and ARP requests will normally be replied intermediately and the BOOTP service only needs to be run once every several seconds. Therefore, the arbiter follows a simple fixed priority algorithm with the highest priority given to the time server service, as it is the most time-critical service, and the lowest priority assigned to the BOOTP service. Service blocks are expected to remain inactive until triggered by the event’s triggering signal trig_i. Triggering signals for the time server and ARP services come from the bits of the recept_st bus that indicate the arrival of an NTP or ARP request, respectively. Hence, these services act as server services because they are triggered by a request packet coming from the outside. On the contrary, the triggering signal for the BOOTP service comes from the timers block output bus t_out, causing the BOOTP service to be activated every time the BOOTP timer expires. As mentioned in Section 6.3, the BOOTP timer can be configured through the static parameter BOOTP timeout, with reasonable values ranging from one to several minutes. The BOOTP service plays the role of a client service since it is initiated within the P&CI itself. Figure 15. Time server service control algorithm. Error handling intentionally omitted for simplicity. Figure 16. Protocol and configuration interface control unit service architecture. The handshake mechanism between services and the arbiter is very simple: if Service iwants to access the processing blocks, then it will assert the request signal rq_i and will wait for signal gt_i to be asserted by the arbiter granting the requested access. When the service block finishes its tasks, it de-asserts the request signal and the arbiter may grant access to another service. Collision of services (two or more services requesting access at the same time) during the normal operation of the system is very unlikely to happen, since NTP and ARP requests will normally be replied intermediately and the BOOTP service only needs to be run once every several seconds. Therefore, the arbiter follows a simple fixed priority algorithm with the highest priority given to the time server service, as it is the most time-critical service, and the lowest priority assigned to the BOOTP service. Service blocks are expected to remain inactive until triggered by the event’s triggering signal trig_i. Triggering signals for the time server and ARP services come from the bits of the recept_st bus that indicate the arrival of an NTP or ARP request, respectively. Hence, these services act as server services because they are triggered by a request packet coming from the outside. On the contrary, the triggering signal for the BOOTP service comes from the timers block output bus t_out, causing the BOOTP service to be activated every time the BOOTP timer expires. As mentioned in Section 6.3, the BOOTP timer can be configured through the static parameter BOOTP timeout, with reasonable values ranging from one to several minutes. The BOOTP service plays the role of a client service since it is initiated within the P&CI itself. It is worth mentioning that the proposed service architecture allows for the easy inclusion of additional client and/or server services in future revisions of the systems. An additional server service Electronics 2019,8, 528 20 of 28 might provide time synchronization utilizing a different protocol, otherwise a new client service could broadcast state information. Obviously, adding additional services is likely to require additional functionality in the P&CI or the whole system. All services together with the arbiter have been implemented in HDL through a standard FSM description technique. Furthermore, they have been validated by simulation in the Xilinx’s design framework. 7. Results and Discussion To validate the proposed design and to estimate its performance, different aspects of the system have been evaluated: (i) Resources used by the implementation; (ii) Accuracy of the NTP core’s local clock with respect to time reference; (iii) Client-side precision measured by a standard NTP client; And (iv) performance of the server core operating under different load conditions. All accuracy and performance tests have also been carried out, side-by-side on a state-of-the-art high-performance NTP server: a LANTIME M600 from Meinberg [48]. The results are detailed in the following subsections. 7.1. Implementation Results The NTP server core has been implemented on a XC3S500E chip: a low-range, now obsoleted, Xilinx FPGA device, with a master clock working at 50 MHz. The use of an old device helps to highlight the efficiency of the design and has also been useful to track the improvements with respect to the earlier designs developed by the authors in the same platform. Porting to newer chips and even a different FPGA vendor is simple and is planned as future work. Table 5summarizes the implementation results. The NTP core uses less than 50% of the chip’s resources, except for the number of slices, which is higher. Table 6displays the number and percentage of hardware resources per module. Slices, flip-flops, Look-Up Tables (LUTs) and BRAM are shown separately. As expected, the P&CI uses more resources than the time reception module (TR mod. in Table 6) and the synchronization module (Sync. mod. in Table 6). The share of the MAC controller is also important while the impact of the UART is negligible. Less than 1% of the available slices and LUTs are consumed by miscellaneous logic (Misc. in the figure) including glue logic, start-up circuitry, as well as top-level routing and interconnections. Considering that the device is a low-range FPGA chip, it can be stated that the hardware footprint is reasonably low and that the overall resources occupation can easily go below 10% in any mid-range state-of-the-art FPGA chip. Table 5. Hardware NTP server resources utilization (Field programmable gate array (FPGA) Spartan-3E XC3S500E chip). Resource No. % Slices 3606 77 Slice Flip Flops 2927 31 4-input look-up table (LUT) 3749 40 RAMB16 5 25 Table 6. Number of internal resources per module. Module Slices Flip Flops LUT’s BRAM’s No. % No. % No. % No. % Universal asynchronous receiver-transmitter (UART) 43 1.2 39 1.3 91 2.4 0 0 Time reception module (TR Mod.) 272 7.5 195 6.7 496 13.2 1 20 Protocol and configuration interface (P&CI) 1488 41.3 1288 44.0 1016 27.1 2 40 Synchronization module (Sync. mod.) 931 25.8 589 20.1 1307 34.9 0 0 MAC cont. 865 24.0 807 27.6 839 22.4 2 40 Miscellaneous logic (Misc.) 7 0.2 9 0.3 0 0 0 0 Total 3606 100 2927 100 3749 100 5 100 Electronics 2019,8, 528 21 of 28 7.2. Synchronization Accuracy to Time Reference To measure the accuracy of the timekeeping subsystem, the PPS signal generated by the NTP core is compared to the PPS signal generated by a GPS receiver. The PPS of the LANTIME M600 high-performance NTP server (NTP server in the following) is also included as a reference. The measurement procedure takes place as follows: the NTP core and NTP server are run until synchronization is established and the offset value with respect to the GPS PPS signal is stable. Then, 1000 consecutive offset values are measured at the active edge of the GPS receiver’s PPS signal for both systems using a digital oscilloscope, which automatically computes the mean and standard deviation (SD) of the measurements (see Figure 17). As the results may vary depending on the state of the GPS constellation, the process is repeated four times at different times of the day and results are averaged (see Table 7). Figure 17. Sample PPS accuracy measurement. PPS signal edge differences between the GPS (top), core (middle) and reference server (bottom) are measured several times using a digital oscilloscope. Statistics are automatically calculated by the instrument. Table 7. Accuracy to the reference GPS PPS signal in nanoseconds. Test NTP Server NTP Core Mean Standard deviation (SD) Mean SD 1 104 42 −47 233 2 127 37 −32 227 3 114 36 −32 214 4 114 34 −37 216 Average 115 37 −37 223 Although the average measured offset of the NTP core is better than the NTP server’s ( − 37 ns vs. 115 ns), it gives a larger dispersion represented by the SD (223 ns vs. 37 ns), which is of the same order as that of the maximum internal built-in resolution of the local clock (238 ns), as discussed in Section 5.3 above. It can be inferred that the techniques and timekeeping algorithms used in the synchronization module are performing extremely well, being only limited by the time resolution of the local clock. As a result, the NTP core’s local clock can easily maintain an accuracy below the microsecond, in the same range as the high-performance NTP server. 7.3. Client-Side Accuracy The setup in Figure 18 has been used with the aim of measuring the achievable precision of a standard NTP client connected to the NTP core, as well as to compare it with a high-performance NTP server. A standard NTP software client version 4.2.6 [ 49 ] running in a desktop computer is synchronized with the high-performance NTP server and the NTP server core. Both are connected Electronics 2019,8, 528 22 of 28 to the same LAN switch as the testing software client. The traffic generator in the picture is used to obtain additional results detailed below. Figure 18. Testing set-up showing connections between NTP clients and servers. When synchronized with a server, the NTP client will continuously calculate the delay, offset and jitter associated to the server, as introduced in Section 3. These parameters, especially the offset and jitter, are good estimators to compare different servers performance in terms of the client’s perspective. An equal procedure has been developed to test both servers: the NTP client is set to use a standard 64 s polling interval and delay, offset and jitter values are read every hour for nine hours by the standard NTP query program (ntpq). Table 8shows the obtained results. After four hours, the values calculated by the client have stabilized (NTP clients are very conservative and they synchronize very slowly), and mean and SD statistics have been calculated through the values registered from 4 h to 9 h. These statistics are shown in Table 9. The delay of the NTP core is about half of the NTP server (97.8 µ s vs. 191 µ s), that is to say, keeping other conditions the same, the packet processing of the NTP core is faster than that of the NTP server, as expected from a hardware implementation. Although a faster response does not imply better accuracy, the smaller dispersion of the delay represented by the SD highlights the fact that NTP core responses happen in a more predictable way. Table 8. Delay, offset and jitter evolution with time (µs). Time (h) Delay Offset Jitter NTP Server NTP Core NTP Server NTP Core NTP Server NTP Core 0 251 98 2849 −6240 1086 1166 1 194 96 630 −303 116 56 2 233 102 40 −103 29 10 3 206 95 −5−20 12 6 4 176 103 1 −2 13 4 5 225 100 6 2 22 3 6 175 101 −4−12 12 2 7 209 101 −9 1 18 3 8 182 89 11 −13 7 4 9 179 93 0 −18 14 5 Electronics 2019,8, 528 23 of 28 Table 9. Accuracy parameters statistics (µs). NTP Server NTP Core Mean SD Mean SD Delay 191.0 20.9 97.8 5.5 Offset 0.8 7.1 −7.0 8.4 Jitter 14.3 5.2 3.5 1.0 Offset and jitter values represent accuracy and stability of the server conforming to the client’s point of view. Figure 19 depicts these values with error bars representing a 95% confidence interval (1.96 ×SD ). Offset confidence intervals are very similar for both, the NTP server and the NTP core, while the jitter interval of the core is much more narrow than that of the NTP server. It means that the client obtains similar accuracy when synchronized with either server in the band of ± 10 µ s, as expected in a software client [ 17 ]. However, the offset is more stable in the short term, when the client is synchronized with the NTP core, as the lower jitter highlights. Then, it can be concluded that accuracy and stability of the NTP core, as perceived by a standard NTP client, is at least comparable to that of a high-performance NTP server. The offset value is also in agreement with case 3 mentioned in [ 20 ] where hardware-assisted PTP client and server are connected through a regular switch and the client-server offset is measured with an oscilloscope giving an offset of 3.7 µ s . Since the measurements of the NTP core are obtained by a less accurate software-only NTP client, it can be said that the accuracy of the core is also comparable to previous SoC approaches when using standard network equipment. Figure 19. Client statistics for offset and jitter. 7.4. Performance under Load A key performance figure in an NTP server is the amount of NTP traffic that it can manage before NTP request packets start being missed and then, synchronization accuracy is affected. The response of the high-performance NTP server and the NTP core have been analyzed for different NTP network loads, ranging from 32 to 10, 000 NTP requests per second (rq/s), which is the maximum load most manufacturers report for their NTP server products [ 8 – 10 ]. At a typical poll interval of 64 s, it is equivalent to serving more than half a million NTP clients. The test scenario is the same as the one shown in Figure 18, although this time the NTP traffic generator is enabled. The generator uses the tcpreplay tool [ 50 ] to inject NTP network traffic in the LAN at a controlled data rate. It should be noticed that the LAN bandwidth used at the maximum NTP load tested is below 20 Mbps and it is far from the full-duplex 200 Mbps bandwidth of the standard switched 100 Mbps Ethernet LAN used in the setup. Furthermore, this time the poll interval in the client is set to 16 s in order to speed-up the data acquisition process. Both packet loss and load impact on servers accuracy, have been analyzed. Electronics 2019,8, 528 24 of 28 7.4.1. Packet Loss Figure 20 summarizes the percentage of missed NTP responses (requests without an answer from the server) as a consequence of an increasing NTP network load. For every server load tested, the count of lost NTP packets is collected from the ntpq program, after making 750 requests in a time span of 200 min. The high-performance NTP server starts missing some NTP requests at about 1000 rq/s load. At 3000 rq/s, the percentage of lost requests increases sensibly to 3%, reaching a maximum around 12% for 6000 rq/s or more, up to the maximum tested load of 10, 000 rq/s. In contrast, no packet loss has been registered in the NTP core at any of the tested server load conditions. Figure 20. NTP servers missed requests percentage vs. network load. 7.4.2. Load Impact on Server Accuracy Equally as done in Section 7.3, 100 consecutive samples of delay, offset and jitter values have been obtained at one-minute intervals using the ntpq program. In addition, mean values and SDs for each parameter have been calculated. The process has been repeated for different NTP loading conditions and results are shown in Figures 21–23. Each graph includes error bars indicating 95% confidence intervals. The synchronization parameters registered by the client when synchronized to the high-performance NTP server degrade as the server is under a higher load. Delay becomes less predictable, as Figure 21 represents, and the offset value increases from about 5 µs, for very light load, to about 20 µ s, for a server load of 3000 rq/s or higher (Figure 22). The fact that the offset confidence interval spreads from about 20 µ s to more than 60 µ s as the load increases is even more noticeable. A similar behavior is observed in the jitter estimated by the client when synchronized with the high-performance NTP server. However, the most remarkable point is that the NTP core yields better results than the high-performance server in all parameters and tested conditions, and it is not significantly affected by the load in the tested range. In particular, the offset is always around ± 10 µ s range and the jitter is mostly between 5 µs to 15 µs interval (not appreciated in Figure 23). The authors assume that the ability of the NTP core to provide clients with similar accuracy regardless of the loading conditions is a direct consequence of decoupling between the timekeeping subsystem and the communication subsystem present in the core’s architecture, as commented in Section 4. Electronics 2019,8, 528 25 of 28 Figure 21. Delay vs. server load. Figure 22. Offset vs. server load. Figure 23. Jitter vs. server load.