Upgrading SPHERE with the second stage AO system SAXO+: Hardware interface integration

Upgrading SPHERE with the second stage AO system SAXO+: Hardware interface integration P. Fontanillasa, F. Ferreiraa, A. Sevina, D. Gratadoura, G. Chauving, E. Diolaitib, R. Grattonf, M. Loupiasc, J. Millid, F. Wildie, A. Boccalettia, and P. Vermota aLIRA, Observatoire de Paris, Universit´e PSL, CNRS, Sorbonne Universit´e, Universit´e Paris Cit´e, 5 place Jules Janssen, 92195 Meudon, France bINAF - Osservatorio di Astrofisica e Scienza dello Spazio (OAS), 40129 Bologna, Italy cUniv. Lyon, Univ. Lyon1, ENS de Lyon, CNRS, Centre de recherche Astrophysique de Lyon UMR5574, F-69230, Saint-Genis-Laval, France dUniversit´e Joseph Fourier-Grenoble 1 /CNRS-INSU, Institut de Plan´etologie et d’Astrophysique de Grenoble (IPAG), Saint Martin d’Heres, France eObservatoire de Gen`eve, CH-1290 Sauverny, Switzerland fINAF, Osservatorio Astronomico di Padova (OAPd), 35122 Padova, Italy gLaboratoire Lagrange, Universit´e Cote d’Azur, CNRS, Observatoire de la Cote d’Azur, 06304 Nice, France ABSTRACT SAXO+, the upgrade of the SPHERE adaptive optics (AO) system, is about to enter its Assembly, Integration, and Testing (AIT) phase following the Final Design Review (FDR) at ESO in May 2025, with first light scheduled for 2028 on the VLT. The SAXO+ project aims to enhance the performance of SPHERE, particularly in terms of contrast, by implementing a second-stage adaptive optics system that will operate in conjunction with the existing first-stage system (SAXO). The Real-Time Controller (RTC) is a key component of the SAXO+ system. It is based on the COSMIC platform for the hard real-time layer and on the ESO RTC Toolkit for the soft real-time layer. The SAXO+ RTC has been designed to be flexible and modular, supporting the implementation of various control strategies, including techniques based on artificial intelligence (AI). This second stage will feature a pyramid wavefront sensor (PWFS) operating at high frame rates—up to 3 kHz—, a new deformable mirror (DM) with 952 actuators and a sFPDP link to ensure coordination with the first stage. The integration of these interfaces remains a significant challenge, requiring careful coordination between hardware timing, data formats and real-time processing constraints to guarantee overall system stability and performance. In this paper, we present the SAXO+ RTC software development current status and the hardware interface integration. We also present preliminary performance results of the RTC, including latency and jitter measurements for sFPDP and camera acquisition. Finally, we discuss the next steps in the development of the SAXO+ RTC, focusing on system integration and testing during the project’s AIT phase. Keywords: Adaptive optics, SPHERE+, SAXO+, VLT, RTC, COSMIC, sFPDP, Camera Link Further authors information: (Send correspondence to P. Fontanillas or F. Ferreira) P. Fontanillas: paul.fontanillas at obspm.fr F. Ferreira: florian.ferreira at obspm.fr, +33 (0)1 45 07 71 78 1. INTRODUCTION Adaptive optics (AO) systems are a cornerstone technology for high-contrast imaging on ground-based telescopes. SPHERE1has delivered remarkable results in the direct imaging and characterisation of exoplanets and circumstellar disks. SAXO+2is conceived as a second-stage AO upgrade for SPHERE that will extend and amplify these capabilities: by adding a high-order, fast-response corrective stage operating downstream of the existing SAXO system, SAXO+ aims to improve the instrument sensitivity at smaller angular separations. Figure 1: Schematic representation of the SAXO+ control system As showns fig. 1the SAXO+ second stage complements the first-stage correction performed by SAXO. It integrates a 34 by 34 actuators Boston Micromachine deformable mirror (DM) and a C-RED One camera running at 3kHz. The second-stage architecture targets residual aberrations left by SAXO, enabling deeper contrast in the final science image. The Real-Time Controller (RTC) is central to SAXO+’s operation. The SAXO+ RTC will control only the second stage, while the Standard Platform for Adaptive Optics Real-time Application (SPARTA3) will continue to be in charge of the first stage. SPARTA and the SAXO+ RTC will be synchronized via a sFPDP link, running at 1kHz. The RTC is implemented as a two-layer system: a hard real-time computer (H-RTC) based on the COSMIC4platform to guarantee deterministic, low-latency processing, and a soft real-time computer (S-RTC) built on the ESO RTC Toolkit to provide monitoring, supervisory functions and flexible experiment orchestration. This design aims to give SAXO+ the low-jitter performance required for kHz-class operations while remaining modular enough to host multiple control strategies. To fully exploit the capabilities of the hardware, SAXO+ will support a suite of six control laws that span classical and modern approaches, with the ability to compare them in operation. The six laws are:  Integrator with Optimized Modal Gain Integrator (OMGI)  Inverse problem approaches5  LQG controller6  Data-driven controller7  PO4AO controller8  Dual-Stage Supervised and Reinforcement Learning controller9 This paper provides an update on the RTC software development status and focuses on the hardware interface integration of the SAXO+ RTC. Firstly, in Sec.2we describe the software development current situation. Secondly, we present the sFPDP and camera interfaces integration and their performance results in Sec.3and Sec.4respectively. Finally, in Sec.5we will introduce a GPU pipeline test validating the entire RTC loop. 2. SOFTWARE DEVELOPMENT CURRENT STATUS This section aims to provide an overwiew of the current RTC architecture. 2.1 Overview On one side, the SAXO+ Hard Real-Time Controller (H-RTC) handles critical real-time tasks with stringent timing requirements. Since some of the SAXO+ control methods rely on artificial intelligence, which is highly demanding in terms of bandwidth, the H-RTC has been designed to use GPU as its computing unit. The H-RTC consists of two NVIDIA H100 GPUs, each providing 3 TB/s memory bandwidth while available CPUs only offer 400 GB/s. SAXO+ H-RTC is built upon the COSMIC plateform, that guarantees deterministic performance and low-latency processing. On the other side, the SAXO+ Soft Real-Time Controller (S-RTC) manages non-critical real-time tasks, including system monitoring, supervisory control, and non-critical computation. The S-RTC is split into 4 components:  the H-RTC Gateway: the main interface between the H-RTC and the S-RTC, responsible for data exchange and command relay  the Compute Node: responsible for executing non-critical Datatasks and offloading work from the H-RTC  the S-RTC Gateway: the main interface between the S-RTC and external systems, responsible for data exchange and command relay  the Storage Node: responsible for data storage and retrieval, ensuring that all relevant data is accessible to the S-RTC In order to reduce the overall cost and increase maintainability, the Storage Node and the S-RTC Gateway will be hosted on the same physical machine. Figure 2: RTC communication Figure 2highlights interactions between H-RTC and S-RTC components. S-RTC servers communicate through Data Distribution Service (DDS) topics, using DDS publishers and subscribers. SPHERE INS can interact with the RTC system through the S-RTC Gateway. The S-RTC is built upon the ESO RTC Toolkit, a robust framework designed for the development of real-time controllers. It interfaces with the H-RTC to ensure seamless operations, to retrieve and store telemetry. It is composed of several types of components:  RTC Supervisor: provides an entry point for high-level control, monitoring of the RTC system, error recovery and is used to access the Runtime Configuration Repository (RTR). There is only one RTC supervisor in SAXO+ S-RTC.  Telemetry Republisher: reads telemetry data in MUDPI format from the HRTC, and republishes it to one or more S-RTC nodes, using DDS reliable Multicast. There are three Telemetry Republisher components: one handling raw data, another one treating calibrated data and a last one managing loop data, which will be specific to each control law.  Telemetry Subscriber: receives data from one or more Telemetry Republishers and write them into a shared memory, to then be consummed by datatasks. As for republishers, there are three subscribers: one for raw data, one for calibrated data and one for loop data.  Telemetry Recorder: reads telemetry data from specific shared memory queues and store it into FITS files. Recorders can also read data from other sources and store them into FITS or CSV files. There are two recorders in SAXO+ S-RTC: one to store pixels telemetry data and another one to store loop telemetry data.  Datatasks: responsible for various non-critical computations and data processing tasks that support the overall operation of the RTC system. They receive telemetry and RTR data. Datatasks results are stored in the RTR after processing. There are ten Datatasks for the integrator with OMGI control law and more to come for the other control laws. H-RTC and S-RTC architectures and behaviour have already been described in detail in Ref.10 We will focus here on the current development status. 2.2 Current status The development of SAXO+ RTC is well underway. The three S-RTC servers are operational and in use. Each of them is composed of two AMD EPYC 9354 processors each containing 32 physical hearts. For H-RTC, a call for tenders is scheduled for early 2026. A H-RTC prototype composed of two NVIDIA A100 GPUs is curently in use. The operating mode involving the integrator with OMGI control law and associated tasks has been defined as the baseline; it is the implementation priority. The other controllers and their associated Datatasks will be developed at a later stage. Recent developments have been made on the S-RTC side. Most of the baseline Datatasks have been implemented, including:  Interaction matrix Calibration: calibrates the interaction matrix, linking each adaptive mirror actuator to the Pyr-WFS response.  Command matrix computation: generates the command matrix from the interaction matrix.  Reference slopes computation: calibrates the Pyr-WFS reference slopes using illuminated and dark images.  Modal gain computation: computes modal gains based on measured turbulence parameters.  Detector Calibration: measures detector background and flat, corrects dead pixels, and calibrates the valid pixel map.  ROI Calibration: defines and calibrates the Region Of Interest (ROI) of the camera sensor (128x128 pixels square).  Pupils Position: monitors the stability of the 4 pupils in the camera image.  Tip-Tilt Offloader: extracts from the deformable mirror commands the averaged tip, tilt and defocus. These datatasks, as the other S-RTC components are designed independently, to ensure modularity and ease of maintenance. As shown in fig. 3, components communicate through shared memory topics and a runtime repository. In the S-RTC architecture, the datatasks lifecycle and data-flow orchestration are implemented in C++. The algorithm kernels are developed in Python and invoked from the C++ side. This separation combines the performance and predictability of C++ for hot paths and system integration with the flexibility and rapid prototyping afforded by Python for algorithm development. Therefore, each datatask includes two kinds of unit tests:  One Python test based on Pytest (pytest): checks the Python algorithm correctness, the numerical accuracy of the output, the handling of edge cases and exceptions.  One C++ test based on Google Test and Google Mock (gtest): checks the datatask computation behaviour, it verifies that the Python algorithm is correctly called via a Python wrapper and then it checks if the results are well saved. The gtest also test if the warning and error messages are well triggered. Figure 3: Datatask testing architecture This dual testing strategy ensures that both the algorithmic integrity and the component computation behaviour are rigorously validated. A functional test pipeline using COMPASS simulator has been implemented on the H-RTC. This test strategy will be presented in Sec. 6. In addition of the RTC baseline development, the other priority is to integrate the hardware interfaces enabling communication with the other components of the AO system. 3. sFPDP INTERFACE INTEGRATION In this section we will describe the sFPDP interface integration into the H-RTC and its performance results. 3.1 Setup As the SAXO+ RTC needs to communicate with the first stage, an interface needs to be implemented. For non-disruptive integration, the link between the two stages will be made using the existing Serial Front Panel Data Port (sFPDP) switch embedded in SPARTA. The sFPDP protocol is a high-speed communication standard optimized for real-time data transfer between systems. It enables low-latency, deterministic transmission of large data streams without the overhead of complex network protocols. Thus, the RAVEN-I board has been selected to be this sFPDP interface. It is a high-performance data acquisition and transmission solution developed by TECHWAY. Designed for flexible and simple integration into complex architectures, the RAVEN-I stands out for its ability to process high-speed analogue signals with high reliability. It is designed to interface with digital equipment using the sFPDP protocol in demanding real-time applications, which fits perfectly with SAXO+ requirements. The RAVEN-I card has been integrated into the SAXO+ H-RTC prototype. The board includes 4 sFPDP channels, each configurable for reception or transmission. The maximum data rate per channel is 2.5 Gb/s, in conformity with VITA 17.1 and 17.3 standards. The card uses optical links via SFP connectors. Its host interface is based on PCIe Gen2 x4 connector, offering a maximum link speed of 6.2 Gb/s. Several tests have been conducted to ensure the board meets SAXO+ requirements. The first goal of these tests was to validate that the board was able to grab the data coming from SPARTA CODE Test Tool. The second goal was to check if the the board was capable to acquire data at 1KHz, according to SAXO frequency. SPARTA CODE Test Tool is a software application developed by ESO to facilitate the testing and validation of systems that use SPARTA. For this test, the Test Tool is used as a real-time frame generator. As shown in Figure 4, it sends sFPDP packets via a SFP optical link to the RAVEN-I board. The sFPDP packets contains a structured payload of 2768 bytes including: Figure 4: sFPDP integration Setup  1377 16-bit for HODM  2 16-bit values for ITTM  2 16-bit values for DTTM  2 16-bit values for PTTM The signal sent is a sinusoid. As its stroke is 1 and the bit range is 16, the signal is centered symmetrically around 32767 with values ranging from 16384 to 65534. The sFPDP acquisition application developed for this test was written in C++ using the Raven API provided by TECHWAY. The application offers a comprehensive set of features to process data in real time. The system automatically reads and analyses sFPDP packet headers, extracting the sequence counter and verifying the integrity of 2768-byte payload. The application retrieves and structures the data. Each packet received is time-stamped with nanosecond precision, enabling fine temporal traceability of measurements. The system automatically generates JSON files containing timestamps correlated to sequence counters, facilitating postacquisition analysis. The application configures Techway board channel 0 in receive mode, disables the Cyclic Redundancy Check (CRC) to optimize performance, and then starts a real-time acquisition loop. Then, the system performs a sequence check to detect lost packets, timestamps each reception with nanosecond accuracy, and can process data either continuously or for a defined number of packets. As SPARTA CODE Test Tool is not able to produce data at 1kHz, we have implementated a new setup to check whether the board is capable of acquiring data at 1kHz. Figure 5: sFPDP real time acquisition setup As shows Fig. 5, for this new setup the RAVEN-I board operates in loopback mode, using a fibre optic cable physically connecting channel 0 (receiver) to channel 1 (transmitter). In this configuration, the data transmitted on channel 1 is immediately received on channel 0, enabling an isolated evaluation of the board’s real-time acquisition performance. For this purpose, we developed a custom data emulator specifically for this test, which respects the SPARTA sFPDP packets structure described above. This internal emulator generated a continuous stream of ramp signals, managing the timing autonomously and precisely. 3.2 Results As illustrates Fig. 6, the signal received by the RAVEN-I board with the inital configuration, corresponds to the input. The sinusoid is perfectly reconstructed and centered around 32767, with values between 16384 to 65534. We receive all the signals we send with the test tool, without any loss and in the right order. This demonstrates the ability of the TECHWAY Raven-I board to acquire data from SPARTA Figure 7shows the timestamp measured at each iteration, with the board in loopback mode. We can see that the reception frequency is 1000Hz, as expected, and it is also perfectly stable. Figure 6: sFPDP received signal Figure 7: sFPDP real-time acquisition results The average jitter is 0.11 µ s and the maximum measured is 0.17 µ s, which illustrates the timing stability of the acquired data. These tests confirm the board’s capability to handle real-time data acquisition in a stable manner. 4. CAMERA INTERFACE INTEGRATION In this section we describe the camera interface integration into the H-RTC and its performance results. 4.1 Setup C-RED One is an infrared camera developed by First Light Imaging. We chose it for its high-speed, highsensitivity and ultra-low noise. It can takes more than 3500 images per second at the full 320x256 resolution. The camera will be linked to the H-RTC via camera link and a frame grabber will acquire the camera frames. The EDT VisionLink F4 frame grabber camera link has been selected and integrated on a H-RTC prototype PCIe slot. This frame grabber has been designed for scientific applications where high resolution, high speed and low latency are required, making it suitable for SAXO+ needs. The camera acquisition test aims to validate that the frame grabber is able to collect frames coming from the C-RED One camera at 3 kHz. Figure 8: Camera interface setup As shown in Figure 8, the camera is running at 3 kHz with a cropped frame size of 128x128 pixels, which is the region of interest (ROI) defined for SAXO+. The C-RED One is operating in global reset Correlated Double Sampling mode (CDS). This mode halves the maximum acquisition rate compared with simple read mode, but allows to compensate for the different exposure times between pixels caused by rolling shutter mode. A COSMICS task has been implemented to acquire frames from the C-RED One camera through the EDT frame grabber. On startup, it loads a configuration file and programs camera settings over the device control channel (image size, framerate, cropping range). Then, it starts a real-time loop that blocks for the next frame, validates its size, and checks the camera’s frame counter to detect drops. Frames are published into shared memory for downstream processing. 4.2 Results Figure 9, shows the camera acquition times measured on reception by COSMIC. The mean acquisition time is 333.44 µ s, corresponding to the 3 kHz frame rate. On the top of that, the very low average jitter measured of 0.58 µ s supports that the frame grabber is capable of acquiring frames in a very stable way. Figure 9: Camera acquisition latency The results demonstrates that the frame grabber is capable of stably acquiring frames at 3000Hz, without any loss. 5. END-TO-END REAL-TIME GPU PIPELINE In this section we will describe the RTC pipeline test setup and its performance results. 5.1 RTC pipeline test setup The final test is to validate the entire real-time GPU pipeline, with the frame grabber acquiring frames at 3 kHz from the C-RED One camera while the sFPDP transmission is running at 1 kHz. As shows Figure 10, HRTC operates in baseline mode. It includes the following steps for the camera frames:  the C-RED One camera frames are acquired at 3 kHz  the frames are copied from CPU to GPU  the frames are calibrated and the slopes are extracted on GPU  a linear reconstruction is performed to reconstruct wavefront from slopes  the integrator control law is applied to compute DM commands  the DM commands are copied from GPU to CPU to be sent to the mirror Figure 10: RTC pipeline setup At the same time, the sFPDP acquisition is running:  the sFPDP acquisition is running at 1 kHz  the first stage data is copied from CPU to GPU  a matrix vector multiplication is performed to synchronize both stages  the data is copied on CPU to be exported for telemetry 5.2 Results The latency presented in Figure 11 has been measured each time a frame was available on CPU. The full chain achieves a mean latency of 78.53 µ s, which is well below the 360 µ s maximal time budget. On top of that, the average jitter measured is 2.27 µ s which demonstrates the system reliability and stability. Figure 11: RTC latency results 6. CONCLUSION AND PERSPECTIVES On the software development side, most of the S-RTC datatasks and H-RTC supervisors have been implemented. Unit tests have also been developed, the next step is to proceed with functional tests. For this purpose we have integrated the Compass simulator11 into the H-RTC in order to generate realistic wavefront inputs. This will allow us to validate the RTC’s behaviour on representative data. Future developments on the H-RTC will aim to implement the remaining five controllers. On the hardware interface integration side, the sFPDP and camera interfaces have been successfully integrated into the H-RTC prototype. The tests performed demonstrate that both interfaces are capable of acquiring data at the required frequencies (1 kHz for sFPDP and 3 kHz for the camera) with very low average jitter. The final test validates that the entire RTC pipeline can run on GPU with a mean latency well below the 360 µ s time