Developing telemetry tools for Alveo Accelerator cards

id192251   DEVELOPING TELEMETRY TOOLS FOR ALVEO ACCELERATOR CARDS JOAN TERUEL JURADO Thesis supervisor XAVIERMARTORELLBOFILL(DepartmentofComputerArchitecture) Thesis co-supervisor BEHZADSALAMI(BarcelonaSupercomputingCenter) Degree Bachelor'sDegreeinInformaticsEngineering(ComputerEngineering) Bachelor's thesis Facultat d'Informàtica de Barcelona (FIB) Universitat Politècnica de Catalunya (UPC) - BarcelonaTech 21/01/2025  Contents 1. Context...........................................................................................................................................10 1.1. Introduction.............................................................................................................................10 1.2. Problem to solve.....................................................................................................................11 1.3. Stakeholders............................................................................................................................11 1.4. Previous concepts...................................................................................................................12 2. Justification.....................................................................................................................................13 3. Scopes.............................................................................................................................................14 3.1. Objectives...............................................................................................................................14 3.2. Requirements..........................................................................................................................14 3.2.1. Functional:......................................................................................................................14 3.2.2. Non-Functional:..............................................................................................................15 3.3. Obstacles and risks..................................................................................................................15 3.3.1. Not enough information..................................................................................................15 3.3.2. Not enough time..............................................................................................................15 3.3.3. Unfeasibility....................................................................................................................15 3.3.4. Hardware/software not available....................................................................................16 4. Methodology and rigour.................................................................................................................16 4.1. Methodology...........................................................................................................................16 4.2. Tools........................................................................................................................................16 5. Project planning..............................................................................................................................18 5.1. Tasks.......................................................................................................................................18 5.1.1. Project management tasks...............................................................................................19 5.1.2. Development Tasks.........................................................................................................20 5.1.3. Writing the memory........................................................................................................22 5.2. Resources................................................................................................................................22 5.2.1. Human resources.............................................................................................................22 5.2.2. Software resources..........................................................................................................23 5.3. Risk management and impact.................................................................................................23 5.3.1. Already encountered problems........................................................................................23 5.3.2. Potential risks..................................................................................................................24 5.4. Table of tasks..........................................................................................................................25 6. Gantt diagram.................................................................................................................................26 7. Budget.............................................................................................................................................27 7.1. Personnel costs........................................................................................................................27 7.2. Generic costs...........................................................................................................................29 7.2.1. Software..........................................................................................................................29 7.2.2. Hardware.........................................................................................................................29 7.2.3. Generic costs...................................................................................................................30 7.3. Estimated budget.....................................................................................................................31 7.3.1. Potential risks cost..........................................................................................................31 7.3.2. Final budget with contingencies.....................................................................................32 7.4. Budget Control........................................................................................................................32 8. Milestone tracking..........................................................................................................................33 8.1. Modifications done.................................................................................................................33 8.2. Alternatives explored..............................................................................................................35 9. Sustainability report........................................................................................................................36 9.1. Environmental:........................................................................................................................36 9.2. Economic:...............................................................................................................................36 9.3. Social:.....................................................................................................................................37 10. Development of the project..........................................................................................................38 Developing telemetry tools for Alveo accelerator cards 11. Out-of-Band..................................................................................................................................40 11.1. Introduction...........................................................................................................................40 11.2. Implementation.....................................................................................................................43 11.2.1. Baseboard management controller workflow...............................................................43 11.2.2. Baseboard management controller-less workflow........................................................44 11.3. Issues and conclusions from Out-of-band.............................................................................48 12. In-Band monitoring......................................................................................................................51 12.1. Introduction...........................................................................................................................51 12.2. Implementation.....................................................................................................................52 12.2.1. Xilinx Run-Time libraries.............................................................................................52 12.2.2. Base Address Registers.................................................................................................54 13. Characterizing and profiling.........................................................................................................63 13.1. Characterizing.......................................................................................................................63 13.2. Profiling................................................................................................................................73 14. Conclusions..................................................................................................................................76 14.1. Satisfied objectives...............................................................................................................76 14.2. Technical skills......................................................................................................................77 14.3. Integration of knowledge......................................................................................................78 15. Laws and regulations....................................................................................................................79 16. Glossary........................................................................................................................................80 APPENDIX A)...................................................................................................................................82 Report on telemetry capabilities on Versal V80............................................................................82 APPENDIX B)...................................................................................................................................83 Table: Supported sensors per Alveo card.......................................................................................83 APPENDIX C)...................................................................................................................................85 Table: Resource utilization for CMS.............................................................................................85 APPENDIX D)...................................................................................................................................86 Table: Compilation of known sensors, offsets, presence mask, and division factor......................86 17. References....................................................................................................................................89 Figure Index Figure 1: Trello webpage, example status for current project; Own compilation..............................15 Figure 2: Trello card example for task called: 'Deliverable1: Context and scope of the project'; Own compilation.........................................................................................................................................15 Figure 3: Gantt initial temporal diagram; Own compilation (using Google Drive Sheets)...............24 Figure 4: Gantt revised temporal diagram; Own compilation (using Google Drive Sheets).............31 Figure 5: AlveoTM U2XX, U50X, U55X, UL3524 block diagram; Modified Xilinx image...........38 Figure 6: Alveo satellite controller inferred internal structure; Own compilation.............................39 Figure 7: Requirements from host system; own compilation.............................................................40 Figure 8: Mock version of OOB program; Own compilation............................................................45 Figure 9: AlveoTM U2XX, U50X, U55X, UL3524 block diagram; Modified Xilinx image...........49 Figure 10: Card Management Solution IP interface image for U280 and U55C; Source Xilinx.......52 Figure 11: Final implementation of the FPGA Shell with CMS, block design view; own compilation ............................................................................................................................................................54 Figure 12: CMS output excerpt on three different nodes of the Meep Workstations; own compilation ............................................................................................................................................................56 3/91 Developing telemetry tools for Alveo accelerator cards Figure 13: Excerpt of the program with 8 FPGAs on one node simultaneously on the Makinote node twelve; own compilation....................................................................................................................57 Figure 14: Figure showing a successful connection with a XDMA FPGA; own compilation...........60 Figure 15: Area and resource utilisation of Wrapper shell, Lagarto Hun with 4 cores and 2 VPUs; Own compilation................................................................................................................................61 Figure 16: Fan speed over time in revolutions per minute; Own compilation...................................62 Figure 17: Temperature over time (U280); Own compilation............................................................62 Figure 18: Total power over time (U280); Own compilation.............................................................62 Figure 19: Temperature over time (U55C); Own compilation...........................................................64 Figure 20: Total power over time (U55C); Own compilation............................................................64 Figure 21: Total power over time using eight U55C FPGAs; Own compilation...............................66 Figure 22: Temperature over time using eight U55C FPGAs; Own Compilation.............................66 Figure 23: Power over time for stressed U55Cs; Own compilation...................................................69 Figure 24: Temperature over time for non-stressed U55Cs; Own compilation..................................69 Figure 25: Standard cell schematic; source........................................................................................71 Figure 26: FPGA cell schematic; source............................................................................................71 Figure 27: Image showcasing the V80 SMBus implementation; source Xilinx................................80 Table Index Table 1: Risk table with probabilities and their impact; own compilation.........................................22 Table 2: Describing tasks, hours invested, dependencies, and resources needed; own compilation..23 Table 3: hourly cost per personnel role; own compilation.................................................................25 Table 4: Table defining tasks and personnel roles; own compilation.................................................26 Tabla 5: Software cost table; own compilation..................................................................................27 Table 6: Table with Hardware devices related expenses; own compilation.......................................28 Table 7: Table exposing generic expenses; own compilation.............................................................29 Table 8: Table defining potential risks and their costs; own compilation..........................................29 Table 9: Final table with the project estimated cost; own compilation..............................................30 Table 10: Table defining deviation formulas; own compilation.........................................................30 Table 11: Revised table defining tasks and personnel roles (incomplete); own compilation.............32 Table 12: Revised final table with the project estimated cost; own compilation...............................32 Table 13: Benefits per telemetry method; own compilation...............................................................36 Table 14: Output from command 'sudo i2cdetect -l'; Own compilation.............................................43 Table 15: Output from command 'sudo i2cdetect -y 0'; Own compilation.........................................43 Table 16: Supported I2C/SMBus commands; source Xilinx..............................................................44 Table 17: Code excerpt of the platform identification parameters; Own compilation.......................58 Table 18: Table with the basic data output; own compilation............................................................59 Table 19: Excerpt of extended data using an Alveo U55C; own compilation....................................60 Table 20: Analysis of power and temperature increment; Own compilation.....................................67 Table 21: Average absolute values for previous table; Own compilation..........................................67 Table 22: Compilation of standard deviation (SD) per sensor; Own compilation.............................68 Table 23: Power increase due to stressing emulated processors on FPGAs; own compilation..........70 Table 24: Utilization of resources; Own compilation.........................................................................73 Table 25: Supported sensors per Alveo card; Source Xilinx..............................................................83 Table 26: Resource utilization for card management solution; Source Xilinx...................................83 Table 27: Compilation of known registers, their names, offsets, presence mask, division factors and location; Own compilation, excerpt from custom C library, inferred from Xilinx sources...............86 4/91 Developing telemetry tools for Alveo accelerator cards ABSTRACT The project would propose solutions and develop tools for getting telemetry data, such as temperature, power, current, or voltage values, from the embedded sensors inside the AMD/Xilinx Alveo FPGAs accelerator cards. We will use this data to track the power/energy consumption and temperature of different FPGA components at run-time, e.g., memory, IOs, and logics. This analysis will help to better understand the performance of the application running on FPGA and thus, propose solutions for improving the power/energy efficiency with lower temperature impacts. We plan to develop these tools on top of Makinote (i.e., BSC’s FPGA cluster) to collect results on a massive number of devices. The variation across different FPGA cards would also provide useful information to better understand the design placement and optimise it. The telemetry toolset will be integrated in FPGA Shell, which is a tool to generate FPGA designs automatically by using the Xilinx IPs for connecting the hardware design to peripherals. 5/91 Developing telemetry tools for Alveo accelerator cards RESUMEN Este proyecto tiene la intención de proponer soluciones y desarrollar herramientas para conseguir información de telemetría cómo la temperatura, potencia, corriente o voltajes, estos valores se obtienen de los sensores embebidos dentro de las Alveo de aceleración FPGA de AMD/Xilinx. Usaremos estos datos para el seguimiento de la energía/potencia consumida y los valores de temperatura en tiempo real de módulos como la memoria, puertos de entrada/salida o secciones lógicas. Este análisis ayudará a comprender el rendimiento de una aplicación dentro de la FPGA y por tanto, proponer mejores soluciones para incrementar la eficiencia en términos de consumo y temperatura. Planeamos desarrollar esta herramienta dentro de Makinote (es decir el clúster de FPGA del BSC) para recolectar los resultados en cantidad. La variación entre diferentes tarjetas FPGA también nos puede ser útil para el entendimiento del diseño interno y optimizarlo. El conjunto de herramientas de telemetría será integrado dentro del FPGA Shell, que es una herramienta que genera diseños para FPGA automáticamente usando las IP de Xilinx para conectar nuestros diseños con sus respectivos periféricos. 6/91 Developing telemetry tools for Alveo accelerator cards RESUM Aquest projecte té la intenció de proposar solucions y desenvolupar eines per aconseguir informació telemétrica com la temperatura, potencia, corrent o voltatges, aquests valors s’obtenen dels sensors encastats dins de les FPGAs d’acceleració Alveo de AMD/Xilinx. Utilitzarem aquestes dades pel seguiment de la energia/potencia consumides y els valors de temperatura en temps real dels mòduls com les memòries, ports d’entrada i sortida o seccions lògiques. Aquest anàlisi ens ajudarà a comprendre el rendiment d’una aplicació dins de les FPGAs i per tant, proposar millors solucions per incrementar l’eficiència en termes de consum i temperatura. Planajem desenvolupar aquesta eina dins de Makinote (un clúster de FPGAs del BSC) per a recolectar els resultats en quantitat. La variació entre diferents targetes FPGA també ens pot ser útil per al enteniment del disseny intern i optimitzar-lo. El conjunt d’eines de telemetría serà integrat dins del FPGA Shell, que és una eina que genera dissenys per a FPGAs automàticament utilitzant IPs de Xilinx per connectar-los amb els seus respectius perifèrics. 7/91 Developing telemetry tools for Alveo accelerator cards 1. Context This is a bachelor thesis of the Computer Science degree at the Facultat d’Informàtica de Barcelona, Universitat Politècnica de Catalunya, on the specialisation of Computer Engineering, directed by Xavier Martorell Bofill with co-supervisor Behzad Salami, head of the FPGA Technologies team at the Barcelona Supercomputer Center (BSC). The project will work within the boundaries of the computer science department, working on the new computer architecture projects, such as the Zettascale, which is one of the most important sources of financing. In this project, BSC works to develop a RISC-V processor entirely designed locally; additionally, there are also different projects of international cooperation with the European Union and private enterprises. 1.1. Introduction A Field Programmable Gate Array (FPGA) is a type of device that, using an array made of basic elements of logic, is capable of emulating the behaviour of defined hardware [1]. The FPGA market was valued in 2020 at 6 billion US dollars, and some forecast studies expect it to grow to 13.6 billion dollars by the end of 2031 [2]. Increasingly FPGAs are being used in data centers and cloud computing, as they can be dynamically modified for compute-intensive tasks such as analysis of data, machine learning, network processing and other specialised computing-intensive tasks. FPGAs are gaining space in the global market, not as emulation platforms for hardware design, but as final computing platforms. In fact, BSC is using the FPGA not for the intended role for the Alveos which in their name specifies (accelerator tasks), but for emulation. Contrary to this, big companies like Amazon or Microsoft are buying an increasing amount of FPGA for their data centers to provide new computation platforms to their customers and themselves [3]. FPGAs are crucial where a hardware product is being designed. These devices are used for emulation purposes, which help provide debugging data to the design team. Without FPGAs, the time that would take for the development of such products would increase significantly, due to the fact that simulations that are run on classic computers can take a lot of hours, if not days, in the case of big designs [4]. As the importance of the productivity of the Register Transfer Level (RTL) team is of the utmost importance, developing tools that focus on the status of those machines is a key service that helps to improve the reliability of those expensive devices, which translates to better efficiency and management of this hardware resources. 8/91 Developing telemetry tools for Alveo accelerator cards •Done: Tasks that are 100% finished. •Review/Improve: Tasks that are almost finished but require testing, review from other actors or some level of polishing. •Discarded: Tasks that were started but no longer are actively follow through due to problems, obstacles or mistakes. Figure 1: Trello webpage, example status for current project; Own compilation Each vertical column is referred to as a ‘list’ which contains a set of ‘cards’ and each card can have an amount of different tasks. In the Figure 2 at the immediate right we can see exactly how we have defined a due date for the task and a checklist, in this case, to write the self-report and write the document to deliver. See that I have also added a tag with a colour, this I use it a little different from the suggested method explained at [9], rather I use the colours to express the severity of the task, red with high degree of importance, green for a low priority. 15/91 Figure 2: Trello card example for task called: 'Deliverable1: Context and scope of the project'; Own compilation Developing telemetry tools for Alveo accelerator cards - Gitlab: One of the most important tool in the active development will be Gitlab, which is a powerful tool that enables constant development and constant integration [10]. Apart from the repositories which save the progress, Gitlab enables new methods to actively report issues and recommendations from the different actors. - Zoom: To communicate between the director and co-director of this project, apart from the physical reunions we have and the standard communication channels such as Google Mail or Drive, Zoom takes a big role, the efficiency in interchanging mails cannot be compared to the one of people actively interacting with each other, when the different actors are not able to reunite together it is decided that remote meetings will be the second option. 5. Project planning This project is made with a BSC contract, although the modality of it is type AI. Initially, the number of hours dedicated to the project were five a day, five days a week; this at least was the initial proposal; the daily hours dedicated to the project have been modified as the project progressed. The contract was signed on the 1 st of March 2024, and the intention is to get it done before the 13th of January. This project officially started as a TFG when it was inscribed and matriculated in the first week of July. That means some of the tasks described that are planned are tasks already completed, and the definition of risks and problems encountered are obstacles that already happened or can possibly happen, so this section will look in retrospective at the work that has already been done, the work that is being done actively, and planned work to do in the future. 5.1. Tasks In this chapter, the tasks of the project will be defined between those considered project management, development, redacting the memory, and finally the project monitoring. Each task will define the time to complete, what the task consists of, and what will be done. At the end of the description in Chapter 5.4. Table of tasks, where additionally the dependencies between tasks are defined and the resources needed. I Modality A indicates that the project is developed as an university project. 16/91 Developing telemetry tools for Alveo accelerator cards 5.1.1. Project management tasks Here, all tasks are defined that are related to project management (PM). •PM.1: Context and scope of the project (25h) Here the project scope, context, and justification of the project are clarified, identifying the problems and solutions the project has to solve, the requirements of the project, the parties involved, and the methodology. A document is written and delivered. •PM.2: Project planning (20h) The planning of the project is defined by the task decomposition, the schedule of each task with the Gantt diagram, the resources needed, and probable risks the project might encounter. A document is written and delivered. •PM.3: Economic and sustainability management (20h) An economic and sustainability report is made, describing the total budget of the project and its environmental consequences. A document is written and delivered. •PM.4: Final integration (15h) All the previous documents are integrated into one, which will be the basis for the memory of the project. A document is written, reviewed, and delivered. •MT: Meetings for monitoring (45h) An amalgamation of all the meetings and reunions made is defined as a task to account for all the hours invested. Since the 1st of March, we defined one reunion of one hour a week with the thesis co-supervisor, and since the first of September, an additional meeting with both the supervisor and co-supervisor of also one hour. The team meetings take approximately one hour each; these meetings are taken every Tuesday from eleven o’clock to twelve o’clock. 17/91 Developing telemetry tools for Alveo accelerator cards 5.1.2. Development Tasks A list of the tasks that are related to the development of the project is defined. The project development tasks will be subdivided into those related to Out-of-band, In-Band and writing tasks. The order of the list does not specify the order in which the tasks are completed. •RT.1: General research on telemetry (25h) A general research on Xilinx’s documents is performed on how to get telemetry data out of the Alveo cards. This introductory task will lead to the subsequent decisions on the development of the project. •RT.2: Investigating XRT methodology (25h) The XRT libraries and APIs are investigated to discover whether those tools are useful or not. Also, a small program in bash scripts is developed. - Out of Band development group of tasks: •OOB.1: Research on OOB methodology (25h) A specific research on OOB methodology is done, investigating exactly the internal structure of the Alveo Accelerator cards and how to communicate with that channel. •OOB.2: Initial development of OOB tool (25h) Here a program that uses the OOB interface is developed, this initial task consists of installing necessary tools and libraries, configuring the FPGAs, and an initial Python program. •OOB.3: Additional research to solve problems (25h) In retrospective, we encountered different problems that made us stop developing the OOB tool and forced us to search for possible solutions. •OOB.4: Baseboard Manager Controller Development (25h) Includes the research of what is a BMC, the testing of BMC inside our hardware, and finding solutions to faulty BMC behaviour. 18/91 Developing telemetry tools for Alveo accelerator cards •OOB.5: Question to Xilinx and conclusions (15h) Having observed a multitude of problems almost everywhere when developing OOB related telemetry, we established a channel with Xilinx engineers through emails and Xilinx’s forum where we asked for solutions to the problems encountered, upon Xilinx answering, we came to some conclusions and made a change in the course of the development of OOB. - In-Band development: •IB.1: Research on IB methodology (25h) Researching IB methodology with Xilinx documentation. •IB.2: Development of a simple Vivado project with CMS (25h) With Vivado IDE, develop a simple design using various types of FPGAs, which enable CMS. •IB.3: Integration of CMS into Wrapper-Shell (25h) Include and integrate the CMS into Wrapper-Shell, additionally, check that this new feature does not break the previous infrastructure and CMS can still be addressed. •IB.4: Devolpment of CMS Python Program (25h) Development of a CMS program that uses the previous designs loaded into the FPGAs and reads the data and writes it into CSV format for later analysis. The program is made in Python but also calls a C executable. •IB.5: Early recopilation of telemetry data (10h) From Alveo U280 and U55c models, extract data for later analysis. •IB.6: Integration of CMS into Embedded-Shell (25h) Expand the CMS to be included inside Embedded-Shell, making sure the addition of CMS does not brake the standard pipeline, and works for U200/U250, U280, and U55c FPGAs. •IB.7: Improvement of telemetry program (15h) Improve the telemetry in-band program so that it includes support for Makinote deployment and multiple FPGAs can be connected to one instance of the program at once. 19/91 Developing telemetry tools for Alveo accelerator cards •IB.8: Development of Dfx and profiling of Cincoranch (50h) Develop Dynamic Function eXchange for embedded-shell so we can collect data individually from each Cincoranch core. •IB.9: Extended recopilation of telemetry data (20h) From previous additions, extend the amount of data taken with the addition of Cincoranch support and DFx for better profiling. 5.1.3. Writing the memory These definitions of tasks correspond with the writing of a document called memory and the defence preparation before the academic tribunal. •WM.1: Concurrent Writing (50h) This task defines the time used when writing the memory while developing the project. •WM.2: Final Writing of the Memory (40h) The tasks correspond to the final synthesis of the memory to deliver once all tasks and recompilation of data are done. •WM.3: Preparing the defence of the thesis (20h) Creating a PowerPoint presentation, possible questions and those answers, and a script that makes up for the defence of the thesis. 5.2. Resources 5.2.1. Human resources - [AP] Author of the project: student who develops the project. - [TS] Thesis supervisor: UPC tutor, who monitors the project. - [TC] Thesis co-supervisor: BSC tutor, who monitors the project. Additional human resources that are more generalistic, like: - [SA] BSC Servers administrators: administrators who help us install required tools. - [XP] Xilinx personnel: Xilinx engineers, which answer our petitions. 20/91 Developing telemetry tools for Alveo accelerator cards - [CW] BSC coworkers: When we require additional help from BSC companions. 5.2.2. Software resources [GD] Google Drive: defined a conglomerate of different tools like PowerPoint, Draw, Sheets, and more. [LO] Libreoffice: LibreOffice IDE is used for writing the official documents to be delivered. [TR] Trello: Trello is the tool that is used with Kanban methodology and monitoring. [VD] Vivado: Vivado is an IDE tool developed by Xilinx in which we create designs and use for loading them into the FPGAs. [VS] Visual Studio Code: Will be the main IDE tool to use when developing Python and C programs to be used on remote clusters. [ZM] Zoom: This tool is used for remote meetings. 5.3. Risk management and impact During the course of the project there will be obstacles and problems that would impact the development time, for this reason the planning is done with a margin of time so that in the worst-case scenario the schedule would still be met. In fact, some of the risks below were already found when developing the project, so I will divide them between the already-existing and potential new ones: 5.3.1. Already encountered problems 1. Not enough information: When developing a telemetry tool, we encountered a set of documents that explained different ways of accessing monitoring data, but the problem is that they are not precise enough or there are missing clarifications. For example, the Vivado IDE, when showing all available FPGAs to load the bitstream, lets you also view a chart with the temperature of the card, but the official documentation does not clarify how to access through that same method the data. Additionally, some information misses steps, or even worse, the documentation is wrong. Experimentally, we have proven that some of the statements of the OOB methodology are not true. The impact on hours can be stablished as 25, as we had to contact multiple times with Xilinx to get answers and also to start looking for information from unofficial sites. 21/91 Developing telemetry tools for Alveo accelerator cards 2. Unfeasibility: Initially in this project we weren’t searching for an in-band solution, because from both (OOB and IB) the best is without a doubt OOB, but for the OOB task, we have seen that the documentation is wrong in some key specifications, without entering the specific details, the wrong statements make OOB totally unfeasible, thus OOB was abandoned and the integration for IB began. The consequences of unfeasibility were far reaching, OOB method was abandoned and the research and development of a tool had to be restarted, we have stablished that from this problem, the impact on additional hours is the total amount of time dedicated to OOB minus the research time, which is 90 hours. 5.3.2. Potential risks 1. Not enough time: As the project started a long time ago, the schedule planned for the missing tasks and features already has a margin for possible additional problems, which we believe is a low risk. But, we consider an additional 10 hours for possible deviations. 2. Hardware/software not available: The project is developed on BSC infrastructure, and sometimes it is needed some maintenance and meetings with the operations team, the team that oversees the Makinote cluster, and petitions have to be made through them, which takes time like installing new programs, downloading libraries, or accessing the hardware physically. The risk is considered 'medium’ and the impact on hours in a worstcase scenario is 25 (because meetings with operations are made each Friday). Risk Probability/Severity Impact Not enough information Medium 25 hours Unfeasibility High 90 hours Not enough time Low 10 hours Hardware/software not available Medium 25 hours Table 1: Risk table with probabilities and their impact; own compilation 22/91 Developing telemetry tools for Alveo accelerator cards 5.4. Table of tasks Table 2: Describing tasks, hours invested, dependencies, and resources needed; own compilation Code Task Dura tion Depend encies Resources (human and technological) Project Management (80 hours) PM.1 Context and scope of the project 25 h --- LO, TR, AP PM.2 Project Planning 20 h PM.1 LO, GD, TR, AP PM.3 Economic and sustainability management 20 h PM.2 LO, GD, AP PM.4 Final Integration 15 h PM.3 LO, AP MT Meetings for monitoring 45 h --- ZM, TR, AP, TS, TC Project Development (375 hours) RT.1 General research on telemetry 25 h --- AP RT.2 Investigating XRT methodology 25 h RT.1 AP OOB.1 Research on OOB methodology 25 h RT.1 AP OOB.2 Initial development of OOB tool 25 h OOB.1 VS, VD, AP OOB.3 Additional research to solve problems 25 h OOB.2 AP, CW OOB.4 Baseboard Manger Controller development 25 h OOB.3 AP, SA OOB.5 Question to Xilinx and conclusions 15 h OOB.4 AP, XP, TC IB.1 Research on IB methodology 25 h RT.1 AP IB.2 Development of simple Vivado project with CMS 25 h IB.1 VS, AP, VD IB.3 Integration of CMS into Wrapper-Shell 25 h IB.2 VS, VD, AP, CW IB.4 Devolpment of CMS python program 25 h IB.3 VS, VD, AP IB.5 Early recopilation of telemetry data 10 h IB.4 VS, AP IB.6 Integration of CMS into Embedded-Shell 25 h IB.5 VS, VD, AP IB.7 Improvement of telemetry program 15 h IB.5 VS, VD, AP IB.8 Development of Dfx and profiling of Cincoranch 50 h IB.3 VS, VD, AP, CW IB.9 Extended recopilation of telemetry data 20 h IB.8 VS, AP Writing the memory (110 hours) WM.1 Concurrent Writing 40 h OOB.4 or IB.2 LO, GD, AP WM.2 Final Writing of the memory 50 h WM.1 LO, GD, AP 23/91 Developing telemetry tools for Alveo accelerator cards WM.3 Preparing the defense of the thesis 20 h WM.1 GD, AP The total amount of time dedicated to the project is around 550 hours (without risks), having in mind that officially the TFG accounts for 18 credits and each takes 30 hours, for a total of 540 hours, we surpass the expected amount of time invested in the project. 6. Gantt diagram In the following Figure 3, we can see the Gantt temporal diagram. Each week is defined as five work days, for example, ‘W01’ only is contabilized as the first of March, and ‘W10’ has two April days and three May days, so is accounted as a May week; as time went by, the hours dedicated to the project were modified, and so the number of weeks needed to complete a task increased but not the total hours to such a task. The figure below was created on the 29th of September, which corresponds with W31. The methodology used is Agile with a custom Kanban; some modifications have been made, so, for example, even though we define each sprint as a two-week task, each week we do the agile review, and as mentioned before, on later stages of the project, the quantity of sprints needed for a task increased as the daily hours dedicated decreased. Individually for each sprint, it is not specified that the planning is an individual task; rather, it is implicit on [MT] meetings for monitoring; when reviewing the progress made, a replanning is done if needed. This project elapsed almost 10 months, the entire month of August there was no progress due to summer vacations. Additionally, we can see that the monitoring for the project stopped from W10 to W15 because we encountered several problems for developing OOB, and it was decided that we would stop until getting answers from Xilinx; in the meantime, the author would work on tasks that are not related to this project. 24/91 Figure 3: Gantt initial temporal diagram; Own compilation (using Google Drive Sheets) Developing telemetry tools for Alveo accelerator cards 8. Milestone tracking 8.1. Modifications done The milestone tracking is done some weeks before the presentation of the technical memory, which is given before the presentation of the bachelor’s thesis. In this case, this review has been done on 3rd of December 2024. In this section, only the changes made are indicated, like a changelog. Contextualization: A new legal note has been added in chapter ‘1.1. Introduction’ where it is specified the almost nonexistent legal constraints of the project. Under ‘3.3. Obstacles and risks’ a new risk has been added, ‘3.3.4. Hardware/software not available’ because there were only three risks described, but in chapter ‘5.3. Risk management and impact’, four appeared. Temporal planning: A restructure has been made of the previous ‘6. Gantt diagram’, where the planification has been modified to fit reality; tasks IB.8 and IB.9 have been revised to the left of the diagram. The time to complete Task IB.8 has been doubled, to fifty hours. Temporally, the ratio of time dedicated between October and November to tasks related to BSC has increased, which means less time is invested per week into this project. Budget: The budget has been modified due to the task IB.8 taking longer than usual; this revision costs an additional 665.25€ to the project. Following the protocol mentioned in ‘7.4. Budget Control’ we have to calculate the deviation; in this case, the deviation is the increase of costs. The percentage of the contingency money used to pay for these costs is (-665.25) ÷ 2947.54 = -22.53%; as it is under the -50% limit, it is acceptable. The following table 11 depicts only the changes made on the respective rows. It has been added a note on ‘Budget Control’ that to calculate the deviation, it is needed to divide by the contingency. 31/91 Figure 4: Gantt revised temporal diagram; Own compilation (using Google Drive Sheets) Developing telemetry tools for Alveo accelerator cards Code Task Dura tion Hours per Role Cost (€) SM SD JR DA Project Management (125 hours) ... Project Development (385 hours) ... IB.8 Development of Dfx and profiling of Cincoranch 50 h 50 1330,50€ ... Writing the memory (110 hours) ... Total: 18537,49€ Table 11: Revised table defining tasks and personnel roles (incomplete); own compilation The final budget is updated as follows : Source Cost (€) Costs related to personnel 18537,49€ Software 73,70€ Hardware 112,51€ Generic costs 1591,83€ Total before contingency: 20315,53€ Contingency (15%) 3047,33€ Potential risks 359,24€ Total: 23722,10€ Table 12: Revised final table with the project estimated cost; own compilation Conclusions: A new chapter called ‘Conclusions’ has been written with subsections. Bear in mind the following section cannot be totally completed until the final delivery. 14.1. Satisfied objectives: This chapter indicates upon the initial scopes suggested, if those have been successfully completed. 14.2. Technical skills : The technical skills associated with the project and its achievements. 14.3. Integration of knowledge: Which UPC subjects have been beneficial to this project and the reason why. 32/91 Developing telemetry tools for Alveo accelerator cards Other minor changes such as grammatical errors, extending some paragraphs to better explain the situation are not indicated here. 8.2. Alternatives explored The exploration of alternatives has been concurrent with the development of the project. The alternatives we have explored are focused on the reduction of power consumption. Which initially was not inside the scopes of the project but was added by my own initiative, not as a product, rather as a theorised exploitation of the program we have been developing. These explorations have not been accounted for in the Gantt diagram, nor in the task decomposition of the project, because it is centred around the experimental usage of the FPGA and some manipulations of current designs. The main alternative was to develop yet another shell, called ‘lite shell,’ which could be able to give telemetry data to the host while using the least amount of power consumption, because, unlike other shells, telemetry does not need memory controllers, the full AXI interface, or additional peripherals like UART, JTAG, or others. The main problem with this idea is that it will likely never get implemented (at least not in BSC’s clusters), because it relies on the development of a ‘daemon,’ a background task with privileges, which requires extensive and careful programming; otherwise it could become a cyber-attacker vector. The main idea is that, upon a user leaving the server, this ‘lite shell’ would get loaded into the FPGA, providing telemetry. Another issue is that a user that wants a program to keep running after their departure from the server would get their FPGA disabled. Alternatives to provide telemetry data are bad; out-of-band would never be able to work on our current servers. Sysmon data through Xilinx’s Vivado GUI is not a solution either, primarily because there is not enough data to know its inner workings. XRT drivers would not be able to bind to our designs, and even if they did, other functionalities like profiling wouldn’t work. Sincerely, I think we chose the best decision among the bad ones, even though it has seemed to be somewhat impractical. 33/91 Developing telemetry tools for Alveo accelerator cards 9. Sustainability report Increasingly, companies all over the world include in their projects a sustainability report, which measures the impact of their products and services not only in ecological but also economic and social matters too. 9.1. Environmental: In chapter 7.2.3. Generic costs I estimated the consumption of electricity as I do not have any car, the CO2 emissions related to this project are approximately of 273 gCO2 / kWh[17] per 1,1665kW times 550 hours; resulting in 175,17 Kg of CO2. Project put into production: It is difficult to accurately estimate the real impact the development of the project will have. Compared to the macro-economic scope, a lone student working with a pair of screens and a laptop is negligible. But the completion of the project could lead to benefits, such as the control of cooling systems to improve their efficiency and so their electrical consumption. Useful life: Currently, there is no solution that suits the BSC as well as mine. Yes, Vivado offers a the XRT tools but only works on their infrastructure, which we currently do not support, so the answer is that is there is no solution right now that suit us. This tool gives the designer the ability to use two new metrics, consumption and temperature, with which to choose the designs that have the best values. Risks: We don’t foresee any type of risks related with the environment due to this project, in fact if there are any it would provide tools to decrement them such as FPGA optimization. 9.2. Economic: Project put into production: I have estimated the costs related to human resources, hardware and software usage, generic costs such as internet, electrical consumption, or water, and also taking into account possible risks and contingencies for the project. The contingency is around 15% of the initial budget, and although there have been certain risks, these have only consumed a quarter of the contingency, this returns a positive difference of €2282.29. 34/91 Developing telemetry tools for Alveo accelerator cards Useful life: This project resolves one of the problems of the actual state of the art programs we used, that aren’t suited for the custom BSC flow. We do not expect additional expenses regarding this project, there is almost no maintenance or updates related costs. Risks: The main risk of this project is that is only compatible for Ultrascale FPGAs and thus only useful as long as the BSC has those FPGAs in their inventory. Risk which is very low as the FPGAs have still a good useful life ahead. 9.3. Social: Project put into production: The development of this project will give me a huge amount of knowledge, firstly because this is my first job, so knowing how is the environment inside a company, how to better communicate with our partners and supervisors, and obviously the knowledge of developing a project, what is a FPGA, how does it work, what is their internal structure, develop projects for FPGAs, different protocols and tools... Useful life: The current programs are only used to know the temperature of the machines without doing anything with them, so they are a kind of entertainment tool because the practicality is limited. Additionally, the users have to oversee the programs (with a GUI) and are not made to run independently in the background. This tool aims to be as easy to use as possible with the minimum user to oversight, only requiring an initial installment and configuration. This project has drawn the interest of my supervisor since a long time ago; he already wrote an academic paper about a simple telemetry tool [6], and we have been talking about writing a paper about this project when it ends. There is also interest from the maintenance team because, as we do not know the temperature of the FPGAs, we have to put their cooling systems at 100%, which is not optimal. Additionally, we want to see to which level this could be used for profiling the cores we develop at BSC. Risks: We do not expect any scenario in which this could be harmful to anyone or any collective. And the project is developed to be as self-contained as possible with the least external dependencies as possible. 35/91 Developing telemetry tools for Alveo accelerator cards 10. Development of the project As it is established on 1.4. Previous concepts, there are two main methods to acquire telemetry data from an FPGA: one is the out-of-band method and the other is the in-band method, which differ in how the data transactions are done, either independently from the data transfers over PCI or with some side channels. OOB does not consume bandwidth over the PCIe even though the pins themselves are found in the PCI interface because they are reserved for that special purpose only, and so a host system would never use them for memory accesses, for example. As In-Band is achieved thanks to additional logic in the programmable logic on the FPGA, it occupies resources. Those resources consume power, and thus, using the in-band method is less efficient at profiling or characterising the FPGA. On the other hand, OOB uses the embedded processor, which has its own power source, and it is always operational. With IB we need a bitstream with that specialised IP, although the golden/factory images, which are the standard bitstreams that get loaded when there is a cold reboot, carry that IP. Luckily, all the Ultrascale devices (Alveo cards) carry the same embedded processor, meaning that OOB has wide support. IB is also supported, but the user has to configure the IP for each target device. Table 13 sums up all the benefits from each method: Out-of-Band In-Band Consumption of bandwidth: Not for PCIe transactions, only for host read/write operations. Yes, It uses PCIe memory accesses Disturbance in readings: No Yes, It is part of the logic design Availability when plugged: Always Only when a suitable bitstream is loaded Multiplatform (among Alveo): Yes Yes, but reconfiguration is needed for each Alveo Table 13: Benefits per telemetry method; own compilation. 36/91 Developing telemetry tools for Alveo accelerator cards The embedded processor, also known as satellite controller is supported on these Alveo Cards: U200, U250, U280, U50, U50LV, U55C, U55N, UL3xxx, U30, V70 and MA35D. Mainly we are going to work using the U55C, U280 and U250. The Versal V80s have a new interface and do not house a satellite controller. FPGAs of the BSC: •96 Alveos U55C on Makinote (Cluster) divided into 12 nodes •2 Alveos V80 on Jonu (Meep workstations) •4 Alveos U280 on Nanu, Picu and two on Raju (Meep workstations) •2 Alveos U55C on Teru and Cucu (Meep workstations) •1 Alveos U250 on Satu (Meep workstation) •2 Alveos U250 on Quar •2 Alveos U50LV unplugged In the following chapters the reader will find an in depth explanation of all the methods to develop telemetry for the Ultrascale FPGAs and what can be done with that information. 37/91 Developing telemetry tools for Alveo accelerator cards 11. Out-of-Band 11.1. Introduction The most sensible solution to provide telemetry is Out-of-band and in this chapter all the implementation details will be explained. The satellite controller is an embedded processor made by Texas Instruments, type MSP432. The SC is connected to the voltage regulators through a power management bus, to an EEPROM, thermal and electrical sensors through I2C protocol. To access the SC data an I2C bus is routed to the PCIe edge connector; four dedicated pins are used for transmitting data: SMDat, SMClk, ground, and Vcc. This bus does not use a pure I2C protocol directly; rather, it uses a modified version with its own standard, SMBus, which stands for System Management Bus. SMBus uses additional working conditions such as a minimum clock frequency or a timeout. The protocol uses master and slave devices that are connected to the bus; each slave has a dedicated address and should not be used by another device in the same bus. These devices sniff the bus to know if anyone has a petition with their own address. If two or more devices use the same address, then the bus would be rendered useless whenever there is a petition to any of them. From figure 5 it can be observed that the main way to access the SC is through the called Baseboard Management Controller (BMC). The reason this is recommended is because the BMC which is an independent chip that has already all the firmware needed to send all requests to the FPGA. This could become a problem (like in our case) if there are not any BMCs housed inside the motherboards. 38/91 Figure 5: AlveoTM U2XX, U50X, U55X, UL3524 block diagram; Modified Xilinx image Developing telemetry tools for Alveo accelerator cards Figure 6 explains the OOB path, it can be observed how the satellite controller has four different slaves, this image makes it clearer how to access the SC, we can see that the slaves are connected to the same bus, each having their own I2C address. That bus begins at the Alveo card and it connects to the PCIe slot, from then it has a bifurcation going to the host system (i.e. the CPU) controller or the Baseboard Management Controller (BMC). The controller is part of the platform controller hub (PCH), which is a dedicated chip that manages all peripheral connections, protocols and communication in general. Over the PCI the pins which control the SMBus are Smdat pin 6, SMClk pin 5, additionally, pin 7 and 8 provide ground and 3.3V respectively all from face B [18]. These four slaves which can be accessed by the user, are dedicated each to a different protocol, the slaves are IPMI FRU at address 0x50, Alveo I2C at 0x65, SMBus (ARP) at 0x61 and finally MCTP/PLDM at 0x18 [19]. Also we can access the data automatically if we have a baseboard management controller available on the motherboard which the Alveo cards are connected to, with the BMC there is no need to create a dedicated program as it uses a standard IPMI protocol. Inside the Satellite Controller, there is a memory module. Connected to the same bus as the EEPROM, is the sensors' module, which is a simplification for all the sensors that can be connected and house the Alveo card. On the other side, we can find the Voltage 39/91 Figure 6: Alveo satellite controller inferred internal structure; Own compilation Developing telemetry tools for Alveo accelerator cards Regulator Module, which is controlled by the SC and makes sure all power requirements are met. The device has four available slaves, each answering to a different address as per the standard of each protocol. Each slave behaves differently, as there are different constraints, such as the size of the packets of data or the available space for reading. The basic characteristics of these protocols are the following: - IPMI FRU: Usually used for reading and writing data to EEPROMs. Supports multi-byte read/write transactions. - SMBus: A subset of I2C designed for system management communication. Supports enables multi-byte read/write transactions. - I2C: General-purpose communication protocol for connecting multiple devices on a twowire bus. The packet size is flexible. - PLDM: Intended for managing platform-level data. Has heathers and payloads with different fields to configure the transaction. From these protocols, we understand that the minimum requisites to totally exploit the satellite controller sensor architecture are a controller sixteen bits wide with the capabilities of slave and master. The width will constrain the amount of logical addresses a device over SMBus can perform. As the following figure explains, this controller is embedded into the host system inside the peripheral controller hub (PCH). The PCH is where all the controllers of the different interfaces from and to the processor are positioned. 40/91 Figure 7: Requirements from host system; own compilation Developing telemetry tools for Alveo accelerator cards We tried on the following motherboards: -MSI MAG Z490 TOMAHAWK [MSI TOWER] -Supermicro X9DR3-F [Abuelo] -Dell Inc. OptiPlex 9020 (00V62H) [Sandy] -Dell Inc. PowerEdge T640 (07978V) [Quar] -Lenovo ThinkSystem SR670 V2 Server [MAKINOTE] -Gigabyte X570 AORUS PRO [MEEP workstations] After a manual update of the firmware from the driver’s repository available on Linux and an update to the BIOS in which no option or configuration had to do with SMBus enabling or disabling, the same faulty behaviour was observed. Other problems observed were that eight of the servers from the MEEP workstations, which had one FPGA each, were that the SMBus controller did not bind itself properly with the drivers, and it seems there are not any available drivers for our controller, “FCH SMBus Controller '1022:790e' ”. Even after opening tickets with Gigabyte, no solution could be found. Finally, there is another problem: in the official documentation from Xilinx, it is stated that the SMBus has ARP (can be seen at figure 5). ARP stands for address resolution protocol, a feature that differentiates it from I2C. The main problem with I2C is the collision of addresses, and so ARP was designed to detect and avoid collisions, changing the addresses of some of the slaves connected to the bus. Experimentally, it was tested on Universitat Autonoma de Barcelona’s servers, which were the only ones capable of i2c/SMBus interchanges, that when two or more Alveos were connected in parallel to the PCIe connectors the programs written in python failed and even the i2c-tools, as when running the ‘i2cdetect’ command those addresses disappeared, one of those FPGA was one of BSC’s U50LV the other an U55C. Additionally, in those servers, we tested some of the functionalities like the cold and warm FPGA reset command (see table 16). It did not work. Theoretically, it was proved with the documentation of the actual manufacturer of the SC chip, Texas Instruments, model MSP-432. In their forum appeared a question asking for the implementation of the actual ARP protocol inside their libraries; their answer was negative. Additionally, Xilinx states that the SC firmware is proprietary and would not share it. The conclusion is then that either Xilinx's statement about the existence of ARP is 47/91 Developing telemetry tools for Alveo accelerator cards incorrect, some prior management has to be done that is not explained/documented, or it does not work properly. The last hypothesis is that being the SMBus a very infraused bus, the different manufacturers do not include the support necessary for the transmission of the data between the chipset and the PCIe edge connector; this would be done to reduce the costs of the manufacturing process by reducing the amount of cables routed in the printed circuit board. This was tested for the ‘MSI MAG Z490 TOMAHAWK’ with an oscilloscope, and while the Alveo was connected, no signal from the SMBus clock or data pin was observed. This reason might explain why, even though successfully enabling the BMC of the ‘Supermicro X9DR3-F’ motherboard and connecting to the webpage, no data from the Alveo accelerator cards were detected inside the list of sensors available to read information except for the native CPU and GPU. Surely the motherboard does not have that SMBus routing. This belief was reinforced when we read the standard for SMBus implementation on PCI [22]. The PCI specification revision 3.0. states (page 271) “The SMBus interface connections on the PCI connector are optional” and additionally the ARP is considered an optional feature, which is also said at the specification of the SMBus [23]. Normally, the I2C protocol has either some jumpers or a logical address that can be set. This is true for the MSP430, at page 8 [24], Texas Instruments indicates that there is a function for the firmware that sets the slave’s address. This does not mean it has the ARP feature, which it does not. Also looking at page 5 we can see in the table that there is no ARP for MSP430, this information is not available for the MSP432 but in the forums of Texas Instruments it is said: “some features of SMBUS (like ARP, which requires slaveside arbitration not supported directly in our I2C peripheral) aren't available in hardware and thus must be bit-banged” [25]. In conclusion, although the official documentation from Xilinx states that the SC has the ability to perform ARP, it is erroneous. This limits the number of FPGAs capable of being connected to a suitable motherboard to perform OOB to one, impractical in our case. Supposing the host is capable of at least accessing 16-bit addresses and behaves like a master and/or slave, and that it has support for SMBus. In our case, as no SMBus support was observed and we could not unplug 84 FPGAs from the servers (leaving one per motherboard). For that reason, the development of the OOB solution was stopped, and we began the implementation of the in-band solution. 48/91 Developing telemetry tools for Alveo accelerator cards 12. In-Band monitoring 12.1. Introduction Data is read through a Xilinx IP core called Card Management Solution (CMS), which in turn accesses the SC with some dedicated pins designed to interchange communication with that module. With the CMS we can read a lot of registers that are housed inside the SC and are updated continuously by the different sensors dispersed in the card and communicate internally with the I2C protocol. The following figure 9 shows the In-Band method. Whenever a device is connected with PCI, it can ask the host for some memory space to allocate data. The CMS is accessed by the dedicated reserved memory space inside the PCI sessions. Those sessions are subdivided into functions, and each function has different Base Address Registers (BAR). Also, they can be read through what are called Direct Memory Access (DMA) commands over the internal AXI protocol; although it may work, we have found some issues, and we do not recommend it. The physical functions (PF) are used to differentiate functionalities and normally are reserved for different drivers. XRT libraries can use PF 0 or PF 1. Apart from the PF we also have an amount of BARs inside a PF. As the access to the CMS is done through a special Axi Lite bus, it is compulsory that it resides at BAR 2. The CMS is configured through the Vivado workflow, a program designed by Xilinx to create all the necessary infrastructure to develop the bitstreams, which would be later inserted inside the FPGA region. Once a bitstream has been created and loaded into a suitable FPGA, we would then have only to develop a program that accesses that memory space, transforms it, and analyses the data. 49/91 Figure 9: AlveoTM U2XX, U50X, U55X, UL3524 block diagram; Modified Xilinx image Developing telemetry tools for Alveo accelerator cards With a C or Python program, those BAR regions can be read directly. Knowing from the Vivado address editor the direction in which the CMS is housed, we can then perform readings inside the BAR2 region given all prior configuration has correctly been satisfied. Except for the Xilinx libraries, there is a small protocol by which to read the data successfully, which is common for all the different methods explained in the subsequent chapters. First, we perform a write operation with a value of just one into the register called “MB_RESETN_REG”, as this given register is active low (activates reset on zero), we must make sure its content is different from zero. Once the write operation has ended, we can perform all the read operations from the list of available sensors specified in the documentation [26]. We can toggle additional configuration to read the HBM temperature sensors or the QSFP low-speed management feature, although strange behaviour has been found on some sensors. 12.2. Implementation To implement the In-Band telemetry, there are two distinct ways. First, using the Xilinx Run-Time libraries, which are already available, reliable, and easy to use if used on Xilinx platforms. And secondly, when actually integrating the IP core used for the In-Band module, we can do it through the BAR region of the PCIe. 12.2.1. Xilinx Run-Time libraries Xilinx Run-Time libraries have already made all necessary software infrastructure available for obtaining all the desired information required for the analysis of data; the issue resolves in the actual workflow at the Barcelona Supercomputing Center. This center works with an already established infrastructure, such as the OpenPiton project. OpenPiton is the world's first open-source, general-purpose, multithreaded manycore processor. It is a tiled manycore framework scalable from one to 1/2 billion cores. It is a 64-bit architecture using SPARC v9 ISA with a distributed directory-based cache coherence protocol across on-chip networks, and it is highly configurable. When designing a processor, it is vital to actually test it. Computer simulations could take up to three days to get completed, so the main way to debug the processors is to emulate them on FPGAs. The problem comes when uploading a design into the FPGA; if done directly, a lot of issues may arise about the designing of a socket for the processor, and for every FPGA, a different socket would be needed to design. So, to abstract the register transfer level (RTL) team, all the necessities of each FPGA, such as all the different pins and interfaces contained, the OpenPiton project was born. A subproject currently used by BSC is the wrapper shell, OpenPiton, plus ACME, and later the embedded shell project, known also as the "fpga shell, was born. 50/91 Developing telemetry tools for Alveo accelerator cards The main issue resides in the fact that all these shells revolve around the use of the QDMA engine, which stands for ‘Queue Direct Memory Access.’ This engine provides a different interface from its competitor, XDMA, made by Xilinx. These two engines are not directly compatible because they use different interfaces, and so for this project, although it has been researched, the potential of XRT libraries has not been used in the final program, as XRT only works with XDMA, which the BSC does not use. These libraries should get installed alongside Vivado or Vitis. On Ubuntu-based environments, it can be found in the ‘/opt/xilinx/xrt/bin’ folder. Before proceeding into the commands, there are some considerations. The following useful commands are from the XRT libraries: $ sudo /opt/xilinx/xrt/bin/xbmgmt examine Outputs the information about the amount and which Xilinx FPGAs devices are connected, giving the states of the cards and its BDFs. $ sudo /opt/xilinx/xrt/bin/xbmgmt examine -r all Outputs the information about the FPGA configuration such as temperature thresholds and more. $ /opt/xilinx/xrt/bin/xbutil examine -d <BDF> -r electrical thermal Outputs the information regarding electrical and thermal, such as voltages, currents, power consumption and temperature of some sensors. $ sudo /opt/xilinx/xrt/bin/xbutil validate -d <BDF> -r all Performs all available tests for the given device. Some commands require the use of privileges. Before executing these commands, make sure that Xbmgmt and Xocl are the same version, if not incompatibilities may arise and also the latest version is installedV. There should not be any problem in developing a tool that from the previous commands, extracts and parses the information into a graphical user interface, but as mentioned before, the final implementation of the telemetry tool does not use the XRT. The reason XRT works is thanks to the so-called ‘platforms’ which is a comprehensive ecosystem of tools, software, hardware and methodologies provided to support all the workflow, such as designing, developing, programming and deploying unto the FPGAs. The platforms when loaded with the golden bitstream come already with the CMS integrated, that's why overwriting these platforms disables the XRT environment and thus the monitoring tools. V Additional trobuleshooting on: https://xilinx.github.io/Alveo-Cards/master/debugging/build/html/docs/xrt-troubleshooting.html 51/91 Developing telemetry tools for Alveo accelerator cards 12.2.2. Base Address Registers The path we have decided is best is through the BAR regions, as they can be accessed in a more straightforward manner. The interfaces declared in the CMS are: •s_axi_ctrl: Axi-Lite control slave interface. •satellite_gpio: GPIO signals from SC. •aclk_ctr: Source clock, make sure it is a 50MHz Clock. •aresetn_ctrl: Reset pin active low reset. (We recommend to connect it to PCI reset) •interrupt_hbm_cattrip: Ored HBM cattriped signals. •Hbm_temp_1 and 2: HBM temperature sensor bus. •satellite_uart: UART signals to SC. •interrupt_host: Interrupt generated by CMS. In the following explanation it will only be described the connection with the CMS from the U55C and U280 which are equivalent. 52/91 Figure 10: Card Management Solution IP interface image for U280 and U55C; Source Xilinx Developing telemetry tools for Alveo accelerator cards Exhaustive steps to build successfully a CMS project: Create or load a project for Vivado, preferably with an already existing DMA engine such as the QDMA/XDMA IP, an HBM controller, and also a PCI core. Easily you can add a new IP with a right click into the block design; search for the ‘Card Management Solution Subsystem.’ This exact module does not need any additional configuration, only the placement and routing of its interfaces. When the CMS is instantiated into the block design, a board preset is applied; this means the board-specific CMS will get selected as differences exist among these modules. Additionally, add a ‘Processor System Reset,’ the same way as the CMS. Make sure double-clicking the QDMA IP, on the top button 'PCIe: BARs,' the size of the 'AXI Lite Master' is at least 256KB; otherwise, problems should be expected. Connect both HBM’s DRAM_STAT_TEMP to the CMS’s ones. Or all the signals from the HBM’s cattrip and place it to interrupt_hbm_cattrip. The clock source of the CMS must be a 50 MHz clock and also the slowest_sync_clk from the reset controller, and finally, we recommend placing the PCIe reset to the ext_reset_in. If there exists an AXI interconnect lite connected to the QDMA M_AXI_LITE port, create an additional master output, with the clock connected to the same source as the aclk_ctrl and the reset of the master interface to the processor system reset interconnect_aresetn. The new master interface routes it to s_axi_ctrl. From the Processor System Reset, connect the ‘ext_reset_in’ with the ‘axi_aresetn’ from the QDMA IP. Basically, this module will synchronise all resets with the clock source from the CMS (QDMA and CMS have different clock sources). Once all intra-signals have been connected, we must make two external interfaces for the satellite_gpio and satellite_uart, left-click those interfaces and click ‘make external.’ To connect the interfaces, it is needed to modify the ‘system_top.sv’ file, written in System Verilog, there is needed to add the CMS new signals to the interfaces section as: input [0:0] satellite_uart_rxd, output [0:0] satellite_uart_txd, input [3:0] satellite_gpio, And finally, to end the implementation, add the specific constraints for the FPGA signals to the constraints folder, those can be extracted from the Xilinx website for each specific AlveoVI. FPGA pin constraints define the rules and requirements for how FPGA pins are connected, configured, and utilised in hardware designs. VI Example for U280: https://www.xilinx.com/products/boards-and-kits/alveo/u280.html#vivado 53/91 Developing telemetry tools for Alveo accelerator cards Figure 11 shows the final block design for the socket. 54/91 Figure 11: Final implementation of the FPGA Shell with CMS, block design view; own compilation APB_rst_and Utility Vector Logic Op1[0:0] Op2[0:0] Res[0:0] Eth100GbSyst_w_hbm m_axi_rx m_axi_sg m_axi_tx qsfp_4x qsfp_4x_grx_n[3:0] qsfp_4x_grx_p[3:0] qsfp_4x_gtx_n[3:0] qsfp_4x_gtx_p[3:0] s_axi qsfp_refck qsfp_refck_clk_n qsfp_refck_clk_p intc[1:0] rx_clk rx_rstn[0:0] s_axi_clk s_axi_resetn tx_clk tx_rstn[0:0] MEEP_uart_0 S_AXI interface_uart s_axi_aclk s_axi_aresetn ip2intc_irpt rin sin sout SHELL_ROM S_AXI s_axi_aclk s_axi_aresetn axi_gpio_0 AXI GPIO S_AXI GPIO gpio_io_o[4:0] s_axi_aclk s_axi_aresetn axi_xbar_pcie AXI Interconnect S00_AXI M00_AXI M01_AXI ACLK ARESETN S00_ACLK S00_ARESETN M00_ACLK M00_ARESETN M01_ACLK M01_ARESETN axi_xbar_pcie_lite AXI Interconnect S00_AXI M00_AXI M01_AXI ACLK ARESETN S00_ACLK S00_ARESETN M00_ACLK M00_ARESETN M01_ACLK M01_ARESETN bscan_prim Debug Bridge m0_bscan m1_bscan bscan2jtag BSCAN to JTAG Converter S_BSCAN M_JTAG clk_wiz_1 Clocking Wizard CLK_IN1_D clk_out1 clk_out2 locked cms_reset Processor System Reset slowest_sync_clk ext_reset_in aux_reset_in mb_debug_sys_rst dcm_locked mb_reset bus_struct_reset[0:0] peripheral_reset[0:0] interconnect_aresetn[0:0] peripheral_aresetn[0:0] cms_subsystem Card Management Solution Subsystem s_axi_ctrl satellite_uart satellite_gpio[3:0] aclk_ctrl aresetn_ctrl interrupt_host interrupt_hbm_cattrip[0:0] hbm_temp_1[6:0] hbm_temp_2[6:0] dbg_jtag debug_hub Debug Bridge S_BSCAN clk eth_axi eth_axi_arstn[0:0] eth_irq[1:0] hbm_cattrip[0:0] hbm_cattrip_or Utility Vector Logic Op1[0:0] Op2[0:0] Res[0:0] hbm_0 HBM IP SAXI_00 SAXI_01 SAXI_02 SAXI_29 SAXI_30 SAXI_31 HBM_REF_CLK_0 HBM_REF_CLK_1 AXI_00_ACLK AXI_00_ARESET_N AXI_00_WDATA_PARITY[31:0] AXI_01_ACLK AXI_01_ARESET_N AXI_01_WDATA_PARITY[31:0] AXI_02_ACLK AXI_02_ARESET_N AXI_02_WDATA_PARITY[31:0] AXI_29_ACLK AXI_29_ARESET_N AXI_29_WDATA_PARITY[31:0] AXI_30_ACLK AXI_30_ARESET_N AXI_30_WDATA_PARITY[31:0] AXI_31_ACLK AXI_31_ARESET_N AXI_31_WDATA_PARITY[31:0] APB_0_PCLK APB_0_PRESET_N APB_1_PCLK APB_1_PRESET_N AXI_00_RDATA_PARITY[31:0] AXI_01_RDATA_PARITY[31:0] AXI_02_RDATA_PARITY[31:0] AXI_29_RDATA_PARITY[31:0] AXI_30_RDATA_PARITY[31:0] AXI_31_RDATA_PARITY[31:0] apb_complete_0 apb_complete_1 DRAM_0_STAT_CATTRIP DRAM_0_STAT_TEMP[6:0] DRAM_1_STAT_CATTRIP DRAM_1_STAT_TEMP[6:0] mc_rstn[0:0] mem_axi mem_calib_complete[0:0] ncmem_axi pci_express_x16 pcie_gpio[4:0] pcie_refclk ExtArstn chipset_clk eth_axi_aclk mc_clk pcie_perstn qsfp_ref_clk_n qsfp_ref_clk_p resetn rs232_rxd rs232_txd uart_irq vpu_clk proc_sys_rst_pcie Processor System Reset slowest_sync_clk ext_reset_in aux_reset_in mb_debug_sys_rst dcm_locked mb_reset bus_struct_reset[0:0] peripheral_reset[0:0] interconnect_aresetn[0:0] peripheral_aresetn[0:0] qdma_0 Queue DMA Subsystem for PCI Express M_AXI M_AXI_LITE pcie_mgtusr_irq dsc_crdt_in tm_dsc_sts tm_dsc_sts_rdyusr_flr qsts_out qsts_out_rdy sys_clk sys_clk_gt sys_rst_n user_lnk_up axi_aclk axi_aresetn soft_reset_n phy_ready qsfp_4x_grx_n[3:0] qsfp_4x_grx_p[3:0] qsfp_4x_gtx_n[3:0] qsfp_4x_gtx_p[3:0] rst_ea_CLK0 Processor System Reset slowest_sync_clk ext_reset_in aux_reset_in mb_debug_sys_rst dcm_locked mb_reset bus_struct_reset[0:0] peripheral_reset[0:0] interconnect_aresetn[0:0] peripheral_aresetn[0:0] rst_ea_CLK1 Processor System Reset slowest_sync_clk ext_reset_in aux_reset_in mb_debug_sys_rst dcm_locked mb_reset bus_struct_reset[0:0] peripheral_reset[0:0] interconnect_aresetn[0:0] peripheral_aresetn[0:0] satellite_gpio[3:0] satellite_uart sysclk0 sysclk1 uart_axi util_ds_buf Utility Buffer CLK_IN_D IBUF_OUT[0:0] IBUF_DS_ODIV2[0:0] util_ds_buf_hbm_clk Utility Buffer CLK_IN_D IBUF_OUT[0:0] vdd_0 Constant dout[0:0] Developing telemetry tools for Alveo accelerator cards The CMS project has been incorporated into the FPGA-SHELL; this enables all processors made at BSC that use this wrapper to provide telemetry on FPGAs without any change in the current workflow, as the CMS has its own embedded processor called MicroBlaze. Prior to any reading, we must disable the reset register; as it is an active low, we must write a '1' to that register, positioned on offset 0x20000 from the starting position of the CMS module. Once written, we can read the registers, taking into account the starting address and the offset of the given registers. We can guarantee the CMS works by performing a read into the 0x28000 which is the Register Map ID, with a fixed value of 0x74736574 or to the 0x28014 which stores the type of Alveo card we are using (e.g. 0x0x5535354E for U55C). The information is read thanks to the different libraries in C and Python. The Python library in question is ‘pypcie’. The inputs required to make it work are the BDF and the initial address where the CMS is placed. The BDF can be known using the ‘lspci’ command and searching for ‘Processing accelerators: Xilinx Corporation Device’; that identifier is located on the right side. If the ‘- vvv’ option is added, we can observe additional important data, such as the regions, also known as BAR. It must be seen in the following “Region 2: Memory at … [size=...]” the size at least being 256Kb if done correctly (in FPGA-SHELL the minimum is 512Kb). And as already mentioned, the initial address can be fetched from the Vivado address editor. The program is self-contained; that means it will automatically detect which type of Alveo card is connected to the host system. After the initialisation of the data structures, it will output whether it has been connected successfully. If no errors are shown, the program will create a list of the available sensors each card has, and every half a second it will query all of those, normalise them (i.e., change the units to standard), and output the timestamp once all values are shown on the screen. In the figure 12 is an example of a run made in three different nodes. Two of those house a U280 and the rightmost an U55C. The program on the header shows the Alveo type, its BDF and the vendor and the device number. Xilinx specifies that the total power consumption is the sum of the powers consumed in the PEX and AUX channels and for each voltage: Total power = Power(12V_PEX + 12V_AUX + 3V3_PEX + 3V3_AUX) Obviously, each power value is the multiplication of x_<PEX/AUX> * x_<PEX/AUX>_I where ‘x’ is either 3V3 or 12V and it comes from PEX or AUX sensors (‘I’ meaning Intensity/Current). 55/91 Developing telemetry tools for Alveo accelerator cards For example, on the U55C the power consumption can only be calculated from the PEX channels, as there aren’t auxiliary ones, and for the U280 there are only sensors for the 12V channels (12V_AUX+12V_PEX). Strange behaviours have been observed, such as missing or disabled sensors, as can be seen in figure 12. For example, when the program was run on different nodes with the same type of Alveo cards connected, there were sensors whose reading was zero on one and data on the other; maybe some sensors have been damaged or rendered inoperable. In this case, the PICU node ‘CAGE_TEMP0’ is zero, and in NANU with the same card, its value is 32ºC. Additionally, even after doing the manual enabling of the HBM sensors, the program has been able to read values from those sensors only when the processor has been booted. The enabling of these is also included in the program. As the official documentation says, the 27th bit must be written with one; after writing it and waiting some time, there are also different types of register, and the type of register that holds the average value can be reset to zero if necessary. 56/91 Figure 12: CMS output excerpt on three different nodes of the Meep Workstations; own compilation Developing telemetry tools for Alveo accelerator cards We can observe some behaviours that we did not know. The U280 fan behaves with a fixed constant velocity, as we can see in figure 16, there is no increase in revolutions despite the increase in temperature of the FPGA (figure 17). The red vertical line at figure 17 and 18 is due to the ‘Boot acme’ event, that is, the loading of the operating system on the FPGA. When loading the operating system, the restart signal ceases to be operational, and the processor begins to consume power almost immediately; the main consumer is the HBM memory. In figure 17 and 18 there is a trend lineVIII[33] generated automatically by Google Drive Sheets; this way we can perceive better the increase of the values. The main difference between those two figures is that consumption rises abruptly due to the enabling of some of the modules and stabilizes. Temperature increase is much slower. The main problem with all the figures is the amount of noise and poor precision. Temperature is truncated to the integer value, thus we can see some small rises in figure 17. Noise is also present at the fan speed and power sensor; in the case of the latter, the precision gets worse because the power data is not a result of a direct reading of a sensor but rather the multiplication and addition of the data from the current and voltages, which increases noise and decrements precision. Voltages and currents are interpreted as millivolts and milliamperes. VIIIR² indicates the determination coefficient. Which indicates how a model (in this case the black line) is as close as the real data. The closer the R², the closer to the real data. 63/91 Developing telemetry tools for Alveo accelerator cards 13.1.2. Second test; One U55C FPGA The second test was launched with the Wrapper-Shell using HBM memory alongside four cores from the Lagarto Hun plus two vectorial processing units on a U55C from the Meep workstations (CUCU). The U55C is a passively cooled FPGA. The test lasted approximately two hours. 64/91 Figure 19: Temperature over time (U55C); Own compilation Figure 20: Total power over time (U55C); Own compilation Developing telemetry tools for Alveo accelerator cards With the U55C the behaviour observed in temperature and power differs greatly from those in the previous figures. First we observe in figure 19 that the initial temperature of the U55C is much higher than that of the U280, that it is because they have a fan and the U55C is passively cooled, also perhaps because in the data centre, they are closer to the ventilation. The methodology we’ve followed is to first load the bitstream of the design and let the temperature stabilise, then boot the FPGA and wait some time, and finally, after it has stabilised as well, reload the same bitstream again. Following this procedure we can make apparent the relation between the temperature and power. Temperature increases more sharply, from the initial 64ºC to about 85ºC. What we can observe is that as the temperature increases, so too does the power consumed. This cannot be observed in the U280 (figure 18), at least as clearly as in figure 20. “The relationships between power and temperature are complex: an increase in dynamic power increases the total power, which increases the temperature, which increases the leakage power, which increases the total power, which increases the temperature (and so on). There is thus a cyclic dependence between power and temperature, which means that a small increase in power leads to a small increase in temperature, which further leads to a small increase in power, and so on. In most cases, this process converges; however, in some cases it does not and this leads to a very dangerous condition called thermal runaway.” [34]. A difference between the initial consumption of the U55C (figure 20) and U280 (figure 18) can be explained primarily by the additional consumption of the fan, the difference is approximately of four watts, which makes sense; the U280 specification doesn’t state the fan consumption [35]. We must remember that two implementations runs of the same design on the same tool can result in different bitstreams. As some of the stages in the FPGA design flow are non-deterministic when running the flow on more than one thread [36]. In the reloading event the FPGA is virtually eliminated from the PCI devices until the bitstream is completely loaded, that is why there is an approximate two minute gap in the data (figure 20, second vertical red line). 13.1.3. Third test; Eight simultaneous U55C FPGAs The third test was made on the Makinote node twelve with eight FPGAs. Loaded with the Wrapper-Shell alongside four cores from the Lagarto Hun plus two vectorial processing units into eight U55C FPGAs. The test lasted forty-five minutes. 65/91 Developing telemetry tools for Alveo accelerator cards In the following figures 21 and 22 there are different FPGAs running different tests based on the same bitstream. In the legend on the right of the figures, you can see the BDF identifier, along with the type of test that has been performed. 'Booted' implies that the FPGA has only loaded the operating system. 'Stressed' indicates that tests have been launched within the operating system session that stress the cores. Finally, 'idle' indicates that only the bitstream has been loaded into the FPGA and nothing else. The most curious feature is that the initial consumption among different FPGAs differs with a high degree of variance. Between the least (yellow line) and the highest (pale blue) initial consumptions, there is approximately a whole watt. 66/91 Figure 21: Total power over time using eight U55C FPGAs; Own compilation Figure 22: Temperature over time using eight U55C FPGAs; Own Compilation Developing telemetry tools for Alveo accelerator cards When launching the procedure to boot the different FPGAs, it has to be done one by one, therefore there is a delay between them. To ensure that there is a correlation between the temperature increase and the power, we have calculated the value. As it is very close to '1' we understand that it is practically perfect. Power 12V PEX Power 12V AUX Power 3V3 PEX Total power Temperature Correlation between power and temperature increase Increase Increase Increase Increase Increase BOOTED 19:00.0 41.23% 42.01% 6.71% 34.67% 24.67% 99.24% 1a:00.0 37.46% 48.64% 6.84% 33.94% 26.32% 99.48% BOOTED + STRESSED 33:00.0 39.41% 52.26% 8.00% 36.08% 26.83% 99.78% 34:00.0 40.31% 46.69% 7.33% 35.30% 22.67% 98.62% Table 20: Analysis of power and temperature increment; Own compilation For table 21, ‘pre’ means before and ‘post’ after the booting event. Power 12V PEX Power 12V AUX Power 3V3 PEX Total power Average pre Average post Average pre Average post Average pre Average post Average pre Average post BOOTED 19:00.0 9.54 13.48 4.19 5.96 3.34 3.56 17.07 22.99 1a:00.0 10.53 14.48 3.61 5.37 3.33 3.56 17.48 23.41 BOOTED STRESSED 33:00.0 10.05 14.00 3.52 5.36 3.22 3.48 16.79 22.85 34:00.0 10.12 14.21 3.61 5.30 3.28 3.52 17.02 23.03 IDLE b3:00.0 10.44 3.75 3.28 17.47 b4:00.0 9.84 4.59 3.26 17.68 cc:00.0 10.18 4.58 3.25 18.01 cd:00.0 10.31 4.08 3.30 17.69 Table 21: Average absolute values for previous table; Own compilation 67/91 Developing telemetry tools for Alveo accelerator cards A final observation that we made is that we saw, as seen in table 22, that the standard deviation of some records tends to be zero, which is very surprising to us since it is very rare for the values not to vary over the whole time the test was running. These values are indicated with a red cell. Power 12V PEX Power 12V AUX Power 3V3 PEX Total power Temperature SD pre SD post SD pre SD post SD pre SD post SD pre SD post SD pre SD post BOOTED 19:00.0 0.031 0.145 0.001 0.062 0 0.008 0.031 0.213 0 0.608 1a:00.0 0.047 0.143 0.023 0.064 0.004 0.009 0.050 0.216 0.456 0.412 BOOTED STRESSED 33:00.0 0.001 0.137 0.004 0.080 0 0.009 0.004 0.218 0 0.442 34:00.0 0.021 0.141 0.028 0.087 0.013 0.009 0.031 0.215 0.488 0.649 IDLE b3:00.0 0.003 0.044 0 0.044 0.146 b4:00.0 0 0.001 0 0.001 0.443 cc:00.0 0.048 0.004 0 0.049 0 cd:00.0 0.051 0.037 0 0.055 0.482 Table 22: Compilation of standard deviation (SD) per sensor; Own compilation 68/91 Developing telemetry tools for Alveo accelerator cards 13.1.4. Fourth test; Eight simultaneous U55C FPGAs The last test was made on the Makinote node twelve with eight FPGAs. All the FPGAs have a bitstream that uses a different shell called Embedded-shell with only one core of SargantanaIX. From these two groups, the first was booted and stressed, the second only booted. To stress the cores, a series of concurrent programs were executed continuously that do not access memory. Otherwise, as the HBM memory consumes a lot of power, it would add too much noise, up to eighteen watts, according to a power report from Vivado. The important data from this test is to see the increase of power consumption when running software on the emulated processors. The period between the vertical black lines of both figure 23 and 24 indicates the process of loading the stress test on each FPGA; the data between the two black lines has not been used for the calculations on table 23. IX Sargantana is an in order processor with a RISC-V architecture made at the BSC. 69/91 Figure 23: Power over time for stressed U55Cs; Own compilation Figure 24: Temperature over time for non-stressed U55Cs; Own compilation Developing telemetry tools for Alveo accelerator cards Additionally, for each figure, there is a pale line colored which indicates the tendency of the values. In figure 24 it is almost impossible to see due to values not changing. Temperature in this test was monitored too, but in this case the Makinote FPGAs are sufficiently regulated, maintaining a steady average among all FPGAs, which can be seen in table 23. When running this test, we found that, while it is common sense to observe an increase in power consumption when running software on an emulated platform, as there is an increase in the activity factor of the transistors, in previous tests, it was not shown as clearly as in this test (figure 23). Although there is not a strong increase in temperature as a result (figure 24). AVG Temperature (T) AVG Total Power Cons. (W) Pre stress Post stress Pre stress Post stress Temperature inc. Total Power inc. 19 38.00 38.00 18.80 18.99 0.00% 1.01% 33 38.13 38.22 18.57 18.56 0.23% -0.04% B3 38.03 38.07 19.24 19.43 0.12% 0.99% CC 39.01 39.00 19.82 19.82 -0.02% 0.01% 1A 37.00 37.00 19.29 19.29 -0.01% 0.02% 34 38.00 38.00 18.76 18.92 0.00% 0.85% b4 38.00 38.00 19.44 19.44 0.00% -0.04% CD 38.70 38.79 19.50 19.69 0.21% 0.98% Table 23: Power increase due to stressing emulated processors on FPGAs; own compilation Vivado power report estimates the Sargantana core to consume 0.405. From table 23 the average increase in temperature for said HDL design is estimated to be 0.95%, 180mW. 70/91 Developing telemetry tools for Alveo accelerator cards 13.2. Profiling Telemetry would not provide insights for emulated processors, since there are no direct correlations between the consumed power from an FPGA and the final processors on a printed circuit board. Briefly below I will explain the reason. Printed circuit boards use standard cells, which are a grouping of transistors and an interconnecting structure that provides Boolean logic functions or storage. The boolean logic can then create complex behaviours, which are what almost all modules are made of [37]. FPGA, in contrast, works with lookup tables, which are memories filled with all the different inputs and, for each input, its output. The principle is that the behaviour of an emulated hardware is calculated and stored in the memories. From figures 25 and 26, the differences can be seen between a standard cell and an FPGA cell. Note that not all FPGA cells are equal; some have different types of resources like BRAM tiles, arithmetic units, LUTRAM, among others [38]. Although we can profit from the fact that any FPGA would be less efficient than any ASIC one. Our process to develop any hardware design is to create the most efficient function that provides the required functionalities. That is done through Karnaugh[39], which simplifies the algebraic expressions into Boolean and then outputs the most efficient solution. FPGAs do the opposite; they provide all the solutions, even the ones we do not care for. Not only that, the structure of the FPGA significantly increases the leakage power due to the extra routing and structures needed. 71/91 Figure 25: Standard cell schematic; source Figure 26: FPGA cell schematic; source Developing telemetry tools for Alveo accelerator cards The question is then, if an FPGA is less efficient, then calculating the consumption of the FPGA provides a higher reading and thus a top margin for power consumption once it is printed. The standard flow to calculate the consumption of any chip is through simulators, which calculate the total consumption based on the simulation runs and architecture. The formula for the power consumption is the following: Pstatic = VDD * IDD Pdynamic = 1 2 C * VDD² * f * α Pleakage = VDD * Ileak Pshortcircuit = t * VDD * Ishort * f The key problem comes from the uncertainty of the values from the previous formulas. To extrapolate the FPGA design to the ASIC design is to make a lot of assumptions, and the accumulation of error will inevitably induce a lot of difference between the calculated values and the real ones: capacitance, activity factor, technological differences from one scale to another. Even with the data from the synthesising simulations for ASICs, the difference can be of several factors. Now, doing profiling on an accelerator design intended for a specific FPGA can give better results. The user can use this tool to calculate the differences between versions of the HDL code and the different Vivado strategiesX. To perform better profiling for the accelerators on the FPGA, we explored the called DFx, which stands for (Dynamical Function eXchange). Using this functionality an user can load on the fly different segments of the FPGA without the need to rewrite all the FPGA bitstream [40]. This can decrement the time for synthesis and implementation because the user can specify a static partition and a reconfigurable one. The static partition can be saved once implemented with a checkpoint file, which enables reuse of the netlists [41]. When profiling designs for the FPGA, one of the problems is that all the interfaces from the Shell (including the CMS) consume their own power. To differentiate from the consumption from the accelerator, shell DFx is used. Independently implementing the Shell and measuring its consumption as a standalone bitstream wouldn’t be precise enough because the netlists of the Shell with the accelerator (only taking the section regarding the shell) and the standalone shell would be different. X Vivado strategies are a set of configuration that constraint the synthesis and implementation algorithms to suit a specific need, like power consumption, utilisation, run-time, timing enclosure optimisations. 72/91 Developing telemetry tools for Alveo accelerator cards - ARP: Address Resolution protocol, a feature in SMBus designed to solve I2C address collisions. - XRT: Xilinx Run-Time, these are a set of libraries implemented by Xilinx to abstract the hardware from the user and obtain some useful services. - XDMA: Xilinx Direct Memory Access engine, which provides direct transfers between FPGA devices and host memory without CPU involvement [47]. - QDMA: It is an engine which stands for ‘Queue Direct Memory Access’, the main differences is that it has a set of queues [48]. - IP: Intellectual property core; a reusable logic unit which is used in a given design to add a functionality which is licensed from a vendor. - VPU: Vectorial Processing Unit. - ASIC: Application-Specific Integrated Circuit. 79/91 Developing telemetry tools for Alveo accelerator cards APPENDIX A) Report on telemetry capabilities on Versal V80 Recently the BSC acquired two V80 FPGAs from Xilinx. These are the most powerful ones we have, and the question of whether we could also add telemetry capability arose. That is why this appendix only wants to report the characteristics and how a future program on the V80 would have to be implemented. I will also talk about some differences and advantages. The V80 doubles the logic density, and the PCIe bandwidth compared to the U55C, almost doubles the memory bandwidth and quadruples the network bandwith [49]. Additionally the V80 internal structure is divided into two parts, PL ‘programmable logic’ and PS ‘processing subsystem’. Compared to the Ultrascale devices the greatest advantage is that they have managed to solve the I2C collision of addresses [50], but it has some downsides, the first being that everything related to the SMBus has to be implemented as an IP, that currently is available ajd still being developed by Xilinx, figure 27 shows the schematic of the system. This IP can be instantiated into the user design. Xilinx already wrote a library with the APIs for the telemetry feature[51]. Therefore, one of the great advantages that the version with an independent controller had is lost, since if the user wants telemetry, he will always have to make sure that he has this module included in a bitstream and loaded. In my opinion it is not very ideal. 80/91 Figure 27: Image showcasing the V80 SMBus implementation; source Xilinx Developing telemetry tools for Alveo accelerator cards APPENDIX B) Table: Supported sensors per Alveo card. Sensor Name U200/U250 U280 U50 U55 U45N X3 UL3422 UL3524 1V2_VCCIO 2V5_VPP23 3V3_AUX ✓ ✓ ✓ ✓ 3V3_PEX ✓ ✓ ✓ ✓ ✓ ✓ ✓ 3V3PEX_I_IN ✓ ✓ ✓ ✓ ✓ 12V_AUX ✓ ✓ ✓ ✓ ✓ ✓ 12V_AUX1 12V_AUX_I_IN ✓ ✓ ✓ ✓ ✓ ✓ 12V_PEX ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ 12V_SW ✓ ✓ 12VPEX_I_IN ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ AUX_3V3_I ✓ ✓ CAGE_TEMP0 ✓ ✓ ✓ ✓ ✓ ✓ CAGE_TEMP1 ✓ ✓ ✓ ✓ ✓ CAGE_TEMP2 CAGE_TEMP3 DDR4_VPP_BTM ✓ ✓ DDR4_VPP_TOP ✓ ✓ ✓ ✓ DDR4_VTT ✓ DIMM_TEMP0 ✓ ✓ DIMM_TEMP1 ✓ ✓ DIMM_TEMP2 ✓ DIMM_TEMP3 ✓ FAN_SPEED ✓ ✓ FAN_TEMP ✓ ✓ FPGA_TEMP ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ GTAVCC GTVCC_AUX HBM_1V2 ✓ ✓ HBM_1V2_I ✓ 81/91 Developing telemetry tools for Alveo accelerator cards Sensor Name U200/U250 U280 U50 U55 U45N X3 UL3422 UL3524 HBM_TEMP1 ✓ ✓ ✓ HBM_TEMP2 ✓ ✓ ✓ MGT0V9AVCC ✓ ✓ ✓ ✓ ✓ ✓ MGTAVCC ✓ ✓ MGTAVCC_I ✓ ✓ MGTAVTT ✓ ✓ ✓ ✓ ✓ ✓ MGTAVTT_I ✓ PEX_3V3_POWER ✓ PEX_12V_POWER ✓ POWER_GOOD ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ SE98_TEMP0 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ SE98_TEMP1 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ SE98_TEMP2 ✓ SYS_5V5 ✓ ✓ ✓ ✓ V12_IN_AUX0_I V12_IN_AUX1_I V12_IN_I VCC0V85 ✓ ✓ VCC1V2_BTM ✓ ✓ VCC1V2_I VCC1V2_TOP ✓ ✓ ✓ ✓ VCC1V5 ✓ VCC1V8 ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ VCC3V3 ✓ ✓ ✓ ✓ ✓ ✓ VCC_5V0 VCCAUX ✓ ✓ VCCAUX_PMC VCCINT ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ VCCINT_I ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ VCCINT_IO ✓ ✓ VCCINT_IO_I ✓ ✓ VCCINT_POWER VCCINT_TEMP ✓ ✓ ✓ ✓ ✓ ✓ ✓ VCCINT_VCU_0V9 82/91 Developing telemetry tools for Alveo accelerator cards Sensor Name U200/U250 U280 U50 U55 U45N X3 UL3422 UL3524 VCCRAM VCCSOC VPP2V5 ✓ ✓ Table 25: Supported sensors per Alveo card; Source Xilinx. Indicates the sensor is supported. Empty cells indicate that the sensor is not supported.✓ APPENDIX C) Table: Resource utilization for CMS. Part Information Resource Utilization Device Package Speed-Grade LUTs FFs DSPs 36k BRAMs 18k BRAMs xcu280 fsvh2892 2L 3702 3700 4 34 0 xcu50 fsvh2104 2L 3702 3700 4 34 0 xcu50 fsvh2104 2LV 3701 3700 4 34 0 xcu55c fsvh2892 2L 3697 3700 4 34 0 xcu200 fsgd2104 2 3820 3883 4 34 0 xcu250 figd2104 2L 2818 3883 4 34 0 Table 26: Resource utilization for card management solution; Source Xilinx 83/91 Developing telemetry tools for Alveo accelerator cards APPENDIX D) Table: Compilation of known sensors, offsets, presence mask, and division factor. //RESET_REG (CMC/XMC/CMS) HAS A DIFFERENT OFFSET #define RESET_REG 0x0 #define DISABLE_RESET 0x0 #define ENABLE_RESET 0x1 //THESE REGISTERS HAVE A DIFFERENT OFFSET #define FIRMWARE_SIZE 0x20000 #define REG_MAP_1_SIZE 0x2000 #define REG_MAP_ID_REG_VALUE 0x74736574 // Magic value #define REG_MAP_ID_REG 0x000 // Magic register #define FW_VERSION_REG 0x004 #define STATUS_REG 0x008 #define ERROR_REG 0x00C #define PROFILE_NAME_REG 0x014 #define CONTROL_REG 0x018 //FPGA BOARD TYPE (read from PROFILE_NAME_REG) #define U200_U250 0x55325858 #define U280 0x55323830 #define U50 0x55353041 #define U55 0x5535354E #define U45N 0x55323641 #define X3 0x58334100 #define UL3422 0x55333432 #define UL3524 0x55333234 #define MASK_U200_U250 0b10000000 #define MASK_U280 0b01000000 #define MASK_U50 0b00100000 #define MASK_U55 0b00010000 #define MASK_U45N 0b00001000 #define MASK_X3 0b00000100 #define MASK_UL3422 0b00000010 #define MASK_UL3524 0b00000001 //Offsets for MAX, AVG and INS regs #define MAX 0x0 //Maximum value register #define AVG 0x4 //Averaged value register #define INS 0x8 //Instantaneous value register //SENSORS NAME, Sensor string offset, Mask, DIV_Factor, Known Location #define REG_12V_PEX "12V_PEX (V)", {0x020, 0b11111111, 1000} #define REG_3V3_PEX "3V3_PEX (V)", {0x02C, 0b11111110, 1000} #define REG_3V3_AUX "3V3_AUX (V)", {0x038, 0b11000101, 1000} #define REG_12V_AUX "12V_AUX (V)", {0x044, 0b11011011, 1000} #define REG_DDR4_VPP_BTM "DDR4_VPP_BTM (V)", {0x050, 0b11000000, 1000} //DDR voltage ¿BANK 1? #define REG_SYS_5V5 "SYS_5V5 (V)", {0x05C, 0b11110000, 1000} #define REG_VCC1V2_TOP "VCC1V2_TOP (V)", {0x068, 0b11000011, 1000} 84/91 Developing telemetry tools for Alveo accelerator cards #define REG_VCC1V8 "VCC1V8 (V)", {0x074, 0b11111111, 1000} #define REG_VCC0V85 "VCC0V85 (V)", {0x080, 0b11000000, 1000} #define REG_DDR4_VPP_TOP "DDR4_VPP_TOP (V)", {0x08C, 0b11000011, 1000} //DDR voltage ¿BANK 2? #define REG_MGT0V9AVCC "MGT0V9AVCC (V)", {0x098, 0b11110011, 1000} #define REG_12V_SW "12V_SW (V)", {0x0A4, 0b11000000, 1000} #define REG_MGTAVTT "MGTAVTT (V)", {0x0B0, 0b11110011, 1000} #define REG_VCC1V2_BTM "VCC1V2_BTM (V)", {0x0BC, 0b11000000, 1000} #define REG_12V_PEX_I_IN "12V_PEX_I_IN (A)", {0x0C8, 0b11111111, 1000} #define REG_12V_AUX_I_IN "12V_AUX_I_IN (A)", {0x0D4, 0b11011011, 1000} #define REG_VCCINT_V "VCCINT_V (V)", {0x0E0, 0b11111111, 1000} // LUTs + DSP ... #define REG_VCCINT_I "VCCINT_I (A)", {0x0EC, 0b11111111, 1000} #define REG_FPGA_TEMP "FPGA_TEMP (C)", {0x0F8, 0b11111111, 0x01} #define REG_FAN_TEMP "FAN_TEMP (C)", {0x104, 0b11000000, 0x01} //FAN #define REG_DIMM_TEMP0 "DIMM_TEMP0 (C)", {0x110, 0b11000000, 0x01} #define REG_DIMM_TEMP1 "DIMM_TEMP1 (C)", {0x11C, 0b11000000, 0x01} #define REG_DIMM_TEMP2 "DIMM_TEMP2 (C)", {0x128, 0b10000000, 0x01} #define REG_DIMM_TEMP3 "DIMM_TEMP3 (C)", {0x134, 0b10000000, 0x01} #define REG_FAN_SPEED "FAN_SPEED (RPM)", {0x164, 0b11000000, 0x01} //FAN #define REG_SE98_TEMP0 "SE98_TEMP0 (C)", {0x140, 0b11111111, 0x01} //PCB TOP FRONT #define REG_SE98_TEMP1 "SE98_TEMP1 (C)", {0x14C, 0b11111111, 0x01} //PCB TOP REAR #define REG_SE98_TEMP2 "SE98_TEMP2 (C)", {0x158, 0b10000000, 0x01} //PCB BTM FRONT #define REG_CAGE_TEMP0 "CAGE_TEMP0 (C)", {0x170, 0b11111100, 0x01} //QSFP 0 #define REG_CAGE_TEMP1 "CAGE_TEMP1 (C)", {0x17C, 0b11011100, 0x01} //QSFP 1 #define REG_CAGE_TEMP2 "CAGE_TEMP2 (C)", {0x188, 0b00000000, 0x01} //QSFP 2 #define REG_CAGE_TEMP3 "CAGE_TEMP3 (C)", {0x194, 0b00000000, 0x01} //QSFP 3 #define REG_HBM_TEMP "HBM_TEMP (C)", {0x260, 0b01110000, 0x01} #define REG_VCC3V3 "VCC3V3 (V)", {0x26C, 0b00111111, 1000} #define REG_3V3_PEX_I "3V3_PEX_I (A)", {0x278, 0b00111110, 1000} #define REG_VCC0V85_I "VCC0V85_I (A)", {0x284, 0b00000000, 1000} #define REG_HBM_1V2 "HBM_1V2 (V)", {0x290, 0b00110000, 1000} //HBM voltage #define REG_VPP2V5 "VPP2V5 (V)", {0x29C, 0b00110000, 1000} #define REG_VCCINT_BRAM "VCCINT_BRAM (V)", {0x2A8, 0b11111111, 1000} #define REG_HBM_TEMP2 "HBM_TEMP2 (C)", {0x2B4, 0b01110000, 0x01} //HBM #define REG_12V_AUX1 "12V_AUX1 (V)", {0x2C0, 0b00000000, 1000} #define REG_VCCINT_TEMP "VCCINT_TEMP (C)", {0x2CC, 0b10111111, 0x01} #define REG_3V3_AUX_I "3V3_AUX_I (A)", {0x2F0, 0b00000101, 1000} #define HEARTBEAT 0x2FC // System registers, present on all FPGAs #define HOST_MSG_OFFSET 0x300 // System registers, present on all FPGAs #define HOST_MSG_ERROR 0x304 // System registers, present on all FPGAs #define HOST_MSG_HEADER 0x308 // System registers, present on all FPGAs #define STATUS2 0x30C // System registers, present on all FPGAs #define HEARTBEAT_ERR_CODE 0x310 // System registers, present on all FPGAs #define REG_VCC1V2_I "VCC1V2_I (A)", {0x314, 0b00000000, 1000} // HBM current #define REG_V12_IN_I "V12_IN_I (A)", {0x320, 0b00000000, 1000} #define REG_V12_IN_AUX0_I "V12_IN_AUX0_I (A)", {0x32C, 0b00000000, 1000} #define REG_V12_IN_AUX1_I "V12_IN_AUX1_I (A)", {0x338, 0b00000000, 1000} #define REG_VCCAUX "VCCAUX (V)", {0x344, 0b00000011, 1000} #define REG_VCCAUX_PMC "VCCAUX_PMC (V)", {0x350, 0b00000000, 1000} #define REG_VCCRAM "VCCRAM (V)", {0x35C, 0b00000000, 1000} #define REG_VCCINT_VCU_0V9 "VCCINT_VCU_0V9(V)", {0x380, 0b00000000, 1000} #define REG_1V2_VCCIO_MAX "1V2_VCCIO (V)", {0x38C, 0b00000000, 1000} #define REG_GTAVCC_MAX "GTAVCC (V)", {0x398, 0b00000000, 1000} #define REG_VCCSOC_MAX "VCCSOC (V)", {0x3B0, 0b00000000, 1000} #define REG_VCC_5V0_MAX "VCC_5V0 (V)", {0x3BC, 0b00000000, 1000} #define REG_2V5_VPP23_MAX "2V5_VPP23 (V)", {0x3C8, 0b00000000, 1000} #define REG_GTVCC_AUX_MAX "GTVCC_AUX (V)", {0x3D4, 0b00000000, 1000} 85/91 Developing telemetry tools for Alveo accelerator cards #define REG_HBM_1V2_I_MAX "HBM_1V2_I (A)", {0x410, 0b00010000, 1000} //HBM current #define REG_VCC1V5_MAX "VCC1V5 (V)", {0x41C, 0b00000001, 1000} #define REG_MGTAVCC_MAX "MGTAVCC (V)", {0x428, 0b00000011, 1000} #define REG_MGTAVTT_I_MAX "MGTAVTT_I (A)", {0x434, 0b00000010, 1000} #define REG_MGTAVCC_I_MAX "MGTAVCC_I (A)", {0x440, 0b00000011, 1000} #define CMC_HOST_MSG_REG 0x1000 // System registers, present on all FPGAs #define CMC_SOC_OPCODE_REG 0x1004 // System registers, present on all FPGAs #define CMC_SOC_LENGTH_REG 0x1008 // System registers, present on all FPGAs #define CMC_SOC_PAYLOAD_REG 0x100C // System registers, present on all FPGAs Table 27: Compilation of known registers, their names, offsets, presence mask, division factors and location; Own compilation, excerpt from custom C library, inferred from Xilinx sources. 86/91 17. References [1]. GEEKSFORGEEKS. Xilinx FPGA Architecture - GeeksforGeeks. GeeksforGeeks [online]. 7th March 2022 [viewed 7th October 2025]. Available from: https://www.geeksforgeeks.org/xilinx-fpga-architecture/ [2]. Field-Programmable Gate Array (FPGA) Market Analysis till 2031. Market Research Reports, Business Consulting | In-depth Insights | TMR [online]. [Consulted 22nd September 2024]. Available at: https://www.transparencymarketresearch.com/fieldprogrammable-gate-array.html [3]. MOOR INSIGHTS AND STRATEGY. Amazon's Xilinx FPGA Cloud: Why This May Be A Significant Milestone. Forbes [online]. 13th December 2016 [Consulted 22nd September 2024]. Available at: https://www.forbes.com/sites/moorinsights/2016/12/13/amazons-xilinx-fpga-cloud-whythis-may-be-a-significant-milestone/ [4]. Helena Krupnova, Gabriele Saucier, FPGA-Based Emulation: Industrial and Custom Prototyping Solutions [online]. Institut National Polytechnique de Grenoble/CSI, 1st of January 2002 [Consulted 23rd September 2024]. ISBN : 978-3-540-67899-1. Doi: 10.1007/3-540-44614-1_8 [5]. BALKIND, Jonathan et al. OpenPiton. En: ASPLOS '16: Architectural Support for Programming Languages and Operating Systems [online]. New York, NY, USA: ACM, 2016 [Consulted 23rd September 2024]. ISBN 9781450340915. Doi: 10.1145/2872362.2872414 [6]. SALAMI, Behzad, Osman S. UNSAL y Adrian CRISTAL KESTELMAN. Comprehensive Evaluation of Supply Voltage Underscaling in FPGA on-Chip Memories. En: 2018 51st Annual IEEE/ACM International Symposium on Microarchitecture (MICRO) [online]. IEEE, 2018 [Consulted 23rd September 2024]. ISBN 9781538662403. Doi: 10.1109/micro.2018.00064 [7]. Introduction — Satellite Controller documentation. GitHub Pages [online]. [Consulted 23rd September 2024]. Available at: https://xilinx.github.io/Alveo-Cards/master/management-specification/oob-intro.html [8]. Trello - Wikipedia. [online]. 16th September 2011 [Consulted 23rd September 2024]. Available at: https://en.wikipedia.org/wiki/Trello [9]. Ferran Sabaté. 2.1.2. Gestión ágil de proyectos. Campus Virtual UPC [online]. [Consulted 23rd September 2024]. Not available to everyone, at: https://atenea.upc.edu/pluginfile.php/6119723/mod_folder/content/0/Mòdul%202.1.b %20-%20Gestion%20agil%20de%20proyectos.pdf?forcedownload=1 [10]. GitLab - Wikipedia. [online]. 25th October 2014 [Consulted 23rd September 2024]. Available at: https://en.wikipedia.org/wiki/GitLab [11]. Jooble [online]. [Consulted 8th October 2024]. Available at: https://es.jooble.org [12]. Amortización de inmovilizado. Sistemas y programas informáticos. SuperContable.com - Contabilidad, Plan contable, asesoría, asesor contable y Programas. [online]. [viewed 9th October 2024]. Available from: https://www.supercontable.com/informacion/impuesto_sociedades/Amortizacion_de_in movilizado.Sistemas_y_programas_.html 17. References [13]. Cómo calcular la amortización de equipos informáticos. Holded [online]. [viewed 9th October 2024]. Available from: https://www.holded.com/es/blog/amortizacion-deequipos-informaticos [14]. ¿Cuánto cuesta el kilovatio hora de luz (kWh) en España? tarifaluzhora.es [online]. [viewed 9th October 2024]. Available from: https://tarifaluzhora.es/info/precio-kwh [15]. Oficinas en Pedralbes. Idealista [online]. [viewed 9th October 2024]. Available from: https://www.idealista.com/inmueble/102386101/ [16]. ¿Cuánto se paga al mes por los suministros básicos? | Rastreator. Rastreator [online]. 19th January 2022 [viewed 9th October 2024]. Available from: https://www.rastreator.com/tarifas-energia/guias/factura-mes-suministros-basicos [17]. Factor de emisión de la energía eléctrica: el mix eléctrico. Cambio climático [online]. [viewed 8th January 2025]. Available from: https://canviclimatic.gencat.cat/es/actua/factors_demissio_associats_a_lenergia/ [18]. CONTRIBUTORS TO WIKIMEDIA PROJECTS. PCI Express - Wikipedia. Wikipedia, the free encyclopedia [online]. 2nd November 2002 [viewed 4th January 2025]. Available from: https://en.wikipedia.org/wiki/PCI_Express [19]. Introduction — Satellite Controller documentation. Site not found · GitHub Pages [online]. [no date] [viewed 4nJanuary 2025]. Available from: https://xilinx.github.io/AlveoCards/master/management-specification/oob-intro.html [20]. AWATI, Rahul. What is a baseboard management controller (BMC)? | Definition from TechTarget. Search Networking [online]. 11th January 2024 [viewed 20 December 2024]. Available from: https://www.techtarget.com/searchnetworking/definition/baseboard-management-controller [21]. Alveo I2C Telemetry — Satellite Controller documentation. GitHub Pages [online]. [viewed 4th January 2025]. Available from: https://xilinx.github.io/Alveo-Cards/master/management-specification/alveo-I2Ctelemetry.html [22]. PCI-SIG. PCI Local Bus Specification Revision 3.0. [online]. 3rd February 2004 [viewed 21 November 2024]. Available from: https://lekensteyn.nl/files/docs/PCI_SPEV_V3_0.pdf [23]. System Management Bus (SMBus) Specification. [online]. 12th May 2024 [viewed 21 November 2024]. Available from: https://smbus.org/specs/ [24]. SMBus Library for MSP430G2xx3 Devices; User's Guide. SMBus Library for MSP430G2xx3 Devices [online]. 13 February 2015 [viewed 3 January 2025]. Available from: https://software-dl.ti.com/msp430/msp430_public_sw/mcu/msp430/SMBusLib/ latest/exports/msp430_smbuslib_1_10_00_00/docs/MSP430G2xx3/ MSP430_SMBus_Users_Guide-1_10_00_00.pdf [25]. MSP432P401R: SMBUS library for MSP432 - MSP low-power microcontroller forum - MSP low-power microcontrollers - TI E2E support forums. TI E2E support forums [online]. 6th September 2018 [viewed 4th January 2025]. Available from: