Enhancing a UVM-based Test Bench for Functional Verification of a RISC-V Vector Accelerator

id192782   ENHANCING A UVM-BASED TEST BENCH FOR FUNCTIONAL VERIFICATION OF A RISC-V VECTOR ACCELERATOR ELOI MERINO SERRAT Thesis supervisor OSCARPALOMARPÉREZ(BARCELONASUPERCOMPUTINGCENTER-CENTRONACIONALDE SUPERCOMPUTACION) Tutor:MIQUELMORETÓPLANAS(DepartmentofComputerArchitecture) Degree Bachelor'sDegreeinInformaticsEngineering(ComputerEngineering) Bachelor's thesis Facultat d'Informàtica de Barcelona (FIB) Universitat Politècnica de Catalunya (UPC) - BarcelonaTech 21/01/2025  Acknowledgements I would like to thank my thesis supervisor, Oscar Palomar, and my thesis tutor, Miquel Moretó, for the opportunity to develop this project, and their invaluable guidance through its blossoming. I am also deeply thankful for all professors that have devotedly transmitted their passion and knowledge during my time at FIB. Last, but definitely not least, thanks to my family, friends, and my partner, whose constant support and love sustain me day after day. Your unwavering encouragement and belief in me keep me going, and I am forever grateful for each and every one of you. III Abstract This thesis aims to show the completion of the verification environment of a RISC-V vector accelerator known as eAccelerator, part of the eProcessor project. This verification environment revolves around an incomplete functional verification test-bench written in SystemVerilog, using the Universal Verification Methodology (UVM) framework. From creating custom components, to implementing custom instructions, and tying everything together through the collection of three types of coverage; this project puzzles together missing pieces of the test-bench. The final result is an upgrade of the verification environment, and the detection of multiple bugs on the implementation of eAccelerator, that would otherwise have flown under the radar. Key Words: RISC-V, Vector Accelerator, Functional Verification, UVM, Coverage, eProcessor. V Resum Aquesta tesi té com a objectiu mostrar la finalització de l’entorn de verificació d’un accelerador vectorial RISC-V anomenat eAccelerator, que forma part del projecte eProcessor. Aquest entorn de verificació gira al voltant d’un banc de proves de verificació funcional incomplet, escrit en SystemVerilog i que utilitza el marc de treball Universal Verification Methodology (UVM). Des de la creació de components personalitzats fins a la implementació d’instruccions especialitzades, acabant amb la integració de tot mitjançant la recopilació de tres tipus de coverage; aquest projecte encaixa les peces que faltaven al banc de proves. El resultat final és una millora de l’entorn de verificació i la detecció de diversos errors en la implementació de l’eAccelerator, que d’una altra manera haurien passat desapercebuts. Paraules Clau: RISC-V, Accelerador Vectorial, Verificació Funcional, UVM, Coverage, eProcessor. VI Contents Acknowledgements III Abstract V Resum VI 1 Introduction 1 2 Background 3 2.1 Vectoraccelerators ............................ 3 2.2 RISC-V .................................. 3 2.3 RISC-VVextension ........................... 4 2.4 The eProcessor project . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2.5 eAccelerator................................ 5 2.6 NoC .................................... 5 2.7 AMBA5CHI............................... 5 2.8 UVM: Universal Verification Methodology . . . . . . . . . . . . . . . 5 2.9 BaseUVMtest-bench .......................... 6 2.10 eAccelerator test-bench . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2.11 Memory Operations interface working principle . . . . . . . . . . . . 8 2.12 Control Status Register instructions . . . . . . . . . . . . . . . . . . 10 2.13Coverage.................................. 10 2.14 Random test generation . . . . . . . . . . . . . . . . . . . . . . . . . 11 3 Verification plan 12 3.1 DUTdefinition .............................. 12 3.1.1 Context .............................. 12 3.1.2 EPAInterface........................... 12 3.1.2.1 Sub-Interface: insn .................. 13 3.1.2.2 Sub-Interface: cdb ................... 14 3.1.2.3 Sub-Interface: cpu_memop ............... 15 3.1.2.4 Sub-Interface: result ................. 16 3.1.2.5 Sub-Interface: commit_ri ............... 16 3.1.2.6 Sub-Interface: roll_back_ri ............. 16 3.1.2.7 Sub-Interface: control ................ 16 3.1.2.8 Sub-Interface: flush .................. 17 3.2 Verification environment . . . . . . . . . . . . . . . . . . . . . . . . . 18 3.2.1 Description of verification environment . . . . . . . . . . . . . 18 3.2.1.1 Components....................... 18 3.2.1.2 Reference model . . . . . . . . . . . . . . . . . . . . 18 VII 3.2.2 Coverageplan........................... 19 3.2.2.1 Code coverage . . . . . . . . . . . . . . . . . . . . . 19 3.2.2.2 Functional coverage . . . . . . . . . . . . . . . . . . 19 3.2.3 Testplan ............................. 21 3.2.3.1 Test cases definition . . . . . . . . . . . . . . . . . . 21 3.2.3.2 CI/regressions . . . . . . . . . . . . . . . . . . . . . 21 3.3 Startingpoint............................... 22 3.4 Plannedwork ............................... 24 3.4.1 Memory coherence interface stimulus . . . . . . . . . . . . . . 24 3.4.2 CSRoperations.......................... 24 3.4.3 Custom extensions support . . . . . . . . . . . . . . . . . . . . 24 3.4.4 Functional coverage . . . . . . . . . . . . . . . . . . . . . . . . 25 4 Implementation 26 4.1 Memoryoperations............................ 26 4.1.1 Exploration of eprocessor_agent ............... 27 4.1.1.1 Code of begin_protocol ............... 29 4.1.1.2 Code of do_protocol ................. 29 4.1.2 Basemechanism.......................... 30 4.1.3 Queuesystem........................... 30 4.1.3.1 Delay checker . . . . . . . . . . . . . . . . . . . . . . 33 4.1.3.2 VIF interactor . . . . . . . . . . . . . . . . . . . . . 34 4.1.4 Multiple loads and stores per cycle . . . . . . . . . . . . . . . 34 4.1.5 Mechanism configuration . . . . . . . . . . . . . . . . . . . . . 35 4.1.6 Factorypattern.......................... 35 4.1.7 Memop generation modes . . . . . . . . . . . . . . . . . . . . 38 4.1.7.1 ALWAYS mode generator . . . . . . . . . . . . . . . 39 4.1.7.2 VEC_MEMOP mode generator . . . . . . . . . . . . 40 4.1.8 Proper control of commit_ri ................... 40 4.1.9 Monitoring ............................ 41 4.2 CSRoperations.............................. 44 4.2.1 Reference model changes . . . . . . . . . . . . . . . . . . . . . 44 4.2.2 Test-bench changes . . . . . . . . . . . . . . . . . . . . . . . . 46 4.3 Random tests generation . . . . . . . . . . . . . . . . . . . . . . . . . 49 4.3.1 Bioinformatics........................... 49 4.3.2 Format converting vfcvt.f.f.v ................. 51 4.4 Coverage.................................. 54 4.4.1 Overview of existing coverage module . . . . . . . . . . . . . . 54 4.4.2 Implementation of new instructions to coverage module . . . . 55 5 Evaluation of results 58 5.1 Memopsimulations............................ 58 5.1.1 Execution with drive mode DISABLE . . . . . . . . . . . . . . 59 5.1.2 Execution with drive mode VEC_MEMOP . . . . . . . . . . 59 5.1.2.1 Older scalar memop . . . . . . . . . . . . . . . . . . 59 5.1.2.2 Younger scalar memop . . . . . . . . . . . . . . . . . 60 5.1.3 Execution with drive mode ALWAYS . . . . . . . . . . . . . . 61 5.2 CSRinstructions ............................. 61 5.3 Coverageresults.............................. 62 5.3.1 Codecoverage........................... 63 VIII 5.3.2 ISACoverage ........................... 64 5.3.3 Functional coverage . . . . . . . . . . . . . . . . . . . . . . . . 65 5.4 Bugsfound ................................ 65 5.4.1 Lack of control of cpu_memop lines ............... 65 5.4.2 Overlap specification mismatch . . . . . . . . . . . . . . . . . 66 5.4.3 Scalar memop timing stalls VPU . . . . . . . . . . . . . . . . 67 5.4.4 Bioinformatic instructions ISS implementation bug . . . . . . 68 5.4.5 Illegal CSR operations not signaled . . . . . . . . . . . . . . . 68 5.4.6 Implementation errors of vfcvt.f.f.v instruction . . . . . . . 68 6 Conclusions and future work 70 6.1 Futurework................................ 71 References 72 A Context and scope 75 A.1 Context .................................. 75 A.1.1 Introduction............................ 75 A.1.2 Thetask.............................. 75 A.1.3 Mainconcepts........................... 75 A.1.4 Stakeholders............................ 76 A.2 Justification................................ 76 A.2.1 Alternatives............................ 77 A.3 Scope ................................... 78 A.3.1 Objectives............................. 78 A.3.2 Requirements ........................... 78 A.3.3 Risks and obstacles . . . . . . . . . . . . . . . . . . . . . . . . 78 A.3.3.1 Software and licenses . . . . . . . . . . . . . . . . . . 78 A.3.3.2 Teamwork and time management . . . . . . . . . . . 78 A.3.3.3 Lack of theoretical knowledge . . . . . . . . . . . . . 79 A.4 Methodology and rigour . . . . . . . . . . . . . . . . . . . . . . . . . 79 A.4.1 Meetings and status updates . . . . . . . . . . . . . . . . . . . 79 A.4.2 Versioncontrol .......................... 79 A.4.3 Issue tracking system . . . . . . . . . . . . . . . . . . . . . . . 80 A.4.4 Automation............................ 80 B Planning 81 B.1 Descriptionoftasks............................ 81 B.1.1 Project management . . . . . . . . . . . . . . . . . . . . . . . 81 B.1.2 Project development . . . . . . . . . . . . . . . . . . . . . . . 81 B.1.2.1 Verification plan analysis and extension . . . . . . . 81 B.1.2.2 Test bench extension and development . . . . . . . . 82 B.1.2.3 Test generation and running . . . . . . . . . . . . . . 82 B.1.2.4 Coverage extension . . . . . . . . . . . . . . . . . . . 83 B.1.3 Project documentation . . . . . . . . . . . . . . . . . . . . . . 83 B.2 Resources ................................. 83 B.3 Estimates and the Gantt . . . . . . . . . . . . . . . . . . . . . . . . . 84 B.4 Risk management: alternative plans and obstacles . . . . . . . . . . . 85 C Budget and Sustainability 86 C.1 Budget................................... 86 IX Chapter 1. Introduction open European full-stack ecosystem based on this new RISC-V CPU [10]. As with many other projects of EuroHPC JU, eProcessor takes advantage of past experience, and closed projects that predate it. This means that projects pertaining to these initiatives never start from absolute 0; there is always some other past project that sets the basis for growth and research. eProcessor is one of the latest iterations of said cycle. Aside from the main OoO CPU this project gives life to, there is also a vector accelerator called eAccelerator. This piece of co-processing hardware combines two main models: its direct memory access reminds us of a Loosely-Coupled Accelerator (LCA), but being program-controlled by the scalar core reminisces a TightlyCoupled Accelerator (TCA)[4]. The idea is to include pairs of 4-way RISC-V Out of Order (RVOOO) processors and eAccelerator accelerators, interconnected by a coherent Network on Chip (NoC). The first tape-out included a single RVOOOeAccelerator pair, and was completed back in 2023. The second tape-out aims to replicate this architecture, creating a multicore layout. For the verification of this vector accelerator, BSC-CNS opted for a functional approach to optimise time-wise, and aid implementation teams verify their design of this vector accelerator. The Design Verification (DV) team inside BSC-CNS developed a UVM test-bench (TB) (seen in section 2.8) surrounding this accelerator. Alternatives to this approach are discussed in appendix A. This test-bench was developed with expandability and adaptability in mind, and porting was trivial for main functionalities. However, every single vector accelerator has its own set of variations that other vector processing units do not possess. An example of this in eAccelerator is the memory disambiguation protocols. Other projects do not have the need for such mechanisms, and therefore their verification needs the development of new custom components in the verification environment to support the needed interfaces. Another example lies in the presence of custom instructions with own details to compare to when developing the verification environment’s golden reference model. More absent features are discussed further down this work. This thesis aims at covering the efforts directed at completing functionality holes present throughout the verification environment of this accelerator. The following sections give much needed context to understand what this means, and describe in great detail the efforts to puzzle together the missing pieces of this project. 2 Chapter 2 Background To properly understand the lines of work of this thesis, some basic concepts surrounding the test-bench and its components must be understood first. 2.1 Vector accelerators In the context of modern computing, where technology is unable to keep up with Moore’s Law[23], computing performance does not come from central singular cores. Latest trends go after accelerators; co-processing pieces of hardware that serve a specific purpose. These accelerators alleviate computing load from the central multipurpose cores, and perform specific tasks far faster and efficiently, both time and resource wise. In 1975, a company named Cray Research developed the architectural basics of what we now know as supercomputers, materializing the historical Cray-1[28]. These computing machines developed some ground-breaking concepts in the field of High Performance Computing (HPC): pipelined functional units, a shared memory model, and multiprocessor concepts later on. However, the most radical concept they pioneered was vector processing. This architecture allowed processors to streamline data throughout replicas of functional units. This meant that operations were applied on big batches of data with a single instruction. Joining these two concepts, we conceal the Vector Processing Unit (VPU). They fall right under the Single Instruction stream, Multiple Data streams (SIMD) category, of Flynn’s Architecture Taxonomy classification[14]. Today, the concepts of these calssical architectures live on, and make it all the way into eAccelerator, which contains the Vitruvius[22] VPU, and expand around it. 2.2 RISC-V RISC-V is an open-source instruction set architecture (ISA) based on the reduced instruction set computer (RISC) principles. Developed in 2010 by researchers at the University of California, Berkeley, RISC-V was initially designed as an academic ISA, but its simplicity made the proposal very appealing. eProcessor uses RISCV as its ISA[37][38], obeys the standard accordingly, and adds custom instructions for the HPC specific computations mentioned further down this introduction (2.5). 3 Chapter 2. Background 2.3 RISC-V V extension The RISC-V Vector extension (V) is the additional ISA functionality that goes beyond the base RISC-V scalar instructions, and gives control over vector operations. This means that by design, one is able to produce way denser code, and way more efficient algorithms where scalar operations would create inefficient and repetitive solutions. The RISC-V V implementation has gone through multiple revisions. The latest and greatest is version 1.0.0, known as RVV1.0. In spite of this, the creation of eAccelerator predates this latest version, and implement 0.7.1; which was a previous revision that gained popularity and standardized throughout the market. Ongoing efforts towards a bump up in vector extension revision are currently active on the implementation team side. Despite this being the case, this thesis will not consider such upgrade. 2.4 The eProcessor project The eProcessor project is a European-funded project that aims at creating a full stack high performance processor ecosystem for HPC[1][10]. Developing both hardware and software solutions, this project is based on the RISC-V open source ISA and features an out of order (OoO) core, with a mixed-coupling vector accelerator called eAccelerator. This whole configuration is named The eProcessor architecture. As seen in [1], this architecture contains the RVOOO (RISC-V Out of Order) processor, eAccelerator, L2 cache, Network on Chip (NoC), IOMMU (Input/Output Memory Management Unit), and coherent C2C (chip-to-chip) link. Two tape-outs were planned for this project. The first one includes only one RVOOO core; the second one includes two cores. Figure 2.1: Overview of eProcessor’s layout. Taken from [1] The RVOOO is a 4-way out-of-order scalar RISC-V core that supports the RISCV64GCV ISA. eAccelerator is composed by a Vector Processing Unit (VPU), reduced and mixed-precision functional units, and an Artificial Intelligence accelerator. This project has received funding from the European High-Performance Computing Joint Undertaking (JU) under grant agreement No 956702. The JU receives support 4 Chapter 2. Background from the European Union’s Horizon 2020 research and innovation programme and Spain, Sweden, Greece, Italy, France, Germany[10]. 2.5 eAccelerator Particularly, eAccelerator is the grain of salt that BSC-CNS brings to the table: a RISC-V coprocessor with custom instructions to support various computationally intensive scientific fields; like Artificial Intelligence (AI), Machine Learning (ML), Deep Learning (DL) and Bioinformatics. There is also a vector processing unit (VPU) inside the accelerator, which supports the majority of the standard RISCV V extension[27] version 0.7.1 instructions. 2.6 NoC eProcessor’s chip layout interconnects the different core-accelerator pairs with the rest of the system (L2 cache + home nodes, coherent chip to chip link, IOMMU) through a crossbar Network on Chip. Based on FastTrackNoC[7], this network is configured to use the AMBA5 CHI interface specification. 2.7 AMBA 5 CHI AMBA 5 CHI (Coherent Hub Interface)[2] is a specification from ARM that defines a high-performance, low-latency, and coherent interconnect for complex SoC (System-on-Chip) designs. It is part of the ARM AMBA (Advanced Microcontroller Bus Architecture) family, which provides interconnect solutions for system buses in embedded systems. This specification is topology-independent, but some optimisations are included in the official documentation to cue effective performance upgrades for certain arrangements. eProcessor’s NoC uses five distinct physical channels to accomodate different CHI message classes and avoid protocol-level deadlocks. It also implements a Home Node Funnel (HN-F), that manages coherent memory transactions; Request Node IO (RNI), of which eAccelerator possesses one to interconnect with the rest of the NoC; and Subordinate Node Funnel (SNF), which the test-bench uses as a backdoor to interact with the NoC itself. 2.8 UVM: Universal Verification Methodology Acronym for Universal Verification Methodology[18], it is a standardized verification methodology. It sets the bases for a reusable and highly configurable tool to fabricate flexible test benches. Although ported as a framework to many Hardware Description Languages (HDL), SystemVerilog is the preferred choice for the eAccelerator’s test-bench. This allows for seamless compatibility between design (Verilog) and verification (SystemVerilog) code. 5 Chapter 2. Background 2.9 Base UVM test-bench Every test-bench revolves around a design to be tested, named Design Under Testing (DUT). This is the module to be verified. A base UVM test-bench consists of a set of basic components to stimulate the DUT, listed as follows. •Test is the component that acts as the root class for the test-bench environment. It is responsible for creating and configuring all environments of the test-bench. •Environment is the next in line and pertains to Test. All subsequent classes pend from this instance. This component is in charge of instantiating and connecting all agents present in the environment •Agent acts as a container for the set of components that are needed to carry out operations on the design. Theory says one agent is to be utilised per interface, or any other classification criteria over the design itself. However, this is an implementation choice made by the architect of the test-bench, and other configurations are possible. •Sequence item Consist of data fields required for generating the stimulus. In order to generate the stimulus, the sequence items are randomized in sequences. Therefore, data properties in sequence items should generally be declared as rand and can have constraints defined. •Sequencer is responsible for receiving (or creating) sequence items and serving them to the next component in line: the driver. •Driver is the main component that interfaces with the DUT. It gets sequence items from the sequencer, and interprets them as time-consuming stimulus on the virtual interface. •Monitor gathers all information resulting from the driver stimulus, acting as a passive observer. •Scoreboard Interprets the results of the sequences observed by the monitor. Since this component is outside the agent, it can combine input from multiple monitors to interpret correlation amongst all interfaces in the DUT. Uses these results to extend the report of the run. Figure 2.2: Base verification UVM test-bench As seen in Figure 2.2, top components of the TB are instantiated in the top_tb module, and properly connected to the DUT using a virtual interface. This is often done 6 Chapter 2. Background by putting and instance of such interface in the globally accessible uvm_config_db, and using such to interact with the DUT physical interface. Another very important concept that UVM brings to the table is phasing, or phasebased execution. A simulation is an execution flow that has a beginning and an end. This may seem trivial, but by defining a finite set of ordered steps, we create a consistent and robust simulation environment in which redefining behaviour becomes very easy. Three main categories of phases are defined, each of which encompasses a subset of sub-phases themselves. •Build phase (3 sub-phases) is the entry point of every simulation. This phase aims to construct, configure and connect the test-bench component architecture. Instantiating subcomponents, importing configurations, and many more initialization tasks are carried out throughout this phase group. •Run phase (13 sub-phases) includes all simulation sub-phases. This phase group was a concept present in UVM’s predecessor, OVM, with some additional phases that were added as a result of this adaptation. Sub-phases that consume simulation time are all part of this phase group. •Clean-up phase (4 sub-phases) prepares everything for an eventual shutdown. Extracting information from scoreboards, collecting data from functional coverage, and cleaning up resources are all part of this phase-group. This phase is also used to determine the final result of the simulation, and report any and every issue found. No sub-phases in this phase consume any time at all. Figure 2.3: Phases of the UVM execution flow. Taken from [42] 7 Chapter 2. Background 2.10 eAccelerator test-bench Figure 2.4: Base test-bench architecture overview. Created by the BSC-CNS verification team In the case of eAccelerator, the test-bench possesses a single test class, and a single environment. Inside this environment, 4 individual agents are instantiated: one for the EPA interface, another for the L2 TLB interface, another for an SNF backdoor, and one final that interacts with NoC’s CHI. This test-bench does interact with an Instruction Set Simulator (ISS) called Spike [31]. This connection is established using Direct Programming Interface (DPI-C); a feature of SystemVerilog to interact with Cor C++ binaries. Spike is run to fetch instructions present in the loaded test binary, and then issued on eAccelerator. Instructions are sent alongside the expected result produced by the ISS, and compared inside a scoreboard with the result that the DUT produces. Instructions are sent one at a time, and wait for the previous instruction to complete before sending the next one. 2.11 Memory Operations interface working principle As mentioned before, eAccelerator access to memory is direct; only arbitrated by the underlying AMBA5 CHI[2] request node, part of the Network on Chip (NoC) interconnecting the whole system. RISC-V V specifications define concrete requirements on the memory ordering of single execution units. Since both eProcessor and eAccelerator pertain to the same hart, we need coherence communications between both components. This is the reason of the existence of the cpu_memop interface and the synchronization mechanism tied to it. Out of all sub-interfaces, the memop mechanism revolves around cpu_memop and commit. It also reads information from insn in certain modes of operation (see Table 3.1.2 for further detail on EPA interface). 8 Chapter 2. Background Figure 2.5: Memop message passing sequence The cpu_memop workings operate as follows. Whenever the scalar core issues a scalar memory operation, it must communicate such event though the cpu_memop_issue sub-interface. This will signal the VPU of a possible conflict with vector memory operations that might be in-flight. In an out of order fashion, extra information about this scalar memory operation is provided through cpu_memop_pad. Parameters like physical address of the access, size of the operation and operation type, are all provided though this interface. After this, cpu_memop_overlap comes from the coprocessor back to the scalar core. This interface contains information about the aforementioned possible conflict between the scalar memory operation and any currently in-flight vector operations. If this signal is active, eAccelerator will wait for the completion of this scalar memory operation, and then proceed with normal operation. The completion of any scalar memory operations is signalled through cpu_memop_done. In the case of scalar loads that do not generate any conflict, there is no need for done signalling, and operation ending is inferred. The conflict signal, which is part of the cpu_memop_overlap sub-interface, is active only when a scalar load is issued while an older vector store is in-flight. This is sufficient condition to activate such signal, and is a valid implementation. However, more optimised implementations of this mechanism calculate overlap of addresses between operations to trigger conflicts only when addresses coincide. Therefore, eAccelerator only looks for timing-based conflicts on memory operations, and not on real memory address overlap. Providing certain parameters out of order is a consequence of having direct access to physical memory. If virtual memory is enabled, TLB accesses are necessary, and such might take several cycles. This out of order parameter issuing is used to support these delays. Being an out-of-order super scalar core, eProcessor may issue multiple scalar memory operations per cycle. The parameter that defines this is called NLS (Number of Load Stores). At the time of developing this project, RVOOO supports 2 LD/ST operations per cycle. This means that every signal of the cpu_memop sub-interface 9 Chapter 2. Background is duplicated, and can work in parallel. 2.12 Control Status Register instructions Another feature missing from this interface are CSR operations. RISC-V ISA defines in its specification a series of Control Status Registers (CSR), which define and reflect current system state and behaviour. Acting like parameters of a highly configurable machine; or scoreboards that reflect what is happening inside. Operations exist in RISC-V ISA to read, write, set and clear bit fields in these registers. Atomic instructions to read and modify registers at the same time exist, and are part of the Zicsr RISC-V extension. Some of these CSRs live inside a register bank located in the scalar core. Therefore, all operations targeting these CSRs execute inside said processor. However, some of these registers are mapped inside eAccelerator, and these operations need to be sent to the VPU for these changes to take effect. The interface of choice is the same one used to issue vector instructions, and read results are reported through results interface. Address Privilege Name Description 0x008 URW vstart Vector start position 0xC20 URO vl Vector length 0xC21 URO vtype Vector data type register Table 2.1: eAccelerator-mapped CSRs Figure 2.12 shows all three CSR mapped inside eAccelerator. One of these registers has user permissions to read and write (URW), and the other two have write exclusivity for higher privilege levels, being user read-only (URO). Another 2 CSRs that the VPU interacts with are vxsat and vxrm. None of these values are mapped inside eAccelerator, but are instead communicated using EPA interface. This is done because their value is instruction-dependent, and sending an instruction to supply said value before every operation would be too much of an overhead. Address Privilege Name Description 0x009 URW vxsat Fixed-Point Saturate Flag 0x00A URW vxrm Fixed-Point Rounding Mode Table 2.2: Other eAccelerator CSRs vxrm is supplied with every instruction issue. vxsat is the value returned with every instruction result. This is the case for all operations, even if such value does not have anything to do with the instruction itself. Value is used or ignored based on the type of instruction. 2.13 Coverage Coverage in hardware functional verification refers to a measure of how thoroughly the design of a hardware system has been exercised by the verification tests. It 10 Chapter 2. Background helps to ensure that the design has been fully tested and that any potential bugs or corner cases are identified before the hardware is fabricated. Several types of coverage metrics do exist, and are often classified into two main categories: •Code coverage evaluates how much of the actual hardware design code has been executed during simulation. •Functional coverage measures whether specific functionality or behaviors that are important to the design have been tested. It is user-defined and focuses on verifying specific design intent. eAccelerator’s project in particular, ported coverage modules from the project that set the base for the VPU used in this accelerator, which is the European Processor Initiative (EPI)[9]. The main idea was to extend upon this coverage, but efforts never included custom instructions, and only eliminated unsupported operations. 2.14 Random test generation eProcessor’s project VPU, eAccelerator, implements a set of custom instructions that target scientific research fields, such as bioinformatics. In the current verification stack, the test-bench is periodically run with different sets of tests. This is done to extend functional coverage of the design. Stressing it to find edge cases helps uncover bugs and design problems that would otherwise have gone unnoticed. Finding the most basic bugs is done by executing a control set of ISA tests: baseline simple tests that make sure the minimum specification functionality is satisfied. To introduce test variety, and achieve such coverage executing those periodic runs, a random test generator is used. This piece of open-source software, called riscv-dv, was initially created by Google all the way back in 2018. It was then published to be co-owned by a bunch of corporations. This group is called ChipsAlliance[3], and it is now FOSS, and accepts contribution from the community. BSC-CNS manages forks of the repository, adapting the tool to the necessities of every project. This SystemVerilog code uses UVM to generate compilable source files of randomly generated instruction sequences. Coded with expandability in mind, adapting it to the needs of the test-bench is something achievable, and a task overhauled in this thesis. 11 Chapter 3. Verification plan 3.2 Verification environment An overview of the verification environment in which eAccelerator is verified. All components and verification flows are detailed in following sections. 3.2.1 Description of verification environment 3.2.1.1 Components 3.2.1.1.1 Stimulus Generation To generate stimulus for this DUT, the test-bench uses binaries as an input, and runs them using both the golden reference model, and the DUT itself. To generate such binaries, two main sources are used: •riscv-dv Generates random binaries with a highly parametric environment to fine-tune results. Creating by hand all tests necessary to stimulate all corner cases would take very long. Also, creating random tests helps find corner cases unconsidered at specification time. •ISA tests Sets of simple tests following examples from RISC-V V specification document that test basic functionality. Manually created, these tests aim at being a baseline for checking if changes break the most basic parts of the design. 3.2.1.1.2 Checkers Scoreboard Two scoreboards check of the operations for the entire test-bench. •isa_scoreboard aims to ensure correct functionality of isa-related details. Receives data traces from monitor_pre and monitor_post, which capture information on the result given by the golden model (pre) and DUT (post). •protocol_scoreboard aims to ensure correct functionality of protocol details. Gets information from monitor_protocol whenever an instruction finishes. Monitor creates transactions for the scoreboard to compare with the reference model. Assertions Some assertions were used as part of the verification of the AMBA5 CHI connection node interface. These serve as a control barrier between eAccelerator, and the underlying NoC simulator used to verify this piece of the design. 3.2.1.2 Reference model As a reference model, the open source Instruction Set Simulator (ISS) Spike[31] is used. This functional model implements simulation of one or more harts, and allows for nice and easy implementation of co-simulation functionalities. This is what BSC-CNS verification team did on a fork of this project. The intention of the modifications are to allow spike to run until a target instruction is found. This way, using DPI-C to communicate with the test-bench, we allow a stepped execution with “breakpoints” that returns information of the binary and the hart state only when reaching desired instructions. In the case of eAccelerator, this means running only until vector instructions are reached, but requirements may vary from project 18 Chapter 3. Verification plan to project. This fork also contains modifications to implement custom eProcessor features, like NarrowInt, FP8 operations, and custom instructions. 3.2.2 Coverage plan Two types of coverage are collected to quantify the level of exercising of this design. 3.2.2.1 Code coverage Code coverage is collected automatically using simulator capabilities, enabled through the environment. It activates every type of code coverage metric supported by the simulation tool. That is: statement, branch, condition, expression, toggle, and FSM code coverage. 3.2.2.2 Functional coverage Functional coverage in eAccelerator is implemented using various coverpoints grouped in covergroups and intertwined using covercrosses. Two distinct kinds of functional coverage can be distinguished in our project: ISA coverage, and internal-module coverage. 3.2.2.2.1 ISA coverage Moreover, components exist to collect functional coverage of the design. First and foremost, cov_rvv_instr takes care of defining coverage points of the stimulus received on the different types of instructions. This takes care of detecting instructions, and making sure specific bins are covered. Cases are defined for: 19 Chapter 3. Verification plan Coverpoint Description cp_vd Coverpoint for destination vector register (vd) cp_even_vd Coverpoint for even destination vector register (vd) cp_vs1 Coverpoint for source vector register (vs1) cp_vs1_0 Coverpoint for source vector register (vs1) being v0 cp_vs2 Coverpoint for source vector register (vs2) cp_even_vs2 Coverpoint for even source vector register (vs2) cp_vs3 Coverpoint for destination vector register in a store operation cp_data Coverpoint for scalar data in non-load operations cp_mask Coverpoint for vector mask in non-vn_vw and unmasked conditions cp_vwr_mask Coverpoint for vector mask in vn_vw and unmasked conditions cp_sew Coverpoint for vector element width (SEW) in non-vectorized and non-vn_vw conditions cp_sew_vf Coverpoint for vector element width (SEW) in vectorized conditions cp_sew_vn_vw Coverpoint for vector element width (SEW) in vn_vw conditions cp_sew_vnvw_vf Coverpoint for vector element width (SEW) in vn_vw and vf conditions cp_vstart Coverpoint for vector start index (vstart) in a load operation cp_vl Coverpoint for vector length (VL) in non-only_vl0 conditions cp_vl0 Coverpoint for vector length (VL) in only_vl0 conditions cp_vlmul Coverpoint for vector length multiplier (VLMUL) with illegal bins cp_frm Coverpoint for floating-point rounding mode (frm) in vectorized condition cp_stride_64 Coverpoint for data stride in SEW64 conditions cp_stride_32 Coverpoint for data stride in SEW32 conditions cp_stride_16 Coverpoint for data stride in SEW16 conditions cp_stride_8 Coverpoint for data stride in SEW8 conditions cp_stride_64_n_op Coverpoint for data stride in SEW64 conditions cp_stride_32_n_op Coverpoint for data stride in SEW32 conditions cp_stride_16_n_op Coverpoint for data stride in SEW16 conditions cp_stride_8_n_op Coverpoint for data stride in SEW8 conditions cp_stride_0_n_op Cover non-optimized strides cp_vl_64 Coverpoint for vector length (VL) in SEW64 conditions cp_vl_32 Coverpoint for vector length (VL) in SEW32 conditions cp_vl_16 Coverpoint for vector length (VL) in SEW16 conditions cp_vl_8 Coverpoint for vector length (VL) in SEW8 conditions Table 3.13: Instruction-related coverpoints. Part of cg_instr covergroup All of these coverpoints are part of a covergroup called cg_instr. Once declared, an instance of this covergroup is created per each instruction existent. The new() function enables or disables some of these coverpoints, to modulate around each possible instruction. Every time an instruction is completed, and the scoreboard deems its result correct; the coverage module is invoked, and the corresponding instruction is sampled. We only sample the input variables to reduce casuistic, and simplify coverage runs. Observing the output is something the test-bench takes care of. Coverage for instructions only looks at input variations that produce correct results. 3.2.2.2.2 Internal module coverage Some modules inside eAccelerator are verified more thoroughly using functional coverage. In particular, by binding coverage modules “inside” of their RTL code, the test-bench gains access to the inner-working of said module. This facilitates snooping corner cases and targeting interesting internal states, instead of only targeting interface results. The following modules contain a probe inside when coverage is active: unpacker, renaming logic, queue demux, reorder buffer, ring interconnection, valid bit, valid mask bit, vector length finite state machine, vector length virtual routing function, source buffer, arithmetic logic unit (ALU), write-back buffer, store buffer, index buffer and load buffer. 20 Chapter 3. Verification plan 3.2.3 Test plan 3.2.3.1 Test cases definition In order to cover as many individual cases as possible, a strategic approach is applied. There exist 4 kinds of test in this verification environment. •ISA tests: Directed tests created to verify correct behaviour of each instruction. They act as a baseline for development of new features. If ISA tests do not succeed, then the most basic cases of the specification are failing. Since they include an instance of all possible instructions in one of all possible SEW, they are ideal to include in coverage regressions runs. •Constrained random tests: Random sequences of instruction streams put all together and constrained to comply with certain minimum requirements. riscv-dv is used to generate brand new tests every run (and combine with passing tests from past occurrences). •Benchmarks: Representative set of common HPC kernels with self-checking implementations to test performance in real-world applications. These tests are: Axpy, Matmul, SpMV, Somier and FFTW. •Directed tests: Manually crafted tests, aimed directly at known prone-to-fail corner cases. 3.2.3.2 CI/regressions Continuous Integration (CI) pipeline are used to periodically stimulate the design. Whenever a job is scheduled, or new code changes are merged into specific branches, these pipelines trigger and execute a set of test with different purposes. Three distinct kinds of pipelines are used: •Smoke regression: When we push an update in to the RTL or verification environment, this pipeline triggers. It executes a small selected test set, which include sanity ISA and three kinds of ISA test regressions (basic ISA, extended ISA, and custom ISA). •Large complete regression: Long regression run scheduled on the weekends. It executes a large set of previously passing random tests. This generates a coverage report, which can be then included into the main coverage database of the design. •Random tests Pipeline scheduled to execute on a daily basis. Executes a new batch of riscv-dv generated tests and classifies them as passing or failing. Some of these are then used to populate the weekend’s large regression. 21 Chapter 3. Verification plan 3.3 Starting point At the start of this project, there exists a full-fledged UVM test-bench that supports instruction-by-instruction level co-simulation, in order to retrieve the internal state of the accelerator, and match it against a reference-model simulator. The UVM uses what we call the Base VPU verification environment (vpu-dv), which is shared with other projects at BSC that use the VPU. This code includes implementation independent details (e.g. the RVV-0.7.1 scoreboard, the connection with spike, etc.). Such abstractions and generalities make it easier to re-use code. Figure 3.2: Components of eAccelerator’s TB. Created by BSC-CNS verification team A single environment is instantitated inside a single test class. This environment is subdivided into agents that encapsulate the verification of a feature or interface: •eprocessor_agent is the main agent of the testbench. Takes care of the main EPA interface, and is used to issue and retrieve instructions. Creates a base class called protocol_base_class from which its functionality can be derived. This agent is part of vpu-dv, and therefore contains very reusable generic code. •vpu_l2_tlb_agent is an agent that generates and monitors transactions at L2 TLB level. This agent contains a set of components to randomly generate transactions with eAccelerator’s TLB, and compare output to what the golden reference model is simulating. It does this connecting with spike, and translating the requests to compare against the response that the VPU generated. These transactions do not necessarily reflect what the binary says to execute, and sequences are randomized. •snf_backdoor_agent, or fully-coherent Subordinate Node Full (SNF) backdoor agent, receives all scalar store-byte instructions from the co-simulator, and interacts with eAccelerator’s Remote Node IO-Coherent (RNI), simulating behaviours similar to those found in the NoC when more components are present in the system. This tests the protocols underlying AMBA5 CHI, and 22 Chapter 3. Verification plan stimulates the network on chip simulator instantiated with the rest of the test-bench. •chi_agent contains only a monitor component. Therefore, it does not create any stimulus, but rather reads the state of a certain interface. The idea behind this monitor is to surveil CHI transactions on the VPU Request Node Interface (RNI). For simplicity reasons, this monitor is stand-alone. This means that no scoreboard subscribes to its transactions, and the monitor itself acts a bit like a scoreboard. It interprets interfaces, and reports errors when necessary. Spike connection is done using Direct Programming Interface C (DPI-C), which allows the test-bench to execute code from spike as a shared object library. Two distinct seq components interact with a wrapper component that acts as a controller for the DPI-C calls. This verification environment also contains some parametrized features controlled by configuration objects. These are classes of uvm_objects randomized with constraints at the beginning phases of the simulation. 23 Chapter 3. Verification plan 3.4 Planned work 3.4.1 Memory coherence interface stimulus When this project was started, every signal of the cpu_memop interface was grounded to avoid ambiguous states. This meant that the mechanism was never tested in the holistic environment that this test bench provides. The initial plan for this project was to make a mechanism to issue and control garbage scalar memory operations. This means generating, issuing and monitoring their route through the interface, while making sure correct operation is being carried out. This will allow us to simulate the presence of a scalar core next to our VPU. Up until now, eAccelerator never received stimulus through said sub-interface (cpu_memop, described in section 3.1.2.3). The making of this mechanism aims to uncover bugs that this interface could cause. 3.4.2 CSR operations As mentioned in section 2.12, some CSR are mapped inside eAccelerator. This makes it a must to access eAccelerator with instructions to interact with said CSRs. Two approaches to tackle this task are considered. Similar to the creation of stimulus for scalar memory operations, a mechanism could be set-up to send garbage sequences of CSR instructions at random times. This could be useful to test instructions sequences found in a context switch, where we read and write to registers at the same time. This approach’s main downsides are the lack of control. In any case, controlling the timing of these sudden bursts would pose quite a challenge for little to no reward. Instead, another approach would be to enable the environment to generate tests with said instructions. Then, we could modify the test-bench to issue CSR instructions as it would do with regular vector instructions. This approach allows much more control of the behaviour of the tests, and is a lot simpler to implement, with little difference in functionality. 3.4.3 Custom extensions support Custom instructions are something the test-bench partially supports at the beginning of this project. This is because they are vector instructions, but they do not follow RISC-V formatting standards on certain fields of instruction encoding. However, almost full support of these instructions has been implemented in spike. The main idea of this task is to aid achieving full support of these instructions in constrained random test generation. This task would be trivial if an assembler was in place to complete the software stack from top to bottom. As mentioned in section 2.14, this is not the case, but support of these instructions needs to be implemented anyway. To circumvent this issue, we plan to modify the random test generator to directly encode instructions, instead of using regular instruction mnemonics. 24 Chapter 3. Verification plan 3.4.4 Functional coverage Functional coverage is a very side-kicked feature at the start of this project. Instructionlevel coverage was ported from another project with similar specifications. Adding new instructions to this coverage will ensure our implementations of other tasks are well tested. Functional coverage of internal modules were part of the initial planning, but the time frame given for this thesis does not fit such work. Some code coverage is planned to be also collected to further gain metrics on the state of the project. 25 Chapter 4 Implementation 4.1 Memory operations So far, the memop interface was pulled down, to not cause any interference with the execution of the rest of the test bench. This was possible because the processor is not really there; it is a simulated environment where no scalar stores are really executed. That is something the test bench may control at any given time. What follows is a general outline of the steps to be carried out: •Generate randomized scalar memops through the interface at randomized instants during the execution. The objective is to create some initial stimulus, and build an infrastructure to extend upon. Make sure the protocol is followed according to specification, and check if the module is responding accordingly. The specific sub-interfaces to be used are the following: –Scalar memop issue. –Scalar memop physical address asynchronous issue. –Overlap result receiving and interpretation. –Done signaling. •Generate scalar memops in a directed fashion. This will allow to trigger the main interface mechanism: the overlap check. When a vector store is issued, and a younger scalar load is also issued, eAccelerator will report an overlap, and act accordingly. This step should focus on targeting the correct address, and scalar memop generation delay control. •Parameterize the implementation of this mechanism. This will allow behavioural modifications at runtime, and fine control over test results. Changes to be applied could start at different components. A couple of them were discussed at the planning phases of this task, and are now revisited. •Create a whole new agent with all necessary components to control memory operations. This would be great to separate from other agents, with own parameters, and specific necessities to be solved. •Embed all this mechanism inside the execution flow of the main agent. This will give us easy access to all other signals of that same interface. Since 26 Chapter 4. Implementation cpu_memop is part of said EPA interface, it makes perfect sense to include such control in the main agent. Therefore, the second approach was selected. Building a monolithic agent to control all signals of an interface as big as EPA IF, would be inefficient and time-consuming if all those signals were completely unrelated. But since the main execution flow of instructions could directly affect the behaviour of other mechanisms (like cpu_memop), having all processes be part of a single agent actually simplifies implementation. Therefore, code of this feature will live inside eprocessor_agent (see figure 3.2). 4.1.1 Exploration of eprocessor_agent To implement this mechanism, first logical step is to find out what role does every part of the selected agent develop. eprocessor_agent consists of seven main components: •agent_cfg Configuration of the agent itself. Set of parameters that change the way this agent behaves. •seqr The sequencer of the agent itself. Takes instructions received from spike’s sequencer, and sends them to either the driver, of the snf_backdoor_driver, based on the type of instruction received (if it is a vector instruction, the sequence item is sent to driver; else, it is sent to the backdoor driver). •driver Only driver of the agent. Receives instructions from the sequencer, and enqueues them to protocol_base_class issue queue. •protocol_base_class Interprets sequence items enqueued by the driver, and consumes simulation time by following the protocols of EPA interface; issuing and retrieving instructions one at a time. •monitor_protocol Captures instructions that returned result, and sends them to a blocking put port for scoreboards (like protocol_scoreboard) to interpret them. •monitor_pre Puts into a blocking port information about state of the golden reference model about an instruction. This is what ISS simulated on that instruction. •monitor_post Sends into a blocking port what the DUT sent about the executed instruction. This information is used with what monitor_pre captures inside isa_scoreboard, to report errors in result. The memop mechanism code will be sub-divided into two main tasks: generating scalar memops metadata (random or directed), and driving the interface using that information. Two main options are considered to architect the mechanism’s code location. First, creating a new sequencer to decide when to generate randomized scalar memops, and then use that metadata to issue memops at will in the driver. This option gives a purpose to every component, and neatly separates responsibilities to each bit of the agent. On the other hand, including the scalar memop generation into the driver itself gives us the option of interacting with the instruction issue mechanism. As we will later discover, this small detail opens up some interesting possibilities. Therefore, the later alternative is selected. Based on what we know, let’s now analyse coproProtocol internals. coprocessor 27 Chapter 4. Implementation 4.1.3.2 VIF interactor 1if (‘VIF.signal_ready[0] && prev_queue.size()) begin 2scalar_memop_t tmp = prev_queue.pop_front(); 3‘VIF.signal[0].ri <= tmp.ri; 4// ... 5‘VIF.signal_valid[0] <= 1’b1; 6next_queue.push_back(tmp); 7end else begin 8‘VIF.signal[0].ri <= ’{default:’0}; 9// ... 10 ‘VIF.signal_valid[0] <= 1’b0; 11 end Figure 4.8: Generic implementation of a VIF interactor component This component starts by asserting that the interface is ready (signal set by the VPU), and that the feeder queue contains memops. When these two conditions meet, the driver pops the front operation of the FIFO queue (first inserted), and feeds the interface with the required parameters. All interfaces use reorder buffer index as an identifier for the memop, so its signaling is represented in the example. Setting the valid signal is also necessary to complete the handshake. When the first condition does not match, the interface being driven is brought to a default zero state, with the signal valid also being lowered. 4.1.4 Multiple loads and stores per cycle As mentioned in section 3.1.2.3, all signals of the memop interface are duplicated. This is because the number of loads and store that the scalar core can issue every cycle is parametrised (NLS). Our mechanism at the moment does not contemplate this possibility, and uses only the first line of the NLS lines that could exist. To solve this, a small modification in the driver’s side must be applied. 1for (int line = 0; line < NLS; ++line) begin 2if (‘VIF.signal_ready[line] && prev_queue.size()) begin 3scalar_memop_t tmp = prev_queue.pop_front(); 4‘VIF.signal[line].ri <= tmp.ri; 5// ... 6‘VIF.signal_valid[line] <= 1’b1; 7next_queue.push_back(tmp); 8end else begin 9‘VIF.signal[line].ri <= ’{default:’0}; 10 // ... 11 ‘VIF.signal_valid[line] <= 1’b0; 12 end 13 end Figure 4.9: Generic implementation of a VIF interactor component with NLS With this for loop, we force the driver to repeat the process for every interface instance there is. If only one scalar memop is found in the queue, only the first line of the interface is used. If more than one struct reach the queue at the same cycle, more than one line will be used to send stimulus. 34 Chapter 4. Implementation 4.1.5 Mechanism configuration To craft a highly adaptative system, we introduce parameters. To do so, we use SystemVerilog’s plusargs. These command line arguments can be read at run time, and used to modify behaviour. The following parameters are added to the mechanism. •max_count Maximum number of scalar memory operations to be created during the whole simulation run. This parameter is user defined, and gets default value 2048 if the user does not define it. •prob Probability of a memop creation. Value between 0 and 1000. Unless user defined, its value is randomized. •drive_mode Memop generation mode. Seen in 4.1.7. •ranges is a set of configurations that determine the random values contstraints of the creation of memops. –issue_delay_range determines the maximum and minimum values of the issue delay range. Range can include negative values, which represent an inversion in the order of issuing. A negative issue delay when using drive_mode VEC_MEMOP means that the scalar memop will be issued -issue_delay cycles in advance to the vector memop. Default value is 1:1. –pad_delay_range determines the delay range of the second delay of a scalar memop: the pad_delay. This range can only contain positive numbers and zero. Defaul value is 0:0. –done_delay_range defines the delay range of the last delay of a scalar memop: the done_delay. Just like pad_delay, it can only contain positive values and zero. Default value is 1:1. –paddr_offset_range gives the range of physical address offset to be randomized. This value is later added to the targeted physical address. Not constrained to any way inside the integer numbers domain. Default value is 0:0. 4.1.6 Factory pattern To properly unify the creation of these scalar memory operations, a factory pattern is put in place. The main idea is to create a SystemVerilog class that is able to gather all parameters about scalar memops, and generate structs when requested to. This class is a singleton, since the needs of this component make it useful to gather arguments just the first time it is created. There exists another option, which is to create a generic UVM component, which will be able to reimplement the framework’s base phases. This option was discarded because it added unnecessary complexity. The following methods are implemented in the aforementioned factory class. •protected function new() Protected new function. Reads plusargs to configure randomisation ranges. Triggers an error if any problem is found with the input, and stops the simulation. Initializes all other variables if needed. 35 Chapter 4. Implementation •function scalar_memop_factory get_instance() Get the one and only static instance of this class. Create it if necessary. •function new_scalar_memop(RobIndex ri, Paddr paddr) Returns a new scalar memop struct, with properly randomised fields. •function int get_memops_left() Get number of memory operations left to be created. •function memop_drive_mode_t get_memops_left() Retrieve drive mode to be used. Used to activate or desactivate certain generators depending on the selected drive mode. •function int get_prob() Get selected probality (user defined or randomly selected) for scalar memop creation. It is used by randomising a number between MAX_PROB and 0, then seeing if its value is smaller than the returned probability. If the randomisation function is homogeneus, this should assert true prob 10 %of the time. Arguments of the memop mechanism configuration live inside this class. Their definition can be observed in figure 4.10. 1// Unique instance of this class in the whole system 2static local scalar_memop_factory m_self; 3 4// Count of all the memops created. Internally managed 5local int m_count; 6 7// Max count of all the memops created. User defined. 8local int m_max_count; 9 10 // Probability of a memop creation. User defined. 11 local int m_prob; 12 13 // Configuration for randomization. No need to be accesible from outside the class 14 local rand_config_t m_rc; 15 16 // How the drivers should behave when creating memops 17 local memop_drive_mode_t m_drive_mode; Figure 4.10: Member arguments of class scalar_memop_factory To ensure correct parsing of aforementioned user input plusargs, we define a format to input values. For all numerical arguments, plusargs should only contain decimal characters [0-9], with only "+" or "-" allowed to denote sign. For ranges, two decimal values must be supplied: the minimum and the maximum, in that order, separated by a single colon. For discrete type inputs (enumerations), a string must be supplied in upper case, which will be matched and parsed to the value of the enumeration with the same name. The following snippets of code show examples of how parsing goes about in each case. 36 Chapter 4. Implementation 1if ($value$plusargs("MEMOP_PROB=%s", tmp)) begin 2if ($sscanf(tmp, "%d", m_prob) != 1) begin 3‘uvm_fatal("scalar_memop_factory", $sformatf("Cannot parse string \"%s\" as MEMOP_PROB", tmp)); 4end 5end else begin 6m_prob = $urandom_range(MAX_PROB, 0); 7end 8‘uvm_info("scalar_memop_factory", $sformatf("MEMOP_PROB value of %d", m_prob), UVM_NONE); Figure 4.11: Parser for numeric parameter prob 1if ($value$plusargs("MEMOP_ISSUE_DELAY_RANGE=%s", tmp)) begin 2if ($sscanf(tmp, "%d:%d", m_rc.issue_delay_range.min, m_rc.issue_delay_range.max) != 2) begin 3‘uvm_fatal("scalar_memop_factory", $sformatf("Cannot parse string \"%s\" as MEMOP_ISSUE_DELAY_RANGE", tmp)); 4end 5end else begin 6m_rc.issue_delay_range = ’{DEFAULT_MEMOP_ISSUE_DELAY_MIN, DEFAULT_MEMOP_ISSUE_DELAY_MAX}; 7end Figure 4.12: Parser for range parameter issue_delay_range 1if ($value$plusargs("MEMOP_DRIVE_MODE=%s", tmp)) begin 2case (tmp) 3"ALWAYS": m_drive_mode = ALWAYS; 4"VEC_MEMOP": m_drive_mode = VEC_MEMOP; 5"DISABLE": m_drive_mode = DISABLE; 6default:begin 7‘uvm_fatal("scalar_memop_factory", $sformatf("MEMOP_DRIVE_MODE \"%s\" is not valid. Supported modes are: ALWAYS, VEC_MEMOP and DISABLE", tmp)) 8end 9endcase 10 ‘uvm_info("scalar_memop_factory", $sformatf("MEMOP_DRIVE_MODE %0s selected", m_drive_mode.name()), UVM_NONE); 11 end else begin 12 m_drive_mode = DISABLE; 13 ‘uvm_info("scalar_memop_factory", $sformatf("MEMOP_DRIVE_MODE not specified. Defaulting to %0s", m_drive_mode.name()), UVM_DEBUG); 14 end Figure 4.13: Parser for enumeration parameter memop_drive_mode_t Range plusargs specifically are used to constraint randomisation. For that, SystemVerilog’s built-in capabilities are used to randomise the structs. What follows is the code of new_scalar_memop that takes care of creating and randomising these structs. 37 Chapter 4. Implementation 1function scalar_memop_t new_scalar_memop(RobIndex ri, Paddr paddr); 2scalar_memop_t sm; 3sm.ri = ri; 4sm.paddr = paddr; 5std::randomize(sm) with { 6sm.issue_delay inside {[m_rc.issue_delay_range.min:m_rc.issue_delay_range.max]}; 7sm.pad_delay inside {[m_rc.pad_delay_range.min:m_rc.pad_delay_range.max]}; 8sm.done_delay inside {[m_rc.done_delay_range.min:m_rc.done_delay_range.max]}; 9sm.paddr_offset inside {[m_rc.paddr_offset_range.min:m_rc.paddr_offset_range.max]}; 10 }; 11 m_count += 1; 12 return sm; 13 endfunction Figure 4.14: Implementation of new_scalar_memop As observed in figure 4.14, parameters ri and base paddr are not randomised. Instead, the caller supplies these parameters according to their needs. Specifically for delays and offset, constrains are put in place to limit the range of possible values that these variables can take. Returning to plusargs, a string of structure like +argument=value is required on the command line of the simulator to supply the arguments. Launching the testbench is done using a Makefile, where we can assign names for these parameters, and create sort of an interface for the user to control. These parameters directly correlate with the ones specified in subsection 4.1.5 (which live inside the memop factory) and described as follows. Argument Default value Possible values Comments MEMOP_DRIVE_MODE DISABLE VEC_MEMOP, ALWAYS, DISABLE Selects the operation mode for the scalar memop mechanism MEMOP_PROB *random [0,1000) Defines the probability of creating a memop at each evaluation. *If not defined, a random value inside the possible range is used MEMOP_MAX_COUNT 2048 NDenotes the maximum number of memops to be created. Negative values are interpreted as 0. MEMOP_ISSUE_DELAY_RANGE 1:1 ZDelay between vec. memop and scalar memop when VEC_MEMOP drive mode is used. Negative values denote an issue of the scalar memop before the vector memop that triggered it MEMOP_PAD_DELAY_RANGE 0:0 NDelay between issue and physical address signals. If negative, the value is interpreted as a 0 MEMOP_DONE_DELAY_RANGE 1:1 NDelay between physical address and done signals. MEMOP_PADDR_OFFSET_RANGE 0:0 NOffset applied to base address when drive mode is VEC_MEMOP Table 4.1: Table of user arguments for Makefile CLI 4.1.7 Memop generation modes Looking into generators, we can easily modify the behaviour of the mechanism depending on the position inside the code, and conditions followed to trigger them. Three main modes are discussed in this section. 38 Chapter 4. Implementation 1typedef enum { 2VEC_MEMOP, 3ALWAYS, 4DISABLE 5} memop_drive_mode_t; Figure 4.15: Enumeration memop_drive_mode_t VEC_MEMOP revolves around the idea of directed issues around vector memory operations. This created scalar memops only when a vectorial instruction that accesses memory is issued. ALWAYS acts as a stress test mode. Creates instructions every cycle, using a probability as a “limiter”. DISABLE does just that. Disables the generation of memops when other mechanisms want to be observed with the test-bench. 4.1.7.1 ALWAYS mode generator 1if (m_scalar_memop_factory.get_drive_mode() == ALWAYS) begin 2if (m_scalar_memop_factory.get_memops_left() > 0 && $urandom_range(MAX_PROB, 0) <= m_scalar_memop_factory.get_prob()) begin 3scalar_memop_t new_memop; 4 5new_memop = m_scalar_memop_factory.new_scalar_memop(select_next_ri(), 0); 6pending_garbage_memops.push_back(new_memop); 7‘uvm_info("copro_protocol", $sformatf("Created memop with the following parameters: %p", new_memop), UVM_DEBUG) 8end 9end Figure 4.16: Implementation of scalar memop generator in ALWAYS mode This generator is directly inserted into the task created at the beginning of this section: issue_garbage_memop (see figure 4.3). Since we want to evaluate the generation of memops each and every cycle of the clock, this is the perfect place to put the generator. When the generator is active, each cycle we generate and evaluate a probability. When the maximum number of scalar memops has not been reached yet and probability asserts, we use the scalar memop factory, and generate a new scalar memory operation. Finally, the new simulated operation is pushed into the pending memop operations queue. 39 Chapter 4. Implementation 4.1.7.2 VEC_MEMOP mode generator 1if (m_scalar_memop_factory.get_drive_mode() == VEC_MEMOP) begin 2if (‘IS_MEMOP(m_instr.iss_state.instr) && m_scalar_memop_factory.get_memops_left() > 0 && $urandom_range(MAX_PROB, 0) <= m_scalar_memop_factory.get_prob()) begin 3// Create and store new memop 4scalar_memop_t new_memop; 5new_memop = m_scalar_memop_factory.new_scalar_memop(select_next_ri(), m_instr.iss_state.src1_value); 6if (new_memop.issue_delay < 0) begin 7// Swap ri’s 8RobIndex tmp; 9tmp = new_memop.ri; 10 new_memop.ri = m_instr.rob_index; 11 m_instr.rob_index = tmp; 12 end 13 pending_garbage_memops.push_back(new_memop); 14 ‘uvm_info("copro_protocol", $sformatf("Created memop with the following parameters: %p", new_memop), UVM_DEBUG) 15 if (new_memop.issue_delay < 0) begin 16 ‘uvm_info("copro_protocol", $sformatf("Negative issue_delay with VEC_MEMOP drive mode. Delaying vector instruction issue by %d cycles", -new_memop.issue_delay), UVM_DEBUG) 17 wait_for_clk(-new_memop.issue_delay); 18 end 19 end 20 end Figure 4.17: Implementation of scalar memop generator in VEC_MEMOP mode This mode of generation wants to create a scalar memory operation when a vector memop is detected. To do so, this generator is inserted in the code of issue_instr, inside the main function of the coprocessor protocol class (seen in figure 4.3). With this, a condition to assert when the vector operation being issued accesses memory is placed in the wrapping if-clause. This condition uses the compiler-expandable macro ‘IS_MEMOP, which tells us if the vector instruction instr accesses to memory. Up until here, creation of the memop is almost exactly like figure 4.16. Now, if issue_delay is negative in this case, some extra steps are taken. First, reorder buffer indexes of the scalar a vector instructions are inverted. This causes the internal logic of the VPU to consider the scalar memop as issued before the vector memop. Secondly, issuing of the vector instruction is delayed as many cycles as issue_delay negated. This will buy time for the scalar memop to advance in the queue system, simulating the desired delay. 4.1.8 Proper control of commit_ri For a moment during implementation of memops, adding the reorder buffer index to the control queue of commit_ri was considered. The test generated problem with the control of the signal itself because of the very different nature of both signals. The main obstacle was that, although scalar memops use the same reorder buffer index as vector instructions to identify themselves, the mechanism targets parallel issuing of both instruction types. This is the way to trigger overlaps, and issuing 40 Chapter 4. Implementation both kinds of instructions sequentially would not work. On top of that, when using commit_ri to control issuing of scalar memops, simulation hangs. Diagnosing this problem took collaboration with the RTL team, and it was found that the test-bench was changing commit_ri in erratic ways. Besides, the queue of values to put on the interface only advanced when a vector instruction result was received. This meant that including scalar memops in this queue would erase the parallel objectives of this mechanism. Simply leaving the commit_ri controller as-is makes all instructions (including scalar memops) before a certain vector instruction to stop being speculative. Therefore, the mechanism is left untouched, and only vector instructions affect the control of this parameter. The only thing to consider is that reorder buffer indexes still denote "order". This means that scalar memops identify themselves using this same identifier. That suffices to ensure correct operation of the mechanism. 4.1.9 Monitoring Last step to compete everything related to scalar memops is control of correctness. To do so, a couple of options arise. On the one hand, we could implement a monitor-scoreboard system to capture and analyse on different components of the test-bench. On the other hand, we could observe and analyse the interface on one single monolithic component. The latter option is prefered, to reduce the number of components, and allow for future improvements. A monitor and an agent UVM components are created to implement this feature. The stand-alone agent is created following the same design philosophy as chi_monitor and chi_agent (see figure 3.2), where the monitor deals with the responsibility of both components, and eliminates any need for interchanging sequences. To implement mif_monitor, we must first know what this checker needs to look for. Extending from uvm_monitor, of which only one phase is reimplemented: the run_phase. In the task that represents said phase, this monitor looks for errors in repeated transactions, transaction order, and overlap conflict activity not matching specification. It does this by saving a copy of all transactions observed in the cpu_memop interface, and grouping them by reorder buffer index. This way, it can trace the progress of the scalar memop in its passing through the interface. 1struct { 2realtime issue_realtime; 3CPUMemOpIssue data; 4} memop_issue[RobIndex]; 5CPUMemOpPad memop_pad[RobIndex]; 6Overlap memop_overlap[RobIndex]; 7realtime memop_vec_store_infl[RobIndex]; Figure 4.18: Data structures of mif_monitor To be able to precisely control any possible miss-matches in protocol functionality, we store all the kinds of memop messages. We use SystemVerilog’s associative arrays to identify all captures with their RobIndex, which should be uniquely representing a single instruction every instant. Moreover, we capture any vector memory 41 Chapter 4. Implementation instructions in-flight that store data. Storing a timestamp of the event gives us the possibility of comparing events order, which can help detect errors in timing specifications. With this data system, we have a representation of what the system looks like at each moment. Whenever an operation finalises without errors, their data structs are erased, and the monitor “frees” the RobIndex for another instruction to come. The reimplemented task run_phase consists of a forever clause, which runs until the end of simulation. We bound this running by using @(posedge m_mif.clk), which instructs the task to run every time there is a raising edge on the clock signal of the interface. 1if (m_mif.insn_ready && m_mif.insn_valid) begin 2RobIndex ri = m_mif.insn.ri; 3if (‘IS_MEMOP(m_mif.insn.iword) && ‘IS_STORE(m_mif.insn.iword)) begin 4memop_vec_store_infl[ri] = $realtime; 5end 6end 7 8if (m_mif.result_ready && m_mif.result_valid) begin 9RobIndex ri = m_mif.result.ri; 10 if (memop_vec_store_infl.exists(ri)) begin 11 memop_vec_store_infl.delete(ri); 12 end 13 end Figure 4.19: In-flight vector store control of mif_monitor From here, we capture activity in the insn interface (figure 4.19), where we capture vector stores issued to the VPU, and eliminate them when their result is retrieved. During that period of time, the instruction is in-flight. For the next monitoring tasks, we iterate through all lines of the interface (NLS). In each iteration, we check possible errors whenever a transaction of the interface arrives. •Whenever an issue transaction is captured, log the $realtime in which the message is observed, and the data that comes through the interface. –If an active transaction exists with the same RobIndex as the transactions just observed, a uvm_fatal is reported saying that the issue signal with said ri was sent twice. –If a vector memory operation is in-flight at the moment, and RobIndex coincides, we trigger an error that communicates that such RobIndex is being used by a vector memop. •When a pad transaction arrives, log the information into the corresponding associative array. –Check for duplication errors. Fail if another active pad message is found with the same RobIndex. –If the pad message is received without any corresponding issue message, then fail and report such casuistic. 42 Chapter 4. Implementation •If an overalap transaction is observed, log the data into the array and check for the following errors. –Duplication of overlap message. Scalar memop overlap (with same RobIndex) is active in the system. –Nonexistence of preceding issue message. –Nonexistence of preceding pad message. –Low conflict signal when conflict exists. This is triggered when the following conditions are met: The scalar memop is a pure load (rd is high), the conflict signal is low, and there are in-flight vector store instructions with an issue time older than the scalar memop. •When the eventual done signal is sent, a single condition is checked. This condition asserts that all previous messages of the referenced scalar memop exist. If so, we erase all of them from their respective associative arrays. Else, an error is triggered communicating the missing messages. Note that errors retrieved in issue and pad interfaces would indicate problems in the implementation of the UVM agent. However, this kind of monitoring could ideally be used in a passive verification environment, where no extra stimulus is sent, and observations on the core+eAccelerator interface could help up indicate problems with RVOOO’s understanding of the protocol. 43 Chapter 4. Implementation throughout the specification of RISC-V Vector extension. Specification tells us that these instructions are part of the OPV opcode group. Therefore, the encoding always contains 1010111 as their opcode. Looking at the funct6 available space of OPV opcode, 111001 does not collide with any other instruction. In funct3 is where specification of bioinformatics instructions deviates from what is instructed in RISC-V V specification. For OPV instructions, funct6 is the only term used to specify an operation further. Funct3 should only specify what the opcode class indicates. In the case of OPV, these three bits indicate the source of operands (i.e., combinations of vector, scalar, and immediate sources). Instead, bioinformatic instructions work with a single funct6, and use funct3 to further specify one of six possible instructions. Going back to the test-bench, we find that the main candidate to extend this new class from is riscv_vector_instruction. It has vector registers, masks, and other common vector operation amongst its inheritable parameters. 1class riscv_custom_instr extends riscv_vector_instr; 2 3‘uvm_object_utils(riscv_custom_instr) 4‘uvm_object_new 5 6function bit [2:0] get_func3(); 7case (instr_name) 8VBPCK_VV: return 3’b000; 9VBUNPCK_VV: return 3’b001; 10 VBPCNT_VV: return 3’b010; 11 VBBCNT_VV: return 3’b011; 12 VBMAX3_VV: return 3’b100; 13 VBMIN3_VV: return 3’b101; 14 default: ‘uvm_fatal("riscv_custom_instr", $sformatf("get_func3. Bad custom instruction %s", instr_name.name())) 15 endcase 16 endfunction : get_func3 17 18 // Convert the instruction to assembly code 19 virtual function string convert2asm(string prefix = ""); 20 string asm_str; 21 string disasm_str; 22 bit [31:0] code; 23 bit [5:0] func6; 24 bit vm; 25 bit [2:0] func3; 26 bit [6:0] opcode; 27 28 disasm_str = {get_instr_name(), " ", vd.name(), ", ", vs1.name()}; 29 func6 = 6’b111001; 30 vm = 1’b1; 31 func3 = get_func3(); 32 opcode = 7’b1010111; 33 34 case (instr_name) 35 VBPCK_VV, VBUNPCK_VV: begin 36 vs2 = V0; 37 end 38 VBPCNT_VV, VBBCNT_VV, VBMAX3_VV, VBMIN3_VV: begin 39 disasm_str = {disasm_str, ", ", vs2.name()}; 40 end 41 default: ‘uvm_fatal("riscv_custom_instr", $sformatf("convert2asm. Bad custom instruction %s", instr_name.name())) 42 endcase 43 44 code = {func6, vm, vs2, vs1, func3, vd, opcode}; 45 asm_str = format_string($sformatf(".4byte 0x%0x", code), MAX_INSTR_STR_LEN); 46 comment = {disasm_str, " (bioinformatics instruction)", comment}; 47 48 if (comment != "")begin 49 asm_str = {asm_str, " #",comment}; 50 end 51 return asm_str.tolower(); 52 endfunction : convert2asm 53 54 endclass : riscv_custom_instr Figure 4.27: Code of class to represent bioinformatics instructions As seen in figure 4.27, a single virtual function is needed to be reimplemented from our base class: convert2asm. This method is queried whenever the main test generation engine wants a valid assembly representation of the instruction. Thanks to RISC-V’s standards, this method can be implemented once, and reading funct3 and 50 Chapter 4. Implementation funct6, used with every child class. The expected return value is a string containing the mnemonic of the instruction. What we do instead is assemble the instruction ourselves. We then end up with the encoded value of the instruction itself (variable code in aforementioned code). With this, we introduce the string ".4byte 0x" with the resulting value appended, and use it as the return value. This representation of the instruction is still valid. Oblivious to the existence of custom instructions, the assembler will simply place this value in the position specified. The instruction will then end up in the final generated test binary. Observe the use of the helper function get_func3. This function returns the identifying parameter of bioinformatics instructions. We match to the mnemonic of the instruction type, and return a unique value for every instruction. After this, we declare as many extensions of the class as instructions there are. This is done with a compiler macro, to not repeat code. This way, we possess a class for every possible instruction, all inheriting directly from a custom common class. Finally, we delare an instruction group to which this instruction group pertains. This will allow us to enable and disable the generation of these instructions at will. Only modifying the target file will be needed to do so. For custom bioinformatics instructions, the name of the instruction group assigned is RVVBIO. 4.3.2 Format converting vfcvt.f.f.v vfcvt is an instruction part of the RISC-V V extension used to perform vectorized floating-point conversion operations. Variations of this instruction change the operand kind (e.g., vfcvt.f.x.v converts all floating point elements to the equivalent integer representation). eAccelerator implements multiple formats of floatingpoint representation, and so arise the need of an instruction to convert between them same-size conversions: vfcvt.f.f.v. To be exact, eAccelerator implements two formats for 8-bit floating point representation (FP8 E4M3 and FP8 E5M2), and two more for 16-bit representation(FP16 and BF16)[20][21]. Figure 4.28: Narrow floating point formats. Taken from [39] 51 Chapter 4. Implementation These narrow formats are the natural progression of deep learning acceleration, always striving for better performance by sacrificing precision. All representations have their use, and eAccelerator implements them with the idea to support heavy machine learning workloads. Specifically vfcvt.f.f.v is the instruction of choice to convert between FP8 E4M3 and FP8 E5M2. Mnemonic funct6 vm vs2 vs1 funct3 vd opcode vfcvt.f.f.v 100010 vm vs2 00100 001 vd 1010111 Table 4.5: Custom convert instruction disassembly vfcvt.f.f.v has no extra space in its instruction encoding, so to decide the direction of the conversion, we must set a bit of the vtype CSR: altfp. At the same time, this bit is used to indicate the current type beign worked with for other instructions, so it represents the destination format. Since this operation only converts between 8-bit formats, CSR vsew must be set to SEW8. Argument vs1 holds the value that represents the encoding space of this instruction, which refers to the type of conversion that the instruction does. This is part of the VFUNARY0 encoding space, and the different values represent different conversions. vs1 Name single-width converts 00000 vfcvt.xu.f.v 00001 vfcvt.x.f.v 00010 vfcvt.f.xu.v 00011 vfcvt.f.x.v 00100 vfcvt.f.f.v widening converts 01000 vfwcvt.xu.f.v 01001 vfwcvt.x.f.v 01010 vfwcvt.f.xu.v 01011 vfwcvt.f.x.v 01100 vfwcvt.f.f.v narrowing converts 10000 vfncvt.xu.f.v 10001 vfncvt.x.f.v 10010 vfncvt.f.xu.v 10011 vfncvt.f.x.v 10100 vfncvt.f.f.v Table 4.6: VFUNARY0 encoding space As seen in table 4.6 the custom instruction vfcvt.f.f.v uses the first available slot of said encoding space, directly after the last single-width convert instruction. Also observe that it shares all 3 Least Significant Bits (LSb) with widening and narrowing instructions of the same type (f.f.v). This is a conscious design choice to simplify decoding. To implement this instruction in random test generation, we find the simplest route to be slightly modifying the base class of riscv_vector_instr to account for the existence of this instruction. 52 Chapter 4. Implementation 1virtual function string convert2asm(string prefix = ""); 2 3// (...) 4 5if (instr_name == VFCVT_F_F_V) begin 6bit [31:0] code; 7bit [5:0] funct6; 8bit [2:0] funct3; 9bit [6:0] opcode; 10 bit [4:0] vs1_encoding_space; 11 12 funct6 = 6’b100010; 13 funct3 = 3’b001; 14 opcode = 7’b1010111; 15 16 vs1_encoding_space = 5’b00100; // Encoding space VFUNARY0 17 18 code = {funct6, vm, vs2, vs1_encoding_space, funct3, vd, opcode}; 19 asm_str = format_string($sformatf(".4byte 0x%0x", code), MAX_INSTR_STR_LEN); 20 comment = {"Custom f to f instruction ", get_instr_name(), ", ", vd.name(), ", ", vs2.name()}; 21 if (!vm) begin 22 comment = {comment, ", v0.t"}; 23 end 24 end 25 26 if(comment != "")begin 27 asm_str = {asm_str, " #",comment}; 28 end 29 return asm_str.tolower(); 30 endfunction : convert2asm Figure 4.29: Addition to riscv_vector_instr to represent vfcvt.f.f.v Using a similar method as bioinformatic instructions (see figure 4.27), this mechanism generates the ".4byte 0x" sequence to insert this specific instruction. The difference in this case is that mask bit could be valid and such is taken as the randomized value. Randomized vs1 has no meaning in this context since it encodes a completely different thing. This means that the only thing this piece of code does is modify the original result of convert2asm in case the instruction is our custom conversion. We then define the RVVCVT instruction group that we will use in target files to include or exclude our newly implemented instruction. One last detail to finish implementing this instruction lies in correctly specifying what SEW is the environment set to at the instant this instruction is generated. To ensure this instruction only appears when SEW8 is set, we must define a constraint that limits the generation of this instruction under certain conditions. Figure 4.30 shows a snippet of code doing just that. 1// Filter unsupported instructions based on configuration 2virtual function bit is_supported(riscv_instr_gen_config cfg); 3// ... 4if (cfg.vector_cfg.vtype.vsew != 8) begin 5if (instr_name inside {VFCVT_F_F_V}) 6return 1’b0; 7end 8// ... 9return 1’b1; 10 endfunction : is_supported Figure 4.30: Constraint of vfcvt.f.f.v generation This function is used by the main test generator when deciding if an instruction should be included in the final program or not. Therefore, if a vfcvt.f.f.v is generated out of the scope of SEW8, it will be discarded, and a new instruction will be generated. 53 Chapter 4. Implementation 4.4 Coverage 4.4.1 Overview of existing coverage module To implement functional coverage of our recently implemented custom instructions, we must look at past efforts. ISA-level functional coverage was ported from the EPI project, and adapted to the specific needs of eAccelerator. Implementation wise, it defines a module to harbor all coverage logic surrounding coverage of vector instructions. 1parameter CORE_INSTR = INSTR_WIDTH; 2parameter CORE_DATA = XREG_WIDTH; 3 4module cov_rvv_instr ( 5input logic clk, 6input logic rsn, 7input logic issue_valid, 8input rvooo_coprocessor_it::Insn issue_instr, 9input logic [CORE_DATA-1:0] issue_data, 10 input vpu_pkg::vcsrs_t issue_csr 11 ); Figure 4.31: Interface for cov_rvv_instr coverage module This module is declared in test_harness, where it will pend from the top module of the test-bench. From here, a generic covergroup is declared to adapt around every possible vector instruction and their parameters. The name is cg_instr, and the coverpoints it contains can be checked out in table 3.13. An instance of this covergroup is declared for every instruction. Using the constructor, we modulate the covergroup with parameters that activate and deactivate coverpoints. It does this by setting option.weight to either 0 or 1, using a binary evaluation of parameters that can activate said coverpoint (e.g., if a coverpoint has option.weight = x & !y, this will be active whenever option x is active and y is not). Figure 4.32 shows the header of the definition of cg_instr covergroup, with all its parameters listed. 1covergroup cg_instr (bit mask=1, 2bit vf=0, 3bit scalar=0, 4bit load=0, 5bit indexed=0, 6bit store=0, 7bit immediate=0, 8bit vn_vw=0, 9bit no_vs2=0, 10 bit even_vs2=0, 11 bit even_vd=0, 12 bit vs1_0=0, //!vs1 is always v0 13 bit no_vs1=0, 14 bit no_vd=0, 15 bit unmasked=1, 16 bit only_vl0=0, 17 bit vwr_mask=0, 18 string name) with function sample(cov_instr_t instr); Figure 4.32: Covergroup cg_instr definition header After this, the module runs on every clock rising edge of the clk input signal monitoring if issue_valid is high. When this happens, and instruction has been issued, 54 Chapter 4. Implementation and a sample is captured on the appropriate covergroup. This is done with a case clause, and the match is done with the mnemonic of the instruction. 1//! Coverpoint for destination vector register (vd) 2cp_vd: coverpoint instr.vd { 3option.weight = !store & !even_vd & !no_vd; 4bins v0 = {V0}; 5bins other = default; 6} Figure 4.33: Example coverpoint cp_vd Code in figure 3.2.2.2.1 shows a coverpoint that is active when the instruction is not a store (therefore, destination vector register vd is relevant), option even_vd is inactive (which is another coverpoint that takes hits only when the destination register is even), and option no_vd is inactive too (for instructions that do not have a destination register). Bins represent specific named values of the variable that are a target to hit. We may also specify illegal bins, and define sequence of bins to match desired values. In the case of the example above, two bins are specified. Bin v0 collects hits whenever the value is 0. This is interesting due to V0 being the only register in RISC-V V that instructions can use as a mask register. Bin others collects hits whenever the previous condition does not match. 4.4.2 Implementation of new instructions to coverage module Functional coverage of vector instructions is extended for custom ones. All bioinformatics are included, as well as format converting. Another important instruction which was left aside was vsetvl, which is the operation that permits altering vector CSR. To begin implementation, we need to specify one covergroup instance for every new instruction we want to include. 1cg_instr vbpck_vv_cg; 2cg_instr vbunpck_vv_cg; 3cg_instr vbpcnt_vv_cg; 4cg_instr vbbcnt_vv_cg; 5cg_instr vbmax3_vv_cg; 6cg_instr vbmin3_vv_cg; 7cg_instr vbmin3_vv_cg; 8cg_instr vfcvt_f_f_v_cg; Figure 4.34: Instantiation of covergroups Afterwards, inside an initial begin clause of cov_rvv_instr module, we include the constructor assignation with all instructions, setting needed parameters for each one of them. 55 Chapter 4. Implementation 1vbpck_vv_cg = new(.name("vbpck_vv"), .mask(0), .no_vs2(1)); 2vbunpck_vv_cg = new(.name("vbunpck_vv"), .mask(0), .no_vs2(1)); 3vbpcnt_vv_cg = new(.name("vbpcnt_vv"), .mask(0)); 4vbbcnt_vv_cg = new(.name("vbbcnt_vv"), .mask(0)); 5vbmax3_vv_cg = new(.name("vbmax3_vv"), .mask(0)); 6vbmin3_vv_cg = new(.name("vbmin3_vv"), .mask(0)); 7vfcvt_f_f_v_cg = new(.name("vfcvt_f_f_v"), .no_vs1(1), .vf(1)); Figure 4.35: Constructor assignation of covergroups Observe how bioinformatics instructions have mask deactivated, and pack and unpack instructions activate option no_vs2. For format convert, we activate no_vs1 (because of VFUNARY0 encoding space taking that place) and vf (which indicates this is a vector instruction that operates with floats). Next, we must add our instructions to the general switch statement that matches instruction mnemonics each issue cycle. 1"vbpck.vv": vbpck_vv_cg.sample(instr); 2"vbunpck.vv": vbunpck_vv_cg.sample(instr); 3"vbpcnt.vv": vbpcnt_vv_cg.sample(instr); 4"vbbcnt.vv": vbbcnt_vv_cg.sample(instr); 5"vbmax3.vv": vbmax3_vv_cg.sample(instr); 6"vbmin3.vv": vbmin3_vv_cg.sample(instr); 7"vfcvt.f.f.v" : vfcvt_f_x_v_cg.sample(instr); Figure 4.36: Sampling of custom instruction covergroups To do this match, a function to get the mnemonic of incoming instructions is used. This function does not contemplate our custom instructions, so modified shall it be. Including bioinformatics instructions requires setting an exception to the norm. This function returns early whenever it finds the instruction matching. We can take advantage of said early return, and put a small piece of conditional code on top of the vector arithmetic opcode group. 1// ... 2OP_VEC_ARITH: begin 3if (i26_arith == 6’b111001) begin 4case (i12) 53’b000: return "vbpck.vv";// New instruction 63’b001: return "vbunpck.vv";// New instruction 73’b010: return "vbpcnt.vv";// New instruction 83’b011: return "vbbcnt.vv";// New instruction 93’b100: return "vbmax3.vv";// New instruction 10 3’b101: return "vbmin3.vv";// New instruction 11 default: return "MalformedBioinfo"; 12 endcase 13 end 14 case(i12) 15 3’b101: begin 16 case(i26_arith) 17 // ... Figure 4.37: Mnemonic identification on bioinformatics instructions 56 Chapter 4. Implementation For format converting, adding vfcvt.f.f.v supposes finding the case clause that matches with cvt instructions func6, and match of vs1 encoding space. We append this instruction at the end of the switch, with its identifier 00100, as seen in code figure 4.38. 1// ... 23’b001: begin 3case(i26_arith) 46’b000000: return "vfadd.vv"; 5//... 66’b101110: return "vfmsac.vv"; 76’b101111: return "vfnmsac.vv"; 86’b100010: case(i15) 95’b00000: return "vfcvt.xu.f.v"; 10 5’b00001: return "vfcvt.x.f.v"; 11 5’b00010: return "vfcvt.f.xu.v"; 12 5’b00011: return "vfcvt.f.x.v"; 13 5’b00100: return "vfcvt.f.f.v";// New instruction 14 5’b01000: return "vfwcvt.xu.f.v"; 15 5’b01001: return "vfwcvt.x.f.v"; 16 5’b01010: return "vfwcvt.f.xu.v"; 17 5’b01011: return "vfwcvt.f.x.v"; 18 5’b01100: return "vfwcvt.f.f.v"; 19 5’b10000: return "vfncvt.xu.f.w"; 20 5’b10001: return "vfncvt.x.f.w"; 21 5’b10010: return "vfncvt.f.xu.w"; 22 5’b10011: return "vfncvt.f.x.w"; 23 5’b10100: return "vfncvt.f.f.w"; 24 5’b10101: return "vfncvt.rod.f.f.w"; 25 endcase 26 // ... Figure 4.38: Mnemonic identification on format converting instructions 57 Chapter 5 Evaluation of results This section is aimed at reporting results gathered off of previous section’s efforts. This is meant to showcase the results obtained, and discuss on final results. All except the last section of this chapter discuss set-up and results of runs. Section 5.4 goes over found bugs with all simulations and problems uncovered during the development of this project. 5.1 Memop simulations To illustrate how the mechanism works with different configurations, let’s take a closer look at a simulation waveform. The test used is a simple ISA test to try vector memory instruction with unit strides. This means that all elements read are next to each other in memory. This program contains a load and a store instruction for every SEW from 64 down to 8, doubling vector length vl each step of the way, to maintain total bit count in all tests. Both masked and unmasked versions of the instructions are executed. Figure D.1 shows the code of the ISA test used to run this simulation. Some macros are used to set up the environment of the test before executing the target instructions. INIT_TEST initializes registers, CSR registers and memory. After that, it sets vl and sew using vsetvl/vsetvli.TEST_vl_op mainly executes necessary instructions to change selected element width and vector length before next execution. END_TEST loops infinitely around a set if instructions that write to a certain symbol. This way, the executor knows when the test is finished. RV_TEST_DATA places data in argument into symbol work_region, for anyone to access. Name Value max_count 2048 prob 750 issue_delay_range -10:10 pad_delay_range 0:20 done_delay_range 1:20 paddr_offset_range -1024:1024 Table 5.1: Parameters for all memop simulation runs To make this process easier to understand and eliminate variability between tests, 58 Chapter 5. Evaluation of results parameters are fixed to the values shown in table 5.1. The only parameter that does change between simulation runs is drive_mode, which determines generator placement. 5.1.1 Execution with drive mode DISABLE In this drive mode setting, all generators are disabled, and so is the mechanism itself. No scalar memop stimulus is observed in the interface. Figure 5.1: Resulting waveform with default parameters We see all stimulus is insn interface, where loads and stores are issued interchangeably. After 8 issued instructions, we observe trios of vector instructions being issued. This is because the instruction being used to fabricate the mask of the second half of the test (vid) is a vector instruction sent to the VPU too. Simulation finishes without errors, and no other remarkable things happen on the interface. 5.1.2 Execution with drive mode VEC_MEMOP This mode of execution evaluates generation of memops whenever a vector instruction is issued. The generator is placed in the issuer of the protocol class. With this, 3 out of 4 evaluations should assert true, and generate a scalar memop. After running the simulation, some interesting scenarios appear. 5.1.2.1 Older scalar memop The first scalar memop of the simulation is created on the first instruction of the test. This vector load creates a scalar load on the same physical address. With a negative issue delay of -8, the scalar memop issue is sent immediately, and the issuing of this instruction is delayed by 8 cycles. Having a pad_delay of 11 means that our vector load is issued before the pad of the scalar load. On the same cycle, the test-bench sends pad, and an overlap report arrives saying that the operation did not provoke any overlap conflicts. Although not necessary, after 6 more cycles, done is sent to the VPU, signaling the end of the scalar operation, and the execution of the vector load continues. 59 Chapter 5. Evaluation of results RTL team, a bugfix was submitted to patch the problem. It so happened that the interface was being controlled as if only one replication existed. This meant that even after seemingly signaling all previous scalar memops as done, handshake readiness never returned. One could reason that all instructions that signaled done through the second interface, never really took any effect on the internal state of eAccelerator. This meant that the queue of permitted in-flight instructions filled-up, and revoked permission to further issue new instructions. Figure 5.6: Replication issue bug Figure 5.6 shows how issue valid was set low in cpu_memop_issue_ready interface, never to return again. 5.4.2 Overlap specification mismatch Specification of eAccelerator states that overlap appears only when a vector store is issued, and a younger scalar load is issued then; only if the addresses of both instructions overlap. This means that the destinations address of the vector instruction contains the address of the scalar load. When running simulations, this was found to not be the case. Figure 5.7 shows a sequence where a unit-stride vector store targets address 0x080004000 and a scalar load targets address 0x180001BC4. Vector length is not long enough to cover the distance between both addresses, but the overlap message signals a conflict. 66 Chapter 5. Evaluation of results Figure 5.7: Overlap without real address overlap After opening an issue, implementation team explained that the specification’s overlap functionality was left out of the current release of the VPU’s load store unit (LSU), which is the component responsible for checking overlaps and memory conflicts (amongst other functionalitites). On the next release cycle, this component will contain a memory request buffer, to maintain some sort of in-flight memory request record, which will then be used for comparing operations, and doing memory disambiguation tasks. A content addressable memory (CAM) will be used to detect these memory overlap conflicts. Another possibility presented as a solution is a coarse-grain implementation of the overlap detection, meaning that address overlap can have some offset value that reduces the combinatory logic of this challenge. For strided memory accesses, calculating overlap may require a lot of logic, and these optimisations would reduce such computational need. 5.4.3 Scalar memop timing stalls VPU Sub-interface cpu_memop has no timing restrictions in their specification. After randomising parameters in generated memops, we found out that whenever issue and done messages coincide on the same cycle, an overlap report is returned, but instructions stop returning results at all. At first glance, this seemed to be a problem with the test-bench itself, and the way it controlled commit_ri: jumping between bigger and smaller values alternatively. This was fixed by excluding scalar memops from commit_ri, but the issue was still unsolved. This issue is still open, and part of an ongoing task. 67 Chapter 5. Evaluation of results Figure 5.8: Timing issue bug Seen in figure 5.8 is the aforementioned bug. From the red cursor onwards, eAccelerator never returns instruction results, and simulation times-out. 5.4.4 Bioinformatic instructions ISS implementation bug The golden reference model of bioinformatics instruction was already implemented before this thesis began. Although this was the case, it was not merged to the main branch, and became a feature left aside. When this project started, the generated constrained random test were used to find uncovered corner cases. In one of such executions, an unexpected never-ending simulation was found with instruction vbpcnt.vv. The problem ended up being an unpredicted integer overflow that made the execution of the instruction in spike run practically forever. The bug was solved, and random test continued to be used to find more corner cases in the design. 5.4.5 Illegal CSR operations not signaled CSR instructions that do not try to modify user read-only CSR execute gracefully. However, when we try to force an illegal instruction exception by generating faulty CSR accesses, we find a mismatch between the DUT and the ISS, stating that spike detected and illegal instruction (correct) that the VPU did not signal. This bug has been placed in an issue, pending to be revised by someone from implementation. 5.4.6 Implementation errors of vfcvt.f.f.v instruction When exercising the model with our newly added vfcvt instructions, a mismatch between the VPU and ISS was found. First, the ISS was misinterpreting the instruction itself as illegal. This is because checks were being done on the vs1 register to avoid overlap with vd destination register. As explained in implementation sections of this instruction, vs1 register encoding space of this specific instruction was instead used to embed type of conversion. 68 Chapter 5. Evaluation of results After promptly solving this issue, the mismatch did not disappear. When looking deeper into it, we found that the implementation of the conversion in spike and in the DUT did not coincide. We later got hints that the implementation of the VPU was incorrect. As this is something that was found at late stages of the project, such issue is yet to be solved. This is why vfcvt.f.f.v instruction is not included into the test results when looking at coverage specifically. An issue has been opened to further analyse this issue. 69 Chapter 6 Conclusions and future work This project has covered all verification efforts done to eProcessor’s VPU, eAccelerator. We successfully included custom verification components for the verification environment to test memory coherence transactions. The test-bench was also adapted to implement issuing CSR instructions for the sake of testing operations on mapped registers inside the VPU. We can also say we implemented custom instructions into the random test generator of the environment, all without the corresponding software stack, which helps uncover yet more bugs by generating corner cases in a controlled repeatable way. All of this was then used to collect coverage, seeing what parts of the main module were directly affected by the integration of these changes. The creation of custom verification components to stimulate untested sub-interfaces helped uncover a series of bugs that would otherwise have gone right under the radar. Even if future projects do not need a coherence sub-interface, many others will have custom solutions that require the constant stimulus of "garbage" sequences. Being able to implement our custom instructions independently of software teams doings is something very valuable too. Depending on such software stack to enable testing is very limiting. With this, independence is gained on that front. On the back-side, if this project re-started today and we could do things differently, we would probably consider pushing earlier on finding coverage metrics. The integration of all features on the main branches of the repository has not been possible in most cases. Including all of them earlier in the project could have given us an evolution of coverage over time. Giving time to pipelines to incrementally show the progress in coverage could have been a very valuable metric, and one that we did not take advantage of early enough. After everything, automating verification of designs is the most important task that a verification engineer develops. Having a self-sufficient system that is able to tell how much a design has been exercised is the ultimate goal. However, tests showed in section 5.3 indicate that when such inclusion is performed, our coverage metrics will increase promptly. To conclude, the development of this project was strategically planned at the initial stages with a wide-scope of possibilities for easy branching and adaptability. Such planning has proved useful, ending with a more than satisfactory result. All planned tasks have been executed in a way or another, and that is to be acknowledged. 70 6.1 Future work As mentioned in the introduction, eProcessor project will cease operations on March 31st, 2025. This means that efforts made in this thesis will not only go towards the project itself, but also as a contribution to future projects branching from it. The idea is to be able to recycle any and every useful work that this project has brought up on the table. With that said, the architecture of eProcessor will serve as inspiration for future projects that may try to replicate a VPU with direct L2 cache access. This is a somewhat different architecture, which promises great advantages over other co-processing paradigms. •Solving all found bugs. Working hand-in-hand with all teams involved in this project is necessary to solve bugs that appeared while developing this thesis. Making sure that standards coincide is a perfect opportunity to improve on specifications and documentation, which this project critically needs. •Adding test generation into CI pipelines. As said earlier in this chapter, integrating all work done into the project’s pipelines will help us quantify how much has this project really affected our design verification status, allowing RTL team developers to create new features, as well as easily checking whether fixing others may break already existing mechanisms. •Extending coverage of internal modules. This thesis has been centered a lot around extending instruction-level coverage. However, developing new coverage modules to assert correct functionality of eAccelerator’s internal components is desired. An example of a relevant uncovered module is the Load Management Unit, or LMU: a module that processes different load instructions, like strided loads. Placing new coverage inside this component would help us ensure greater understanding of the VPU’s level of stimulus. •Adding CSR instructions to coverage. Adding these instructions to ISA coverage would promote the completion of instruction-level coverage, adding every possible instruction that can be issued to the VPU. Albeit not crucial, this would help recreate another real-life scenario that eAccelerator finds itself into when integrated with the real eProcessor. •Verification of other projects. There exists other projects in similar situations as eProcessor, where the software stack limits the quantity of testing that DV team is able to do. With the know-how to implement these custom instructions, the implementation can be replicated in those other projects. Implementing mechanisms similar to memop is also valuable for those tasks, as mentioned at the start of this chapter. Garbage sequences are yet another way to direct stimulus to desired parts of the design. 71 References [1] Lluc Alvarez et al. “eProcessor: European, Extendable, Energy-Efficient, ExtremeScale, Extensible, Processor Ecosystem”. en. In: Proceedings of the 20th ACM International Conference on Computing Frontiers. Bologna Italy: ACM, May 2023, pp. 309–314. isbn: 9798400701405. doi:10.1145/3587135.3592178.url: https://dl.acm.org/doi/10.1145/3587135.3592178 (visited on 09/24/2024). [2] AMBA®CHI Architecture Specification. English. Mar. 2024. url:https:// developer.arm.com/documentation/ihi0050/latest/. [3] CHIPS Alliance. en-us. url:https://chipsalliance.org/ (visited on 01/09/2025). [4] Emilio G. Cota et al. “An analysis of accelerator coupling in heterogeneous architectures”. In: 2015 52nd ACM/EDAC/IEEE Design Automation Conference (DAC). ISSN: 0738-100X. June 2015, pp. 1–6. doi:10.1145/2744769. 2744794.url:https://ieeexplore.ieee.org/abstract/document/7167228 (visited on 12/08/2024). [5] Coverage Cookbook | Cookbook | Siemens Verification Academy. en-us. Dec. 2013. url:https://verificationacademy.com/cookbook/coverage/ (visited on 09/25/2024). [6] TJ Devries. Neovim.url:https://neovim.io/ (visited on 10/01/2024). [7] A Ejaz and I Sourdis. “FastTrackNoC: A DDR NoC with FastTrack Router Datapaths”. en. In: Technical Report (). [8] Energía en edificios de Oficinas. es. June 2015. url:https://www.enectiva.cz/ es/blog/2015/06/ideas-energia-edificio-de-oficinas/ (visited on 10/09/2024). [9] EPI SGA1: Specific Grant Agreement 1 of the European Processor Initiative (EPI). en. url:https://www.bsc.es/research-and-development/projects/ epi - sga1 - specificgrantagreement - 1 - the - europeanprocessor (visited on 09/25/2024). [10] eProcessor. en-US. url:https://eprocessor.eu/ (visited on 09/25/2024). [11] eProcessor: European, extendable, energy-efficient, energetic, embedded, extensible, Processor Ecosystem. en. url:https : / / www . bsc . es / research - and - development / projects / eprocessor - european - extendable - energy - efficient - energetic-embedded (visited on 09/25/2024). [12] Infoadmin-infoadmin [at] rediris [dot] es. RedIRIS - Welcome to RedIRIS. en. Publisher: RedIRIS. Spain. Oct. 2024. url:https://www.rediris.es/ (visited on 10/08/2024). [13] EuroHPC JU. en. Sept. 2024. url:https://eurohpc-ju.europa.eu/index_en (visited on 09/25/2024). [14] Michael Flynn. “Flynn’s Taxonomy”. en. In: Encyclopedia of Parallel Computing. Ed. by David Padua. Boston, MA: Springer US, 2011, pp. 689–697. isbn: 978-0-387-09766-4. doi:10.1007/978-0-387-09766-4_2.url:https: //doi.org/10.1007/978-0-387-09766-4_2 (visited on 09/26/2024). 72 [15] Formal Verification - an overview | ScienceDirect Topics.url:https://www. sciencedirect.com/topics/computerscience/formalverification (visited on 09/25/2024). [16] Functional Verification. en. url:https://www.cadence.com/en_US/home/ explore/functional-verification.html (visited on 09/25/2024). [17] History – RISC-V International. en-US. url:https : / / riscv . org / about / history/ (visited on 12/03/2024). [18] “IEEE Standard for Universal Verification Methodology Language Reference Manual”. In: IEEE Std 1800.2-2020 (Revision of IEEE Std 1800.2-2017) (Sept. 2020). Conference Name: IEEE Std 1800.2-2020 (Revision of IEEE Std 1800.22017), pp. 1–458. doi:10 . 1109 / IEEESTD . 2020 . 9195920.url:https : / / ieeexplore.ieee.org/document/9195920 (visited on 01/11/2025). [19] Learn the architecture - An introduction to AMBA AXI.url:https://developer. arm.com/documentation/102202/0300/Channel-transfers-and-transactions (visited on 12/19/2024). [20] Paulius Micikevicius et al. FP8 Formats for Deep Learning. en. arXiv:2209.05433 [cs]. Sept. 2022. doi:10.48550/arXiv.2209.05433.url:http://arxiv.org/abs/ 2209.05433 (visited on 01/08/2025). [21] Paulius Micikevicius et al. Mixed Precision Training. en. arXiv:1710.03740 [cs]. Feb. 2018. doi:10.48550/arXiv.1710.03740.url:http://arxiv.org/abs/1710. 03740 (visited on 01/08/2025). [22] Francesco Minervini et al. “Vitruvius+: An Area-Efficient RISC-V Decoupled Vector Coprocessor for High Performance Computing Applications”. In: ACM Trans. Archit. Code Optim. 20.2 (Mar. 2023), 28:1–28:25. issn: 1544-3566. doi:10.1145/3575861.url:https://dl.acm.org/doi/10.1145/3575861 (visited on 01/11/2025). [23] Moore’s Law. en. url:https : / / www . intel . com / content / www / us / en / newsroom/resources/moores-law.html (visited on 09/25/2024). [24] oficinesilocals.amb.cat. Evolución del mercado inmobiliario de oficinas i locales - AMB. es. url:https://oficinesilocals.amb.cat/ (visited on 10/09/2024). [25] OMIE | Publicaciones.url:https://www.omie.es/es/publicaciones/informeanual (visited on 10/09/2024). [26] Questa Advanced Simulator | Siemens Software.url:https://eda.sw.siemens. com/en-US/ic/questa/simulation/advanced-simulator/ (visited on 10/01/2024). [27] RISC-V "V" Vector Extension Working Draft. en. url:https://github.com/ riscv/riscv-v-spec/blob/master/v-spec.adoc (visited on 09/25/2024). [28] Richard M. Russell. “The CRAY-1 computer system”. In: Commun. ACM 21.1 (Jan. 1978), pp. 63–72. issn: 0001-0782. doi:10.1145/359327.359336.url: https://dl.acm.org/doi/10.1145/359327.359336 (visited on 01/11/2025). [29] BarD Software s.r.o. GanttProject: free project management tool for Windows, macOS and Linux. en. url:https://www.ganttproject.biz (visited on 09/30/2024). [30] Salary: Verification Engineer in Spain 2024. en. Feb. 2024. url:https://www. glassdoor.com/Salaries/verificationengineersalarySRCH_KO0, 21.htm (visited on 10/08/2024). [31] Spike, a RISC-V ISA Simulator. original-date: 2011-08-26T20:00:24Z. Sept. 2024. url:https://github.com/riscv-software-src/riscv-isa-sim (visited on 09/25/2024). 73 [32] Sueldo: Rtl Design Engineer en España 2024. es. url:https://www.glassdoor. es / Sueldos / rtl - design - engineer - sueldo - SRCH _ KO0 , 19 . htm (visited on 10/08/2024). [33] Sueldo: Senior Project Manager en España 2024. es. url:https : / / www . glassdoor.es/Sueldos/senior-project-manager-sueldo-SRCH_KO0,22.htm (visited on 10/08/2024). [34] SystemVerilog DPI Tutorial.url:https : / / www . doulos . com / knowhow / systemverilog/ systemverilog - tutorials / systemverilogdpi - tutorial/ (visited on 10/02/2024). [35] Tarifas de Fibra óptica para empresas - Telefónica. es. url:https://www. telefonicaempresas.es/internet-empresas (visited on 10/09/2024). [36] TeX Live - TeX Users Group.url:https : // tug. org /texlive/ (visited on 09/30/2024). [37] The RISC-V Instruction Set Manual: Volume II - Privileged Architecture.url: https://drive.google.com/file/d/17GeetSnT5wW3xNuAHI95-SI1gPGd5sJ_ /view?usp=drive_link&usp=embed_facebook (visited on 09/25/2024). [38] The RISC-V Instruction Set Manual: Volume II - Unprivileged Architecture. url:https://drive.google.com/file/d/1uviu1nH-tScFfgrovvFCrj7Omv8tFtkp/ view?usp=drive_link&usp=embed_facebook (visited on 09/25/2024). [39] Using FP8 with Transformer Engine — Transformer Engine 1.13.0 documentation.url:https://docs.nvidia.com/deeplearning/transformer-engine/userguide/examples/fp8_primer.html (visited on 01/08/2025). [40] UVM Cookbook | Cookbook | Siemens Verification Academy. en-us. Sept. 2018. url:https://verificationacademy.com/cookbook/uvm-universal-verificationmethodology/ (visited on 09/24/2024). [41] Verification Planning. en. url:https://www.cadence.com/en_US/home/ explore/verification-planning.html (visited on 09/24/2024). [42] vlsiverify. UVM Phases. en-us. May 2021. url:https://vlsiverify.com/uvm/ uvm-phases/ (visited on 12/18/2024). [43] Zoom | One platform to connect. en. url:https://www.zoom.com (visited on 09/30/2024). [44] Zotero | Your personal research assistant.url:https ://www. zotero.org/ (visited on 09/30/2024). 74 Appendix A Context and scope A.1 Context A.1.1 Introduction This project is part of my Computer informatics thesis. Through an academic agreement with Universitat Politècnica de Catalunya (UPC), I have been working as an intern at Barcelona Supercomputing Center - Centre Nacional de Supercomputació (BSC-CNS). Here, I was introduced to the hardware design and verification workflow, and was given an opportunity to gain hands-on experience. The eProcessor initiative has been one of the latest projects to which I have been included, the verification of which is the basis of this thesis. A.1.2 The task The task of this project is to extend and develop an already-existing verification environment for eAccelerator; a RISC-V vector accelerator developed in the context of the eProcessor project. Make sure all the remaining interfaces and sub-interfaces work properly, and set forth a quality verification plan for verification efforts to come. Extend coverage efforts, and work towards a full-fledged test bench where tests can be run with confidence and efficiency, making proper use of the tools at hand. Past development of this project has been steady in time, but many modules are still pending to be thoroughly verified, and their functionality to be correctly checked. The efforts of this thesis are directed at accomplishing as much of that as possible. A.1.3 Main concepts This section aims to provide much needed context to understand everything that is talked about in this document. However, the needs of the project at its closing stages benefit from having those explanations closer to the beginning. Since appendixes A, B and C were part of a stand-alone document, maintaining this structure was necessary. However, they have now been relocated to chapter 2, where they are extended and completed, in order to avoid repetition and craft a nice introduction for the reader. 75 Chapter B. Planning •VP2 - Extend and prune verification plan Modify the shared document, or create a new one to describe the verification plan to be followed. Revise with the eProcessor validation team, to clear any corner cases and put forth a quality document. B.1.2.2 Test bench extension and development There are two main parts that conform this section: development of vector memory operation interface support and verification of CSR accesses and interactions. •TB1 - Read documentation about the memop interface Understand the necessity of memory operation signalling, and comprehend the protocol followed by the EPA interface. •TB2 - Create mechanism to overlap memory operations Combining scalar and vector memory operations, trigger the mechanism at will (randomly or directed). •TB3 - Add parameters to test bench configuration Parametrize mechanism created in TB2, to enable reconfiguration. •TB4 - Shuffle parameters, and test multiple configurations Report results on all possible timing cases, and compare with specification. •TB5 - Read documentation about CSR module Understand the handling of CSR registers inside the eAccelerator, and how to interact with them. •TB6 - Mimic context switch behaviour with CSR interactions Test CSR interactions inside of eAccelerator. •TB7 - Report results Report TB4 and TB6 results. B.1.2.3 Test generation and running Identifier: TG# •TG1 - Gather context on current test generation status Gather context on current test generation status for the eAccelerator test bench. •TG2 - Gather context on current test golden reference model status ISS (Instruction Set Simulator) Spike[31] details are to be studied. Understand how to co-simulation mechanism enables the test bench to reliably determine correctness, and how the connection through SystemVerilog Direct Programming InterfaceDPI-C[34] works. •TG3 - Make ISS support custom instructions Implement the subset of custom instructions defined in VP2 into the ISS. This will enable the test bench to check correct functionality of such instructions in the design. •TG4 - Make test generator work with custom instructions Implement the subset of custom instructions supported on the ISS in TG3. •TG5 - Generate and run tests Use the before mentioned tools to create tests with the newly implemented instructions. •TG6 - Report results Report TG* results; specially TG5 tests status and custom instructions support. 82 Chapter B. Planning B.1.2.4 Coverage extension •CV1 - Gather context on past coverage efforts Read past coverage reports and understand progress attained. Meet with people involved in such tasks and ask for details and recommendations. This task is compatible with other knowledge gathering tasks (such as VP1, TB1 and TG1). •CV2 - Attain further detail on module specifications Meet with RTL implementation team and request detailed specifications of every unspecified module. Ask for guidance and modules where coverage would be necessary or useful. •CV3 - Define coverage points and add assertions Take information extracted from CV2, and define cover points, assertions, and other rules to prepare functional coverage extension. •CV4 - Collect and analyse coverage data Collect coverage data, and extract conclusions. CV3 and CV4 are susceptible to be cycled multiple times, if coverage results suggest so. B.1.3 Project documentation Group of tasks to document everything the project produces. Produces the final report of the project. •PD1 - Collection of events Intermittent task performed at the end of every task subgroup. Annotate results non-formally to accumulate them in a centralised document. •PD2 - Redact final documentation Gather all annotations and create final document. Create conclusions, and polish details. •PD3 - Prepare presentation Ensemble a presentation, and practice public speech presenting the project’s main points. B.2 Resources Resources are an essential part of a project’s success. Correct resource management can lead the way to a success. Lack thereof will condemn any project to desperation and unnecessary stress. Resources used for this project are neatly subdivided into Human resources (HR), Hardware resources (HW) and Software resources (SW). Group ID Description HR DEV Main project developer. Part of the DV team. HR DV Design Verification Team working on eProcessor. HR PM Project manager (thesis supervisor). HR RTL Register Transfer Level development team. HW LT Laptop for developing the project; Dell Latitude 7440 with 32GB and Intel i7-1365U. HW HP Headphones used to attend online meetings. SW NV NeoVim[6]; Text editor used to modify code and documents. SW GP Tool to create Gantt diagrams; GantProject[29]. SW ZT Used to collect and neatly store bibliography entries; Zotero[44]. SW ZO Videoconferencing tool to carry out meetings; Zoom [43]. SW QS Siemens tool to run RTL simulations; QuestaSim [26]. SW TX latex distribution to locally compile documents; TeX Live [36]. Table B.1: Resources used by the project 83 Chapter B. Planning B.3 Estimates and the Gantt ID Name Time(h) Predecessors Resources Human Hardware Software Project management 115 PM1 Context and scope 30 DEV, PM LT NV, TX, ZT PM2 Project planning 30 DEV, PM LT NV, TX, ZT, GP PM3 Budget and sustainability 20 DEV, PM LT NV, TX, ZT PM4 Final Document 5 PM1, PM2, PM3 DEV, PM LT NV, TX, ZT PM5 Meetings, sync and planning 30 DEV, PM, DV, RTL LT, HP ZO Project development 525 Verification plan 60 VP1 Study existing verification plan 20 DEV, DV LT NV VP2 Extend and prune verification 40 VP1 DEV, DV LT NV Test bench extension 210 TB1 Study documentation on memop interface. 30 DEV, RTL LT NV TB2 Create memop overlap mechanism 50 TB1 DEV LT NV, QS TB3 Parametrize mechanism 20 TB2 DEV LT NV, QS TB4 Test configurations 15 TB3 DEV LT NV, QS TB5 Study documentation on CSR module 30 CV2 DEV, RTL LT NV TB6 Mimic context switch CSR interactions 50 TB5 DEV LT NV, QS TB7 Report results 15 TB4, TB6 DEV LT NV, TX Test generation and running 172 TG1 Study current test generation status 30 DEV LT NV TG2 Study current ISS status 30 DEV LT NV TG3 Make ISS support custom instructions 50 TG2 DEV LT NV, QS TG4 Make test generator work with custom instructions 50 TG1, TG3 DEV LT NV, QS TG5 Generate and run tests 30 TG4 DEV LT NV, QS TG6 Report results 10 TG3, TG5 DEV LT NV, TX Coverage extension 83 CV1 Study past coverage efforts 30 DEV LT NV CV2 Attain further detail on module specifications 8 CV1 DEV, RTL LT ZO CV3 Define coverage points and add assertions 30 CV2 DEV LT NV, QS CV4 Collect and analyse coverage data 15 CV3 DEV LT NV, QS Project documentation 140 PD1 Collect events 30 DEV LT NV PD2 Redact final documentation 80 PD1 DEV LT NV, TX, ZT PD3 Prepare final presentation 30 PD2 DEV, PM LT NV, TX, ZT Total 780 Table B.2: Time estimations per task Figure B.1: Gantt chart 84 Chapter B. Planning B.4 Risk management: alternative plans and obstacles From September 16th, 2024 until October 2nd, 2024, there was an outage of the software resource SW-QS. This problem is solved now, but affected all tasks that depended on the tool. To be exact, task TB3 was affected and could not progress. This lack of licenses caused other HM-DV sub-teams to put efforts into trying to run the projects with other similar tools, but no luck was found in most of them. Efforts were put into that too, but got nowhere. This led to the start of two other simultaneous tasks: VP2 and CV1 where commenced before initial planning, in order to support such delay. This whole situation altered the project duration by around 60 hours, which are accounted in the final hours sum. Other risks to take into account are hardware related ones. In case something happened to HW-LT, the helpdesk department inside BSC-CNS would substitute the equipment, but machine setup time should be taken into account. The effects of this casuistic are much lower than the licenses one, but should be considered nonetheless. About half a work day, or 4 hours of re-setup such be needed to fix such a failure. Tasks like documenting are much less resource-needy, and could be done from another functioning computer, without the need of such set-up time. Human resources are prone to fail too. Although all tasks are mainly dependent on HM-DEV;HM-DV and HM-RTL are needed for information gathering. Communication is never perfect, and always causes some kind of delay. Although minor, time allocated for information requests are elongated with miscommunication situations in mind. Since this thesis has 4 independent working "threads", jumping between them would help support such time delays. 85 Appendix C Budget and Sustainability C.1 Budget C.1.1 Identification of costs To identify the costs of this project, we must re-visit resources mentioned in past deliverables. C.1.1.1 Human resources Human resources identified for task identification were created with work role in mind. In reality, these roles do not correspond with actual position or salary. For this reason, to identify human resources costs, some different roles are identified, and properly mapped to task identification roles. •Verification engineer This role can be mapped to HR-DEV or HR-DV. It takes part in all tasks of the project at some level or another. •Project manager This role is mapped to HR-PM. It takes part on all project management tasks that involve any kind of planning and reunion (PM5 and CV2). •RTL design engineer This role is mapped to HR-RTL, and takes responsibility for all meeting and synchronization tasks (PM5 and CV2) C.1.1.2 General costs In all other cases, resources previously identified do map directly to their identifier (HW-* and SW-*). Hardware resources all carry some kind on cost. Software resources do not; most of the tools used are free and open source software (FOSS) or offer some kind of free tier service. For other costs, electricity, workspace and internet service are considered. C.1.1.3 Contingency and incidentals Task estimates are generous both in time and resources. This means that some kind of cushion exists for small unforeseen events. Nonetheless, considering a contingency margin is important to further ensure the correct development of the project. A 10% contingency margin is defined for such undesired cases. 86 Chapter C. Budget and Sustainability C.1.2 Cost estimates C.1.2.1 Human resources cost In the case of human resources, HR-DV and HR-RTL are teams of people. For this reason, not every role on the following list corresponds exactly to one role mentioned on previous deliverables. For the salaries per hour approximations, 223 yearly working days are assumed (this is because BSC-CNS is located in Barcelona, with 246 working days, and 23 paid vacation days), and a 7:30h daily schedule is used. This gives us a total of 1672.5 yearly working hours. This number will be used for the rest of the project budget estimations. For getting the hourly gross salary + Social Security (SS) tax, the advised 35% increase is applied. Role Annual gross salary (€) Gross salary/hour (€) Gross salary/hour + SS (€) Total hours project Role total cost Verification engineer[30] 35000 20.93 28.26 780 22042.8 Project manager[33] 48000 28.70 38.75 38 1472.5 RTL design engineer[32] 35000 20.93 28.26 38 1073.88 Table C.1: Cost of human resources C.1.2.2 Hardware resources cost To calculate amortization cost we take the proportional part of the cost using the fraction of used hours over total hours. This, will give us the approximate "useful cost" we have taken out of the product during the project, or how much of its retail value has been used. It is worth to emphasize this is an approximation. ID Item Total retail price (€) Useful life (years) Amortization cost of the project LT Dell Latitude 7440 with 32GB and Intel i7-1365U 1500 5 139.91 HP EPOS PC 5 Chat 30 3 0.23 Table C.2: Cost of hardware resources C.1.2.3 Software resources cost Most of the tools used are free and open source software (FOSS), or a free license is provided indefinetly. For the sake of completeness, all listed software in B.1 will be included in the budget calculations. The only paid software, QuestaSim advanced simulator, is only used on some development tasks. To be exact, TB2 to TB4, TB6, TG3 to TG5, CV3 and CV4. All of these rack up 310 hours, which are used to compute the amortization cost of this tool in the project. ID Item Total retail price (€) Useful life (years) Amortization cost of the project (€) NV Neovim 0.00 ∞0.00 GP GanttProject 0.00 ∞0.00 ZT Zotero 0.00 ∞0.00 ZO Zoom 0.00 ∞0.00 QS QuestaSim Advanced Simulator 1.794,58 1 332.63 TX TeX Live 0.00 ∞0.00 Table C.3: Cost of software resources 87 Chapter C. Budget and Sustainability C.1.2.4 Other resources cost Other costs include electricity, workspace and internet access. All of these are paid on a monthly basis, so the duration of the project has a direct effect on the final amount that these resources contribute to general costs. Electricity prices are the 2023 average in Spain [25], and the amount consumed is an approximation seen in [8]. For workspace cost, the official data for average price per square meter in Barcelona between August and September is used[24]. Internet access service cost is calculated considering a symmetric 1Gb connection plan for businesses in Barcelona[35]. Resource Cost calculation Cost(€) Electricity 110.6kW h year×m2×10m2×0.10020 e kW h ×1 12 year months ×6months 55.41 Workspace 16.45 e m2×month ×10m2×6months 987 Internet 38.80 e month ×6months 232.8 Table C.4: Cost of other resources C.1.2.5 Total cost estimations Activity Amount (€) Observations PM1 - Context and scope 847.80 Verification engineer, 30 hours PM2 - Project planning 847.80 Verification engineer, 30 hours PM3 - Budget and sustainability 565.20 Verification engineer, 20 hours PM4 - Final document 141.30 Verification engineer, 5 hours PM5 - Meetings, sync and planning 2858.10 RTL engineer, Verification engineer, Project manager, 30 hours VP1 - Study existing verification plan 565.20 Verification engineer, 20 hours VP2 - Extend and prune verification plan 1130.40 Verification engineer, 40 hours TB1 - Study documentation on memop interface 847.40 Verification engineer, 30 hours TB2 - Create memop overlap mechanism 1413.00 Verification engineer, 50 hours TB3 - Parametrize mechanism 565.20 Verification engineer, 20 hours TB4 - Test configurations 423.90 Verification engineer, 15 hours TB5 - Study documentation on CSR module 847.80 Verification engineer, 30 hours TB6 - Mimic context switch CSR interactions 1413.00 Verification engineer, 50 hours TB7 - Report results 423.90 Verification engineer, 15 hours TG1 - Study current test generation status 847.80 Verification engineer, 30 hours TG2 - Study current ISS status 847.80 Verification engineer, 30 hours TG3 - Make ISS support custom instructions 1413.00 Verification engineer, 50 hours TG4 - Make test generator work with custom instructions 1413.00 Verification engineer, 50 hours TG5 - Generate and run tests 847.80 Verification engineer, 30 hours TG6 - Report results 282.60 Verification engineer, 10 hours CV1 - Study past coverage efforts 847.80 Verification engineer, 30 hours CV2 - Attain further detail on module specifications 762.16 RTL engineer, Verification engineer, Project manager; 8 hours CV3 - Define coverage points and add assertions 847.80 Verification engineer, 30 hours CV4 - Collect and analyse coverage data 423.90 Verification engineer, 15 hours PD1 - Collect events 847.80 Verification engineer, 30 hours PD2 - Redact final documentation 2260.80 Verification engineer, 80 hours PD3 - Prepare final presentation 847.80 Verification engineer, 30 hours Total CPA (Costs per activity) 24589.18 HW-LT - Laptop 139.91 Calculated in table C.2 HW-HP - Headphones 0.23 Calculated in table C.2 SW-TX - TeX Live 0.00 Free to use SW-NV - NeoVim 0.00 Free to use SW-GP - GanttProject 0.00 Free to use SW-ZT - Zotero 0.00 Free to use SW-ZO - Zoom 0.00 Free tier used, or adhered to rediris[12] SW-QS - QuestaSim 332.63 Calculated in table C.3 SW-TX - TeX Live 0.00 Free to use Electricity 55.41 Calculated in table C.4 Workspace 987.00 Calculated in table C.4 Internet 232.80 Calculated in table C.4 Total CG (General Costs) 1747.75 Total Costs (Total CPA + Total CG) 26336.93 Contingency 2633.69 Contingency margin = 10%. Reasoning in C.1.1.3 Total DC+IC+Contingency 28970.62 SW-QS licenses outage 423.90 Verification engineer, 60 hours. Risk 25% Teamwork and time management 70.65 Verification engineer, 10 hours. Risk 25% Lack of theoretical knowledge 141.30 Verification engineer, 20 hours. Risk 25% Total incidentals 635.85 TOTAL 29606.47 Table C.5: Budget structure 88 Chapter C. Budget and Sustainability C.1.3 Management control In order to control all the possible budget deviations, indicators are described to calculate such offsets. The basic idea is calculating the difference between real resources cost consumed minus estimated resources planed, and quantify error. C.1.3.1 Direct costs Possible causes of budget deviations in Direct Costs run on 2 possible scenarios: CPA and GC. For CPA, a salary miscalculation, or a temporal sub-estimation of a task, can lead to a total budget overrun. To report these 2 figures, the following calculations can be used. For a budget increase caused by salary miscalculation. (RealCostP erHour −EstimatedCostP erHour)×P ositionRoleHours For a budget increase caused by time sub-estimation of tasks: X i∈ctsk ((RealT askT imei−EstimatedT askT imei)×P ositionRoleP riceP erHouri) Where ctsk is the set of completed tasks. C.1.3.2 Indirect costs Hardware, Software or Other costs can directly affect IC variance. In case of hardware, useful life estimation deviations can increase the total budget by the following amount: Price ×HoursUsed RealUsefulLife ×1672.5−EstimatedCost Where 1672.5 stands for working hours considered per year, because RealUsefulLife is expressed in whole years. Software resources deviations are not considered, since overtime on direct costs are already considered in Direct Costs, and the formula is exactly the same. C.1.3.3 Contingency and incidentals Contingency overhead considered is relatively low, since such caution for the project is already been accounted for in the general approximations. A 10% increase on cost is easily surpassed by major tool outages, or human resources overtimes. The formula to calculate whether contingency costs have been surpassed is the following: (DCest +ICest −DCact −ICact)×1 + ContingencyP ercent 100  Where act stands for actual real IC and DC costs, and est stands for estimated costs at the planning phase. If the formula returns a number > 0, contingency limit has been surpassed. 89 Chapter C. Budget and Sustainability C.1.3.4 Total costs Total costs estimations are the sum of all cost variations considered before, also taking incidentals into account. Incidentals in this project suppose an increase in hours spent on a specific task. Depending on the role, the incidental will suppose a different budget variation increase. To calculate the total variation: varDC +varIC +varcontingency +varincidentals Where var stands for variability in the cost specified by the subscript. These control mechanisms will be applied both on a regular and sporadic manner. The former (about once a month) will act as an alarm whenever any of the indicators goes over the limit. The latter will serve as a tool to quantify the status of the project whenever HR-PM sees it useful. C.2 Sustainability report C.3 Initial sustainability report For all IT projects to thrive, a sustainability report is most definitely necessary. This thesis aims nothing short of such excellence, and provides these self-reflections in following subsections. The report aims to unveil the level on knowledge and care that has been put into 3 separate points of view. •PPP, or Project Put into Production, is the point of view that includes the planning, development and implementation of the project. •Exploitation refers to the useful life of the project, and the post-development indicators. •Risks inherent to the project during the entirety of its life cycle (development, useful life and dismantling). To further illustrate all different points, the following matrix was supplied by GEP tutors: Figure C.1: Sustainability Matrix for the bachelor’s thesis. Provided by GEP tutors. And a survey was prompted to be answered in order to facilitate this level of introspection. After submitting my response to build some self-awareness, the following questions are to be answered regarding the sustainability of the project at planning stage. 90 Chapter C. Budget and Sustainability C.3.1 Environmental dimension Regarding PPP: Have you estimated the environmental impact of undertaking the project? Have you considered how to minimise the impact, for example by reusing resources? There is a massive amount of factors that should be taken into consideration to make an even slightly educated estimation on the environmental impact of this project. However, knowing that the use of resources, in most cases, is not material; we may be able to narrow down environmental impact factors to resources consumed by the developers themselves. Reducing that should suffice to greatly lower the environmental impact of this project, but trading off comfort. Regarding Exploitation: How is the problem that you wish to address resolved currently (state of the art)? In what ways will your solution environmentally improve existing solutions? The set of tools and techniques to correctly verify all the components of this project are nothing new, and many functional test benches exist, but all of them need specific modifications to grant correct verification of the design under testing (DUT). This thesis will build on top of existing knowledge, and will try to adapt existing solutions to specific problems of this project. This way, currently existing solutions will benefit from further exploration of common (or not so common) problems that may arise during the development of this project. This efforts will surely help save resources when developing future hardware. C.3.2 Economic dimension Regarding PPP: Have you estimated the cost of undertaking the project (human and material resources)? Yes, and we are confident in our knowledge on the project’s economic impact and cost. The estimations are made taking human and material resources into account. Final calculations can be checked out in table C.5. Regarding Exploitation: How is the problem that you wish to address resolved currently (state of the art)? In what ways will your solution economically improve existing solutions? In economical terms, the solutions proposed in this thesis may aid future developers of EuroHPC JU projects to sooner understand common mishaps that will incur during verification stages. This will save money, and help redirect funds to other needier tasks. C.3.3 Social dimension Regarding PPP: What do you think undertaking the project has contributed to you personally? Talking from a Undergraduate Bachelors degree student perspective, the undertaking of this project helped me massively understand the tool-chains and workflows of nowadays hardware development processes. The deepening in understanding on topics seen during my studies, and the fulfilling of undiscovered lagoons of knowledge make this project very attractive to me. It has shifted my perspective in many things regarding project organization too, and that is really valuable. Regarding Exploitation: How is the problem that you wish to address 91