How to develop and maintain 1 million lines of code in the SWMF

How to develop and maintain 1 million lines of code in the SWMF? Gabor Toth, University of Michigan NSF NASA Space Weather Modeling Framework SWMF Control & Infrastructure Eruption Generator Solar Corona Inner Heliosphere Global Magnetosphere Polar Wind Ionospheric Electrodynamics Thermosphere & Ionosphere Energetic Particles Plasmasphere 3D Outer Heliosphere Couplers Flare/CME Observations Upstream Monitors Gravity Waves F10.7 Flux Particle in Cell Particle Tracker Inner Magnetosphere BATSRUS PWOM GITM DGCPM RIM AMPS FLEKS AMPS FLEKS BATSRUS BATSRUS GL & TD Kota MFLAMPA BATSRUS BATSRUS RCM CIMI RAM-SCB HEIDI Magnetograms, tomography Radars Magnetometers F10.7 Flux SWMF is open-source at http://github.com/SWMFsoftware and also available via CCMC EXTRAS SWMF_DATA CRASH_DATA SWPCTEST SWMFSOLAR VisanaJulia VisanaMatlab 3 code summary ØSource code (active = tested): •692K lines of Fortran •330K lines of C++ • 54K lines of Perl scripts • 73K lines of Python scripts • 11K lines of Julia scripts • 58K lines of IDL scripts • 32K lines of Makefiles • 26K lines of in the wrappers and couplers • 18K lines of XML description of input parameters ØSWMF is portable: runs on any Unix/Linux/OSX/WSL system (laptop to supercomputer) with Fortran 2008, C++, Perl, Python compilers and optional MPI, OpenMP, OpenACC, AMREX, HYPRE, HDF5 libraries. ØUser manual with up-to-date documentation of input parameters. ØFully automated nightly testing with several machine/compiler combinations. ØThese tests provide working examples for running the code. Weak scaling to 28,600 cores on Frontera ideal scaling ØPhysics •Classical, semi-relativistic and Hall MHD •Multi-species, multi-fluid, 5 and 6-moment •anisotropic pressure for ions and electrons •Radiation hydrodynamics multigroup diffusion •Multi-material, non-ideal equation of state •Heat conduction, viscosity, resistivity •Alfven wave turbulence and heating ØNumerics •Parallel Block-Adaptive Tree Library (BATL) •Cartesian and generalized coordinates •Splitting the magnetic field into B0 + B1 •Divergence B control: 8-wave, CT, projection, parabolic/hyperbolic •Numerical fluxes: Godunov, Rusanov, AW, HLLE, HLLC, HLLD, Roe, DW •Explicit, local time stepping, limited time step, sub-cycling •Point-, semi-, part and fully implicit time stepping •Up to 4th order accurate in time and 5th order in space ØApplications •Heliosphere, sun, planets, moons, comets, high energy density physics experiments Ø215,000+ lines of Fortran 2008 code with MPI + OpenMP + OpenACC parallelization 4 BATS-R-US Weak scaling to 224,000 cores on Frontera Commercial Software Scientific Software developers and users are (very) different developers and users often coincide/collaborate commercial value and financial responsibility scientific value and scientific responsibility multiple use, multiple users, large market few uses, few users, no or very small market well-defined input and output not very well-defined input, unknown output security issues are important security issues are not too important written from scratch or from modern libraries mixture of new and old/legacy software Commercial and Scientific Software are Different While there are significant differences, most software engineering principles apply to both. Scientific Software for Modeling Physical system Verification: solving the equations right Mathematical model Approximations Numerical model Discretization errors Implementation Bugs Numerical Solution Round-off errors Va l i d a t i o n : solving the right equations ØAvailability and Usability: •Open-source repository: including the clearly-defined production version that code developers use •Documentation of code use: comprehensive guide for new users •Documentation of inputs and outputs: full description of input and output files, support for visualization and analysis ØReliability: •Version control: changes are documented, and previous versions are archived •Verification tests: the algorithm implementation is provably correct •Functionality tests: the code produces the expected results on multiple platforms •Validation tests: the code agrees well with the modeled system ØModifiability: •Clearly written software with sufficient comments (in English) •Modularity: code consists of a hierarchy of manageable pieces (functions, modules, classes, objects) •Coding standards: makes code more readable and provides uniformity irrespective of developer 7 Available, Usable, Reliable and Modifiable Software ØOpen-source •Will people steal / misuse the code? •Our experience is that this happens very infrequently, and benefits are larger ØDocumentation •Who has the time to document? •Our experience: overall time is saved by not having to answer repetitive user questions ØTesting •Writing tests is time consuming. If the code reproduces reality, who needs more testing? •Our experience: continuous automated testing is the only way to develop a large reliable code. ØCoding quality •Who cares if the software is ugly if it works perfectly? •Our experience: badly written code is almost always incorrect and impossible to debug/develop. 8 Beautiful Goals but … You want to do this Verification Test Lowrie test 3 for electron energy 0.001 0.010 0.100 Grid resolution 0.01 0.10 Relative error 1st order slope Godunov, cond Linde, cond Linde, radcond Lowrie’s test 3 for electron energy -0.04 -0.02 0.00 0.02 0.04 x 0 1 2 3 4 5 6 ion temperature 0.007 0.008 0.009 0.010 5.0 5.2 5.4 5.6 5.8 6.0 Thermal front Hydro shock relaxation Analytic vs. Numerical solution Grid Convergence Study Why does Formatting Matter? 17 ØThe SWMF is a mature software under continuous development •More than 1 million lines of code •About 5 changes per day •Fully open-source ØWe have limited resources to develop and maintain the code •Establishing good software engineering practices: version control, documentation, coding standards •Fully automated nightly tests provide many benefits: •Code reliability improves, stable branch is generated automatically •Early discovery of errors makes fixing easier •Documentation is kept up to date •Tests provide working examples for all typical applications on many different platforms ØEven with automation, code maintenance requires human time •So far, we have not received any dedicated support for code maintenance. •Still, the alternative is much worse: non-working unreliable software that becomes unused Summary