Observability: A Practical Primer for RSEs
Abstract
When you distribute and run your software, do you have a good idea of how it's functioning "in the wild"? Do your users have to manually report back to you when they find a bug - and do you have access to the software state when it occurred? How do you know if your application has enough resources when running on a cluster or the cloud, and at what rate it's processing datasets/jobs?These questions are some of the motivating use cases underpinning observability, which can broadly be defined as using event-driven data (telemetry) to gain insights into the functioning of both software and infrastructure. Observability is useful for troubleshooting issues on production systems, monitoring performance and security, and for tracking key metrics and events happening within a system.In this introductory walkthrough, I will introduce observability in the context most often used by RSEs - running software in "batch" mode (e.g. running on a cluster) or "server" mode (long-running applications on cloud or workstations). The three main telemetry types will be introduced - metrics, logs, traces - and I will demonstrate how to capture metrics from Python applications in "batch" mode using Microbench (pypi.org/project/microbench) and in "server" mode using OpenTelemetry (opentelemetry.io) and Prometheus (prometheus.io). I will share resources for capturing logs and traces (for later study, in the interests of time). The emphasis will be on simple, practical examples that RSEs can adopt into their own software with minimal fuss.A recording of this session is available on YouTube: https://youtu.be/-FtIMrmHApg
Full text
Observability: A Practical Primer for RSEs RSECon25 Walkthrough Alex Lubbock Rosalind Franklin Institute git clone https://github.com /rosalindfranklininstitute /rsecon25-observability-primer v1.0rc1 (2025-08-22) 1
Learning Objectives By the end of this walkthrough, you should: Understand what observability is and why it matters Know where observability can and should be applied See how to add observability into your own Python code Gain hands-on practice with simple use cases v1.0rc1 (2025-08-22) 2
Audience & Scope Targeted at Research Software Engineers (RSEs) and related roles Focus: workloads in batch/HPC and cloud/server contexts We will use Python, but the principles generalise Introductory level v1.0rc1 (2025-08-22) 3
What is Observability? Definition: Ability to understand the internal state of software and systems via event-driven outputs: logs, metrics, traces Use cases: Troubleshooting failures Performance monitoring & capacity planning Security & auditing Usage analytics v1.0rc1 (2025-08-22) 4
Why Observability? Observability is most valuable when: Systems are complex (many ways to fail) Workloads are remote (HPC, cloud, distributed) High reliability is required (uptime, reproducibility, security) Research computing often checks all three boxes. v1.0rc1 (2025-08-22) 5
Why Observability in Research? Reproducibility: ensure results can be tied to exact code, data, and environment Debugging: batch jobs may fail hours into execution → metadata helps diagnose Performance drift: explain why the same job took 2 hours last month, 6 hours today Collaboration: share structured context with teammates, not just “logs.txt” v1.0rc1 (2025-08-22) 6
Workloads We’ll Focus On Batch / HPC jobs (SLURM, Snakemake, Nextflow, Argo Workflows, Airflow) Cloud / long-running services (Web apps, APIs, servers) Different environments → different tools & strategies v1.0rc1 (2025-08-22) 7
Outputs Used in Observability - Logs Logs → notable events, warnings, errors Usually timestamped Examples: system start/stop, exceptions, requests *** DEBUG | 2025-08-08 10:07:03 | Server started *** INFO | 2025-08-08 10:08:37 | Request from user Alice for IMG01.TIF *** WARN | 2025-08-08 10:08:37 | Memory usage high *** ERROR | 2025-08-08 10:08:38 | Out of memory v1.0rc1 (2025-08-22) 8
Outputs Used in Observability - Metrics Metrics → (numerical) state (e.g. memory usage, throughput, uptime) requests_per_second{url="/process-image"} 12.2 requests_failed{url="/process-image"} 4.0 v1.0rc1 (2025-08-22) 9
00-0 Basic Example from microbench import MicroBench import time bench = MicroBench() @bench def slow_function(x): time.sleep(x) return x slow_function(2) v1.0rc1 (2025-08-22) 16
00-1 Extending with line profiler from microbench import MicroBench, MBLineProfiler, MBHostInfo, MBPythonVersion class MyBench(MicroBench, MBLineProfiler, MBHostInfo, MBPythonVersion): pass bench = MyBench() @bench def compute(n): squares = [i**2 for i in range(n)] return sum(squares) compute(1_000_000) v1.0rc1 (2025-08-22) 17
00-2 Telemetry capture from microbench import MicroBench class TelemetryBench(MicroBench): telemetry_interval = 1 # sample every 1s @staticmethod def telemetry(process): # For values you can capture from "process", see # https://psutil.readthedocs.io/en/latest/#psutil.Process return {'cpu_percent': process.cpu_percent()} bench = TelemetryBench() @bench def busy_work(): ... v1.0rc1 (2025-08-22) 18
Where Microbench Helps Dependency mismatches Confirms which versions of packages were active during execution Performance drift Detects slower runs, different CPU/memory usage HPC reproducibility Records environment + metadata for published results Lightweight monitoring No extra infrastructure, just decorate functions v1.0rc1 (2025-08-22) 19
What Microbench Doesn't Do Microbench captures metadata and telemetry, but it is not a replacement for: Good reproducibility practices: Version control & Git history Documentation of workflows CI/CD testing Reproducible environments (Conda, Poetry, virtualenvs) Containers (Docker/Singularity) for portability Full observability stacks (Prometheus, OpenTelemetry, etc.) Debuggers / profilers for in-depth code analysis Automated monitoring / alerting for long-running services v1.0rc1 (2025-08-22) 20
Observability: A Practical Primer for RSEs Adding Metrics to a Flask Web App v1.0rc1 (2025-08-22) 21
Intro to OpenTelemetry Open-source framework for instrumenting applications Standardises formats for metrics, logs, traces Vendor-agnostic: works with Prometheus, Jaeger, etc. Language-agnostic: works with most popular languages v1.0rc1 (2025-08-22) 22
Intro to Prometheus Open-source monitoring & alerting toolkit Pull-based metrics collection Stores metrics as time-series data Integrates with dashboards like Grafana v1.0rc1 (2025-08-22) 23
Environment Setup Docker Compose (recommended) cd 01-prometheus docker compose up -d Python cd 01-prometheus/app conda create -n prom python=3.13 -y conda activate prom pip install -r requirements.txt python app.py v1.0rc1 (2025-08-22) 24
Motivating Example: Dice Roll App Simple Flask web app Endpoint: / → rolls a dice (1-6) app = Flask(__name__) @app.route("/") def roll_dice(): # Roll the dice (random number 1-6) result = str(roll()) return result v1.0rc1 (2025-08-22) 25
Other Use Cases for Metrics Track HTTP request count & latency Monitor errors & exceptions Database query metrics Custom application events https://prometheus.io/docs/concepts/metric_types/ v1.0rc1 (2025-08-22) 32
Beyond Metrics: Logs & Traces Logs: capture detailed runtime events Traces: visualise request flows across services Complements metrics to fully understand app behavior v1.0rc1 (2025-08-22) 33
Summary Observability Metrics + Logs + Traces Reproducibility, monitoring, troubleshooting, auditing Microbench captures metadata from batch jobs OpenTelemetry standardises instrumentation Prometheus collects and visualises metrics v1.0rc1 (2025-08-22) 34
Useful References Microbench: https://github.com/alubbock/microbench OpenTelemetry Python: https://opentelemetry.io/docs/python Prometheus: https://prometheus.io/docs/introduction/overview/ Metrics & Monitoring: https://prometheus.io/docs/practices/instrumentation/ OpenTelemetry + Prometheus Exporter: https://opentelemetry.io/docs/instrumentation/python/exporters/prometheus/ Loki (log aggregation): https://grafana.com/oss/loki/ Jaeger (trace aggregation): https://www.jaegertracing.io/ Microsoft on Observability: https://microsoft.github.io/code-with-engineeringplaybook/observability/ v1.0rc1 (2025-08-22) 35
Questions? BlueSky: @AlexLubbock.com Slides and example code: https://github.com /rosalindfranklininstitute /rsecon25-observability-primer v1.0rc1 (2025-08-22) 36