Stackless vs. Stackful Coroutines: A Comparative Study for RDMA-based Asynchronous Many-Task (AMT) Runtimes

Stackless vs. Stackful Coroutines: A Comparative Study for RDMA-based Asynchronous Many-Task (AMT) Runtimes Mia Reitz and Jonas Posner #[email protected] www.jonasposner.com 16th November 2025 Motivation Contributions Background Implementation Experiments Conclusion Thanks Motivation Modern supercomputers: Increasing complexity ⇒programmability challenges for MPI+X Asynchronous Many-Task (AMT) Promising abstraction; especially for irregular, dynamic workloads Load balancing via work stealing One key challenge: suspend and migrate tasks efficiently across cluster nodes The efficiency of suspending, migrating, and resuming is critical to application performance ⇒Our Goal: Compare stackful vs. stackless coroutine implementations in Remote Direct Memory Access (RDMA)-based AMT runtimes 1 / 17 Motivation Contributions Background Implementation Experiments Conclusion Thanks Contributions Mechanism for migrating C++20 stackless coroutines across processes (cluster-nodes) via RDMA Two RDMA-based AMT runtimes (prototypical): Stackful (uni-address scheme) Stackless (migratable C++20) Experimental comparison of: Task creation Context switching Communication overhead 2 / 17 Motivation Contributions Background Implementation Experiments Conclusion Thanks Background: AMT Asynchronous Many-Task (AMT) Programmers split the computation into a large number of fine-grained tasks Tasks are dynamically mapped to workers (e.g. processes distributed across cluster nodes) Task Models Dynamic Independent Tasks (DIT) Nested Fork-Join (NFJ) 3 / 17 Motivation Contributions Background Implementation Experiments Conclusion Thanks Background: Task Models Dynamic Independent Tasks (DIT) using child-stealing Parent Child 1 Child 2 spawn spawn Final result reduction Parent tasks complete after spawning. All task results are combined into the final result by reduction. Nested Fork-Join (NFJ) using continuation-stealing Parent (pre-spawn) Child 1 Child 2 Parent (continuation) spawn spawn sync Parent continuation is suspended until children complete. Children return their results to their parent. 4 / 17 Motivation Contributions Background Implementation Experiments Conclusion Thanks Work Stealing Dynamic load balancing: idle workers (thieves) steal from victims RDMA-based stealing bypasses the victim CPU and directly access victim’s memory We use dynamic independent tasks and coordinated random work stealing with continuation-stealing Execute most recently spawned task locally →good cache locality Parent’s remaining work (continuation) becomes stealable Requires suspendable tasks for migration and resume T1 T2 T3 T4 head tail Victim pushes/pops here Thief steals from here 5 / 17 Motivation Contributions Background Implementation Experiments Conclusion Thanks Stackful vs. Stackless Coroutines Suspendable tasks in C++ AMT runtimes are typically implemented as coroutines Stackful Suspend anywhere (flexible) Fast task creation Large migration payload Higher context switch cost Stackless Minimal, constant payload Low context switch cost Slower creation (heap alloc) Restricted suspend points 6 / 17 Motivation Contributions Background Implementation Experiments Conclusion Thanks Stackful Implementation (traditional) Based on uni-address scheme (Shiina & Taura, 2022) Uni-address region: identical virtual address on all processes Execution stack for tasks Migration: Thief performs MPI Get() on victim’s stack region Suspended parent task copied to same address ⇒pointers valid Resume via fast assembly-level context switch 7 / 17 FN FN−1 FN−2 child child . . . recursive call spawn spawn Call Tree Execution Stack Frame for FN Frame for FN−1 Frame for FN−2 . . . Stack Growth Task Size Motivation Contributions Background Implementation Experiments Conclusion Thanks Stackless Implementation (C++20) No own stack Compiler transforms a coroutine into a state machine object Each coroutine = heap-allocated frame object with internal pointers Problem: function pointers invalid after migration Solution: Pointer patching Exchange base addresses between processes Adjust internal function pointers after transfer Enables first migration of C++20 stackless coroutine 8 / 17 Resume Function Pointer Destroy Function Pointer Promise Object Coroutine Parameters Internal State (e.g., local variables) Code Segment points to resume logic points to cleanup logic Invalid after migration Motivation Contributions Background Implementation Experiments Conclusion Thanks Variable Granularity: Average Task Size 32 64 128 256 512 1024 2048 4096 215 216 217 218 219 220 221 222 223 224 225 226 227 228 Stackful Stackless Task Size [byte] Cutoff Parameter (C) Boxes show the range of task sizes Lines show the average task size Size of a stackful continuation is proportional to its recursion depth Stackless has constant size 15 / 17 fine-grained ⇒coarse-grained Motivation Contributions Background Implementation Experiments Conclusion Thanks Conclusion and Outlook Compared stackful and stackless coroutines in RDMA AMT runtimes Both show nearly identical performance Stackful: faster creation (up to 2.4×) Stackless: faster switching (up to 3.5×); smaller task states Smaller migration task size does not reduce communication time →latency-dominated Stackless gains relevance as task state grows Choice depends on task granularity! Future work: Experiments with benchmarks having larger task states Hybrid AMT runtime that adaptively selects the coroutine type 16 / 17 Thank you for your attention! Questions? #[email protected] www.jonasposner.com §github.com/projectwagomu zenodo.org/communities/projectwagomu