Uncertainty Quantification of Prediction Models for Differential Expression Analysis

Uncertainty Quantification of Prediction Models for Differential Expression Analysis Christof Seiler — https://christofseiler.github.io Department of Rheumatology, USZ/UZH, and Department of Advanced Computing Sciences, Maastricht University ZueKoSt: Seminar on Applied Statistics, Zurich, November 6, 2025 1/35 My Plan for Today •Part 1: Modeling •Part 2: Prediction 2/35 Part 1: Modeling Differential Protein Marker Expression cells (rows) celltype donor condition features (columns) marker counts cell info 1 2 1 2 T B NK T B NK T B NK T B NK stim unstim non-gating (~10) 1 5 3 … … … markers to compare cell types between conditions Are there any differentially expressed protein markers between experimental conditions? 3/35 Common Analysis Workflows 1. Response variable: mean counts and cell type abundance •R package diffcyt (Weber et al. 2019): High-resolution clustering and empirical Bayes moderated tests adapted from transcriptomics and univariate marginal analyses •F1000 CyTOF Workflow (Nowicka et al. 2017): Manual gating and univariate marginal analyses 2. Response variable: experimental condition •R package Citrus (Bruggner et al. 2014): Hierarchical clustering and regularized regression to select predictive features •Python package CellCnn (Arvaniti and Claassen 2017): Convolutional neural networks to detect rare cell populations •Limitations: Univariate marginal analyses and/or no statistical guarantees 4/35 Marginal vs. Conditional •Common to analyze biological data with marginal regression (as in diffcyt (Weber et al. 2019)) •Why? •Can produce p-values for each protein marker, then use BH (Benjamini and Hochberg 1995) and BY (Benjamini and Yekutieli 2001) procedures to control false discovery rate (FDR) •Easy to interpret •Fast 5/35 Marginal vs. Conditional •But marginal fitted coefficients might give the wrong answers •Example (Wakefield 2013): Suppose “true” model is E(Y|X=x,Z=z) = β0+β1x+β2z E(Z|X=x) = a+bx •Then what if we regress Yon x? E(Y|X=x) = β0+aβ2+ (β1+bβ2)x •Biased if Corr(X,Z)6=0 and Yconditionally dependent on Z 6/35 Our RPackage cytoeffect Multivariate Poisson Log-Normal Model with Zero Inflation •Model for protein markers Y1,...,YKgiven experimental condition X=x •Zero inflation: •Yijk are counts in cell i, patient j, and protein marker k •Flip a biased coin which lands Heads with probability πjk •If it comes up Heads, then set Yijk =0, otherwise Yijk ∼Poisson(λijk ) •Multivariate Poisson (Chib and Winkelmann 2001): log(λijk ) = βcond[i]k+bik +ujk 7/35 Low-Dimensional Summary of Posterior Samples Step 1: Compute distance matrix for each posterior draw of λ: log 2median distance log i alogit d falls patients logIjn Drs logthin in proteinmarkers Euclidean distance Drs 11 11 protein markers 14/35 Low-Dimensional Summary of Posterior Samples Step 2: DiSTATIS (Abdi et al. 2005) on posterior distance matrices D(1),...,D(n): DISTATIS patient B Jpatient c Dhl points m y hpatient A 15/35 Real Dataset: Pregnancy Study pCREB pSTAT5 pP38 pSTAT1 pSTAT3 prpS6 pMAPKAPK IkB pNFkB pERK1_2 A A B B C C D D E E F F G G HH I I J J K K L L M MN N O OP P −0.25 0.00 0.25 −0.4 −0.2 0.0 0.2 Factor 1 Factor 2 term 1st trimester 3rd trimester donor A B C D E F G H I J K L M N O P PTLG001 PTLG002 PTLG003 PTLG004 PTLG005 PTLG007 PTLG008 PTLG009 PTLG010 PTLG012 PTLG018 PTLG019 PTLG020 PTLG022 PTLG024 PTLG029 Posterior DiSTATIS of Latent Variable λ •Possible problem: MCMC sampler (as in rstan) will take too long 16/35 Parametric Bootstrap DiSTATIS 1) Fit Poisson log -normal model to each patient-condition combination independently using pairwise composite likelihood (Lindsay 1988; Fieuws and Verbeke 2006; Molenberghs and Verbeke 2006): b βand b Σ 2) Generate parametric bootstrap samples from the model: Yik |λik ∼Poisson(λik ), log(λik ) = b βk+bik , bi∼Normal(0,b Σ). 3) Replace posterior samples with bootstrap samples to create DiSTATIS plot 17/35 Comparison: Posterior vs. Parametric Bootstrap DiSTATIS pCREB pSTAT5 pP38 pSTAT1 pSTAT3 prpS6 pMAPKAPK IkB pNFkB pERK1_2 A A B B C C D D E E F F G G HH I I J J K K L L M MN N O OP P −0.25 0.00 0.25 −0.4 −0.2 0.0 0.2 Factor 1 Factor 2 term 1st trimester 3rd trimester donor A B C D E F G H I J K L M N O P PTLG001 PTLG002 PTLG003 PTLG004 PTLG005 PTLG007 PTLG008 PTLG009 PTLG010 PTLG012 PTLG018 PTLG019 PTLG020 PTLG022 PTLG024 PTLG029 Posterior DiSTATIS of Latent Variable λ pCREB pSTAT5 pP38 pSTAT1 pSTAT3 prpS6 pMAPKAPK IkB pNFkB pERK1_2 A A B B C C DD E EF F G G H H I IJ J K KL L M M N N O O P P −0.3 −0.2 −0.1 0.0 0.1 0.2 −0.4 −0.2 0.0 0.2 Factor 1 Factor 2 term 1st trimester 3rd trimester donor A B C D E F G H I J K L M N O P PTLG001 PTLG002 PTLG003 PTLG004 PTLG005 PTLG007 PTLG008 PTLG009 PTLG010 PTLG012 PTLG018 PTLG019 PTLG020 PTLG022 PTLG024 PTLG029 Parametric Bootstrap DiSTATIS of Latent Variable λ 18/35 Conclusions •Multivariate models: Describe marker correlations to avoid biases •Mixed models: Describe individual patient effects to avoid reporting overconfident results •R package cytoeffect with vignettes: https://christofseiler.github.io/cytoeffect/ 19/35 Part 2: Prediction Joint Work with Justine Leclerc (PhD Candidate) 20/35 Counterfactual Prediction Science Table ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! TEH jig Ht 7 Die int y.cc J.LT 44 Science Table cell treated T 1 2 control C cell treatment effect observed predicted predicted observed 21/35 Step 4: Download Model with Calibrated Prediction Intervals ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! A calibrating intervals! on dataset model with! prediction intervals target Y target Y predictor Xpredictor X not covered 27/35 RAI Platform — Reliable AI for Biomedicine ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! mm MM Mm mm amm mm Mm Mm mm MM mmm my M aa it I 1) Upload prediction model 3) Calibrate model on dataset 4) Download model with prediction intervals target variable features target variable features target variable features target variable features 2) Upload dataset Funded by the Digital Society Initiative Infrastructure and Lab Program in Zurich 28/35 Split Conformal Prediction Split the data into two sets: •D1called the proper training set of size n1=|D1| •D2called the calibration set of size n2=|D2| 29/35 Procedure 1. Fit prediction model on D1to obtain b fn1 2. Compute calibration set residuals Ri=|Yi−b fn1(Xi)|,i∈D2 3. Compute conformal quantile b qn2=d(1−α)(n+1)esmallest of Ri,i∈D2 4. Compute conformal set b Cn(x) = hb fn1(x)−b qn2,b fn1(x) + b qn2i 30/35 Finite-Sample Guarantees This procedure guarantees (Vovk, Gammerman, and Shafer 2005): P(Yn+1∈b C(Xn+1)|(Xi,Yi),i∈D1)≥1−α. Remarks: •Replace the residuals with other score functions (lower meaning better) •Only marginal coverage (Foygel Barber et al. 2021) 31/35 Our RPackage conformerR Our Idea 1treated control 23 jpvalues 1Aij Yij TJij c iobserved predicted i 7 Amatrix aconstructed with conformal prediction my samples treated control NOVELTY •R package conformerR under development: https://github.com/juslecl/conformeR 32/35 Open Problems Iposttreatment Idistribution shift Iconditioncoverage variables ya.at gn'Ere niftiest live.ge that tAh 8 33/35 Center of Experimental Rheumatology at USZ/UZH Alexandra Khmelevskaya Alexandre Meier Alissa Weibel Amela Hukara Andrea Laimbacher Andrea Nüesch Anna-Maria Hoffmann-Vold Asimina Kakale Astrid Hofman Bojana Müller Durovic Camino Calvo Caroline Ospelt Celina Geiss Danilo Menghini Ellen Kossmann Elena Pachera Ezen Ege Gabriela Kania Georgina Mathis-Pairo Gino Bonazza Ievgeniia Kocherova Jan Devan Justine Leclerc Katerina Apostolopoulou Laura Much Leyi Zhang Lumeng Li Marija Lugar Maryam Asadikorayem Masoume Halsadat Mirra Muriel Elhai Nicole Schneider Oliver Distler Pamela Bittterli Peter Künzler Phelipe Hatt Phatthamon Laphanuwat Philip Stauffer Pietro Bearzi Przemek Blyszczuk Raphael Micheroli Seyram Duphey Shao Thing Teoh Silja Malkewitz Stefan Dudli Tamara Mengis 34/35 Thanks for Listening! RAI Platform — Reliable AI for Biomedicine: https://rai.uzh.ch Funded by the Digital Society Initiative of UZH Please contact us if you’d like to conformalize your models or help us solve the open problems! Abdi, Hervé, Alice J O’Toole, Dominique Valentin, and Betty Edelman. 2005. “DISTATIS: The Analysis of Multiple Distance Matrices.” In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)-Workshops, 42–42. IEEE. Arvaniti, Eirini, and Manfred Claassen. 2017. “Sensitive Detection of Rare Disease-Associated Cell Subsets via Representation Learning.” Nature Communications 8: 14825. Benjamini, Yoav, and Yosef Hochberg. 1995. “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing.” Journal of the Royal Statistical Society: Series B (Methodological) 57 (1): 289–300. Benjamini, Yoav, and Daniel Yekutieli. 2001. “The Control of the False Discovery Rate in Multiple Testing Under Dependency.” Annals of Statistics, 1165–88. Bruggner, Robert V, Bernd Bodenmiller, David L Dill, Robert J Tibshirani, and Garry P Nolan. 2014. “Automated Identification of Stratifying Signatures in Cellular Subpopulations.” Proceedings of the National Academy of Sciences 111 (26): E2770–77. Carpenter, Bob, Andrew Gelman, Matthew D Hoffman, Daniel Lee, Ben Goodrich, Michael Betancourt, Marcus Brubaker, Jiqiang Guo, Peter Li, and Allen Riddell. 2017. “Stan: A Probabilistic Programming Language.” Journal of Statistical Software 76 (1). Chib, Siddhartha, and Rainer Winkelmann. 2001. “Markov Chain Monte Carlo Analysis of Correlated Count Data.” Journal of Business & Economic Statistics 19 (4): 428–35. Easton, Morris L. 1989. “Chapter 7: Random Orthogonal Matrices.” In Group Invariance in Applications in Statistics, Volume 1:100–107. Regional Conference Series in Probability and Statistics. Haywood CA; Alexandria VA: Institute of Mathematical Statistics; American Statistical Association. https://projecteuclid.org/euclid.cbms/1462061037. Fieuws, Steffen, and Geert Verbeke. 2006. “Pairwise Fitting of Mixed Models for the Joint Modeling of Multivariate Longitudinal Profiles.” Biometrics 62 (2): 424–31. Foygel Barber, Rina, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani. 2021. “The Limits of Distribution-Free Conditional Predictive Inference.” Information and Inference: A Journal of the IMA 10 (2): 455–82. Jauch, Michael, Peter D Hoff, and David B Dunson. 2019. “Monte Carlo Simulation on the Stiefel Manifold via Polar Expansion.” arXiv Preprint arXiv:1906.07684. Lewandowski, Daniel, Dorota Kurowicka, and Harry Joe. 2009. “Generating Random Correlation Matrices Based on Vines and Extended Onion Method.” Journal of Multivariate Analysis 100 (9): 1989–2001. Lindsay, Bruce G. 1988. “Composite Likelihood Methods.” Contemporary Mathematics 80 (1): 221–39. Molenberghs, Geert, and Geert Verbeke. 2006. Models for Discrete Longitudinal Data. Springer-Verlag New York. Nowicka, M, C Krieg, LM Weber, FJ Hartmann, S Guglietta, B Becher, MP Levesque, and MD Robinson. 2017. “CyTOF Workflow: Differential Discovery in High-Throughput High-Dimensional Cytometry Datasets [Version 2; Referees: 2 Approved].” F1000Research 6 (748). https://doi.org/10.12688/f1000research.11622.2. Vovk, Vladimir, Alexander Gammerman, and Glenn Shafer. 2005. Algorithmic Learning in a Random World. Vol. 29. Springer. Wakefield, Jon. 2013. Bayesian and Frequentist Regression Methods. Springer Series in Statistics. Springer, New York. https://doi.org/10.1007/978-1-4419-0925-1. Weber, Lukas M, Malgorzata Nowicka, Charlotte Soneson, and Mark D Robinson. 2019. “Diffcyt: Differential Discovery in High-Dimensional Cytometry via High-Resolution Clustering.” Communications Biology 2 (1): 183. 35/35