Life2vec: Foundation models for registry data @ Complexity Science Hub

Life2vec: Foundation Models for registry data Networks for Healthy Aging (Complexity Science Hub) Presented by: Germans Savcisens (NEU) Collaborators: Tina Eliassi-Rad (NEU) Lars Kai Hansen (DTU) Laust Mortensen (KU) Lau Lilleholt (KU) Ingo Zettler (KU) Anna Rogers (ITU) Sune Lehmann (DTU) Other Contributors: Søren Mørk Hartmann 10.5281/zenodo.17639200 | 18th Nov 2025 Northeastern University About Me PhD in Computational Social Science (DTU, 2024): oMachine Learning in Social Science oAI Fairness and Interpretability Postdoctoral Research Associate at Northeastern University @ Network Science Institute, RADLAB (Tina Eliassi-Rad): oRelevance and Stability of ““Beliefs”” (LLMs) oMechanistic Interpretability (LLMs) and uncertainty in GNNs oSequence analysis with LLMs: o(NYU) Onset of dementia o(Sweden) Epidemiology 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 2 Northeastern University Main contributions: 1. Propose a framework to analyze large-scale registry data 2. Demonstrate the power of dense representations 3. Adapt explainability methods to understand predictions GitHub Repository SocialComplexityLab/life2vec 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 3 Project Page Northeastern University Networks for Healthy Aging | G.Savcisens18 Nov 2025 4 Northeastern University Networks for Healthy Aging | G.Savcisens 518 Nov 2025 Born at Rungstedlund 1885 Accepted to the Royal Danish Academy of Fine Arts’ 1903 Leaves the Academy 1903 Publishes a tale “The Hermits” 1907 Engages to Bror Blixen 1912 Moves to Kenya 1914 Hospitalised at the National Hospital in Copenhagen 1915 Established a coffee farm in Kenya 1917 Divorces Bror Blixen 1925 Sells the coffee farm and permanently returns to Denmark 1931 Mother Dies 1939 Publishes “Winter’s Tales” in Denmark 1942 Awarded a grant by “H.K. Andersen Fund” 1955 Guest of Honour in New York 1959 Co-founder of the Danish Academy 1960 Dies 1962 “Shadows on the Grass” is published in Denmark. 1960 Life of Karen Blixen* (Danish author) * simplified Life Trajectories Northeastern University The Problem Born at Rungstedlund 1885 Accepted to the Royal Danish Academy of Fine Arts’ 1903 Leaves the Academy 1903 Publishes a tale “The Hermits” 1907 Engages to Bror Blixen 1912 Moves to Kenya 1914 Hospitalised at the National Hospital in Copenhagen 1915 Established a coffee farm in Kenya 1917 Divorces Bror Blixen 1925 Sells the coffee farm and permanently returns to Denmark 1931 Mother Dies 1939 Publishes “Winter’s Tales” in Denmark 1942 Awarded a grant by “H.K. Andersen Fund” 1955 Simplifying data How many times admitted to a hospital? Career changes? Traveling abroad? Probability of readmission to a hospital? Model 1 Model N Income level within the next year? Travelled within a year …Married Hospital Admission 1…1 2 Model 2 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 6 * simplified Northeastern University Born at Rungstedlund 1885 Accepted to the Royal Danish Academy of Fine Arts’ 1903 Leaves the Academy 1903 Publishes a tale “The Hermits” 1907 Engages to Bror Blixen 1912 Moves to Kenya 1914 Hospitalised at the National Hospital in Copenhagen 1915 Established a coffee farm in Kenya 1917 Divorces Bror Blixen 1925 Sells the coffee farm and permanently returns to Denmark 1931 Mother Dies 1939 Publishes “Winter’s Tales” in Denmark 1942 Awarded a grant by “H.K. Andersen Fund” 1955 Guest of Honour in New York 1959 Co-founder of the Danish Academy 1960 Dies 1962 “Shadows on the Grass” is published in Denmark. 1960 12 -2 10 539 -30 20 33 General Purpose Model Predict the human behaviour (on an individual level) Study sociological phenomena (on a global scale) We want a single model that takes nuanced life trajectories Compressed representation of life progression Give comprehensive insight into the data 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 7 Northeastern University life2vec Novel way to understand the structure of the data Our Work: life2vec as a proof-of-concept Life Progression from the point of view of Labor and Health Records Process complex-structure such as life-sequences Encoder Model Symbolic representation of data (text-like) Main Components: 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 8 Northeastern University Life-Trajectories and Registry Data Part I 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 9 Northeastern University Power of National Registry The National Registry is a source of fine-grained information about the progression of one’s life. How can we allow for a more nuanced exploratory analysis without trading of on the details? 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 16 Northeastern University Representation Learning and NLP Part II 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 17 Northeastern University Language and Machines “Everything was beautiful and nothing hurt”1 1. Slaughterhouse-Five, Kurt Vonnegut (1969) **AI-Generated Image 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 18 Northeastern University Language Processing “Everything was beautiful and nothing hurt”1 Networks for Healthy Aging | G.Savcisens18 Nov 2025 1. Slaughterhouse-Five, Kurt Vonnegut (1969) Create a numerical representation of the text! Computers can work with numbers a … and … beautiful … everything … hurt … no nothing … was … zyzzyva 0 … 1 … 1 … 1 … 1 … 0 1 … 1 0 19 Northeastern University Language Processing via Counts Networks for Healthy Aging | G.Savcisens18 Nov 2025 1. Slaughterhouse-Five, Kurt Vonnegut (1969) “Beautiful was nothing and everything hurt” “Everything beautiful hurt and was nothing” “Everything hurt nothing and was beautiful” If we reconstruct the sentence a … and … beautiful … everything … hurt … no nothing … was … zyzzyva 0 … 1 … 1 … 1 … 1 … 0 1 … 1 0 20 Northeastern University Language Processing via Counts “Maria likes spaceships”“Viktor prefers apples” “Susanne likes kiwi” apples: 1 prefers: 1 likes: 0 spaceships:0 kiwi: 0 apples: 0 prefers: 0 likes: 1 spaceships:1 kiwi: 0 apples: 0 prefers: 0 likes: 1 spaceships: 0 kiwi: 1 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 1 1 0 0 0 0 0 1 1 0 0 0 1 0 1 Count-based representations 21 Northeastern University Complexity of Language Language is a super complex signal… …and it inherits many issues associated with the longitudinal data. Image: Luchmee, D. (2019, July 25). The Complex Skill of Language. HappyNeuron. Retrieved March 5, 2024, from https://news.happyneuronpro.com/the-complex-skill-of-language/ 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 22 Northeastern University Representations in Geography longitude* latitude* Great Pyramid 31.08 29.58 Petra 30.19 35.26 Machu Picchu 13.09 35.26 Colosseum 12.29 41.53 * simplified Pyramids Petra These values capture spatial location, and allow us to reason about the distances (“similarity”). 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 23 Northeastern University Language Processing via Counts Representations Solution in NLP: Assign coordinates to words liveliness vehicle-(ness) artificiality spaceship 0.0 1.0 1.0 apple 0.3 0.0 0.2 kiwi 0.3 0.0 0.3 dog 1.0 0.3 0.1 18 Nov 2025 Networks for Healthy Aging | G.Savcisens These coordinates represent “meaning” and relations 24 Northeastern University Language Processing via Counts Representations “Viktor prefers apples”“Susanne likes kiwi” 0.2 0.4 0.5 0.1 0.2 0.6 “Maria likes spaceships” 0.2 -.1 0.0 Aggregate word representations to create sentence embeddings 18 Nov 2025 Networks for Healthy Aging | G.Savcisens … aggregate sentence embeddings to create paragraph embeddings and so on These two are closer Representations Of Sentences 25 Northeastern University BERT: Training Paradigm Multi-Genre Natural Language Inference Named-entity recognition Question answering and/or reading comprehension Devlin, Jacob, et al. "Bert: Pre-training of deep bidirectional transformers for language understanding." arXiv preprint arXiv:1810.04805 (2018). 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 32 Northeastern University BERT: Pretraining Encoder Blocks Embedding Layer Everything was beautiful and nothing hurt[CLS] [SEP] •Mask 15% of tokens (not including [PAD],[SEP],[CLS]): –10% unchanged –10% substituted with random tokens –80% substituted with the [MASK] token BERT 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 33 Northeastern University BERT: Pretraining Encoder Blocks Embedding Layer Yoga was [MASK] and nothing hurt[CLS] [SEP] •Mask 15% of tokens (not including [PAD],[SEP],[CLS]): –10% unchanged –10% substituted with random tokens –80% substituted with the [MASK] token 18 Nov 2025 Networks for Healthy Aging | G.Savcisens BERT 34 Northeastern University BERT: Pretraining Encoder Blocks Embedding Layer Yoga was [MASK] and nothing hurt[CLS] [SEP] Masked Language Decoder/Layer “Ask” the model to predict what was originally there? Decoder [CLS] usually has some task assigned BERT 18 Nov 2025 Networks for Healthy Aging | G.Savcisens Everything was beautiful and nothing hurt[CLS] [SEP] Original Sequence What we show to BERT 35 Northeastern University BERT Representation of the word “Everything” is now updated with information from the sequence: •It might contain some aspects of the word “beautiful” 18 Nov 2025 Networks for Healthy Aging | G.Savcisens Encoder Block 1 Contextualised representation Encoder Block N 36 Everything was beautiful and nothing hurt[CLS] [SEP] Northeastern University BERT: Finetuning PRETRAINED Encoder Blocks PRETRAINED Embedding Layer Everything was beautiful and nothing hurt[CLS] [SEP] Task Specific Decoder/Layer BERT Prediction (e.g., Sentiment Classification, next-token prediction) 12 -2 10 539 -30 20 33 Compressed representation of a sequence 18 Nov 2025 Networks for Healthy Aging | G.Savcisens What we show to BERT 37 Northeastern University Large Language Models LIFE2VEC Adapts BERT for life-sequences BERT Create nuanced word embeddings and handle complex sequences Great predictive performance on many NLP tasks General-purpose model, adaptable to new tasks 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 38 Northeastern University Socio-economic and health (symbolic) language Part III 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 39 Northeastern University Unfolding the data * slightly simplified overview Tabular to Textual Representation? 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 40 Northeastern University Forming a Language “In May 2008, Riley received >95k as a manager in Bank.” Language allows for flexible and nuanced communication 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 41 Convey the content in a spoken language Northeastern University * slightly simplified overview Concepts Age Global Time Segment Networks for Healthy Aging | G.Savcisens18 Nov 2025 48 Northeastern University * slightly simplified overview Concepts Age Global Time Segment Networks for Healthy Aging | G.Savcisens18 Nov 2025 49 Northeastern University * slightly simplified overview Concepts Age Global Time Segment Networks for Healthy Aging | G.Savcisens18 Nov 2025 50 Northeastern University * slightly simplified overview Concepts Age Global Time Segment Networks for Healthy Aging | G.Savcisens18 Nov 2025 51 Northeastern University * slightly simplified overview Concepts Age Global Time Segment [CLS] [MALE] [YEAR_1989] [JAN][SEP] 0 0 0 0 0 0 0 0 0 0 A A A A A [INT] 0 0 A The “Background sentence” Start of the sequence word Specification of sex Birth year Birth Month Country of origin A “dot” 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 52 Northeastern University * slightly simplified overview Concepts Age Global Time Segment Individual Life-Sequence Input to the life2vec model 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 53 Northeastern University Vocabulary 18 Nov 2025 Networks for Healthy Aging | G.Savcisens It forms the vocabulary of our artificial language 54 Northeastern University Part IV 18 Nov 2025 Networks for Healthy Aging | G.Savcisens life2vec: Capturing overall Structure 55 Northeastern University life2vec pipeline Concepts Age Global Time Segment Embedding Layer Encoder 1 Encoder N Life Sequence Aggregated numerical representations of concept tokens Contextualised representations of concept tokens Decoders life2vec 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 56 Northeastern University life2vec: Pre-training Concepts Age Segment Global Time •Mask 30% of tokens (not including [PAD],[SEP],[CLS]): –10% unchanged –10% substituted with random tokens –80% substituted with the [MASK] token Encoder Blocks Embedding Layer MLM Decoder 18 Nov 2025 Networks for Healthy Aging | G.Savcisens life2vec Note: we use our artificial language here, not a natural one (e.g., English) 57 Northeastern University Networks for Healthy Aging | G.Savcisens 6418 Nov 2025 Space of Concept Tokens (with PaCMAP) Savcisens, G., Eliassi-Rad, T., Hansen, L. K., Mortensen, L. H., Lilleholt, L., Rogers, A., ... & Lehmann, S. (2023). Using sequences of life-events to predict human lives. Nature Computational Science, 1-14. [INCOME 1] [INCOME 98] Northeastern University Projection to “Income” Direction [INCOME 1] Concept of the 1st quantile [INCOME 99] Concept of the 99th quantile [LF_514] Student [LF_312] Unemployed [LF_110] Self-employed [LF_315] Childcare leave from unemployment [LF_131] Employed with a managerial role [LF_134] Basic Wage earners [POS_1120] Senior Management [POS_2139] IT –Highest Level [POS_5131] Servant [POS_3423] Fitness Instructor 0.0 1.0 .36 Normalized coordinate along the direction .48 .49 .55 .62 .65 .45 .47 .71 .64 LF –Labor Force Status POS –Prof. Position 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 65 Northeastern University Projection to “Year” Direction [YEAR 1946] Concept of the 1946th birth year [YEAR 1991] Concept of the 1991st birth year [T40] Poisoning by narcotics 0.0 [LF_514] Student .72 1.0 [LF_134] Basic Wage earners [LF_611] Maternity/paternity leave from unemployment .94 .90 .89 [LF_313] Upskilling [LF_412] Early Retirement 0.46 LF –Labor Force Status [M81] Age-related osteoporosis without current pathological fracture 0.58 .78 Direction of the “Birth Year” [F32] Depression [I25] Chronic ischaemic heart disease .63 Normalized coordinate along the direction [LF_511] Person in Training 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 66 Northeastern University Projection to “Occupation” Direction (n.d.). What Is Your Opposite Job? The New York Times. Retrieved March 11, 2024, from https://www.nytimes.com/interactive/2017/08/08/upshot/what-is-your-opposite-job.html 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 67 Northeastern University Projection to “Occupation” Direction [POS 5120] “Cooking” [POS 2111] “Physisist” 0.0 1.0 [LF_134] Basic Wage earners [INCOME 5] Income 5th Qt .38 LF –Labor Force Status Normalized coordinate along the direction [INCOME 97] Income 97th Qt .61 [LF_131] Employed with a managerial role .64 [G20] Parkinson’s .60 [R52] Unspecified Pain .40 [IND 5629] Other restaurant businesses .20 [IND 3316] Repair of airand space-crafts .83 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 68 Northeastern University Concept Space: Robustness Models trained on separate datasets and with different initialization 1. Schober, P., Boer, C., & Schwarte, L. A. (2018). Correlation coefficients: appropriate use and interpretation. Anesthesia & analgesia,126(5), 1763-1768. rho > .6 (Strong monotonic correlation)1 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 69 Northeastern University Life2vec as a proof-of-concept •Life-sequences can be encoded as a symbolic “language” •Transformers learn structure and dependencies in this artificial language –Embeds both systems in one shared representation space •Embedding space lets us explore relationships between categories –This space is stable and aligns with our knowledge about the data 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 70 Northeastern University life2vec: Does it carry a predictive signal? Part V 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 71 Northeastern University Life-Summaries • Are learned embeddings “transferable” (can be used to solve many tasks)? •We want high-predictive power and explainability •We fine-tune (i.e., condition) life2vec on two tasks: –Early Mortality Prediction –Self-reported Personality Assessment 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 72 12 -2 10 539 -30 20 33 Prediction Northeastern University life2vec: finetuning Concepts Age Segment Global Time PRETRAINED Encoder Blocks PRETRAINED Embedding Layer Contextualised representations Task Specific Decoder New Task: Mortality Prediction, Questionnaire response 12 -2 10 539 -30 20 33 Compressed representation of a life sequence (conditioned on a specific task) 18 Nov 2025 Networks for Healthy Aging | G.Savcisens life2vec 73 Northeastern University Early Mortality Prediction: Auditing (c) 18 Nov 2025 Networks for Healthy Aging | G.Savcisens Interpretation of MCC: The higher the value, the more ”accurate”/correct the prediction on average (for a specific subgroup) 80 Northeastern University Early Mortality Prediction: Data Use Retrain the model on different variations of the dataset Partial Labor: no industry, sector, position and labour force 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 81 Northeastern University We can look at the low-dimensional space of life-summaries (each dot is a representation of the person 2D Projection 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 82 Northeastern University Global Interpretability •Interpretation of the directions of the personsummary space •Sensitivity of the model towards these directions Explainability with TCAV (Mortality Prediction) 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 83 Northeastern University Local Interpretability If we zoom-in: Maternity Leave Flexjob, receives salary Malignant neoplasm of brain (admitted to hospital) Pyothorax with fistula (admitted to hospital) •Interpretation of the scores: how much does the output likelihood change if we slightly modify the embedding of the token? •Only a local explanation (e.g., per sequence), with a vague interpretation. Sequence of an individual in a textual format Read: left-to-right, top-to-bottom 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 84 Northeastern University Study II: Extraversion Nuance Prediction Networks for Healthy Aging | G.Savcisens 8518 Nov 2025 Northeastern University life2vec and Personality Traits Image source: Wikipedia Inventory Descriptions: The HEXACO Personality Inventory - Revised We focus on Extroversion Facets: •Sociability (tendency to enjoy social interactions) •Liveliness (one's typical enthusiasm and energy) •Self-esteem (tendency to have positive self-regard) •Boldness (comfort within a variety of social situations) Example: 1. In social situations, I'm usually the one who makes the first move 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 86 Northeastern University Extraversion Nuance Prediction Task: “What kind of replies does the person give to the 10 questions evaluating their Extraversion? ” •Multiclass prediction •Ordinal Classification task (i.e., labels have order/hierarchy) •Highly imbalanced data Statement: In social situations, I'm usually the one who makes the first move Strongly agree Strongly disagree Neutral 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 87 Northeastern University Personality Data Questions: 6. Most people are more upbeat and dynamic than I generally am (liveliness) 7. The first thing that I always do in a new place is to make friends (social I) 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 88 Northeastern University What does it tell us? Performance –Pretrained life2vec transfers well to downstream tasks –Finetuned models produce interpretable predictions –Interpretations align with existing literature and domain knowledge Person-summaries –Learned person-summaries form a meaningful, structured space –This space can be used to study diverse social and behavioral phenomena 18 Nov 2025 Networks for Healthy Aging | G.Savcisens 89