Data Cleaning – Data Days Autumn 2025

Universitetsbiblioteket Anna Karpova Christine Dybwad, PhD Data Cleaning Image: Evita L. Bergstad Alternative title: I have messy data, help me! Validation of research methods –Transparency of science Sharing requirements Plan for the session ▪Why Data cleaning? ▪First rule of Data cleaning ▪Second rule of Data cleaning ▪Tidy Data Principles ▪Date Formats ▪Null values ▪Common mistakes ▪Practical techniques 2. Learn more and practice (BYOD or some practical exercises) 1. Introduction part: Validation of research methods –Transparency of science Sharing requirements Why do i have to do Data cleaning? Make your data understandable and reusable (the R in FAIR) Transform your messy excel spreadsheets into machineand human-readable tables Good for publications Easy to read into R, Python, etc. By the end of this session: Create consistent, accurate and analysis-ready tabulated data Validation of research methods –Transparency of science Sharing requirements Go from this.... Validation of research methods –Transparency of science Sharing requirements Ta-da! To this! Problematic null values (White et al., 2013) First Rule of Data Cleaning Always preserve a copy of your Raw data Do all cleaning, processing, working in a separate (nicely named) file Anna_data_2024_raw.txt →Anna_data_2024_processed.xls →Anna_data_2024_clean_v01.xls Problematic null values (White et al., 2013) Second Rule of Data Cleaning Read Me files: Document all cleaning steps here, to inform the user how the data has been modified DataverseNO ReadMe template for datasets: https://zenodo.org/records/10849096 The Tidy Data Principles 1.Every variable must have a separate column. 2.Every observation must have a separate row. 3.Only one data point per cell. Tidy Data Principles Tidy Data Principles (Wikham, 2016) Problematic null values (White et al., 2013) Practical techniques (that I will not speak further about) Sorting and Filtering Conditional Formatting Text to Columns Removing Duplicates Data verification Problematic null values (White et al., 2013) How to get started? Hands-On Work Time Try it out: Did you bring your own data? Fabulous! Get started and ask us along the way if you need help No data? No problem! Here are some practical exercises: Pretty easy: https://datacarpentry.org/spreadsheet-ecology-lesson/index.html Definitely harder: https://carpentries-incubator.github.io/fair-bio-practice/07-data-in-excel.html R: https://datacarpentry.github.io/R-ecology-lesson/working-with-data.html Universitetsbiblioteket Anna Karpova Christine Dybwad, PhD End:Data Cleaning Thank you for your attention! Feedback? Get in touch at anna.karpov[email protected] Information and help Research Data Manag ement at UiT [email protected] Research Data Management at UiT Email: [email protected] References Teal et al., 2019, datacarpentry/spreadsheet-ecology-lesson: Data Carpentry: Data Organization in Spreadsheets for Ecologists, June 2019: Zenodo, doi:10.5281/zenodo.3269869. White et al., 2013. Nine simple ways to make it easier to (re)use your data, Ideas in Ecology and Evolution 6(2): 1-10 Special Issue-Data Sharing in Ecology and Evolution https://ojs.library.queensu.ca/index.php/IEE/article/view/4608 Hadley Wickham, Tidy Data, Vol. 59, Issue 10, Sep 2014, Journal of Statistical Software. http://www.jstatsoft.org/v59/i10.