Resilient Technologies. Why Decades-Old Tools Define the ROOT of Modern Research Data Management

R E SILIENT T E CHNOLOGIES WHY DECADES-OLD TOOLS DEFINE THE ROOT OF MODERN RESEARCH DATA MANAGEMENT LUKAS C. BOSSERT [email protected] �0000-0003-3076-3968 IT Center @ RWTH AACHEN UNIVERSITY Department: Research Process & Data Management Group: Technical Training & Consulting Version: 0.9.0 cba LuaHBTeX, Version 1.22.0 (TeX Live 2025) GNU Emacs 30.2 | org-mode: 9.7.34 DOI: 10.5281/zenodo.17157588 Powered by: Org mode #+TITLE: Resilient Technologies #+SUBTITLE: Why Decades-Old Tools Define the ROOT of Modern Research Data Management #+AUTHOR: Lukas C. Bossert #+DATE: [2025-10-28] #+EMAIL: [email protected] #+LANGUAGE: en #+FILETAGS: poster:coding:reproducibility #+OPTIONS: title:nil author:nil date:nil toc:nil #+LATEX_COMPILER: lualatex #+LATEX_CLASS: article #+LATEX_CLASS_OPTIONS: [12pt] #+LATEX_HEADER: \usepackage[english]{babel} #+LATEX_HEADER: \usepackage[a0paper,top=0cm,bottom=20cm,left=0cm,right=0cm]{geometry} #+LATEX_HEADER: \usepackage{fontspec,longtable,microtype,multicol,titlesec,caption} #+LATEX_HEADER: \captionsetup{labelfont=bf,font={sf,small}} #+LATEX_HEADER: \setmainfont[Numbers = {Monospaced, OldStyle}, Scale=0.90, ItalicFont = Roboto-LightItalic, BoldFont = Roboto-Medium,]{Roboto-Light} #+LATEX_HEADER: \usepackage{enumitem} \setlist{nosep} #+LATEX_HEADER: \usepackage[RGB]{xcolor} #+LATEX_HEADER: \definecolor{rwth-blue-100}{RGB}{ 0, 84,159} #+LATEX_HEADER: \definecolor{rwth-blue-25}{RGB}{199,221,242} #+LATEX_HEADER: \usepackage[minted,most,skins,breakable]{tcolorbox} #+LATEX_HEADER: \RenewTCBListing{minted}{O{}m}{myhbox={fonttitle=\bfseries,coltitle=rwt h-blue-100,}{#2}, breakable, enhanced jigsaw, listing only, listing engine=minted, minted language=#2, boxrule=0.25pt, colframe=rwth-blue-25, colback=rwth-blue-25, opacityback=0.4, minted options = {fontsize=\small,baselinestretch=.75,breaklines,break anywhere,autogobble,style=friendly,#1}} #+LATEX_HEADER: \RenewTCBListing{verbatim}{}{listing engine=minted,opacityframe =0.1,opacityback =0.1,colback=black,colframe=black,breakable, enhanced jigsaw, listing only,minted options = {breaklines,breakanywhere,autogobble,}} #+LATEX_HEADER: \AtBeginEnvironment{longtable}{\footnotesize} #+LATEX_HEADER: \usepackage[norule,marginal,hang]{footmisc} #+LATEX_HEADER: \setcounter{secnumdepth}{-1} \pagestyle{empty} #+LATEX_HEADER: \makeatletter #+LATEX_HEADER: \tcbset{myhbox/.style 2 args={enhanced,breakable,colback=white,colframe=rwth-blue-100,attach boxed title to top left={yshift*=-\tcboxedtitleheight},title={#2},boxed title size=title,boxed title style={sharp corners,rounded corners=northwest,colback=tcbcolframe,boxrule=0pt},underlay boxed title={\path[fill=tcbcolframe] (title.south west)--(title.south east) to[out=0, in=180] ([xshift=5mm]title.east)-- (title.center-|frame.east) [rounded corners=\kvtcb@arc] |- (frame.north) -| cycle;},#1}} #+LATEX_HEADER: \makeatother #+LATEX_HEADER: \newtcolorbox{sectitlebox}[1]{myhbox={frame empty}{#1},nobeforeafter,after=\vspace*{-1.75\baselineskip}} #+LATEX_HEADER: \titleformat{\section}{\normalfont\Large\bfseries}{}{0pt}{\tcbheading} #+LATEX_HEADER: \titleformat{\subsection}{\normalfont\large}{}{0pt}{\tcbheading} #+LATEX_HEADER: \newcommand{\tcbheading}[1]{\nointerlineskip\begin{sectitlebox}{#1} \end{sectitlebox}} #+LATEX_HEADER: \setlength{\parindent}{0pt}\setlength{\parskip}{6pt} #+CITE_EXPORT: biblatex ext-alphabetic-verb #+BIBLIOGRAPHY: references.bib #+PROPERTY: header-args:latex :exports code :mkdirp yes :eval never :tangle "resilient-technologies-poster.tex" #+PROPERTY: header-args:bash :results verbatim :exports both :mkdirp yes #+PROPERTY: header-args:shell :results verbatim :exports both :mkdirp yes #+begin_export latex \begin{multicols*}{5} #+end_export This poster demonstrates how decades-old yet robust digital tools can be combined in a /literate programming/ workflow. All content you see here — text, data, and code — has been created, executed, and rendered within a single source file. The entire poster is thus both a publication /and/ a working research object: every figure, dataset, and code snippet can be regenerated at any time, ensuring complete transparency and reproducibility. #+begin_quote Documentation must be regarded as an integral part of the process of design and coding. A good programming language will encourage and assist the programmer to write clear, self-documenting code, and even perhaps to develop and display a pleasant style of writing. [cite:@Hoare1973] #+end_quote This approach embodies the very idea of the *Resilient Technologies* concept: software and methods that persist not because of trends, but because of their clarity, openness, and lasting utility. Tools like ~Emacs~,~sed~, or ~awk~ remain essential not as relics, but as reliable companions for modern Research Data Management — stable foundations on which reproducible workflows can be built. A small detail worth mentioning: the title font of this poster has been chosen very deliberately. It is called =Trajan= — a typeface inspired by the stone-carved capitals of the Trajan Column in Rome (2nd century AD). Its enduring elegance perfectly reflects the message of this work: that well-crafted designs, whether in stone or in code, can remain relevant for centuries. Just as ~awk~ or ~tar~ continue to serve in today’s data ecosystems, =Trajan= persists as a timeless artifact of precision and clarity. For aesthetic balance, we slightly refined the digital kerning of the font to match the harmony we seek between tradition and technology. For this poster we follow a use case and show how to use the mentioned tools while following the research data life cycle. Going through the various tools we download a file, process it, analyze it and prepare it for proper archiving. * Abstract :noexport: The list below defines the content of the data-driven-visualization circle: #+begin_src csv :tangle ddv-resilient-technologies.csv :eval never category,name,description,startangle,totalangle,color utility,rsync,"Sync and backup (incremental)",-90,60,passive utility,LaTeX,"Typeset reports, posters, DMPs",90,180,passive utility,perl,"Text processing and glue scripting",30,120,passive utility,find,"Discover files (recursive search)",60,90,passive utility,SQLite,"Local catalogs and metadata",30,120,passive utility,cron,"Automate recurring runs",30,180,active pipeline,curl,"Acquire data (HTTP, APIs, repos)",200,160,active pipeline,sed,"Clean and normalize text streams",30,60,active pipeline,grep,"Validate/search (regex)",30,120,active pipeline,awk,"Summarize and analyze (aggregations)",30,120,active pipeline,tar,"Package for archiving; add checksum",-90,60,active pipeline,diff,"Verify and compare against previous/manifest",-30,60,active orchestrator,make,"SOP that orchestrates steps and dependencies",30,300,active hub,emacs / org-mode,"Hub for notes, code, writing; literate workflows",90,360,all #+end_src For the data-driven-visualization we need a package that has not been published, yet. But it is publicly availbale #+name: curl for ddv.sty #+begin_src shell :results none curl -L -O -C - -# "https://git.rwth-aachen.de/dl/templates/data-driven-visualization/-/raw/main/ddv.sty" #+end_src #+begin_src latex :noweb yes :tangle resilient-technologies-abstract.tex :results none :eval never \documentclass{standalone} \RequirePackage{fontspec} % Required for specifying fonts \setmainfont{TeX Gyre Heros} % Setting the main font to TeX Gyre Heros \newfontfamily\ddvHeaderFont{TeX Gyre Cursor} \usepackage[RGB]{xcolor} \definecolor{rwth-blue-100}{RGB}{ 0, 84,159} \definecolor{rwth-blue-25}{RGB}{199,221,242} \definecolor{rwth-blue-75}{RGB}{64,127,183} \definecolor{rwth-bordeaux-100}{RGB}{161,16,53} \colorlet{passive}{rwth-blue-25}\colorlet{active}{rwth-blue-100} %\colorlet{passive}{active} \colorlet{all}{rwth-bordeaux-100} \usepackage[ innercirclesize =5, dataset ={% name = {ddv-resilient-technologies.csv}, sorting = {totalangle}}, stages ={% name = {{Planning,Production,{Analysis},Archiving,Access,Re-use}}, bgcolor = {rwth-blue-100}, fontsize = 50}, slices ={% fontcolor = {white}, fontsize = 22} ]{ddv} \begin{document} \DDV{% stages = {options = {{% % arc data pos =.7, arc data pos={\WCmidangle<180?.4:.6}, radius={1.5+\thestagesRadius}{4+\thestagesRadius}, gap=2}}}, slices = {options = {{% arc data pos={\WCmidangle<180?.4:.6}, radius={1+\thestagesRadius}{2+\thestagesRadius}, gap=0}}}} \end{document} #+end_src * Poster Start :noexport: ** Preamble The preamble is the essential part of defining the outcome and the style of the posteer. We document it as one block since it is of less importance than the actual content. #+name: preamble #+begin_src latex \DocumentMetadata{ pdfversion=1.7, pdfstandard=A-3b} \RequirePackage{snapshot} \documentclass[english,a0paper]{scrartcl} \providecommand\InfoTeX{} \providecommand\InfoOrg{} \providecommand\InfoEmacs{} \providecommand\restWatermark{data_life_cycle_eng_rgb.png} \newcommand\restDOInumber{10.5281/zenodo.17157588} \def\restVersion{0.9.0} \def\restTitle{Resilient Technologies} \def\restSubtitle{Why Decades-Old Tools Define the ROOT of Modern Research Data Management} \usepackage{scrbase,enumitem,adjustbox,babel} \pagestyle{empty} \usepackage[style=ext-alphabetic-verb]{biblatex} \addbibresource{references.bib} \renewcommand\bibfont{\normalfont\footnotesize}% \usepackage{csquotes} \usepackage{qrcode} \usepackage[RGB]{xcolor} \definecolor{rwth-blue-100} {RGB}{0,84,159} \definecolor{rwth-blue-50} {RGB}{142,186,229} \definecolor{rwth-blue-25} {RGB}{199,221,242} \definecolor{rwth-green-100} {RGB}{87,171,39} \definecolor{rwth-bordeaux-100} {RGB}{161,16,53} \definecolor{orcid} {RGB}{166, 206, 57} \colorlet{passive}{rwth-blue-25} \colorlet{active}{rwth-blue-100} \colorlet{passive}{active} \usepackage[final, stretch =10, shrink =10, babel]{microtype} \newenvironment{optlist}[1]{% \begin{labeling}[]{#1}% \setlength{\itemsep}{-2pt}% \setlength{\topsep}{0pt}% \setlength{\parsep}{0pt}% \setlength{\partopsep}{0pt}% }{\end{labeling}} \usepackage{graphicx} \graphicspath{{img/}} \DeclareGraphicsRule{.ai}{pdf}{.ai}{} \usepackage{ccicons,dingbat,academicons} \usepackage{fontspec} \setmainfont[% Numbers = {Monospaced, OldStyle}, Scale = 0.90, ItalicFont = Roboto-LightItalic, BoldFont = Roboto-Medium, ]{Roboto-Light} \setkomafont{labelinglabel}{\ttfamily\bfseries} \setkomafont{labelingseparator}{\normalfont} \usepackage{trajan} \ExplSyntaxOn \NewDocumentCommand{\mytrajan}{O{\linewidth}m} {\lukas_trajan:nn {#1} {#2}} \tl_new:N \l__lukas_trajan_text_tl \cs_new_protected:Nn \lukas_trajan:nn {\tl_set:Ne \l__lukas_trajan_text_tl { \text_uppercase:n {#2} ~ } \tl_replace_all:Nnn \l__lukas_trajan_text_tl { ~ } { \c_space_tl } \tl_replace_all:Nnn \l__lukas_trajan_text_tl { R } { R \__lukas_trajan_r: } \tl_replace_all:Nnn \l__lukas_trajan_text_tl { T } { T \__lukas_trajan_t: } \resizebox{#1}{!}{\texttrjn{\tl_use:N \l__lukas_trajan_text_tl\unskip}} } \cs_new_protected:Nn \__lukas_trajan_r: {\peek_meaning:NF \c_space_tl { \__lukas_trajan_raise:N }} \cs_new_protected:Nn \__lukas_trajan_raise:N {\kern-0.3em\scan_stop: \raisebox{0.25\height}{\scalebox{1}[0.8]{#1}}} \cs_new_protected:Nn \__lukas_trajan_t: {\peek_meaning:NF \c_space_tl { \__lukas_trajan_lower:N }} \cs_new_protected:Nn \__lukas_trajan_lower:N {\kern-0.2em\scan_stop: \scalebox{1}[0.8]{#1}} \ExplSyntaxOff \NewDocumentCommand{\orcidIcon}{} {\begingroup\sffamily\textcolor{orcid}{\aiOrcid}\endgroup} \NewDocumentCommand{\orcid}{m} {\raisebox{-1pt}{\orcidIcon}\,\href{https://orcid.org/#1}{#1}} \usepackage[poster,many,skins,most,minted]{tcolorbox} \tcbset{% listing engine=minted, coverbox/.style={% enhanced, breakable, before skip =15pt plus 4pt minus 2pt, after skip =15pt plus 4pt minus 2pt, left=.1em, right=.1em, bottom=.1em, top=.1em, boxrule =20pt, colframe={rwth-blue-100}, colback ={rwth-blue-100}, coltitle={rwth-blue-100}, fontupper =\color{white}, title filled =false, subtitle style ={% boxrule =0.0pt, opacityfill =0.4, colback=white, colbacktitle ={rwth-blue-100}, coltitle =white,}}} \makeatletter \tcbset{ myboxtitle/.style 2 args={% natural height, enhanced, breakable, colback=rwth-blue-25, colframe=rwth-blue-100, attach boxed title to top right={yshift*=-\tcboxedtitleheight}, adjusted title={\tcbfontsize{1}\bfseries{#1}}, rounded corners, right=0mm, boxrule=0pt, left=12mm, boxed title style={% rounded corners, colback=tcbcolframe, boxrule=1pt, }, overlay={ \node[rotate=90, anchor=north west, inner sep=1mm, text=white] (title@aux) at (frame.south west) { #2}; \path[fill=tcbcolframe] ([email protected] west)--([email protected] east) to[out=90, in=270] ([yshift=5mm][email protected])-- ([email protected]|-frame.north) [rounded corners=\kvtcb@arc] -| (frame.west) |- ([email protected])[sharp corners] -- cycle; \node[rotate=90, inner sep=2mm, text=white] at (title@aux) {#2}; }, underlay boxed title={% \path[fill=tcbcolframe] (title.south east)--(title.south west) to[out=180, in=0] ([xshift=-10mm]title.west)-- (title.center-|frame.west) [rounded corners=\kvtcb@arc] |- (frame.north) -| cycle; }}} \newtcolorbox{coverbox}[1][]{coverbox,#1} \tcbset{ rest/base/.style={ listing only, breakable, top=0.5pt, bottom=0.5pt, left=5pt, right=5pt, colback=rwth-blue-25, colframe=rwth-blue-25, opacityback=0.4, fonttitle=\bfseries, coltitle=rwth-blue-100, sharp corners, boxrule=0pt, bottomrule=2pt, toprule=2pt, enhanced jigsaw, attach boxed title to top left={yshift*=-\tcboxedtitleheight}, boxed title style={ sharp corners, rounded corners=northwest, colback=tcbcolframe, boxrule=0pt, }, underlay boxed title={ \path[fill=tcbcolframe] (title.south west)--(title.south east) to[out=0, in=180] ([xshift=5mm]title.east)-- (title.center-|frame.east) [rounded corners=\kvtcb@arc] |- (frame.north) -| cycle; }, minted options={ fontsize=\small, breaklines, autogobble, style=friendly, breakanywhere }}} \makeatother \NewTCBInputListing{\restCLI}{O{} m}{% rest/base, listing file={snippets/#2}, minted language=bash, title={\tcbfontsize{1} CLI}, #1} \NewTCBInputListing{\restSnippet}{O{} m m}{% rest/base, listing engine=minted, listing file={snippets/#2}, title={\tcbfontsize{1} snippet (#3)}, minted language =#3, #1} \usepackage{hyperref} \hypersetup{ pdftitle = {Resilient Technologies}, pdfsubject = {tools reproducability}, pdfauthor = {Lukas C. Bossert}, pdfcopyright = {Copyright (c) \the\year}, pdfhighlight = /N, pdfdisplaydoctitle = true, pdfkeywords = {emacs awk orgmode curl diff sed grep make tar}, pdfproducer = {LuaLaTeX}, pdflicenseurl = {http://creativecommons.org/licenses/by-nc-nd/4.0/}, plainpages = false, colorlinks = true, linkcolor = black, citecolor = black, urlcolor = black, final= true, } \NewDocumentCommand{\restAdjB}{s O{} m o}{ \IfBooleanTF{#1}{% Starred: text-mode defaults \adjustbox{valign=c,#2}{#3}% }{% Normal: graphics-mode defaults \adjustbox{valign=c,#2}{\includegraphics[height=#4]{#3}}% }} \NewDocumentCommand{\restPosterFrame}{ O{} m m}{% % #1: #2: title #3: subtitle \posterbox[natural height]{name =title, column =1, span =4 }{\resizebox{\linewidth}{!}{\begin{coverbox}[#1] \mytrajan{#2}\\[2em] \resizebox{\linewidth}{!}{\sffamily\MakeUppercase{#3}} \end{coverbox}}} \posterbox{name=author,column=1,above=row2,yshift=-3mm,} { \resizebox{\linewidth}{!}{\scshape\bfseries Lukas C. Bossert}\\ \href{mailto:[email protected]}{[email protected]}% \hfill\orcid{0000-0003-3076-3968}} \posterbox{name=affiliation,column =4, above =row2} {IT Center @ \textbf{\scshape RWTH Aachen University}\\ Department: Research Process \& Data Management\\ Group: Technical Training \& Consulting} \posterbox[sidebyside, valign upper=center, righthand width = 15cm] {column =1, above =bottom, name =qr} {\restAdjB*{\qrcode[height=3.5cm]{https://doi.org/\restDOInumber}} \tcblower \restAdjB*{Version: \restVersion\space\hspace*{5cm} \ccbysa}\\ \restAdjB*{\InfoTeX}\\ \restAdjB*{\InfoEmacs\ | org-mode: \InfoOrg}\\ \restAdjB*{DOI: \href{https://doi.org/\restDOInumber}{\restDOInumber}}} \posterbox{column =2, above =bottom, name =hashtag} {\restAdjB{FDM-Hashtag-FDMRWTH}[3cm]\hfill \restAdjB{powered_by_emacs.png}[3.8cm]\hfill \restAdjB{powered_by_org_mode.pdf}[3.8cm]} \posterbox{column =3, above =bottom, name =dkz} {\restAdjB{logo-dkz2r}[4cm] \restAdjB{qr-dkz2r}[4cm]\hfill \restAdjB{BMFTR}[2cm]\hfill \restAdjB{Förderhinweis-EU_horizontal}[1.5cm]} \posterbox{column =4, above =bottom, name =itc} {\restAdjB[center]{rwth-itc}[5cm]} \coordinate (TCBPOSTER@mybottom) at ([yshift=-1cm][email protected]); } #+end_src ** Body First we need to set the frame: #+name: document-begin #+begin_src latex \begin{document} #+end_src Then we are defining the general setup and layout of the poster #+name: poster-setup #+begin_src latex \begin{tcbposter}[ coverage = {spread, watermark opacity = 0.2, watermark graphics = {\restWatermark}}, poster = { %showframe, spacing =7mm, columns =4, rows=8 }, boxes = {enhanced jigsaw, colframe =white, opacityframe =0.0, opacityback =0.0, colback =white, boxrule =0pt}, fontsize = 24pt] #+end_src Let’s start with the title of the poster. #+name: poster-title #+begin_src latex \restPosterFrame{\restTitle}{\restSubtitle} #+end_src * Tools and Technologies This is now the main part of the document. We will show the various tools and their technologies to improve numerous stages of the research data life cycle. These tools are furthermore assistance along the research cycle, too (fig. [[fig:research-life-cycle]]). # [[][]] HOTFIX FOR MINTED #+name: fig:research-life-cycle #+caption: Life Cycle of Research. #+attr_latex: :float nil :width 0.75\linewidth #+attr_org: :width 700 file:img/research-cycle.png Overall we try to describe a nice concrete use-case on how to use the following tools. As a nice example we are taking a dataset about the NFDI and analyse it further. It is about how the individual NFDI consortia are collaborating with each other (or not) and how it changed over time. This is a common topic and has been discussed before [cite:@Strecker:834322;@Strecker:835121]. ** Abstract-Text :noexport: We also want to have the abstract on the main poster. But there is no need to have it exported, too. #+begin_src latex \begin{posterboxenv}[colback =white,natural height] {column =2, span =2,row =2, name =abstract-circle}% \hfil\includegraphics[width=\linewidth]{resilient-technologies-abstract.pdf} \end{posterboxenv} #+end_src #+begin_src latex \begin{posterboxenv}[colback =white,natural height] {column =1, span =1,row =2, name =abstract-text}% \newcommand\mytext[2]{\textcolor{#1}{\bfseries #2}} Research data management (RDM) today relies on a growing landscape of platforms and specialized tools. While these innovations advance research, they also bring risks: short life cycles, proprietary dependencies, and limited sustainability. In contrast, long-established tools have proven to be remarkably resilient— distinguished by openness, interoperability, and active communities. This poster demonstrates how such “\textbf{resilient technologies}”— despite their age—remain central to sustainable, transparent, and reproducible research data management today. \end{posterboxenv} #+end_src ** emacs (1976) / org-mode (2003) Still Emacs is a great tool and should not be neglected despite its age [cite:@Hahn2016Emacs;@Kitchin2016DataSharing;@Strobel1996Linux]. The first section will discuss this and show the benefits and capabilities. #+begin_src latex \begin{posterboxenv}[myboxtitle={emacs (1976) / org-mode (2003)} {planning, production, analysis, archiving, access, re-use}] {column=1, row=3, name=emacs} \textbf{Emacs} is not just an editor but a fully extensible research and productivity environment. It can be adapted to cover the entire workflow—from notes and coding to analysis and publishing—without switching applications. \medskip The real strength unfolds with \textbf{org-mode}: plain text enriched with \emph{semantic markup} and lightweight metadata. An Org file can act simultaneously as notebook, task manager, analysis script, and manuscript— a robust foundation for \textbf{reproducible research}. #+end_src The relevance of Emacs and org-mode is evident looking at the small peak of its functional tool-set. We start with an overview of how Org-mode structures content using simple metadata lines at the beginning of each file. These simple keyword/value pairs describe the context of your document and can be exported to LaTeX, HTML, Markdown, or other formats without additional effort. With ~org-modes~ ’s metadata system, each file is not just content but a /self-describing research object/. #+name: org-mode-meta #+begin_src org :eval never :tangle snippets/org-mode-meta :exports code #+TITLE: Resilient Technologies #+SUBTITLE: Why Decades-Old Tools Define the ROOT of Modern Research Data Management #+AUTHOR: Lukas C. Bossert #+DATE: [2025-10-28] #+EMAIL: [email protected] #+LANGUAGE: en #+FILETAGS: poster:coding:reproducibility #+end_src #+begin_src latex \tcbline Org-mode supports \textbf{structured metadata} at the top of every file, also for export/provenance tracking. %\restSnippet{org-mode-meta}{orgmode} #+end_src ~org-mode~ supports semantic markup that stays readable as plain text. It is similar to Markdown but more flexible and extensible. #+name: org-mode-markup #+begin_src org :eval never :tangle snippets/org-mode-markup :exports code *bold* /italics/ _underline_ +strike-through+ ~code~ [[https://orgmode.org][Org website]] =verbatim= [cite:@<bibtexkey>] footnote:[fn] [[file.pdf][PDF]] #+end_src #+begin_src latex \tcbline Org-mode has \textbf{semantic markup}, similar but more extensible than markdown. %\restSnippet{org-mode-markup}{orgmode} #+end_src This tool supports research in many ways: you can manage notes and concepts, and build a “second brain” that connects your ideas (e.g. using ~org-roam~ for a Zettelkasten approach). For researchers this is especially important because knowledge work rarely follows a linear path. Instead, concepts, datasets, and publications are interwoven (fig. [[fig:org-roam-ui]]). # [[][]] HOTFIX FOR MINTED ~org-roam~ helps to: -create a knowledge graph of papers, methods, and datasets -discover hidden connections across projects and disciplines -ensure that ideas are not lost but integrated into the broader context -support reproducible writing by linking notes directly into manuscripts #+name: fig:org-roam-ui #+caption: ~org-roam-ui~ visualizes connections as a graph, making it easier to navigate and explore your research knowledge base. Here it shows the node of ~NFDI~ and its connected nodes. #+attr_latex: :float nil :width \linewidth #+attr_org: :width 700 file:img/org-roam-ui.png #+caption: Metadata header of a ~org-roam~ -file. #+begin_src org :tangle snippets/org-roam :results none :eval never :exports code :PROPERTIES: :ID: 6225a860-4c1f-419e-9b67-ae20827f98f9 :ROAM_ALIASES: "RWTH Aachen University" "RWTH Aachen" :END: ,#+title: RWTH (Universität) ,#+filetags: :affiliation: Active participation within the [[id:537cc27f-73c2-4484-af11-e3acfc0dc932][NFDI]]. #+end_src # [[][]] HOTFIX FOR MINTED #+begin_src latex \tcbline \textbf{Personal knowledge management} and linking by creating a network of thoughts, snippets and information with \texttt{org-roam}. #+end_src Org-mode also allows you to keep all your writing, analysis, and results in a single plain text file. This is particularly valuable for *reproducible research*, since every step is documented and can be re-executed at any time. Similar to a Jupyter Notebook, *org-babel* goes even further by integrating multiple programming languages into one unified, version-controlled workflow document. This is also the concept of /literate programming/ and is applied in creating this poster. For further information about the historic context of this approach see [cite:@Knuth1984LiterateProgramming] and for its modern use [cite:@Schulte2012MultiLanguage;@Stanisic2014]. #+begin_src org :tangle snippets/org-mode-babel :results none :eval never :exports none ,#+name: my emacs folder structure ,#+begin_src bash :eval yes :exports both :results verbatim ls -d ~/@mycene/00-09_system/01_emacs/* | sort | awk -F/ '{print $NF}' | head -3 ,#+end_src ,#+RESULTS: my emacs folder structure | 01.00_system | | 01.01_attachments | | 01.02_bib | #+end_src #+begin_src latex \tcbline With \texttt{org-babel} you have embedded, executable code blocks for transparent analyses, best way to do \textbf{literate programming}. #+end_src There is even the possibility to work simultaneously in the same buffer: one editing code the other documenting it (cf. [cite:@HartmanBossert2023org]). #+begin_src org :tangle snippets/org-mode-macro :eval never :exports code #+MACRO: rdlc (eval (concat "*Research Data Life Cycle* (Section: " (capitalize $1) ")")) ... {{{rdlc(analysis)}}} ... #+end_src Macros give you great flexibility for content-dependent phrasing. This enables consistent terminology, DRY authoring, and context-dependent phrasing (e.g.\ localized labels, dynamic dates, or section names). Macros are expanded at export time, so the source stays concise while the output remains explicit and auditable. #+begin_src latex \tcbline \texttt{org-mode} provides a lightweight macro system that lets you define reusable text snippets with parameters. #+end_src Org tables support formulas, cell and range references, and automatic recomputation. Totals, averages, and percentages can be expressed directly in the document; recalculation updates values and derived metrics in place. Because tables are plain text, they are diff-friendly, versionable, and export cleanly to LaTeX, HTML, or CSV—ideal for transparent, FAIR-aligned data summaries within manuscripts. #+name: nfdi-datasets #+begin_src org :eval never :tangle snippets/org-mode-table :exports code | domain | consortia | datasets | avg | % | |------------------+-----------+----------+-----+------| | Life Sciences | 8 | 123 | 15 | 13.6 | | Humanities | 6 | 150 | 25 | 16.6 | | Engineering Sc. | 5 | 173 | 35 | 19.1 | | Natural Sciences | 7 | 458 | 65 | 50.7 | |------------------+-----------+----------+-----+------| | TOTAL | 26 | 904 | | | #+TBLFM: @>$2=vsum(@2$2..@-1$2);%.0f::@>$3=vsum(@2$3..@-1$3);%.0f #+TBLFM: @2$4..@-1$4=$3/$2;%.0f::@2$5..@-1$5=100*$3/@>$3;%.1f #+end_src When the formulas are evaluated you get updated values in columns and totals, making the table an active, reproducible spreadsheet within your document. #+begin_src latex \tcbline Tables are plain-text, too, however you have full control over formulas and calculations. \end{posterboxenv} #+end_src *Summary*: Emacs and Org-mode integrate the complete research workflow— metadata, analysis, documentation, and publication—into a single, version-controllable plain-text ecosystem. ** curl (1997) When it comes to accessing online data sources, ~curl~ is one of the most reliable and flexible tools available. It supports a wide range of network protocols (~file://~,~https://~,~sfpt://~, etc.) and allows automation of file retrieval, metadata inspection, and data publication. It excels because it can be easily combined with other command-line tools such as ~jq~,~awk~, or ~diff~ to build reproducible, scriptable workflows. #+begin_src latex \begin{posterboxenv}[myboxtitle={curl (1997)}{planning, access, re-use}]% {column=1, below=emacs, name=curl} \texttt{curl} is one of the most universal and resilient tools for data transfer. Unlike graphical or browser-based downloads, \texttt{curl} can be scripted and parameterized for reproducibility, transparency, and automation—core principles of \textbf{FAIR} and \textbf{Good Scientific Practice}. #+end_src In the context of *Research Data Management* (RDM)}, ~curl~ serves as a lightweight yet powerful component for: -accessing files and metadata from repositories (e.g.\ Zenodo, Dataverse), -automating REST API queries for research data and publications, -validating file integrity through checksum comparison, -and scripting reproducible data acquisition processes. #+name: curl-pattern #+begin_src bash :tangle snippets/curl-pattern :eval never :exports none curl [OPTIONS] [URLs] #+end_src #+begin_src latex \tcbline The minimal usage pattern: \restSnippet{curl-pattern}{bash} #+end_src We demonstrate this with a small workflow illustrating reproducible data access and validation — a common RDM use case: downloading a dataset and confirming its integrity. First, query Zenodo’s REST API for the file’s checksum: #+name: curl-checksum #+begin_src bash :exports both curl -sL "https://zenodo.org/api/records/15880071" | jq -r '.files[] | select(.key=="Collaboration_of_Consortia_2025.csv") | .checksum' | awk -F: '{print $2}' #+end_src #+RESULTS: curl-checksum : aeb63d7c5fc00bf0f37c14b2f6fe8d6c Now download the file itself. We use ~curl~ to query Zenodo’s REST API for the checksum of our dataset. The following options are used: -~-s~ Silent mode (no progress output) -~-L~ Follow redirects (often necessary for Zenodo) -~-O~ Save the file with the same name as on the server. -~-C -~ Resume a previous download if it was interrupted. An existing file is not overwritten. -~-#~ Show a progress bar instead of the default meter. Since we don’t get feedback when the download has happened successfully we add it manually -- but only after exit code 0 for the ~curl~ part. #+name: curl-download #+begin_src shell :exports both :dir use-case curl -L -O -# "https://zenodo.org/records/15880071/files/Collaboration_of_Consortia_2025.csv" && echo "file downloaded." #+end_src #+RESULTS: curl-download : file downloaded. Once downloaded, calculate the local checksum and compare it with the remote one: #+name:file-checksum #+begin_src shell :exports both :dir use-case md5sum Collaboration_of_Consortia_2025.csv | awk '{print $1}' #+end_src #+RESULTS: file-checksum : aeb63d7c5fc00bf0f37c14b2f6fe8d6c #+begin_src bash :var remote=curl-checksum :var local=file-checksum :exports both :tangle snippets/diff-match [[ "$remote" == "$local" ]] && echo -n "MATCH: $local" || echo "MISMATCH" #+end_src #+RESULTS: : MATCH: aeb63d7c5fc00bf0f37c14b2f6fe8d6c Since the checksums match, we can trust the integrity of our dataset — a key requirement of *Good Scientific Practice* and a simple but powerful example of how resilient CLI tools enable transparent, verifiable data workflows. #+begin_src latex \end{posterboxenv} #+end_src ** sed (1974) In this section we show how to manipulate the downloaded file while making sure we follow Good Scientific Practice. #+begin_src latex \begin{posterboxenv}[myboxtitle={sed (1974)}{production}]% {column=1, below=curl, name=sed} \texttt{sed} is a stream editor for transforming text—fast, scriptable, and reliable even on very large files. It excels at \emph{data cleaning} tasks: normalizing metadata, harmonizing formats, and fixing systematic errors. #+end_src RDM relevance: Because transformations are declared in plain text, they are transparent, auditable, and reproducible. This makes \texttt{sed} a robust choice for FAIR-aligned workflows and provenance tracking. The substitution form is the core building block: -~s~ = substitute (search /pattern/, write /replacement/) -~g~ (all matches per line), or use a number (only the n-th match), -for a flag, e.g. ~I~ (ignore case) The pattern shown below is very basic but in most cases enough to achieve and master the tasks. #+begin_src bash :tangle snippets/sed-pattern :eval never :exports none sed [FLAGS] 's/pattern/replacement/g' [FILE] #+end_src A quick explanation how the snippet has to be read and understood. #+begin_src latex \restSnippet{sed-pattern}{bash} The leading \texttt{s} stands for \emph{substitute}. It tells \texttt{sed} to search for the given \emph{pattern} and replace it with the specified \emph{replacement}. The trailing \verb|g| flag applies the substitution \emph{globally} across the line, so every occurrence of the pattern is replaced, not just the first. #+end_src There are many more prefix-flags for ~sed~, but in our showcase we limit ourselves to the most useful one. Further flags can be: -~p~: print only lines where a substitution occurred (e.g. ~sed -n 's/foo/bar/p' file.txt~) -~I~: case-insensitive matching (e.g. replace ~DOI:~,~doi:~, or ~Doi:~ in one step) -~[0-9]~: apply substitution only to the n-th match in a line (e.g. ~s/foo/bar/2~) -~w~: write all modified lines to a separate file for documentation or audit These options make ~sed~ extremely flexible for *research data cleaning*, for example when normalizing metadata, harmonizing date formats, or validating identifier strings. Coming back to our concrete example. We will fix a misspelling and change ~NFDI-NFDI-MatWerk~ to ~NFDI-MatWerk~ and keep a backup for provenance. To avoid escaping slashes and to protect whole words, use an alternate delimiter and word boundaries. In the example we use an alternate delimiter to avoid escaping ~/~. Additional prefix-flags can be added to control the behavior: -~-i~ edit the file /in place/, replacing the original -~.bak~ create a backup-file with the original content (suffix ~.bak~) #+name: sed-cleaning-delimiter #+begin_src shell :tangle snippets/sed-delimiter :dir use-case sed -i.bak 's|NFDI-NFDI-MatWerk|NFDI-MatWerk|g' Collaboration_of_Consortia_2025.csv && echo "file cleaned." #+end_src #+RESULTS: sed-cleaning-delimiter : file cleaned. Here we show that we do have now two files on our system and they are not the same #+name: sed-files-on-system #+begin_src shell :exports both :dir use-case ls Coll*_of_Con*_2025* #+end_src #+RESULTS: sed-files-on-system : Collaboration_of_Consortia_2025.csv : Collaboration_of_Consortia_2025.csv.bak Great power comes when using /regular expressions/ in the search pattern, which is activated with the flag \verb|-E| (extended regular expression). #+begin_src shell :tangle snippets/sed-pattern-capture :eval never sed -E 's/^([^,]+),[[:space:]]*([^,]+)$/\2, \1/' [FILE] #+end_src In this example two fields separated by \verb|,| are being swopt. -~^ ...$~ it takes the whole line into account (from start to end). -~([ ,]+)~ one or more non-comma characters (non-greedy-by-construction), it also handles optional spaces after the comma. -~\2, \1~ reinsert the captured groups in the new order. #+begin_src latex \end{posterboxenv} #+end_src ** grep (1973) In the following section we talk about a tool that is useful when it comes to /searching and validating data/. It is one of the oldest and most resilient tools. #+begin_src latex \begin{posterboxenv}[myboxtitle={grep (1973)}{production,analysis}] {column=1, below=sed, name=grep} When it comes to \textbf{searching and validating data}, \texttt{grep} is one of the oldest and most resilient tools. It is extremely fast, works on arbitrarily large files, and supports /regular expressions/ for flexible pattern matching. #+end_src The name ~grep~ comes from the original Unix /ed/-command *g/re/p*, meaning ``globally search for a regular expression and print''. In research data management ~grep~ helps to -/Identifier validation/ (e.g.\ DOIs, ORCIDs, grant IDs) and quick conformance checks on headers or required fields. -/Quality gates/ in pipelines (detect missing/forbidden values, out-of-range tokens, or schema violations) before publishing data. -/Provenance & auditing/ by saving search rules and match counts alongside datasets (easy to version-control and diff). -/Interoperability/ in UNIX pipelines: combine with tools like ~sed~,~awk~, or \texttt{sort}/\texttt{uniq} for reproducible, end-to-end checks. - /Portability & longevity/: available on all platforms, no dependencies, future-proof plain-text rules. The following snippet would not work typing it into your CLI but it shows the syntax and gives a good impression how to use the tool. #+begin_src bash :tangle snippets/grep-pattern :eval never :exports none grep [FLAGS] PATTERN [FILE ...] #+end_src We load it into LaTeX for a nice display. #+begin_src latex \tcbline The basic syntax is: \restSnippet{grep-pattern}{bash} #+end_src We can add various flags to specify the search pattern: -~-i~ ignore case (match /NFDI-MatWerk/,/nfdi-matwerk/, etc.) -~-v~ invert match (show lines /without/ the pattern) -~-c~ count the number of matching lines -~-n~ show line numbers where matches occur -~-r~ search recursively through directories -~-E~ use extended regular expressions Just for the sake of doublechecking: Do we still have =NFDI-NFDI-MatWerk= in our ~.csv~-file? Let’s find out: #+begin_src bash :dir use-case grep -ci "NFDI-NFDI-MatWerk" Collaboration_Of_Consortia_2025.csv || echo "file properly cleaned." #+end_src #+RESULTS: : 0 : file properly cleaned. The result is =0= entries of our search pattern are found in the file. Therefor we get the echo =file properly cleaned=. Perfect nice use case for the potential of ~grep~. Like ~sed~ before the power of ~grep~ unfolds when using /regular expressions/ in the search pattern. For example: -~grep -E "10\.[0-9]{4,9}/"~: find Digital Object Identifiers (DOIs) -~grep -E "[0-9]{4}-[0-9]{4}-[0-9]{4}-[0-9X]"~ : match valid ORCID identifiers -~grep -E "ERROR|WARNING"~: scan log files for error /or/ warning messages With these patterns, entire datasets can be validated or filtered in seconds, even when files are gigabytes in size. To give a concrete example we apply a regular expression to the dataset (and for keeping it short we limit it to the first line). #+begin_src shell :tangle snippets/grep-re :dir use-case grep -E "^Event:(.+?)MatWerk(.+?);ongoing$" Collaboration_of_Consortia_2025.csv | head -1 #+end_src #+RESULTS: : Event: conference organisation;NFDI Berlin-Brandenburg network meeting;MaRDI,NFDI4Dat aScience;FAIRagro,FAIRmat,MaRDI,NFDI-MatWerk,NFDI4Biodiversity,NFDI4BIOIMAGE,NFDI4Cat,N FDI4Chem,NFDI4DataScience,NFDI4Earth,NFDI4Energy,NFDI4Immuno,NFDI4Ing,NFDI4Memory,NFDI4 Microbiota,NFDIxCS,PUNCH4NFDI,Text+,KonsortSWD;annually;networking;ongoing We can filter our dataset for specific information: Here the pattern ~^Event:(.+?)MatWerk(.+?);ongoing$~ means that only lines that describe an /Event/ related to /MatWerk/ and that are marked as /ongoing/ at the end of the line are matched. This gives us a precise overview of how many such cases exist in the dataset. #+begin_src latex \end{posterboxenv} #+end_src ** diff (1974) The tool ~diff~ is one of the oldest yet most crucial components for *reproducible research*. It was designed to detect line-by-line differences between text files—an idea that evolved into version control systems such as RCS, CVS, Git, and beyond. #+begin_src latex \begin{posterboxenv}[myboxtitle={diff (1974)}{analysis}]% {column=1, below=grep, name=diff} When it comes to verifying reproducibility and provenance, \texttt{diff} is one of the simplest yet most powerful verification tools. It compares files line by line and reports any deviations. This allows researchers to document data cleaning steps, validate outputs, or monitor versioned transformations. After applying transformations (e.g.\ normalization with \texttt{sed}), \texttt{diff} can confirm the expected changes and provide a readable audit trail. #+end_src In the following block we cannot show in detail how ~diff~ can be used in the context of Research Data Management, but it is essential for: -tracking and documenting changes in datasets, code, or metadata, -validating reproducibility by comparing generated outputs against references, -auditing transformations (e.g. after using ~sed~ or ~awk~), -and verifying the integrity of archived or published files. Even small differences (e.g. an extra semicolon or changed date format) can affect data reproducibility— ~diff~ helps to make those differences transparent and reviewable. #+begin_src bash :tangle snippets/diff-pattern :eval never :exports none diff [FLAGS] [FILE1] [FILE2] #+end_src #+begin_src latex \tcbline The basic syntax looks like: \restSnippet{diff-pattern}{bash} #+end_src We dive right into our example of above and compare two files, the edited one and the backup file of ~Collaboration_of_Consortia_2025.csv~. A structured way for viewing the output of a comparison is to use the \texttt{-u} (unified) flag. There are also some other flags to be used. -~-u~ unified format; shows 3 lines of context for each change. -~-q~ brief output; only report if files differ. -~-y~ side-by-side comparison. -~-U<number>~ produces a diff with <number> of lines of context. Note that some flags above are only exclusively applicable. You cannot combine them. Compare the original and cleaned dataset after processing it with ~sed~. To understand the output of ~diff~ we are happy with only the first couple of lines of changes. #+name: diff-csv #+begin_src shell :tangle snippets/diff-csv :results output :exports both :dir use-case diff -u -U0 Collaboration_of_Consortia_2025.csv.bak Collaboration_of_Consortia_2025.csv | head -5 #+end_src #+RESULTS: diff-csv : --- Collaboration_of_Consortia_2025.csv.bak 2025-10-21 15:51:58 :+++ Collaboration_of_Consortia_2025.csv 2025-10-21 15:52:24 : @@ -11 +11 @@ : -Event: conference organisation;"AI as an enabler for science; 3rd NFDI Berlin Brandenburg Meeting";NFDI4DataScience,MaRDI;NFDI4DataScience,MaRDI,FAIRmat,NFDI4Cat,NFD I4Biodiversity,NFDI-NFDI-MatWerk,FAIRagro;single event;discussions, presentations;completed : +Event: conference organisation;"AI as an enabler for science; 3rd NFDI Berlin Brandenburg Meeting";NFDI4DataScience,MaRDI;NFDI4DataScience,MaRDI,FAIRmat,NFDI4Cat,NFD I4Biodiversity,NFDI-MatWerk,FAIRagro;single event;discussions, presentations;completed Each hunk begins with a header such as ~@@ -11 +11 @@~, meaning that one line (line 11) in the old file was replaced by one line in the new file. Lines marked with ~-~ were removed; those with ~+~ were added. This clear textual provenance leaves every transformation traceable. #+begin_src latex \end{posterboxenv} #+end_src ** make (1976) The tool ~make~ is one of the earliest and most elegant automation systems ever developed in computing. Originally designed for compiling programs, it soon proved to be a *universal workflow manager*: It is totally language agnostic, but a common use is within the realm of \LaTeX{} [cite:@Bossert_dtk_make]. #+begin_src latex \begin{posterboxenv}[myboxtitle={make (1976)}{production,analysis,archiving,access}] {column=4, row=2, name=make} A \textbf{Makefile} (without suffix) is an integral part of every project. It serves as a \textbf{Standard Operating Procedure (SOP)} that defines how individual tasks depend on each other and how they are executed. \tcbline In the context of research data management, \texttt{make} functions as a lightweight yet powerful workflow engine. Each rule represents not just a command, but a clearly defined dependency chain that transparently records how results are generated from inputs. This approach ensures \emph{traceability}, \emph{reusability}, and \emph{automation}—core principles of Good Scientific Practice and reproducible research. #+end_src In the file you can use various variable definitions and placeholder to ease your workflow. -~$<~ refers to the first prerequisite (the source CSV) -~$@~ refers to the target (=$(FOLDER)/clean.csv=) -~=~ /recursive (lazy)/ — RHS is stored as text and expanded when the variable is used. Later changes to referenced vars are seen. -~:=~ /simple (immediate)/ — RHS is expanded now and the result is stored. Later changes elsewhere do not affect it. -~?=~ /conditional default/ — Assign only if the variable is /not defined/ (defined-but-empty still counts as defined). -~+=~ /append/ — Appends to the current value The rule ~all:~ acts as the *default target* of the Makefile. When you execute simply ~make~ on the command line, this rule is triggered first. It typically defines the *main workflow* or entry point — here, the complete data pipeline that downloads, cleans, and summarizes the dataset. This ensures that each dependent step runs in the correct order and only when its inputs have changed. The declaration ~.PHONY~ defines targets that do *not* correspond to real files. If a file named ~all~ or ~clean~ already existed, make would assume that the target is up to date and skip its execution. Declaring these targets as phony tells make that they are procedural rules, not file outputs. #+begin_src makefile :tangle snippets/make-pattern :eval never :exports none VAR =|?=|+=|:= VALUE .PHONY: TARGET TARGET TARGET all: TARGET TARGET: PERREQUISITE(S) COMMAND #+end_src #+begin_src latex Here is the basic design of a \texttt{makefile}: \restSnippet{make-pattern}{make} #+end_src Here comes a more concrete example of how to setup a ~makefile~ according to our usecase. #+begin_src makefile :tangle use-case/makefile :eval never :exports code CONSORTIUM = NFDI-MatWerk CSV ?= Collaboration_of_Consortia_2025.csv FOLDER := results-$(CONSORTIUM) FLAGS += -L FLAGS += -O PREPARATION = $(shell mkdir -pv $(FOLDER)) all: $(PREPARATION) summary @echo "all done." $(CSV): @curl $(FLAGS) -C - -# "https://zenodo.org/records/15880071/files/$(CSV)" @echo "file downloaded." diff: $(CSV) @( diff -u $(CSV).bak $(CSV) > $(FOLDER)/diff-result.txt ) || test $$? -le 1 @echo "diff done." summary: diff @grep -ci "$(CONSORTIUM)" $(CSV) > $(FOLDER)/grep-summary.txt @echo "summary done." #+end_src To execute the defined targets, you mentioned it otherwise to execute the main rule (=all:=), it is enough to write in your CLI plainly #+begin_src bash :dir use-case make #+end_src #+RESULTS: : diff done. : summary done. : all done. and all dependencies will be considered and created if necessary. #+begin_src latex \end{posterboxenv} #+end_src ** cron (1975) The tool ~cron~ is one of the most reliable automation systems ever developed. It allows the scheduling of commands and scripts at specific intervals — from minutes to months — and thus provides a temporal dimension to reproducible workflows. Originally designed for system maintenance tasks, it quickly became the backbone of automation across UNIX systems. In essence, ~cron~ is the *timekeeper* of reproducible research: while ~make~ manages *dependencies*,~cron~ manages *time*. Together, they ensure that tasks are executed automatically, in the correct order and at the correct time — without user intervention. Typical use cases in research data management include: -periodic backups or synchronization of repositories (e.g., using ~rsync~), -scheduled data validation or checksum verification, -regular harvesting of metadata from APIs, -and the automated regeneration of reports or dashboards. #+begin_src latex \begin{posterboxenv}[myboxtitle={cron (1975)}{production,analysis,archiving}] {column=4, below=make, name=cron} When reproducibility meets time, \texttt{cron} provides the temporal backbone of automated research workflows. It executes predefined tasks at regular intervals — hourly, daily, or monthly — ensuring that updates, validations, and synchronizations happen automatically. \texttt{cron} thus complements \texttt{make}: the former manages \emph{when}, the latter \emph{how}. #+end_src Each user can maintain their own ~crontab~ file, defining scheduled jobs. The syntax follows a five-field pattern specifying time and frequency, followed by the command. #+begin_src bash :tangle snippets/cron-pattern :eval never :exports none # |----------- 1: minute (0–59) # | |--------- 2: hour (0–23) # | | |------- 3: day of month (1–31) # | | | |----- 4: month (1–12) # | | | | |--- 5: day of week (0–6, Sunday=0) #12345 * * * * * command_to_execute #+end_src #+begin_src latex \tcbline Here is the explanation and a basic pattern of a cron-file. \restSnippet{cron-pattern}{bash} \end{posterboxenv} #+end_src For instance, to run a nightly validation and synchronization of research data: #+begin_src bash :tangle snippets/cron-rsync-validate :eval never :exports code 0 2 * * * make summary && rsync -av use-case/ backup/ #+end_src This example runs every night at 2 a.m., validating the dataset using the ~make~ rule and synchronizing it afterward. The combination of ~make~ and ~cron~ thus enables a fully automated, time-aware, and reproducible research workflow. ** awk (1977) When it comes to handle structered data (e.g. in columns) then *awk* is your friend. The original handbook of [cite:@awk] is still the most comprehensive book and should always be the first choice of documentation. It is continously maintained and the development goes on. #+begin_src latex \begin{posterboxenv}[myboxtitle={awk (1977)}{analysis,production}] {column=4, below=cron,name=awk} When it comes to data processing and analysis, \textbf{awk} is one of the most remarkably enduring tools in computational research. It combines the features of a \emph{streaming text processor}, a \emph{pattern matcher}, and a lightweight \emph{programming language}. Its concise syntax allows complex data operations to be written in just a few lines of code. #+end_src Unlike shell pipelines that only filter text, ~awk~ can: -perform real-time calculations and aggregations, -handle column-based datasets like CSV files directly, -extract, transform, and summarize data on the fly, -and produce structured output for further analysis. For researchers, awk shines when dealing with raw or tabular data: it bridges the gap between ~grep~/~sed~ filtering and high-level statistical scripting in R or Python. Because it runs in any UNIX environment and needs no dependencies, it remains ideal for *reproducible workflows*:~awk~ scripts are plain text and require no dependencies, they are fully transparent, version-controllable, and reproducible—key requirements for FAIR-compliant workflows. When embedded in Makefiles or shell pipelines, ~awk~ becomes a lightweight yet powerful component of automated and auditable RDM processes. #+name: awk-pattern #+begin_src bash :tangle snippets/awk-pattern :eval never :exports none awk -F',' 'BEGIN { init } pattern { action } END { summary }' [FILE] #+end_src Let’s first start with the basic pattern of ~awk~. -~-F','~ field separator (here =,= for a =.csv=) -~\$1,\$2,...~ columns/fields; =NF= = number of fields, =NR= = current record number - ~BEGIN~ initialization block, executed once before reading input (e.g., set field separators, initialize counters, print headers) - ~pattern~ condition or regex tested per input line (e.g., =$3 > 0= or =/MatWerk/=) -~action~ command executed when the pattern matches (e.g., =sum+=$3; print $1,$3=) -~END~ finalization block, executed once after all lines are processed (e.g., print summary, averages, totals) #+begin_src latex \tcbline Here is the basic pattern: \restSnippet{awk-pattern}{awk} #+end_src For our dataset we want to get an overview of how many times e.g. =NFDI-MatWerk= (defined as variable) is listed in /Affiliated Lead/Chair/ (3rd column). The script is saved as the file =awk-count-by-type= (in the folder =snippets=) and looks like #+name: affiliation-count-by-type #+begin_src awk :in-file Collaboration_of_Consortia_2025.csv :var consortium="NFDI-MatWerk" :results verbatim :exports code :eval never :tangle use-case/affiliation-count-by-type BEGIN { FS = ";"; OFS = "\t" } function norm(s){ s=tolower(s); gsub(/[[:space:]-]+/,"",s); gsub(/\"/,"",s); return s } NR==1 { want = norm(consortium); next } index(norm($3), want) { cnt[$1]++; total++ } END{ for(t in cnt){if(length(t)>m)m=length(t); if(length(cnt[t])>w)w=length(cnt[t])} print "----"consortium"----" for(t in cnt) printf "%-*s %*d\n", m, t, w, cnt[t] } #+end_src In your CLI you execute the file in the following way. Here you can specify the content of the variable, too - with e.g. any other NFDI consortium’s name. -~-f~ specify the path to the ~awk~-file script (e.g. =make -f affiliation-count-by-type= ) -~-v~ declare the variable and its content, see above. #+name: awk-affiliation #+begin_src bash :eval true :dir use-case :results verbatim :var consortium="NFDI-MatWerk" : folder="results-${consortium}" mkdir -pv ${folder} awk -f affiliation-count-by-type -v consortium=${consortium} Collaboration_of_Consortia_2025.csv \ | tee "${folder}/analysis-overview.txt" #+end_src #+RESULTS: awk-affiliation : ----NFDI-MatWerk---- : Joint service 1 : Project 2 : Interest group 1 : Event: workshop 7 : Event: conference participation 2 : Event: local networking 2 We can easily reuse the code snippet and pass a different NFDI consortium to it. #+call: awk-affiliation(consortium="NFDI4ING") #+RESULTS: #+begin_example results-NFDI4ING ----NFDI4ING---- Event: hackathon 1 Event: conference organisation 2 Joint service 4 Interest group 8 Event: workshop 1 Event: lecture series 1 Alliance 1 Event: local networking 5 #+end_example #+begin_src latex \end{posterboxenv} #+end_src ** tar (1979) We conclude our /tour de tools/ with a way to archive all necessary files. #+begin_src latex \begin{posterboxenv}[myboxtitle={tar (1979)}{archiving}] {column =4, below=awk, name =tar} The command-line tool \textbf{tar}—short for \emph{tape archive}—has been the backbone of data packaging and long-term archiving for nearly half a century. Originally designed to write sequential data streams to magnetic tape, it remains one of the most widely used and trusted archiving tools in computing today. #+end_src In the context of research data management, ~tar~ provides: -platform-independent, lossless bundling of files and directories, -reproducible data snapshots for archiving and transfer, -compatibility with checksums and compression utilities -(e.g. ~gzip~,~bzip2~,~xz~), -and transparent integration in automated workflows or HPC pipelines. Because ~tar~ simply concatenates files with metadata, it is robust, transparent, and /future proof/: every UNIX-like system will be able to unpack a tar archive even decades later. #+begin_src shell :tangle snippets/tar-pattern :eval never :exports none tar [FLAGS] [FILEs/PATTERN] [DIRECTORY] #+end_src #+begin_src latex \tcbline Here you see the basic pattern \restSnippet{tar-pattern}{bash} #+end_src Let’s create an archive of cleaned datasets with a dedicated checksum file: #+begin_src bash tar -C use-case -czf NFDI-results.tar.gz . && echo "file tape-archived." sha256sum NFDI-results.tar.gz | tee NFDI-results.tar.gz.sha256 #+end_src #+RESULTS: : file tape-archived. : 6bcf8f4675bb7cf33fbac55a5d09e8c46cc7dd42553685ac8ee10174b6ab11c6 NFDI-results.tar.gz The flags we are using here are: -~-c~: Create a new archive containing the specified items. -~-z~: Compress the resulting archive with gzip. -~-f~: Write the archive to the specified file and of course we want to see if it really worked and look into the generated tape archive. #+begin_src bash :results verbatim tar --list -f NFDI-results.tar.gz | sort #+end_src #+RESULTS: #+begin_example ./ ./affiliation-count-by-type ./Collaboration_of_Consortia_2025.csv ./Collaboration_of_Consortia_2025.csv.bak ./makefile ./results-NFDI-MatWerk/ ./results-NFDI-MatWerk/analysis-overview.txt ./results-NFDI-MatWerk/diff-result.txt ./results-NFDI-MatWerk/grep-summary.txt ./results-NFDI4ING/ ./results-NFDI4ING/analysis-overview.txt #+end_example The resulting ~.tar.gz~ file can be deposited in a repository or sent to collaborators, along with the checksum guaranteeing data integrity and provenance. #+begin_src latex \end{posterboxenv} #+end_src ** Results Providing a condensed overview of tools that can also be considered as /resilient technologies/ but have not been presented in detail above. Furthermore, we introduce the term *ROOT* — a conceptual badge that can be applied to all these technologies. A tool is considered *ROOT* when it is /robust/,/open/,/ongoing/, and /time-tested/. These qualities ensure long-term usability, reproducibility, and resilience in research data management environments. #+begin_src latex \begin{posterboxenv}[colback=white, natural height] {column=2, span=2, below=abstract-circle, name=result}% \vspace*{5cm} \hfil\includegraphics[width=.8\linewidth]{logo-root} \end{posterboxenv} #+end_src The overview below shows the variety and bandwidth of mature, open, and stable technologies that continue to form the foundation of research infrastructures. #+begin_src latex \begin{posterboxenv}[colback=white, natural height] {column=4, above=mybottom, name=result-list}% Not all tools of lasting relevance can be introduced in depth here. The following list presents additional \emph{resilient technologies} that remain fundamental to reproducible and sustainable research workflows. \begin{labeling}{XX} \item[find (1974)] \emph{BSD/GPL}; Recursively searches complex directory structures to locate files, match patterns, or filter metadata. Essential for navigating large research repositories with nested folders or evolving data versions. \item[LaTeX (1984)] \emph{LPPL}; The enduring standard for scientific writing and documentation— used for DMPs, reports, posters, and publications. Combines typographic precision with reproducibility and platform independence. \item[perl (1987)] \emph{Artistic/GPL}; A versatile scripting language for text manipulation, pattern recognition, and metadata transformation. Ideal for parsing logs, transforming XML/JSON, or validating research data. \item[rsync (1996)] \emph{GPL}; Synchronizes and mirrors directories across systems efficiently. Commonly used for backups, replication, and maintaining mirrored repositories. Transfers only differences, saving time and bandwidth. \item[SQLite (2000)] \emph{Public Domain}; A serverless SQL database stored in a single file—lightweight yet powerful. Excellent for metadata catalogs, portable archives, and small-scale analytical datasets. \end{labeling} \end{posterboxenv} #+end_src #+caption: Badge for a *ROOT* technology. #+attr_latex: :float nil :width 0.75\linewidth #+attr_org: :width 700 file:img/logo-root-invers.png #+print_bibliography: #+begin_export latex \end{multicols*} #+end_export #+name: document-end #+begin_src latex :exports none \end{tcbposter} \end{document} #+end_src * Meta and Source Poster :noexport: In this section we create a poster with the source code of this file. For this we take advantage of the ~noweb~-feature of org-mode. ** Meta Poster #+begin_src latex :noweb yes :tangle resilient-technologies-poster-meta.tex :results none \DocumentMetadata{ pdfversion=1.7, pdfstandard=A-3b} <<preamble>> <<document-begin>> <<poster-setup>> <<poster-title>> \begin{posterboxenv}[enhanced jigsaw, colback=white, top=0pt, bottom=0pt, left=0pt, right=0pt, ]{column = 1, row=2,span = 4} \includegraphics[width=\linewidth]{resilient-technologies-code.pdf} \end{posterboxenv} <<document-end>> #+end_src ** Source Poster #+begin_src latex :noweb yes :tangle resilient-technologies-source.tex :results none \DocumentMetadata{ pdfversion=1.7, pdfstandard=A-3b} \mag4000 \documentclass{article} \usepackage[english]{babel} %\usepackage[a0paper,top=0cm,bottom=20cm,left=0cm,right=0cm]{geometry} \usepackage[{a6paper},top=0cm,bottom=2.4cm,left=0cm,right=0cm]{geometry} \usepackage[RGB]{xcolor} \definecolor{rwth-blue-100}{RGB}{ 0, 84,159} \definecolor{rwth-blue-25}{RGB}{199,221,242} \usepackage{multicol} \pagestyle{empty} \usepackage[minted,skins,breakable]{tcolorbox} \NewTCBInputListing{\restCLI}{O{}m}{% listing only, listing file={#2}, breakable, top=0pt, bottom=0pt, left=0pt, right=0pt, colback=white, opacityback=0, sharp corners, boxrule=0pt, bottomrule=0pt, toprule=0pt, enhanced jigsaw, minted language=org, minted options={ breakautoindent=false, breaksymbolleft={}, breakanywheresymbolpre={}, breakanywhere, breaklines, autogobble, style=friendly}, #1 } \begin{document}% \columnsep -5pt% \fontsize{1.25pt}{1.3pt}\selectfont% \begin{multicols*}{5}% \restCLI{resilient-technologies-code.org} \end{multicols*} \end{document} #+end_src #+begin_src latex :noweb yes :tangle resilient-technologies-poster-source.tex :results none <<preamble>> <<document-begin>> <<poster-setup>> <<poster-title>> \begin{posterboxenv}[ colback=white, enhanced jigsaw, top=0pt, bottom=0pt, left=0pt, right=0pt, ]{column=1, row=2, span=4, between=row1 and mybottom} \includegraphics[width=\linewidth]{resilient-technologies-source.pdf} \end{posterboxenv} <<document-end>> #+end_src