Albert's travel diary: The VuFind® Adventure Continues – Scaling and Shaping Discovery
Abstract
This presentation provides an overview of the past year’s developments in ALBERT, a discovery system based on VuFind® of the cooperative library network Berlin-Brandenburg (KOBV). ALBERT runs as a multi-instance platform with a centralized Solr index for several member libraries and research institutions. We recently completed the migration of the majority of our instances to VuFind® as well as the migration from version 7.1 to 10.1. As part of our effort to adopt more standard VuFind® components, we present a prototype of a default VuFind® instance running in Kubernetes. Separately, we are evaluating RecordManager for potential use in backend metadata processing. Finally, we introduce SolrProxy, a gatekeeper middleware developed at KOBV that bridges between ALBERT and the Solr index. It enables flexible parameterization of fine-grained mapping, boosting, and filtering to serve diverse user groups more effectively. Deploying SolrProxy in Kubernetes improves scalability and automatic workload balancing across multiple VuFind® instances.
Full text
Image: StableDiffusion VuFind® Berlin 2025: What’s in it for us? Albert's travel diary: The VuFind® Adventure Continues – Scaling and Shaping Discovery Ying Wang, Martin Weigelt
2 30.09.2025 Kooperativer Bibliotheksverbund Berlin-Brandenburg © 2025#by Ying Wang,#Martin Weigelt# is licensed under#CC BY 4.0 ALBERT Introduction ●The Cooperative Library Network Berlin-Brandenburg (KOBV) is the association of all university libraries, all public libraries and many special libraries in Berlin and Brandenburg. ●ALBERT is a discovery system developed by the KOBV over the past 11 years. It is based on a concept of the library of the Helmholtz Centre Potsdam German Research Centre for Geosciences, the development and pilot user of ALBERT. ●ALBERT has 2 kind of frontends with appropriate backend processing: –Topic based search portals: Selected data sources and/or data excerpts in format Marc21 on a topic of different institutions (e.g. KOBV-Portal, a regional portal with nearly all metadata of catalogs in region Berlin/Brandenburg) –Institutional frontends: Multiple individual sources that are relevant for institute with support of different data-formats.
3 30.09.2025 Kooperativer Bibliotheksverbund Berlin-Brandenburg © 2025#by Ying Wang,#Martin Weigelt# is licensed under#CC BY 4.0 ALBERT with frontend VuFind® SOLRProxy SOLRProxy ALBERTBackend VuFind® Frontends SOLR-Cloud ALBERT FTP/HTTP OAI/SRU SOLRProxy SOLR-Cloud Gemeinsame Verbündeindex (GVI) ●ALBERT has a modularised backend for data processing, an Apache Solr™ in SolrCloud mode as a search index and VuFind® as frontend ●13 project cooperations ●Team of 5 Full-Stack DevOps + 1 Projectmanager
4 30.09.2025 Kooperativer Bibliotheksverbund Berlin-Brandenburg © 2025#by Ying Wang,#Martin Weigelt# is licensed under#CC BY 4.0 ALBERT Keypoint Features ●Integration of multiple search indizes as tabs –Institutional/Portal index –Article index with data from Crossref –GVI index (for german inter-library loan) ●Own design with barrier-free components (e.g. high-contrast view) ●Display of hierarchies of records ●Integration of classifications (e.g. Regensburger Verbund Klassifikation (RVK)) ●Special availability (custom real-time via script, through records or regularly updated availability database) ●Account functionality for SISIS (via NCIP) ●Connection to Solr-Cloud with own scheme
5 30.09.2025 Kooperativer Bibliotheksverbund Berlin-Brandenburg © 2025#by Ying Wang,#Martin Weigelt# is licensed under#CC BY 4.0 ALBERT Project-Timeline 2024 Q1 2024: Release FHP Q2 2024: Release EUV Q3 2024: •Release DLR •Release BTU Q4 2024: •Release TH-Wildau with ILS Koha •Update of VuFind 7.1 to 10.1 with EWW Q2 2023: •Migration to in-house Kubernetes-Cluster (in co-op with external partner) •Set-up of productive environment and establishment of a process for releases (in coop with external partner) Q1 2021: •Requirement analysis •Introductory training on VuFind® Q4 2023: Release TH-Wildau Q4 2021: Set-up of an development and test environment with CI/CD based on Docker and Gitlab Q2 2022: •Kubernetes training •Takeover of project from external development partner •Further development of additional features by ALBERT-Team 2021 2022 2023 Q2 2021: Start migration to VuFind® 7.1 with THWildau with external development partner Q3 2021: Start of development account functionality for SISIS in VuFind® and migration of BTU, FHP und EUV 2025 Q1 2025: Release BTU, IPN and GEOMAR with VuFind 10.1 Q3 2025: •Release WAE and DKFZ with VuFind 10.1 •Update FHP,DLR, THWildau •Evaluation of Solr’s Collapse/Expand Q4 ‘25/Q1 ‘26: •Finalize migration of KOBV-Portal Q2 2025: •Evaluation RecordManager •Start migration of KOBV-Portal to VuFind10.1
6 30.09.2025 Kooperativer Bibliotheksverbund Berlin-Brandenburg © 2025#by Ying Wang,#Martin Weigelt# is licensed under#CC BY 4.0 From ALBERT to standard VuFind® ●To migrate our backend especially solr scheme more to standard VuFind®, we decide to create a default VuFind® in our infrastructure Kubernetes for evaluation. ●To achive this goal we containerize VuFind® –Lightweight infrastructure with Alpine-Linux and Lighttpd webserver –Modularization of each part: PHP, Lighttpd, MariaDB, Apache Solr™
7 30.09.2025 Kooperativer Bibliotheksverbund Berlin-Brandenburg © 2025#by Ying Wang,#Martin Weigelt# is licensed under#CC BY 4.0 Orchestration of VuFind® Web-Server Base-Image Alpine-Linux 3.22.1 App Lighttpd Volume - Ports 80,443, 9000 Image-Size* 22 MB PHP 8 Base-Image Alpine-Linux 3.22.1 App php8-fpm Volume - Ports 9000, 3306, 8983 Image-Size* 353 MB MYSQL Base-Image Alpine-Linux 3.22.1 App MariaDB Volume /var/lib/mysql Ports 3306 Image-Size* 285 MB Solr Base-Image Alpine-Linux 3.22.1 App Apache Solr™ Volume /data Ports 8983 Image-Size* 546 MB VuFind Importer Base-Image Alpine-Linux 3.22.1 App OpenJDK JRE Volume - Ports 8983 Image-Size* 478 MB *Unique Size (additional unique 8 MB for shared Alpine-Linux)
8 30.09.2025 Kooperativer Bibliotheksverbund Berlin-Brandenburg © 2025#by Ying Wang,#Martin Weigelt# is licensed under#CC BY 4.0 VuFind® with Kubernetes Namespace VuFind® Instance 1 Pod Webserver & PHP WebServer Service Webserver Ingress Controller ConfigMap (Key-Value-File) Persistent Volume (MariaDB) Secrets (Key-Base64File) PHP Pod Database MariaDB Service Database Pod Solr Service Solr Solr Persistent Volume (Solr) Job Solr-Volume-Init Job Pod Importer Java
9 30.09.2025 Kooperativer Bibliotheksverbund Berlin-Brandenburg © 2025#by Ying Wang,#Martin Weigelt# is licensed under#CC BY 4.0 Try it out ●2 kind of setups for local use: –Docker (possibly works also with Podman) –Kubernetes within Docker ●„k3d is a lightweight wrapper to run k3s(Rancher Lab’s minimal Kubernetes distribution) in docker“ ●Tested with Debian 13 (also in WSL) ●Includes indexing a file of sample-data ●Status: Experimental, proof-of-concept https://git.zib.de/mweigelt/vufind-default-k8s
16 30.09.2025 Kooperativer Bibliotheksverbund Berlin-Brandenburg © 2025#by Ying Wang,#Martin Weigelt# is licensed under#CC BY 4.0 Enrichment of Crossref Data in VuFind® - On the Fly ●Current Crossref Workflow with Albert Library Holdings Information ●OpenAlex Introduction –ORCID, Open Access information, topics, citation & relevance graph –API access via ●email registration (basic) ●authentication key (premium / institutional users) ●Implementation of the On-the-Fly Enrichment in VuFind® ●Alternative Hosting –OpenAlex data in MongoDB for further analysis and usage
17 30.09.2025 Kooperativer Bibliotheksverbund Berlin-Brandenburg © 2025#by Ying Wang,#Martin Weigelt# is licensed under#CC BY 4.0 Full Crossref-to-VuFind® Enrichment Pipeline Crossref Vollabzug / OAI Update Bibliographical Information • • • • ZDB-ID Library Holdings Classifcation ID Others Angereicherte Crossref in MARC XML Solr Cloud Solrproxy Database Indices Solrproxy with access to instance-specifc holding online articles Solrproxy with access to the complete online articles Crossref Enrichment VUFind VUFind VUFind OpenAlex VUFind OpenAlex Connector
18 30.09.2025 Kooperativer Bibliotheksverbund Berlin-Brandenburg © 2025#by Ying Wang,#Martin Weigelt# is licensed under#CC BY 4.0 OpenAlex Datasets Introduction ●OpenAlex Datasets are structured by –Works(Unpaywall data), Instititutions, Topics, Authors –Additional mapping/clustering, e.g. SDGs ●Data sources –Crossref, pubmed, DOAJ, arXiv, etc. ●Our Interest –ORCID, Open Access information, topics, citation & relevance graph –API access via authentication key (premium / institutional users) ●Interesting Facts (via API filter) –OA articles in Crossref: 54.123.886 (≈ 34.95 %) –German National and alliance licenses are not included
19 30.09.2025 Kooperativer Bibliotheksverbund Berlin-Brandenburg © 2025#by Ying Wang,#Martin Weigelt# is licensed under#CC BY 4.0 OpenAlex Extension of KOBV Record Driver ●Driver Architecture –Derived from the VuFind® SolrDefault driver –KOBV record driver serves as the foundation for further kobv drivers –KOBV Crossref record driver extended with an OpenAlex connector ●On-the-Fly Enrichment –occurs only for the records being displayed, not across the entire indexed dataset. ●Additional Metadata Retrieved from OpenAlex –Open-access details –Citation counts –Citation & relevance graphs KOBV Instance Index Crossref Index Interlibrary Loan Index VuFind/ SolrDefault OpenAlex Connector
20 30.09.2025 Kooperativer Bibliotheksverbund Berlin-Brandenburg © 2025#by Ying Wang,#Martin Weigelt# is licensed under#CC BY 4.0 Open-Access Enrichment Example: KOBV Online Articles ●Linking Identifier with Crossref Data –DOI ●The 1st enrichment for result lists with open access informaton –Open-access type –Best OA location / URL –Citation count ●Further Enrichment Possible –ORCID –Citation metrics & topic clusters
21 30.09.2025 Kooperativer Bibliotheksverbund Berlin-Brandenburg © 2025#by Ying Wang,#Martin Weigelt# is licensed under#CC BY 4.0 Open-Access Enrichment Example: KOBV Online Articles
22 30.09.2025 Kooperativer Bibliotheksverbund Berlin-Brandenburg © 2025#by Ying Wang,#Martin Weigelt# is licensed under#CC BY 4.0 Open-Access Enrichment Example: KOBV Online Articles
23 30.09.2025 Kooperativer Bibliotheksverbund Berlin-Brandenburg © 2025#by Ying Wang,#Martin Weigelt# is licensed under#CC BY 4.0 Example: Further Enrichment with Related Works ●Linking Identifier: –OpenAlex ID ●The 2nd Enrichment for Single Result –Related works retrieved –Entire dataset of related works available –Pagination for large relevance graphs –Citation graph also available ●Backward Linking to Library Own Stocks: –DOI
24 30.09.2025 Kooperativer Bibliotheksverbund Berlin-Brandenburg © 2025#by Ying Wang,#Martin Weigelt# is licensed under#CC BY 4.0 Conclusion: On-the-Fly Enrichment in VuFind® ●Pros: –Lightweight and use-oriented –Ranking and filtering affected on the external dataset as well –Possible bidirectional linking between external data sources and library holdings –Benefits from external development –Supports Open Science Initiatives ●Cons –Dependency on external data and associated costs –Increased Frontend/VuFind® processing load –Some search features limited (e.g., faceting)
25 30.09.2025 Kooperativer Bibliotheksverbund Berlin-Brandenburg © 2025#by Ying Wang,#Martin Weigelt# is licensed under#CC BY 4.0 ALBERT VuFind® outlook ●Update to VuFind® 11 ●Migration of our portal frontends ●Migration of our solr scheme towards VuFind®’s standard scheme ●Further evaluation of RecordManager as part of our backend ●Further evaluation of deduplication with Collapse/Expand feature from Apache Solr™