Exploring the NFDI4Earth portal OneStop4All - Recommendations for conducting a feedback workshop

NFDI4Earth is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – project number: 460036893 NFDI4Earth Report – Measure 2.1 OneStop4All Exploring the information portal OneStop4All Recommendations for conducting a feedback workshop Authors: Ivonne Anders, Marie Ryan, Sibylle Hassler Task Area: 2 2025-07-31 Executive Summary This documentation describes a workshop that serves as a blueprint and should therefore be repeatable. It can also be used to collect continuous feedback in a similar way. First, the objectives of the workshop are defined and the methodology itself is described. The selection of the target group, the general setup of the workshop and the process are described. This is followed by information on documentation during and after the workshop. This description is kept as general as possible and can therefore also be transferred in part to other applications. Furthermore, workshops that have been held are documented and concrete results are shown using an example. NFDI4Earth is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – project number: 460036893 Content WORKSHOP OBJECTIVES 3 METHODOLOGY 3 (1) Participant Selection 3 (2) Workshop Setup 3 (3) Workshop Structure 4 DOCUMENTATION OF OBSERVATIONS AND RESULTS 5 MATERIALS 6 LIST OF WORKSHOPS CONDUCTED BASED ON THESE MATERIALS TO DATE: 6 EXAMPLE 8 NFDI4Earth is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – project number: 460036893 Workshop Objec.ves The aim of this workshop is to introduce and evaluate the OneStop4All portal—one of the flagship products of the NFDI4Earth consortium. OneStop4All serves as a centralized, web-based access point designed for the Earth System Sciences (ESS) community to discover and explore a broad range of research data management (RDM) resources1. Key features include: • Unified search across distributed ESS resources such as data and software repositories, RDM articles, educational content, and training materials2 • User-centered design, supporting different skill levels through guided, hierarchical filters and annotations, based on detailed user scenarios1 • Integrated support, where questions the portal cannot answer are forwarded to the User Support Network (USN) to ensure continued assistance3 The workshop is structured to gather firsthand feedback on usability, navigation, and content clarity. Methodology (1) Participant Selection In the early development phase of a digital product like the OneStop4All portal, it is strongly recommended to begin testing with a small group of friendly users from the immediate work environment. These individuals should be open-minded, patient, and have a basic understanding that the portal is still under construction and not yet fully functional. Working with such a "friendly user group" allows the project team to gather valuable, constructive feedback early on—without the risk of giving a negative first impression to a more critical or external audience. At the same time, it helps build initial awareness and support for the product in a safe, supportive setting. It’s important to remember: Workshops are not only about evaluation; they also serve as a communication and outreach opportunity. That’s why participant selection should be strategic—starting with trusted internal colleagues during development, and expanding to the real target audience once a stable, convincing version is available (2) Workshop Setup The workshop can be conducted flexibly—in person, online, or in a hybrid format. It is recommended that parts of the workshop be conducted in teams of two, as this allows feedback to be documented directly within the team. The interaction encourages engagement with the product. 1 h#ps://www.nfdi4earth.de/2facilitate/onestop4all 2 Anders, I., Braesicke, P., Degbelo, A., Hassler, S. K., Henzen, C., Kleeberg, U., Ryan, M., and Thiemann, H.: Towards a userfriendly NFDI4Earth OneStop4All portal to support researchers in Earth System Sciences in Germany, EGU General Assembly 2024, Vienna, Austria, 14–19 Apr 2024, EGU24-17279, https://doi.org/10.5194/egusphere-egu24-17279, 2024. 3 h#ps://www.nfdi4earth.de/2facilitate/user-support-network NFDI4Earth is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – project number: 460036893 (3) Workshop Structure Before start: target group assessment Before the workshop actually begins, the participants' background information is collected. The open survey that has already been developed is used for this purpose. It contributes to the general survey of the NFDI4Earth target groups. The link to the survey is: https://bildungsportal.sachsen.de/umfragen/limesurvey/index.php/415965?lang=en Part 1: Free Exploration Starting the workshop by providing participants with nothing more than the URL to the OneStop4All portal—without any additional explanation or guidance—offers several key advantages. First and foremost, it allows for an unfiltered first impression: users interact with the portal without any prior knowledge, revealing valuable insights into usability, clarity of content, and intuitive navigation. This approach also closely mirrors a realistic usage scenario, where users may come across the portal via a link or search engine without prior instruction. Additionally, this method encourages exploratory behavior, allowing participants to move through the portal freely. As a result, the feedback is often more authentic and spontaneous, shedding light on how users actually engage with the content, what paths they follow, and where they may encounter confusion. It also helps surface critical areas for improvement—such as a vague landing page or unclear structure—early in the development process. Overall, this open-ended entry point provides a strong foundation for meaningful discussion and targeted refinement of the portal. While this open-ended approach offers useful insights, it also presents some challenges. Without guidance, participants may feel uncertain or frustrated, especially if the portal is still incomplete or lacks clarity. This can lead to feedback that reflects current limitations rather than the portal’s potential. Additionally, with no clear task, some users may only explore superficially, missing key features. There’s also a risk that participants misjudge the portal as a finished product if its early-stage status isn’t clearly communicated. To address this, it's important to clarify expectations in advance and follow up the free exploration with guided tasks and reflection. Part 2: Conducting Task-Based Testing Task-based testing is a usability evaluation method where users are given specific tasks to complete using a product or portal. Instead of simply exploring freely, participants follow realistic scenarios that reflect typical user goals. This approach helps identify usability issues by observing how effectively and efficiently users can accomplish key tasks. When designing task-based tests, it is recommended to prepare 3 to 10 well-defined tasks depending on the test duration (usually 30–60 minutes). Tasks should be clear but not overly detailed—enough to guide users toward a goal without telling them exactly how to complete it. For example, rather than instructing every click, a task might be: “Book a consultation appointment.” This encourages natural user behavior and reveals potential navigation or interaction problems. NFDI4Earth is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – project number: 460036893 Advantages of task-based testing include practical insights into real user behavior, clear identification of pain points, and easy comparison across users. However, there are some disadvantages, such as the risk of limiting creativity if tasks are too scripted, potential variability in user approaches complicating analysis, and the need for careful task design to cover essential functions. Part 3: Closing Discussion A closing discussion is key to gather user impressions and clarify questions after testing a portal. There are several effective ways to structure it: An open feedback session lets participants share their experiences freely, offering authentic insights but sometimes lacking focus. A more structured option is a guided question round, where specific questions help collect focused feedback, though it may limit spontaneity. The Plus-Delta method asks users to list what they liked (“Plus”) and what needs improvement (“Delta”), providing clear and easy-to-analyze input. For deeper engagement, a prioritization exercise lets participants vote on the most important issues, highlighting key user concerns but requiring more time. Alternatively, there are also visual tools such as storyboarding or journey mapping, or written surveys can be used to obtain anonymous feedback. Choosing the right format depends on goals, time, and audience—mixing methods can also enhance feedback quality. Documenta.on of observa.ons and results To effectively document observations and results during usability tests or workshops, a mix of qualitative and quantitative methods can be used. One common approach is taking observation notes or session logs, where observers write down key behaviors, comments, and issues as they happen. Video or audio recordings offer a more detailed capture of user behavior and can be reviewed later, though they require user consent and time to analyze. There is also the option of working with usability labs, which are equipped with the appropriate equipment and knowledge for this type of user feedback. Capturing screenshots during important moments can help illustrate specific problems or successes in reports. Using task completion grids allows teams to track whether users completed tasks successfully, how long they took, and whether they needed help—offering a more structured, comparable view. Collecting user quotes is another powerful method, highlighting emotional reactions or valuable insights in the users’ own words. In hybrid formats, participants can be asked online to express their thoughts in the chat. With the participants' permission, the statements made can also be evaluated. After the sessions, teams often apply affinity mapping or thematic analysis to cluster observations into key themes, such as usability issues, confusing terminology, or navigation problems. For a formal summary, a usability report compiles all findings, including quotes, screenshots, and actionable recommendations. In addition, written feedback forms can be handed out to participants for structured, often anonymous input. Finally, tagging observations with labels like #error or #positive can help with filtering and organizing findings for analysis. In the case of OneStop, users of the portal have the option of contacting a user support (as a User Support Network with a ticket system and delayed written response) via a form if they have any questions. Participants can also use this to provide feedback directly. NFDI4Earth is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – project number: 460036893 In the case of self-documentation, it is recommended to form teams of two among the participants and to suggest that, for example, tickets for feedback in the USN or other written documentation be submitted immediately. By combining several of these methods, teams can ensure a well-rounded, insightful evaluation that supports concrete design improvements. Materials Material 1: Slides Anders, I., Hassler, S., & Ryan, M. (2025). Collecfng feedback on NFDI4Earth OneStop4All - Slides. Zenodo. hhps://doi.org/10.5281/zenodo.16737775 Material 2: Tasks Hassler, S., Ryan, M., & Anders, I. (2025). Collecfng feedback on NFDI4Earth OneStop4All - Tasks. Zenodo. hhps://doi.org/10.5281/zenodo.16737525 Target group assessment: https://bildungsportal.sachsen.de/umfragen/limesurvey/index.php/415965?lang=en List of workshops conducted based on these materials to date: Workshop 1 Workshop 2 Workshop 3 Date 27.05.2025 10.07.2025 14.07.2025 Time / Duration 40min 60min 60min Participants (Roles) Data Stewards, Developer, Meteorologists/Climatol ogists/Oceanographer by training, Library Scientists, Science Communicator Scientists Scientists Number of Participants 22 7 1+3 Organization / Dept. DKRZ, Data Management Department KIT, Institute for Water and Environment (IWU) KIT, Institute for Meteorology and Climate (IMK) Workshop Topic Feedback NFDI4Earth OneStop4All Feedback NFDI4Earth OneStop4All Feedback NFDI4Earth OneStop4All Format hybrid On-site On-Site Methods Used Task-based testing, open discussion Task-based testing, open discussion Task-based testing, open discussion Workshop Goal Identify usability pain points; Understand content process Identify usability pain points; Understand content process Identify usability pain points; Understand content process NFDI4Earth is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – project number: 460036893 Key Findings Users struggle with menu depth Need for editorial templates Confusion over permission levels Documentation Link See USN-Feedback and M2.1 feedback collection See USN-Feedback and M2.1 feedback collection See USN-Feedback and M2.1 feedback collection Facilitator Ivonne Anders Sibylle Hassler Sibylle Hassler Tools Used Zoom, Notes, USN Notes, USN Notes, USN NFDI4Earth is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – project number: 460036893 Example Feedback-“Mini”-Workshop at DKRZ Datamanagement Departement 27.05.2025 (The mini workshop was held as part of the weekly meeting of the Data Management department at the DKRZ.) Workshop Framework • Type of workshop: hybrid • Number of par3cipants: 12 online, 10 onsite • Background Par3cipants: Data Stewards, Developer, Meteorologists/Climatologists/Oceanographer by training, Library ScienSsts, Science Communicator • Dura3on: 40min • Documenta3on: via Moderator taking notes, via Chat-funcSon of the VCTool, via USN-tickets Part 1: Free Exploration (cp. slide 1 in XX) Participants were asked to open their preferred browser and enter the URL of the portal. They then had seven minutes to try out the portal without any further instructions, etc. Questions to the participants afterwards: • Q1: Do you know, what is the portal about? • Q2: What did you klick on first? What did you do first? Answers to Q1: “The portal is offering access to data catalogues, tools and learning materials.”, “Website for everything”, “a portal for scientists interested in “earth science” collaboration”, “All sorts of information and resources for climate / earth system sciences”, “Digital resources”, “It’s a search engine for data + tools and how to use them (learning section)”, “Portal to collect and find resources (info About trusted repos, (meta)data Access and Training materials) for nfdi4earth community.” Q2: Q2-1“About NFDI4Earth (Second: User Support)”, “About NFDI4Earth”(6 participants), “I clicked first on “Discover datasets and repositories” and “Take a guided repository search””, “I browsed through the datasets first”, “Main page and going through the menu” “I searched for “temperature” first” “I clicked first on “Discover datasets and repositories" Q2-2 “Tried to ‘Improve your skills and start learning’” NFDI4Earth is funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) – project number: 460036893 After "Discover datasets and repositories" I have tried to publish my dataset 6 participants online and 2 on-site got lost after clicking the “About NFDI4Earth”-link. They did not recognize they left the OneStop4All. Part 2: Specific Tasks The participants performed the specified tasks one after the other. They spoke aloud what they observed or described, and in the case of online participation, they also did so in the chat. Furthermore, some participants also submitted direct inquiries to the USN using a form. No participant was able to achieve a satisfactory result in task 1. Particular emphasis was placed on the lack of filter options, but also on incorrect results (incorrect content, which led to misleading results). Task 2 was individual. Participants thus had the opportunity to find and try out different resources. Some questions were asked about information on the EduPortal, but incorrect content was also detected in this way. Part 3: Closing Discussion In the final discussion, participants were asked what they particularly liked about the portal. In general, they praised the fact that the portal is set up, that it offers a wealth of content, and that it can be navigated on a mobile phone. Areas for improvement were identified in particular in the content, the search function, and the reduction of results through filtering. It was noted that the portal is geared less toward scientists and more toward service providers. All comments and findings made verbally or in the chat that require action on the part of content management or technical design were subsequently recorded transparently via the USN.