XAI-FUNGI: Dataset from the user study on comprehensibility of XAI algorithms
Abstract
XAI-FUNGI: Dataset from the user study on comprehensibility of XAI algorithms We present the dataset which was created during a user study on evaluation of explainability of artificial intelligence (AI) at the Jagielloninan University as a collaborative work of computer science (GEIST team) and information sciences research groups. The main goal of the research was to explore effective explanations of AI model patterns to diverse audiences. The dataset contains material collected from 39 participants during the interviews conducted by the Information Sciences research group. The participants were recruited from 149 candidates to form three groups that represented domain experts in the field of mycology (DE), students with data science and visualization background (IT) and students from social sciences and humanities (SSH). Each group was given an explanation of a machine learning model trained to predict edible and non-edible mushrooms and asked to interpret the explanations and answer various questions during the interview. The machine learning model and explanations for its decision were prepared by the computer science research team. The resulting dataset was constructed from the surveys obtained from the candidates, anonymized transcripts of the interviews, the results from thematic analysis, and original explanations with modifications suggested by the participants. The dataset is complemented with the source code allowing one to reproduce the initial machine leaning model and explanations. The general structure of the dataset is described in the following table. The files that contain in their names [RR]_[SS]_[NN] contain the individual results obtained from particular participant. The meaning of the prefix is as follows: RR - initials of the researcher conducting the interview, SS - type of the participant (DE for domain expert, SSH for social sciences and humanities students, or IT for computer science students), NN - number of the participant File Description SURVEY.csv The results from a survey that was filled by 149 participants out of which 39 were selected to form a final group of particiapnts. SURVEY_en.csv Content of the SURVEY translated into English. CODEBOOK.csv The codebook used in thematic analysis and MAXQDA coding QUESTIONS.csv List of questions that the participants were asked during interviews. SLIDES.csv List of slides used in the study with their interpretation and reference to MAXQDA themes and VISUAL_MODIFICATIONS tables. MAXQDA_SUMMARY.csv Summary of thematic analysis performed with codes used in CODEBOOK for each participant PROBLEMS.csv List of problems that participants were asked to solve during interviews. They correspond to three instances from the dataset that the participants had to classify using knowledge gained from explanations. PROBLEMS_en.csv Content of the PROBLEMS file translated into English. PROBLEMS_RESPONSES.csv The responses to the problems for each participant to the problems listed in PROBLEMS.csv VISUALIZATION_MODIFICATIONS.csv Information on how the order of the slides was modified by the participant, which slides (explanations) were removed, and what kind of additional explanation was suggested. ORIGINAL_VISUZALIZATIONS.pdf The PDF file containing the visualization of explanations presented to the participants during the interviews ORIGINAL_VISUZALIZATIONS_EN.pdf Content of the ORIGINAL_VISUZALIZATIONS translated into English. VISUALIZATION_MODIFICATIONS.zip The PDF file containing the original slides from ORIGINAL_VISUZALIZATIONS.pdf with the modifications suggested by the participant. Each file is a PDF file named with the participant ID, i.e. [RR]_[SS]_[NN].pdf TRANSCRIPTS.zip The anonymized transcripts of interviews for each given participant, zipped into one archive. Each transcript is named after the particiapnt ID, i.e. [RR]_[SS]_[NN].csv and contains text tagged with slide number that it related to, question number from QUESTIONS.csv, and problem number from PROBLEMS.csv. The detailed structure of the files presented in the previous Table is given in the Technical info section. The source code used to train ML model and to generate explanations is available on Gitlab