Automated generation of balanced sets using Discuit: A tool description and usage example

Discussion With Discuit you can • split your list of materials in multiple balanced sets - based on clustering algorithms;² • obtain automatic set comparisons (statistics); • choose between a web app or a Python package. With the described use case, we show that this also works for actual/complex data. Contact me if you have data-sets I can run this on as further use cases for an upcoming paper! Use case - output Discuit produces the following output: • File with added set distribution (see below); • Statistics for all variables (set comparison, see right). Output file: • Discuit produced two sets that differ in none of the 10 variables controlled for. • It took ~ 1 minute to do so. Verb Log10 freq4 AoA5Concr.6Instr. Transit. Irreg. Past Pre1 Past Pre1 Future Pre2 Past Pre2 Future Set Box 2.20 4.60 3.81 0 0 0 1 0 1 1 2 Break 2.35 5.94 3.71 0 1 1 0 0 1 1 2 Brush 1.15 4.12 4.54 1 1 0 0 0 0 1 1 Build 2.47 4.64 3.71 0 1 1 1 0 1 1 1 Catch 2.32 5.20 4.11 0 1 1 0 0 0 1 2 Cheer 1.35 5.75 2.77 0 0 0 0 0 1 0 2 Clean 1.84 4.22 3.81 1 1 0 1 0 0 1 1 Cook 2.15 4.88 4.32 1 0 0 0 1 0 0 2 … Automated generation of balanced sets using Discuit: A tool description and usage example Dörte de Kok Center for Language and Cognition, University of Groningen, The Netherlands ✉ d.a.de.k[email protected] Get this poster! Introduction Problem: • For experiments or treatment studies balanced sets of items are needed. • Controlled for multiple linguistic variables… • For each participant individually. • Don’t reduce continuous variables to categories (use e.g. actual frequency values). Solution so far: • Make 1 long list. • Use categories such as high vs low frequency. • Split by hand. Proposed solution: • New tool: Dynamic item set clustering UI tool (Discuit).¹ • Works as Python package and as a web app. • Based on k-means and k-mode clustering (from the scikit-learn package).² Use case - method Study into treatment efficacy:³ • Participant DTR: mild aphasia, treatment for verb inflection. • 107 experimental items (filling in verbs in sentences: past & future tense). • Needed to be split in treated and untreated items. • Balanced variables: Log10 frequency4, AoA ratings5, concreteness ratings6, transitivity, instrumentality, regularity of past tense form, accuracy in 2 sessions before treatment (for both past and future). • File with all items and variables passed to Discuit. Input file: Verb Log10 freq4 AoA5Concr.6Instr. Transit. Irreg. Past Pre1 Past Pre1 Future Pre2 Past Pre2 Future Box 2.20 4.60 3.81 0 0 0 1 0 1 1 Break 2.35 5.94 3.71 0 1 1 0 0 1 1 Brush 1.15 4.12 4.54 1 1 0 0 0 0 1 Build 2.47 4.64 3.71 0 1 1 1 0 1 1 Catch 2.32 5.20 4.11 0 1 1 0 0 0 1 Cheer 1.35 5.75 2.77 0 0 0 0 0 1 0 Clean 1.84 4.22 3.81 1 1 0 1 0 0 1 Cook 2.15 4.88 4.32 1 0 0 0 1 0 0 … Discuit Either use Discuit with the web application… … or install the Python package and run locally. https://discuit.streamlit.app https://pypi.org/project/discuit Try Discuit app Try Discuit package References 1 De Kok, D. (2023). Discuit (v0.2.1). Zenodo. https://doi.org/10.5281/zenodo.7839874 2 Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., & Duchesnay, E. (2011). Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12(85), 2825-2830. 3 Cuperus, P. (2023). Aphasia therapy software: Research, development, and implementation [Doctoral Dissertation]. University of Groningen. https://hdl.handle.net/11370/419a2d92-877e49d1-8b69-089383800088 4 van Heuven, W. J. B., Mandera, P., Keuleers, E., & Brysbaert, M. (2014). Subtlex-UK: A New and Improved Word Frequency Database for British English. Quarterly Journal of Experimental Psychology, 67(6), 1176–1190. https://doi.org/10.1080/17470218.2013.850521 5 Kuperman, V., Stadthagen-Gonzalez, H. & Brysbaert, M. (2012). Age-of-acquisition ratings for 30,000 English words. Behavior Reseach Methods, 44, 978–990. https://doi.org/10.3758/ s13428-012-0210-4 6 Brysbaert, M., Warriner, A.B., & Kuperman, V. (2014). Concreteness ratings for 40 thousand generally known English word lemmas. Behavior Research Methods, 46, 904-911. https://doi. org/10.3758/s13428-013-0403-5 Use case - statistics a: Pearson Chi-square tests of independence b: Krusall-Wallis Anova Variable Set 1 Set 2 Test statistic p TransitivityaIntransitive Transitive 26 28 26 27 X²(1) = 0.000 1.000 Instrumentalitya Non-instr. Instrumental Missing data 39 14 1 40 12 1 X²(2) = 0.157 .942 Past tense formaRegular Irregular 35 19 31 22 X²(1) = 0.225 .636 Pretest 1 - PastaIncorrect Correct 32 22 32 21 X²(1) = 0.000 1.000 Pretest 1 - FutureaIncorrect Correct 44 10 43 10 X²(1) = 0.000 1.000 Pretest 2 - PastaIncorrect Correct 15 39 14 39 X²(1) = 0.000 1.000 Pretest 2 - FutureaIncorrect Correct 27 27 27 26 X²(1) = 0.000 1.000 FrequencybMean (SD) 1.947 (0.592) 1.998 (0.653) X²(1) = 0.058 .810 AoAbMean (SD) 5.091 (1.132) 5.283 (1.314) X²(1) = 0.845 .358 ConcretenessbMean (SD) 3.797 (0.591) 3.796 (0.564) X²(1) = 0.049 .824