Supporting Usability and Reusability Based on eLearning Standards

Supporting Usability and Reusability Based on eLearning Standards Rocio Garcia-Robles, Josep Blat, Sergio Sayago, Dai Griffiths, Francis Casado, Juanjo Martinez Interactive Technology Research Group, Department of Technology, Pompeu Fabra University [email protected], [email protected], [email protected], [email protected] Abstract The IMS-QTI, and other related specifications have been developed to support the creation of reusable and pedagogically neutral assessment scenarios and content, as stated by the IMS Global Learning Consortium. In this paper we discuss how current specifications both constrain the design of assessment scenarios, and limit content reusability. We also suggest some solutions to overcome these limitations. The paper is based on our experience developing and testing an IMS QTI Lite compliant assessment authoring tool, QAed. It supports teacher centering, which is quite neglected when designing such tools. In the paper we also discuss how to make compatible standards support and user centering in eLearning applications and provide some recommendations for the design of the user interfaces. 1. Introduction Questions and assessments (Q&A) are very commonly used elements in education. The IMS Global Learning Consortium (www.imsproject.org), which can be considered a de facto standardization body for eLearning, has developed some related specifications. IMS-QTILite is one of them, where QTI stands for Questions and Tests Interoperability, and which is a compact subset of IMS QTI ASI. We decided to start our work around QTILite to have a relatively simple but commonly used testbed for pedagogical approaches, reusability and interoperability. It resulted into a simple open-source, multiplatform eLearning application, for editing question and assessments (Q&A) items, QAed, and which binds the IMS QTI Lite specification. From the point of view of teacher centering, the tool is designed to support the teacher’s workflow. While this seems obvious, it is quite frequently forgotten in tools intended to support reusability and interoperability specifications. When the latter goal is promoted, packages usually adopt a very technical terminology close to the specification, and forget the usual workflow and terminology of teachers when preparing the tests. Other tools take the opposite approach, supporting teachers but using proprietary standards. Canvas Learning (available from http://www.imsproject.org/direct/getproducts.cfm) is an example of tool supporting QTI; Hot Potatoes (available from http://web.uvic.ca/hrd/halfbaked/) is an example of Q&A tool with proprietary format. Even further, strong support of reusability is not taken from the point of view of the teachers, but in terms closer to the specifications and far from the practice. In the paper we show how we have departed from these approaches. We show that the main services of the application support the usual workflow of teachers in this context. We also show the services integrating both the workflow and re-usability in terms of the teacher practice, while preserving interoperability. The recommendations for user interface design are developed in terms of patterns , both to formalize them better and to allow a suitable understanding and wider applicability. We concluded that the UI must support teachers’ usual workflow of Q&A preparation and reflect the essential structure of the standard, e.g. by grouping the elements according to their functionality; but the terminology must not be specialized. Meta-tagging and packaging conceptualization should be invisible to the final user, in order to be effective for both, content creators and authors. Moreover, reusability is promoted by supporting several services such as repository, different granularity levels, and domain classification. On the other hand, the IMS QTILite specification only supports multiple-choice questions and limits the rendering form to the “true response” choice from a set of answers. From the pedagogical point of view, this is very limited, as assessment can be performed in a wide variety of educational scenarios. But even the larger QTI ASI specification has limitations for providing appropriate support to common assessment scenarios such as Question Item Banks (QIB), which are basic for supporting reusability in the teacher’s workflow. Question Item Banks are collection of items which can be used to construct assessments through the selection of questions based on various predefined criteria according to the appropriate assessment scenarios envisaged. [1]. While QIB are supported by the specification, important features allowing their sensible use, such as for instance, the overlap exclusion requirement, is not supported. Overlap exclusion means, in simple terms, to make some questions force removal of other questions. This is acknowledged by the IMS QTI ASI, and is intended to be supported in version 2.0 of the specification. Another QIB common requirement is the overlap inclusion. Nevertheless, we claim that the approach intended to provide support for overlap inclusion is not going to allow for true reusability because the specification suggests the use of the so called “section” entity for encapsulating the dependency. We discuss how this approach hinders reusability, by addressing the level of granularity incorrectly, by not allowing the feature to be included in question items. This approach makes it also backward incompatible with the IMS QTI Lite compliant banks, because this specification only supports the question item object, neither sections nor assessments. We suggest and discuss an alternative approach, based on XLink, , which is a W3C specification. So, the main weakness of the packaging approach underlying current versions of IMS QTI specifications is that it cannot support question items dependency, neither inclusion nor exclusion. It can partially support question items inclusion by packaging dependent question into static sections, but constraining the granularity, and thus content reusability. The proposed linking approach supports items dependency by linking items establishing a relationship between them, and therefore avoiding encapsulating them into closed sections. In the next section we discuss the QAed related issues, in the following, our XLinking approach. We conclude summarising the results and indicating some other perspectives. 2. The support of QAed for both teachers’ workflow and standards based reusability QAed1 is a simple eLearning open-source and multiplatform application developed in JAVA for editing Q&A items binding the IMS QTI Lite 1The tool was developed in the framework of the EU funded project SCOPE www.tecn.upf.es/scope,www.tecn.upf.es/gti/leteos/ specification , i.e. it is a tool to develop Q&A. The IMS-QTI Lite compliancy implies a strong orientation towards reusability and interoperability. But another feature is strong teacher support: we think tools should support the usual workflow of Q&A preparation, and the user should not need to know anything about the standard for his/her work. Supporting the principles of conceptual design as defined by [2], the GUI features a multiple-window paradigm in such a way that each window encapsulates information related to only one part of the standard. It also allows users to decide when, and how interact with what information. It supports varying user roles (question editor, assessment editor and tool user), and the standard specification structure. In practical terms, some times users might prefer to edit the questions first, and others might approach first the edition of the assessment. On the other hand, the standard specification defines the assessments as a container of questions and responses and therefore, from the UI perspective, they can be handled as separate entities. The same flexible teacher workflow approach has been adopted for main services such as saving (in PDF, XML, HTML and ZIP formats2), searching (by date, author and category), pre-visualization (in a HTML customizable style), export and import (to/from XML files binding the IMS QTI Lite specification). QAed has been designed according to an authoring oriented approach, trying to keep the specification complexity invisible to the user. By contrast, most of the already created learning authoring tools complying with IMS QTI specifications have GUIs which resemble very closely specification related concepts such as content packaging process and meta-tagging. This approach may be closer to the educational publishing industry way of doing, but it is far away from normal teaching practices. For that purpose, the application was designed taking usage-centered and usability approaches. Trying to converge the usage with the standardization on eLearning, positive results were obtained with both experienced and inexperienced users, who were both able to use the application successfully. Three factors were identified in this success, and are suggested as UI recommendations. Firstly, the interface reflects the essential structure of the standard grouping the elements according to their functionality; standardization requires that the specification elements and their corresponding relationships must be reflected in the GUI design. Secondly, the terminology used is not specialized; usage requires to translate the 2e.g. compressing HTML and attached images terminology and to enlarge the information available in the specification data model. Thirdly, the GUI reflects information supporting teachers’ usual workflow of Q&A preparation, supporting and promoting to reuse, recombine, share and visualize content. Further usability enhancements could come from a customizable user interface, because it may be useful to show or hide certain type of information according to the user profile.; and for support for collaborative work. In the standardization framework, interoperability and reusability are the main promises for promoting the extended use of this kind of specifications. QAed promotes two of the three key issues identified by [3]: granularity and accessibility. Main services promoting both are the repository, the domain classification and the possibility of supporting different granularity levels. The repository is managed by using a folders tree to organize the structure of the assessment, question and responses. Tree elements are folders, subfolders and Q&A. That folder structure is the main local browsing facility, offering a logical hierarchy on which actions can be undertaken. Moreover, keywords can be used to classify Q & A into domain categories. Finally, an assessment scenario can be created by editing questions and then grouping and/or associating them, or vice versa, and so different granularity levels are supported. A shopping basket facility is also available as persistent storage (the user must update/delete explicitly the items in the basket) of Q&A items supporting the user on pre-selecting and reusing content. The third key issue promoting reusability is selfcontained-ness, intended for resources to be reusable in multiple situations. According to some authors “For maximum reuse, resources should be context free: they should not contain information specific to a particular subject discipline” [4]. However, many other authors recognize that “this contradicts the way the teachers normally modify and adapt resources to fit specific teaching situations” [5]. Because of this controversy, the current implementation of QAed leaves the teacher distinguish context from resources. 3. An Xlinking approach to overcome current reusability limitations The IMS QTI Lite specification supports only multiple-choice questions and limits the rendering form to the classical true response from a set of answers (true/false alternative). It is a compact subset of the IMS QTI ASI specifications, which describes the components required to construct the simplest form of an IMS QTI-compliant system. IMS QTI ASI specifications support eight core data object, which are combinations of Assessment, Sections and Question (ASI) items. IMS QTI Lite supports only two of those core data objects [6], and both of them are based on the question item object, i.e. it doesn’t support the assessment neither the section objects. Conceptually then, the only assessment scenario possible is the QIB. There are many requirements in QIB scenarios. Among them, the overlap exclusion requirement has been identified by CAA experts. To avoid similar items appearing in the same test and a mix of questions where one question provides the answer for another is clearly needed. [7]. Nevertheless, that “overlap exclusion requirement” is not supported by any IMS specification, and this fact is explicitly recognized in the IMS QTI specifications [8] In addition, the complementary requirement, overlap inclusion, is only partially supported and the need for further study in new releases of IMS QTI specifications is recognized. The requirement involves different cases: (i)- If item ‘X’ is presented then item ‘Y’ must also be presented. [8]; (ii)- Item ‘Y’ can only be presented if ‘X’ has already been presented [8]; (iii)- Presentation of item ‘Y’ depends on outcome or response of item ‘X’ [8]. Only case (i) is partially addressed by the current specifications. As indicated above, the QTI ASI intended solution suggests the use of the “section” entity for encapsulating the dependency. Nevertheless, this might lead to several problems: 1Encapsulating the question items dependency by structuring question items into nested sections do not promote reusability, because it compromises the granularity level. For instance, if we want to create an assessment with n question items in which every question item depends on the previous one3 , we will need to create an assessment with one section packaging all the question items, or a package with (n1) nested sections, as it is shown in Figure 1. In that case the granularity will be fixed to the assessment level. 3that use case is very frequent in simulated cases, e.g. in medical assessment SECTION ‘1’ ITEM ‘1’ ITEM ‘2’ ITEM ‘n’ Figure 1. Packaging question items 2The IMS-QTILite specification is restricted to question items only, not dealing with sections or assessments. This means that QIB is the only assessment scenario supported. But, on the other hand, it would not be possible to address overlap inclusion as suggested, because sections are needed to package items dependency. Considering that question items dependency is a common requirement to many QIB assessment scenarios, there should be another mechanism for supporting question items dependency directly related to the question item objects, avoiding the encapsulation of the dependency in aggregated structures like section and assessment which are not supported by the QTILite specification. We suggest an alternative solution to be implemented in the next release of QAed, in order to support both overlap exclusion and inclusion requirements, namely to move from a packaging to a linking approach. We propose supporting items dependency by linking items , allowing an item which depends on another to explicitly reference it, and thus establishing a relationship between them. In the packaging approach there does not exist a relationship among individual items. Linking versus packaging would solve the constraints explained. Other benefits of the linking solution would be: 1Supporting assessment knowledge customizable approaches. The linking approach facilitates to establish items relationships depending on the teacher/tutor’s point of view. In fact, e.g. one teacher could consider question items q1 and q2 exclude each other, while other teacher could disagree. Not only exclusion but also inclusion could be dependent on the teacher’s perspective. 2Taking into account diss-aggregation is considered a previous stage to reusing content [9]. The linking approach promotes reusability because the final user does not need to think in terms of how to dissaggregate a whole section. 3It promotes data mining because it is possible to navigate through the linked structure. In practical terms, the linking approach could be supported by using XLink linkbases for gathering together the information of related linked items4. XLink is a W3C specification which allows elements to be inserted into XML documents in order to create and describe links between resources. Linkbases are a type of XLink link by which relational elements are 4We are specially concerned with question items because it is the only core data object supported by IMS QTI Lite, but also sections and assessment could be enlarged to support linking between them. stored separately from the resources they associate. This makes link management easier, it allows linking read-only resources, and it supports describing different views of the items dependency in terms of different linkbases. XLink has some semantic attributes: role,arcrole and title, which describe the meaning of resources within the context of a link. Arcrole or title can be used for describing the type of dependency (exclusion, inclusion and even the type of inclusion) between linked items. The role attribute of every resource linked, or the directionality of the arc (explicitly described using the from and to attributes of the XLink arc element type) can be used to express the order in the inclusion relationship. More than one title could be used for specifying other semantically relevant information related to the inclusion dependency between the linked elements, as illustrated by the following example. Figure 2. qi1 qi2, exclusion and inclusion <xlink:extended xmlns:xlink=”http:// www.w3.org/1999/xlink/ namespace”> <xlink:locator href=”uri_qi1” role=”question_item_01” title=”first question item”/> <xlink:locator href=”uri_qi2” role=”question_item_02” title=”second question item”/> <xlink:locator href=”uri_qi3” role=”question_item_03” title=”third question item”/> <xlink:arc from=”question_item_01” to= “question_item_03” arcrole=exclusion title=”exclusion”> <xlink:arc from=”question_item_02” to= “question_item_01” arcrole=inclusion title=”inclusion”> </xlink:extended> Different types of inclusion dependency could be specified. For example, in Figure 2, qi2 has an inclusion dependency in relation to qi1, but this could mean at least two things in a QIB scenario: (case 1) qi2 can appear only if qi1 has been also selected, or (case 2) if qi1 appears then qi2 must also appear. Case 2 is solved by the current version of the IMS-QTI specifications by packaging qi1 and qi2 in a section, while case 1 is not supported anyway, i.e. the overlap exclusion scenario is not supported by current version of IMS-QTI. On the other hand, there are two potential disadvantages related to the linking solution. First one is the need of using unique resource identifier (URI) qi1 qi2 qi3 Exclusion Inclusion for each item. This is already solved by adopting the URI identifying naming convention recommended by the IMS specifications [11]. QAed automatically generate unique identifiers for items, reducing the cognitive load on the user. Second disadvantage is the need of solving cyclic dependency, if exists, in runtime. XLink specification addresses that issue in the following terms: “An application should maintain a list of extended links retrieved as a result of processing a linkbase, and should not retrieve duplicate resources or links in the case where a cyclic dependency exists” Therefore, both issues could be better considered as already solved constraints than disadvantages. We conclude that linking can be a feasible solution, as well as the packaging solution, but it enlarges packaging capabilities by supporting inclusion and exclusion requirements between items. 4. Conclusions and perspectives We have discussed two issues for eLearning tools, usability and reusability, arising from our experience developing a standards compliant tool for Q&A authoring5, and have described some of the lessons learned which might have wide applicability. We have not discussed some interoperability problems of current specifications, which have appeared when implementing QTILite compliancy, and which seem to be quite applicable to other eLearning specifications. We intend to do this in a future paper. We have not discussed other improvements we intend to support the use of scientific notation, currently absent. In some fields like Maths, this would mean to use a standard oriented solution like MathML, a product of the W3C Math working group, which is a low-level specification for describing mathematics as a foundation for the inclusion of mathematical expressions in Web pages. A more significant aspect is related to the need of strengthening the pedagogical component in the assessment field, as indicated in [10] which remarks the weaknesses of IMS QTI specifications in order to describe advanced assessment scenarios. Peer to peer, self-assessment or groupwork are not supported. If we consider that assessment should be integrated in the global learning process, then other IMS specifications could be used, such as the recent IMS Learning Design. But when using those types of pedagogically oriented specifications, we think that there is a need for 5Further analysis of use of the QAed tool, including a further comparison to other available tools has been undertaken. For paper page limitations it was not able to include that information in the current paper. an ontological solution supporting assessment experiences in a broad sense. 5. References [1] Bull, J., and Dalziel, J., Reusing online resources: A sustainable approach to e-learning, Kogan Page, UK, 2003. [2] Norman, D.A., The Psychology of everyday things, Basic Book, NY, 1998. [3] Duncan, C., Reusing online resources: A sustainable approach to e-learning, Kogan Page, UK, 2003. [4] Naeve, A., Conceptual Navigation and Multiple Scale Narration in a Knowledge Manifold, Kungl Tekniska Hogskolan, Sweden, 1999. [5] Littlejohn, A., Reusing online resources: A sustainable approach to e-learning, Kogan Page, UK, 2003. [6] IMS QTI Lite v1.2, Figure 2.2., pp. 8. [7] Maughan, S Peet, D Willmott “A On-line formative Assessment Item banking and learning Support”, CAA Conference Proceedings, Loughborough University, 2001. [8] IMS QTI ASI Selection and Ordering v1.2, pp 9-12. [9] Koper, R., Reusing online resources: A sustainable approach to e-learning, Kogan Page, UK, 2003. [10] N. Sclater, B. Low and N. Barr, “Interoperability with CAA: does it work in practice?” , Proceedings of the CAA Conferences, Loughborough University, 2002. [11] IMS QTI Lite, pp. 36.