The importance of user in ISD. Do we really teach?

26TH INTERNATIONAL CONFERENCE ON INFORMATION SYSTEMS DEVELOPMENT (ISD2017 CYPRUS) The importance of user in ISD. Do we really teach? J.M. Sánchez-Begines [email protected] University of Seville Seville, Spain M.J. Escalona m[email protected] University of Seville Seville, Spain O. Strutynska [email protected] National Pedagogical Dragomanov University Kiev, Ukraine M. Umryk y[email protected] National Pedagogical Dragomanov University Kiev, Ukraine T. Wojdynsky [email protected] The School of Banking and Management Cracow, Poland F.J. Domínguez-Mayo [email protected] University of Seville Seville, Spain Abstract In 1999, the European Higher Education Area (EHEA) defined the framework that regulates all degrees,including Computer Engineering (CE), within the environment of the European Union. Each country, in turn, regulated each of these studies on the basis of that common framework. In the case of Computer Engineering there are concrete competences that refer to the effective incorporation of user and need for Information Technology (IT) professionals to learn how to manage and work in multidisciplinary environments. However, these competencies are difficult to teach to students. This paper presents a critical view of the way we are teaching future developers of information systems to work with user. Besides, it analyses the consequences that it may bring to future professionals. For this purpose, an experiment at three different universities in Europe has been carried out. This paper also offers some considerations and intends to motivate a future discussion on this topic. Keywords: user in ISD, European learning, Computer Engineering 1. Introduction The European Higher Education Area (EHEA) is the result of the political will of forty-eight countries that, progressively during the last eighteen years, built an area using common tools [1]. It includes the implementation of Bologna Process [2], which lets countries, institutions SÁNCHEZ-BEGINES ET AL. THE IMPORTANCE OF USER IN ISD... and stakeholders of the European area continuously adapt their higher education systems making them more compatible and strengthening their quality assurance mechanisms. Thus, the European Union (EU) has developed several instruments to support transparency and recognition of knowledge, skills and competences to make it accessible to study and work anywhere in Europe [3]. Computer Engineering (CE) is not an exception. All degrees (bachelor, master and doctorate) in CE have to guarantee that students get a set of concrete competencies defined in the EHEA [4] when they finish their studies. Among these competencies, we can distinguish general transversal skills and job specific skills, but both refer to the necessity our students have to improve their social abilities and competencies such as working in multidisciplinary environments, developing communication capabilities or working in international contexts. Our Computer Engineers, as any others, [5] have to develop these social abilities, including communication, social presentation and personal skills [6], and teachers must offer them a suitable learning environment to achieve this goal. In the case of CE and, mainly in Information System Development (ISD) learning, these social capabilities acquire a critical character, mainly in the first phases of the life cycle. Our ISD experts, who are going to work in requirements or analysis phases, or even, managing teams, defining software projects or even validating ISD results, have to learn how to interact with different users, getting diverse argots and moving suitably in different functional scenarios [7][8]. However, since the Chaos report started to be published by the Standish Group in 1994, the impact of user involvement to get a successful project is always one of the most relevant aspects. The last published edition [9], considered this point as one of the third most important1 regarding this topic. According to this report, we can infer that the number of successful projects was quite stable in the last year and again user involvement definition continues being critical. “Why is so complex to get an effective user involvement?” There are several studies trying to respond to this question [10], [11]. Nonetheless, one of the most key issues to solve is: “Are Computer Engineers acquiring capabilities in their degrees to cope with a right user involvement?” Bearing this question in mind, this paper presents a short and critical analysis concerning how our students perceive the importance of user involvement in ISD. For that purpose, we have made a very short review of the experiences carried out in three different universities: US (University of Seville) in Seville, Spain, WSZIB (School of Banking and Management) in Cracow, Poland, and NTDU (National Pedagogical Dragomanov University) in Kiev, Ukraine. The same workshop was given to students of CE at both bachelor and master levels. After that, students sat a test whose result confirmed the necessity to better provide them with these capabilities. The paper is structured as follows. Section 2 presents the workshop and the questionnaire. Besides, it offers a global view of the different interviewed institutions and students. Section 3 summarises the main lessons and limitations of our study and finally, Section 4 states conclusions and future critical work. 2. The experience on the trenches 2.1 The structure of the experience 1 In this paper we consider user involvement definition as the fact that takes place when users are involved in the project decision-making and information-gathering process, including user feedback, requirements review, basic research, prototyping, and other consensus-building tools, following Chaos report definition [9]. ISD2017 CYPRUS Although we refuse to define our experience as a software experiment, we have followed the structure defined by [12] for software experiments. This technique proposes a life cycle based on the definition of objectives, design of the experimentation, execution of experimentation and analysis of results. In Figure 1, extracted from the work of Juristo and Moreno, it can be observed that in the execution of tasks the working hypotheses derive into the experiment, the design of the experiment, results and the final evaluation of the initial hypotheses. Fig. 1. Experimentation process We are conscious that our work has critical gaps to be considered as an experiment. We will discuss them in Section 2.2. However, it is likely to be a good option to follow a scientific method even for detecting our own future improvements. In this sense, we could consider our case as a pseudo-experiment. Thus, our main objectives in this project have been oriented to solve the research questions listed below: RQ-01: How do students in CE perceive the importance of user? RQ-02: Do students in CE consider that skills in user collaboration are required for their future profession? RQ-03: Do students in CE receive good learning to improve relations with users? To answer the aforementioned questions, we have formulated our hypotheses as follows: H1- Students of CE receive very few suitable lessons for developing their capabilities along their studies. H1- Students of CE bet for the fact that users are necessary in the life cycle, but they are not interested in developing their capabilities with users. According to the previously exposed, we have defined a workshop composed of three main parts: 1. Part A- It deals with some general questions asked to students, mainly led to evaluate how they initially conceive the user and its position in ISD life cycle. 2. Part B- It comprises a talk lasting 1,30 hours that offers a global view about requirements principles and techniques grouped in three phases: capture, definition and validation. Thus, the diagram presented in Figure 2 represents the core of the talk. It is obtained from [13] SÁNCHEZ-BEGINES ET AL. THE IMPORTANCE OF USER IN ISD... Fig. 2. Core of the workshop A set of techniques are numbered and discussed with the students in each phase. For instance, Capture involves techniques such as interviews or brainstorming, among others; definition comprises techniques such as use cases or activity diagrams; and validation utilizes reviews or walkthroughs among other techniques. The set of techniques presented are also obtained and defined in [13]. 3. Part C- It consists in a 16 open-question test related to the talk that is given to students to answer individually with the aim to evaluate their opinion. We have applied our experiment to three different environments in order to defined subjects. 1. In November 2015 and 2016, the workshop was executed at the Computer Science School of the US, Spain. It was performed at master level (concretely within the Master of Software Engineering and Technologies). In general, 10 first-year and 7 second-year students participated in the experience. They all shared a common feature: all of them had work experience in CE. 2. In May 2016, the workshop was carried out with 26 students of Bachelor Degree at the Computer Engineering School of WSZIB in Cracow. They were registered in the last course and only one of them had previous work experience. 3. In March 2017, the workshop was executed at the School of Informatics of the National Pedagogical Dragomanov University in Kiev (NPU), Ukraine. This time, the participants were 16 students of Master Degree and 9 of Bachelor Degree. Among them, only three had previous work experience (particularly, 2 students of Master and 1 students of Bachelor). The execution of each workshop followed the same rules in every situation. All sessions were taught by the same teacher, professor Dr. Escalona, who is an expert in requirements engineering and user involvement from Spain and a temporal lecturer in WSZIB and NPDU under international teacher mobility programmes. ISD2017 CYPRUS 2.2 Limitations of the experience In this section we would like to justify why our experience should not be considered as a software experiment. Basically, our experience has some important limitations that should be addressed before presenting learning lessons, even though we have been following the software experiment process described in Figure 1. Firstly, it seems that the number of interviewed subjects was very limited. In fact, although the experience was run in three different countries under the EHEA, which enriched the experience, we gathered very few subjects. Our conclusions could have been more meaningful, if the number of participants had been higher. We could enhance this aspect by repeating our experience in other universities or even, in the same institutions, but in different courses. Secondly, we did not define a suitable form to capture demographical data from participants. In fact, we have their information (because all of them were registered students) but our experience enabled us to discover that other personal data should be considered to get more significant conclusions. Finally, Part A posed some problems. During the experiences, we discovered that this part was one of the most relevant to validate our hypotheses. However, its presentation was quite informal, mainly discussing and debating with students personally, which would have been bettered, if a written form had been used. Nevertheless, it is worth pointing out that the discussion was really exciting. Probably, the best option could mix both paths: to have a discussion and later to hand out each participant a form to answer individually according to his/her experience. Despite these drawbacks, we are looking at this study as a starting point because the previous experiences showed us that this research line is very relevant and should be marked out as a future work. 3. Main learned lessons 3.1 Learned lessons from Part A In the first part of the workshop, that is to say, the initial discussion, the experiences were surprisingly quite similar in the three scenarios. In Part A, the teacher started talking about ISD life cycle. After a global view of classical agile or iterative life cycles, the teacher wrote a set of phases in the blackboard: requirements and analysis, design, implementation, implantation and maintenance and then, she asked two questions: 1. What is the phase where highest salaries are obtained in CE? 2. What is the most expensive phase for an ISD project? The answer was clear for those students without work experience: implementation. Code and programing is the most relevant aspect in their profession and they feel it is the most important in their future lives. Nonetheless, students with work experience had a different opinion. They stated that the best salaries are commonly in requirements and analysis and the most expensive phase is maintenance. In general, the reality in companies is close to the latter idea [14]. When the teacher explained that to the students without experience and asked for the reason why it happened that way, they were normally unconfident. After discussion, they got to the same conclusion as always: working with user is not easy and demands some skill or social capabilities that are not originally taught in CE degrees. Students with work experience in all our scenarios agreed with that idea. Consequently, we can summarise two main conclusions from Part A: C1.- Students in CE degrees are prepared and they focus on programming or technical phases. The relation with users does not seem to be relevant for them. SÁNCHEZ-BEGINES ET AL. THE IMPORTANCE OF USER IN ISD... C2.- This initial orientation changes when they have work experience. Then, they discover their gaps for user relations. 3.2 Learned lessons from Part B Part B was basically executed as a master class. The teacher explained different techniques and presented their advantages and disadvantages to the students. Obviously, 1,5 hours was not enough to provide them with detailed information about those techniques but, at least there was a discussion about their advantages and disadvantages. Table 1 lists the techniques presented in each phase. They were obtained from [13]. Table 1. Techniques presented in each requirements phase Requirements Capture Requirements Definition Requirements Validation Interview JAD (Joint Application Development) Brainstorming Concept Mapping Sketches and Storyboards Uses cases Questionnaire and Checklist Terminology comparison Looking for Information Prototypes Natural languages Glossaries Patterns/templates Scenarios Use cases Formal Languages Prototypes Reviews or Walk-throughs Auditories Traceability Matrix Prototypes Thesaurus In this part of the experience, the students’ feedback was very poor due to the way the class was taught. However, our experience was satisfactory because students participated very actively; they took part in class and discussed issues with the teacher during all the presentation. 3.3 Learned lessons from Part C The last part of the experience, Part C, consisted in handing out each student a test with a set of questions to answer individually. These 16 questions are listed at the end of the paper in the Appendix. Every student had to respond to all questions. Such questions were open, meaning that there had not close or unique answers. However, they were limited to some space in order to get concrete responses. The most relevant question was the first one. 100% of students in all courses in the three universities answered that the user is really important in ISD. Thus, the first conclusion we can take out from this part is: C3.- Students think that users are important in ISD at all levels. In contrast, when we analysed the second question regarding the reason why they were not properly involved in the life cycle, the answers differed completely between students who had previous work experience and students who did not have it. In the former group, two main reasons were highlighted: - Users did not have enough time to meet or to be engaged. They were a very expensive resource in ISD. - Communication with users was difficult because Computer Engineers do not often easily understood their “argot”. ISD2017 CYPRUS In the latter, the questions were responded in a different way. Therefore, the answers can be summed up in two main reasons: - Users did not know what they wanted. - Users did not know enough about ISD, thus they did not manage to explain what they wanted. As it can be observed, their perception changes completely depending on their previous work experiences. Thus, we can conclude: C4.- Students without prior work experiences perceived that users were an inaccessible resource because they were not able to explain themselves. Students with work experience changed their views and thought that they needed to implement their capabilities to understand users. The remaining set of questions, related to concrete techniques, had different responses, among which we can extract some conclusions: C5.- Interviews, prototypes and natural languages were the most attractive and known techniques for students. They also knew use cases, but they did not consider them to be interesting without previous work experience. In questions concerning examples (from 7 to 10), it was clear that students without previous work experience understand the difference with difficulty. Meanwhile, students with previous work experience have a high knowledge, although some of them had never worked on prototyping. Besides, it is interesting to mention that all of the answers were linked to Apps in students without experience whereas in the other group, they were more focused on Web programs. C6.- Prototypes seemed to be known by students, even though the lack of experience made them poorly understand the different ways to use them. The aforementioned statements constitute the most relevant conclusions obtained in this third part. 3.4 Final global considerations In addition to the obtained conclusions, there was a very important aspect in our experience that should be mentioned. Initially, we thought that selecting students from Master and Bachelor degrees could be interesting for our evaluation. However, after the development of the experiences, we discovered that the factual element that affected the perception of the project was students’ previous work experience. The fact that students had not a real perception of the user role until they started to work constitutes a meaningful source to value. If this hypothesis is true, academia has to make a change, since social capabilities required in Computer Engineers are not being developed sufficiently in the current degrees. Obviously, our work is quite limited to confirm that it is a global problem, but it can be contemplated as an alarm that is completely aligned with the results of Chaos Reports from the beginning. Other important questions than can be raised from our experiences are: Should CE work close to users in the future? How many users do they meet during their degrees? Probably, not too much and they are quite limited to subjects like training in companies. SÁNCHEZ-BEGINES ET AL. THE IMPORTANCE OF USER IN ISD... 4. Final conclusions and future work This paper presents a critical view of user involvement in ISD development. The paper starts with analysing the importance of users and the need of our students to acquire social capabilities along their degrees. After that, the paper presents an initiative that, in spite of being based on software experiment techniques [12], could be defined as a pseudo-experiment because it is quite limited, as presented in the paper. This experience offers a first approach to be aware of students’ perception of final users. To conclude, this paper analyses the results of carrying out the same experience in three different universities in Europe. Main conclusions obtained from these experiences show that the hypotheses of the work could be an interesting research line to be considered. It seems that students of CE are lost in the relation with user, independently of whether they are studying Master or Bachelor Degrees, and they really get right capabilities to work with final users when they have work experience. Our case demonstrates that the academia has to review, or at least consider, how they are developing social skills in students. As a future work, we would like to open three different lines: The first one is proposed from this paper. As the preliminary study tries to show, it could be interesting to extend this pseudo-experiment as well as launch a real experiment that study not only the academia, but also business. In [15], the team closely worked with companies in a similar experience and that could be a relevant path to analyse the experience in both senses. The second one tries to solve the problem presented in this paper. In this sense, in the University of Seville, we have included users participation in a final-course module of the Degree [16]. In this module, students work with requirements in a real case with real users. They have to interview users, validate requirements with them and try to simulate capabilities oriented to deal with requirements and final users since the academic background. This experience has been put in practice for the last three academic years providing good results. Therefore, this initiative could be extended to other subjects and other environments. Finally, the last one intends to confirm if some other learning techniques, such as gamification, could be a good approach to improve the social skills that our students require for user involvement in ISD. For that purpose, the idea is to follow experiences like [17][18][19], where games are successfully used for developing capabilities associated with project management and team management in ISD. We aim to find similar solutions to implement the skills regarding users treatment in ISD in our students. Acknowledgements This research has been supported by Pololas project (TIN2016-76956-C3-2-R) and by SoftPLM Network (TIN2015-71938-REDT) of the Spanish Ministry of Economy and Competitiveness. 5. References 1. http://www.ehea.info/. Last accessed 04/2017 2. http://ec.europa.eu/education/policy/higher-education/bologna-process_en. Last accessed 04/2017 3. http://ec.europa.eu/education/policy/strategic-framework/skills-qualifications_en. Last accessed 04/2017 4. https://ec.europa.eu/esco/portal/home. Last accessed 04/2017 5. de Castro, A., Guerra, D., Soto, J. D., Calle, M. G., & García, D. L. (2014, April). Communications Skills in Senior Engineering Students. In The International Scientific Conference eLearning and Software for Education (Vol. 2, p. 125). "Carol I" National Defence University. ISD2017 CYPRUS 6. Nguyen, D. Q. (1998). The essential skills and attributes of an engineer: A comparative study of academics, industry personnel and engineering students. Global J. of Engng. Educ, 2(1), 65-75. 7. Lingard, R., & Barkataki, S. (2011, October). Teaching teamwork in engineering and computer science. In Frontiers in Education Conference (FIE), 2011 (pp. F1- C1). IEEE. 8. Deiters, C., Herrmann, C., Hildebrandt, R., Knauss, E., Kuhrmann, M., Rausch, A., & Schneider, K. (2011, August). GloSE-Lab: Teaching global software engineering. In Global Software Engineering (ICGSE), 2011, 6th IEEE International Conference on (pp. 156-160). IEEE. 9. https://www.infoq.com/articles/standish-chaos-2015. Last accessed 04/2017 10. Kujala, S. (2003). User involvement: a review of the benefits and challenges. Behaviour & Information Technology, 22(1), 1-16. 11. Kujala 1, S. (2008). Effective user involvement in product development by improving the analysis of user needs. Behaviour & Information Technology, 27(6), 457-473. 12. Juristo, N., & Moreno, A. M. (2013). Basics of software engineering experimentation. Springer Science & Business Media. 13. Escalona, M. J., & Koch, N. (2004). Requirements engineering for web applications-a comparative study. J. Web Eng., 2(3), 193-212. 14. Ardis, M., Budgen, D., Hislop, G.W., Offutt, J., Sebern, M., & Visser, W. "SE 2014: Curriculum Guidelines for Undergraduate Degree Programs in Software Engineering". Computer: Vol. 48: pp. 106-109, Nov. 2015. 15. Escalona, M.J., López, G., Vegas, S. García-Borgoñón, L., García-García, J.A., Juristo, N. A Software Engineering Experiments to value MDE in testing. Learning Lessons. Communication in Congress. XXI Conferences on Software Engineering and Databases. Salamanca, Spain. 2016 16. http://www.lsi.us.es/docencia/pagina_asignatura.php?id=115. Last accessed 04/2017 17. Calderón, A., & Ruiz, M. (2015). A systematic literature review on serious games evaluation: An application to software project management. Computers & Education, 87, 396-422. 18. Valencia, D., Vizcaíno, A., García-Mundo, L., Piattini, M., & Soto, J. P. (2016, August). GSDgame: A serious game for the acquisition of the competencies needed in GSD. In Global Software Engineering Workshops (ICGSEW), 2016 IEEE 11th International Conference on (pp. 19-24). IEEE. 19. Lizcano, D., Soriano, J., López, G., Gutiérrez, J.J.. 2017. Automatic verification and validation wizard in web-centred end-user software engineering. Journal of Systems and Software 125 (March 2017), 47-67.