Social welfare professionals willing to participate in client information system development : Results from a large cross-sectional survey

This is a self-archived version of an original article. This version may differ from the original in pagination and typographic details. Author(s): Title: Year: Version: Copyright: Rights: Rights url: Please cite the original version: CC BY-NC 4.0 https://creativecommons.org/licenses/by-nc/4.0/ Social welfare professionals willing to participate in client information system development : Results from a large cross-sectional survey © 2022 Taylor & Francis Accepted version (Final draft) Martikainen, Susanna; Salovaara, Samuel; Ylönen, Katri; Tynkkynen, Elina; Viitanen, Johanna; Tyllinen, Mari; Lääveri, Tinja Martikainen, S., Salovaara, S., Ylönen, K., Tynkkynen, E., Viitanen, J., Tyllinen, M., & Lääveri, T. (2022). Social welfare professionals willing to participate in client information system development : Results from a large cross-sectional survey. Informatics for Health and Social Care, 47(4), 389-402. https://doi.org/10.1080/17538157.2021.2010736 2022 Social welfare professionals willing to participate in client information system development – Results from a large cross-sectional survey. Susanna Martikainen PhD University of Eastern Finland, Kuopio, Finland Samuel Salovaara, M.Soc.Sc University of Lapland, Rovaniemi, Finland Katri Ylönen, Lic.Soc.Sc. University of Jyväskylä, Jyväskylä, Finland Elina Tynkkynen, M.Soc.Sc University of Jyväskylä, Jyväskylä, Finland Johanna Viitanen, D.Sc. (Tech), Professor of Practice Aalto University, Espoo, Finland Mari Tyllinen, M.Sc (Tech) Aalto University, Espoo, Finland Tinja Lääveri, MD, PhD Inflammation Center, Infectious Diseases and Meilahti Vaccine Research Center MeiVac, University of Helsinki and Helsinki University Hospital, Helsinki, Finland Corresponding author Susanna Martikainen susanna.mar[email protected] Human-centered design methods should be implemented throughout the client information system (CIS) development process to understand social welfare professionals’ needs, tasks, and contexts of use. The aim of this study was to examine Finnish social welfare professionals’ experiences of participating in CIS development. A national cross-sectional web-based survey on the CIS experiences of social welfare professionals (1145 respondents) was conducted in Finland in spring 2019. This study focused on statements concerning the experiences of end users with CIS development and participation. The results are reported by professional and age groups. Half (50%) of the 1145 respondents had participated in CIS development. Half (56%) knew to whom and how to send feedback to software developers, but most (87%) indicated that changes and corrections were not made according to suggestions and quickly enough. The most preferred methods of participation were telling a person in charge of information systems development about usage problems (53%) and showing developers on site how professionals work (34%); 19% were not interested in participating. Social welfare professionals are willing to participate in CIS development, but vendors and social welfare provider organizations are underutilizing this resource. Social welfare informaticists are needed to interpret the needs of end users to software developers. Keywords: Human-centred design, Information systems, Social welfare professionals, Social care, Social work, Client information system, IT system development, Participatory methods; Survey; Informaticist; Subject classification codes: Introduction A thorough understanding of users, their needs, tasks, and contexts of use is necessary for the successful development and implementation of information systems (IS) with good usability1. Human-centered design (HCD) approach contributes to the success of IS by improving software quality and increasing user satisfaction2-7. Early focus on users and end-user involvement are particularly important in the case of complex systems that are used for complicated contexts and tasks,5 such as those in social welfare or healthcare. However, the demanding task of developing IS can be undermined by poor communication and a failure to identify clear responsibilities for various stakeholders, such as social welfare professionals, managers, IT staff, and IS vendors8. HCD has also been suggested as a potential approach to achieve organizational innovation and change9. Surveys, usability tests, and interviews are the most commonly used HCD methods for involving potential end-users in IS development 1,10-11. Moreover, developers may observe and interview end users on site 12. Larger numbers of end users can be engaged in focus groups13. End-user feedback, such as system error reporting, can also be collected with dedicated web-based systems 14,15. In healthcare, many organizations train and employ physician or nurse informaticists who engage in both clinical work and health information system (HIS) development 16-20. However, reports on such roles in social welfare are largely lacking, even though many authors have highlighted the importance of social welfare informaticists who actively participate in the design of technology that will shape, guide, and support their practice of social work21-23. In Finland, IS coverage in healthcare and social welfare has been regularly monitored since the beginning of the 2000s 24. While practically all public and major private healthcare providers have already stored patient data electronically since 2007 and integrated to the national Patient Data Repository (“Kanta”) services since 2015, 24- 26 progress has been slower in social welfare. Only in 2017 did the availability of electronic client information systems (CIS) in public social welfare organizations exceed 90% 24. However, of the 3,971 units registered as private sector social service providers and mainly providing institutional care for the elderly or the disabled, or in child welfare, only half reported electronically storing the majority of client data 24. Perhaps as a consequence of the slower progress in social welfare in comparison with healthcare, the user experiences of physicians have been monitored nationally since 2010 and the experiences of nurses since 2017 27-29, but the first national survey among social welfare professionals was only conducted in 2019 30. Information technology is seen as a means to transform the social and healthcare system 24. In Finland, the strategic aims are to ensure the access of social welfare and healthcare professionals to information systems that support their work, refinement of information and knowledge management, and interoperable and modular architecture 31. This strategy has pushed many social welfare organizations to develop and update their CIS. Moreover, some IS transformation projects seek to combine social welfare and healthcare data into a shared information system 32-34. In recent years, interest in end-user experiences of CIS has grown 35-38. These studies have highlighted usability problems and experiences of managerial and reporting needs ruling over the needs of frontline professionals 8, 35-39. However, research into the experiences of social welfare professionals while participating in CIS development is scarce; the few, mainly qualitative ethnographic investigations have been conducted in Australia, England, New Zealand, and Scotland 21, 35, 36, 40-42 . These studies have only comprised a small number of social welfare professionals and they mainly focus on their user experiences rather than seek to comprehend how the end users could be engaged in CIS development. In a study conducted among physicians and nurses in Finland, the end users were dissatisfied with their ability to influence HIS development 43. While some studies suggest that younger healthcare professionals are more contended with the usability of HIS than the older ones. 44,45, younger age has not been associated with increased satisfaction among Finnish healthcare professionals. Similarly, younger clinicians have not been more contended with the vendors’ responsiveness than the older; however, they are more willing to participate 43. The aim of this study was to explore the experiences of end users participating in CIS development in a large Finnish national survey conducted among social welfare professionals. The research questions were as follows: ● RQ1. What experiences do social welfare professionals have with giving feedback to CIS developers? ● RQ2. Have they participated in CIS development? ● RQ3. How would they prefer to participate in CIS development? Central concepts Since occupational positions in the field of social work and social welfare vary between countries depending on socio-political contexts and other factors, we chose to apply a generic concept of a social welfare professional when describing the respondents, regardless of their occupational position (e.g., social workers, social counselors, or practical nurses). In Finland, licensed social workers have university-level master’s degrees in either the social sciences or political sciences. Social counselors have bachelor’s degrees in social services from universities of applied sciences. Many providers, particularly in the field of institutional care, have degrees in nursing (registered nurses/practical nurses). HCD was used as a theoretical framework for this study. The international standard (ISO 9241-210:2019) describes HCD as “an approach to systems design and development that aims to make interactive systems more usable by focusing on the use of the system and applying human factors/ergonomics and usability knowledge and techniques” 46. Most HCD techniques and methods require the participation of users. Usability refers to the extent to which a system can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use 46. Materials and methods Social welfare organization professionals in Finland In Finland, local public authorities (municipalities or federations of municipalities) are responsible for organizing and funding general social services for the entire population and special services for certain population groups; the Social Insurance Institution of Finland (KELA) manages social income support. Local public authorities may purchase social services from private service providers, including non-governmental organizations (NGOs) (n = 3,971 in 2017) 24. Social welfare and healthcare data are stored in separate registries; client data documented by social welfare professionals of, for example, elderly services are stored in the social welfare registry, but the physician addressing the health problems documents data in the healthcare registry. In many contexts of use, sharing client and patient data across social welfare and healthcare is only allowed after specific consent is given by the client/patient. In 2018, most (3,888; 95%) social workers were employed by municipalities, and fewer (196; 5%) worked in the private sector/NGOs. Fifty-six percent (5,638) of social counselors worked in municipalities, and 44% (4,384) worked in private sector/NGOs. The work tasks of social workers mainly focus on managing cases and making decisions about services and benefits, whereas social counselors usually provide services during scheduled appointments or at housing service units, most often in child welfare and disability services. Moreover, 32,227 and 18,284 practical nurses provide daily care mainly for the elderly and disabled persons in municipalities and the private sector/NGOs, respectively 47, 48. Design of the study This study was part of a large national cross-sectional CIS user experience survey conducted among social welfare professionals 39. The user participation methods in the questionnaire were (Table 1; Q3): observation of/shadowing users (Q3.A), focus/enduser groups (Q3.B), provision of feedback (Q3.C and Q3.D), and communication with an end-user representative in the organization (Q3.E). Table 1. Development-related questions in the questionnaire. Q1. Think of the experiences you have had in providing feedback about your client information systems. Please indicate your responses to the following statements. Scale: Fully agree / Somewhat agree / Neither agree nor disagree / Somewhat disagree / Fully disagree A I know how and to whom I can send feedback about the information system if I wish to do so. B The information system vendor is interested in feedback about the system provided by the end users. C The information system vendor implements corrections and change requests according to the suggestions of the end users. D Corrections and change requests are implemented within a reasonable time frame. Q2. Have you participated in information systems development work? A lot A little Not at all Q3. In which ways would you be interested in participating in information systems development work in the future? You may choose one or more alternatives. Scale: Fully agree / Somewhat agree / Neither agree nor disagree / Somewhat disagree / Fully disagree A I would be interested in showing software developers how I work and describing my software-related needs. B I would be interested in participating in a development work group made up of end users. C I would be interested in providing suggestions and feedback about how the software can be designed and changed to the vendor using a dedicated website. D I would be interested in providing suggestions and feedback about how the software can be designed and changed to the vendor using email. E I would be interested in telling the person in charge of information systems development for the organization about usage-related problems. F I am not interested in participating. G How else would you like to participate? Figure 3. Preferred methods of participation in IS development. Percentage of those who agree with the statement. Percentages for those who disagree and exact numbers of respondents per statement are provided in Supplementary table 2. Relation of age and experiences of CIS development When the age groups were compared, the youngest (under 35 years) and the oldest (over 55 years) appeared to have participated less than the other groups (Figure 4; supplementary table 4). The youngest respondents did not have more positive experiences with vendor reactions to feedback than the older respondents 33% 28% 14% 29% 56% 12% 46% 28% 17% 28% 60% 11% 33% 26% 15% 24% 54% 19% 25% 17% 13% 21% 45% 30% 34% 24% 15% 25% 53% 19% Q3.A I'd be interested in showing software developers how I work and describing my software-related needs. Q3.B I would be interested in participating in a development work group made up of other system end users. Q3.C I would be interested in providing suggestions and feedback about how the software can be designed and changed to the vendor using a dedicated website. Q3.D I would be interested in providing suggestions and feedback about how the software can be designed and changed to the vendor using email. Q3.E I would be interested in telling the person in charge of information systems development for the organization about usage-related problems. Q3.F I am not interested in participating. Leader or manager % Social workers % Social counselor % Other % Total % (supplementary table 5). The most popular participation method for all age groups was communicating with a person responsible for CIS development within the organization (Q3.E.), but those over 55 years of age favored this method the most. Only 25% of respondents over the age of 55, compared with 39% in the two youngest age groups, were interested in showing software developers how they work (Q3.A). The oldest were also the least interested in participating in focus groups (11% vs. 23–30%). The youngest were the most willing to send development ideas to developers via email (34% vs. 20% in the two oldest groups) and on a dedicated webpage (20% vs. 9–14% in the two oldest groups). In total, 17% of the two youngest age groups, but 24% of the oldest respondents, were not interested in participating at all (supplementary table 6). Figure 4. Participation in IS development work by age group. Exact numbers of respondents are provided in Supplementary table 4. Discussion Good usability is essential for developing successful and high-quality IS. Applying a HCD approach is one of the central means for achieving good usability1. Moreover, user involvement in IS development should be active, by participating in design, acting as a source of relevant data, or evaluating solutions1,47,54. The main finding of this study was that social welfare professionals are willing to participate in CIS development, but vendors and social welfare provider organizations are not fully utilizing this resource. The results agree with those of physicians and nurses43. Vendors considered unresponsive to CIS-related feedback When compared with other social welfare professionals, those working in leadership and managerial positions reported more positive experiences regarding their feedback to CIS vendors and developers; however, only 26% and 17% of leaders/managers indicated that their suggestions were implemented according to their wishes and quickly enough, respectively. Interestingly, social workers provided the most negative assessments. There are several possible reasons behind this finding. In recent years, the development of CIS has been heavily directed by national guidelines. It is likely that frontline social workers have not been included in these discussions; therefore, they may not have fully understood why some changes were implemented in a particular manner. Moreover, they may not personally agree with the guidelines. In addition, nationally enforced CIS developments may emphasize data collection over practical informational needs and the objectives of social work 53,55. Presumably the employers of social care workers have not allocated working time for development; this may have led to the underrepresentation of social workers in various focus groups or the overlooking of their viewpoints. Some researchers suggest that end-user tools may have been mainly developed between vendors and IT staff, 8 or end-user participation may have been used to gain buy-in for technology instead of seeking collaboration 54. Since the developers do interact with end users, 56,57 the most probable explanation for end users experiencing unresponsiveness is that the vendors and developers fail to inform end users about developments and corrections in a timely and easily understandable manner. Simply fixing problems/issues is ineffective if users do not know they have been fixed and may lead to end users feeling that their ideas have not been considered at all. Social welfare informaticists and on-site developers highly valued Of the various HCD methods for user participation 1, the respondents of this study favored the most explaining problems/development needs to a dedicated person in charge of CIS development in the organization. A social welfare informaticist could serve as an interpreter between the IT-focused language of the developers and the fieldspecific uses of the language of the end users8. They could also explain to both groups what is possible and what is not, either from a technical or workflow perspective. Indeed, Lagsten and Andersson (see ref. 8) identified that a lack of common language and structures for describing social work to CIS developers hampered critical discussions of CIS improvements. In healthcare, many organizations have employed physician and nurse informaticists who also use the information system themselves in clinical work 16-20. International requirements for such roles in the social welfare sector are lacking, but Gillingham (see ref. 58), for example, has suggested that social workers and their managers should take the lead in CIS development as they know their work best. Similar to health care professionals 9 frontline social welfare professionals need to be supported in their attempts to act as champions of innovation and change. Social welfare informaticists would enable the development of CIS that multilaterally and effectively support the practice of social work, data utilization, and service development for various stakeholders 23. They are also needed when developing processes and workflows alongside information system development. One third of respondents were willing to show their work to developers. On-site visits provide developers with a unique opportunity to understand how an information system is used and to hear immediate feedback on possible development plans. However, an individual end user does not necessarily represent a typical user or may not always be aware of the strategic goals of the organization. Therefore, site visits should always be complemented by collecting feedback from a diverse audience 59,60. One fourth of respondents were interested in participating by attending focus group meetings. It may be difficult to arrange time for frontline social welfare professionals who also see clients and manage cases. Moreover, to work efficiently, the participants of these groups should have a thorough understanding of not only the use contexts and end-user needs but also the CIS itself. We suggest that resource-intensive focus group meetings should be used sparingly. Instead, social welfare informaticists could gather end-user feedback through everyday work contacts. From a vendor/developer perspective, collecting feedback on dedicated webpages or via email is often the most feasible method of obtaining detailed information on individual problems or issues in the CIS. However, providing screenshots and accurate descriptions on use contexts may require a considerable amount of work from the end users. Vendors should develop automated feedback collection tools built into the information systems 61. Leaders and managers had participated more actively than the others Those working in leadership and managerial positions had participated more in CIS development than other social welfare professionals. Accordingly, managerial needs may have become overrepresented in the development of CIS 36. As most leaders/managers in social welfare do not participate in frontline social work, special attention should be paid to routine workflow design by also engaging non-managerial professionals; this applies particularly to usability testing. On the other hand, leaders and managers are likely to be more aware of the strategic and quality goals of the organization, as well as national CIS documentation requirements; indeed, collaboration between the various stakeholders is important 8. Moreover, more attention should be given in social welfare organizations to communicating these goals to frontline professionals. Younger social welfare professionals willing to participate in CIS development The finding that the youngest age groups had participated less than the older groups was not unexpected since CIS development requires an understanding of the work processes and strategic goals of an organization. The more experienced workers have been described as being more able to make constructive suggestions for changes to CIS user interfaces and functionality to serve their purposes more efficiently than novice users 40. However, since (at least in Finland) the majority of older social welfare professional generations have a relatively short history of electronic documentation and utilization of CIS in daily work, they may try to copy paper-based workflows into the CIS instead of being able to re-evaluate the whole process. Moreover, novice users may identify usability problems that may not be noticed by experienced users 40. Except for communicating via a social welfare informaticist, the youngest respondents were more interested than the oldest respondents in all other participatory methods included in our survey; while this has also been reported in studies involving physicians and nurses, 43 we found no studies that would explain this difference. However, vendors or social welfare organizations should not assume that, compared to older workers, the younger have more positive impressions of vendors being responsive to feedback and implementing CIS improvements. Only a few not interested at all in participating One of the central findings of this study was that a relatively small proportion (19%) of social welfare professionals were not interested in participating in CIS development. This is in accordance with the findings involving physicians and nurses 43. Moreover, half of the respondents had participated to at least some extent. It is likely that those who responded to this survey have been more actively involved in CIS development than those end users who did not respond; this may have also resulted in overestimating the proportion of those interested. However, it is evident that a large proportion of social welfare professionals are willing to participate in CIS development. Since the majority were not satisfied with the end results of the interaction between end users and system developers and vendors (changes and corrections were not made according to suggestions and quickly enough), social welfare organizations, CIS vendors, and developers should seek to find more suitable means of improving this experience. The principles and methods of HCD provide tools for the involvement of end users for all stages of the IS design process; this may in turn improve the chances for successful IS development and outcomes 1. Limitations One of the central limitations of our study was that we could not assess the response rate; however, to our best knowledge, with 1145 respondents, this was the largest survey to date among social welfare professionals concerning their experiences with participating in CIS development. Moreover, the results can be considered generalizable since the respondents represented social welfare service lines in similar proportions as reported nationally 48. It is evident that social welfare comprises different services and the social welfare professionals may have different educational levels in different countries; national surveys are needed to best implement HCD methods. Even though we did not ask the respondents open-ended questions about participation, the strength of our survey method was that we were able to also include those who do not usually participate in development; most previous observational studies have focused on those who are already involved, and the numbers of study subjects have been low 21,35,36,40-42. One limitation deserving attention is that our survey did not cover the experiences of the largest group of social welfare professionals in Finland i.e. practical nurses. However, they mainly use HIS instead of CIS, and their information needs differ considerably from those of social workers and social counselors. Moreover, in most international contexts, their tasks would be regarded as healthcare instead of social welfare. We suggest that their experiences should be covered in a separate survey. We also did not ask how the participants had participated in CIS development; this should be addressed in future studies to assess whether, for example, those who have social welfare informaticists in their organizations or have had developers on site are more satisfied. Moreover, the questionnaire did not cover the phase of software development in which the respondents had participated. End users tend to be more active in reporting errors and development ideas during the implementation phase of a new IS; however, at the time of the survey, there had been no major changes in the CIS brands in use in Finland 62. Future research Although the usefulness of end-user participation in IS development is widely acknowledged, there are few studies on the experiences of social welfare professionals. Indeed, we encourage similar surveys in other countries as well. Moreover, large studies, both surveys and experimental studies, are needed to explore the usefulness of the various methods in the field of social welfare. Surveys should be complemented with interviews and usability tests, but the viewpoints of larger audiences are difficult to reach with qualitative ethnographic studies. Conclusion Our findings emphasize the importance of end-user involvement and the deployment of appropriate HCD methods for CIS development. One of the greatest challenges in the relationship between end users and developers is communication. End users must not only verbalize their needs to developers, but developers must also communicate when and how the needs of end users will be implemented; positions for social welfare informaticists should be established in organizations to alleviate this lack of communication. Social welfare professionals are willing to participate in CIS development, but vendors and social welfare provider organizations are not fully utilizing this resource. Acknowledgments We express our gratitude to Professor Sanna Hautala for guidance in the design of the survey. We thank the Talentia Union of Professional Social Workers, the Trade Union for the Public and Welfare Sectors, and the Finnish Society of Social Work Research for kindly sending questionnaire links. Declaration of interest statement Martikainen has been previously employed by one of the CIS software vendors included in the study. Lääveri, Salovaara and Tyllinen are/have been employed by a CIS software vendor not included in the study. The employer did not provide any support, financial or otherwise, for the study. Moreover, the vendor was not involved in the design of the study or in the collection, analysis, and interpretation of the data. Authors’ contributions Susanna Martikainen (PhD) had the main responsibility for the manuscript. Tinja Lääveri (MD, PhD) had the main responsibility for the statistical analysis. All authors contributed to all parts of the article, including interpretation of the data and approval of the manuscript. In addition, the authors Martikainen, Salovaara, Ylönen, Tyllinen, Viitanen, and Lääveri contributed to the design of the survey study. References 1. Maguire M. Methods to support human-centred design. Int. J. Human-Computer Studies. 2001; 55: 587-634 doi:10.1006/ijhc.2001.0503 2. Qin Y, Zhou R, Wu Q, et al. The effect of nursing participation in the design of a critical care information system: A case study in a chinese hospital. BMC medical informatics and decision making. 2017;17(1):165. journal of medical informatics (Shannon, Ireland). 2020;137:104117. doi: 10.1016/j.ijmedinf.2020.104117. 44. Jeddi FR, Nabovati E, Bigham R, Khajouei R. Usability evaluation of a comprehensive national health information system: relationship of quality components to users' characteristics. Int J Med Inform 2020;133:104026. doi: 10.1016/j.ijmedinf.2019.104026. 45. Raglan GB, Margolis B, Paulus RA, Schulkin J. Electronic health record adoption among obstetrician/gynecologists in the United States: physician practices and satisfaction. Journal for Healthcare Quality. 2014. 46. ISO. ISO 9241-210 ergonomics of human-system interaction — part 210: Human-centred design for interactive systems. 2009. 47. Ministry of Social Affairs and Health. Social and health services. https://stm.fi/en/social-and-health-services. Accessed 18.5.2021. 48. Vuorijärvi P, Raappana M, Kinnunen P, Kostamo-Pääkkö K. Sosiaalihuollon ammattihenkilöstö- ja tehtävärakenneselvitys 2018. http://www.sosiaalikollega.fi/poske/tapahtumat/kkp_soshuollonammattihlosto_t ehtavarakenneselvitys2018. Updated 2019. 49. Hyppönen H, Kaipio J, Heponiemi T, et al. Developing the national usabilityfocused health information system scale for physicians: Validation study. Journal of medical Internet research. 2019;21(5):e12875. https://www.ncbi.nlm.nih.gov/pubmed/31099336. doi: 10.2196/12875. 50. Viitanen J, Hyppönen H, Lääveri T, Vänskä J, Reponen J, Winblad I. National questionnaire study on clinical ICT systems proofs: Physicians suffer from poor usability. International journal of medical informatics (Shannon, Ireland). 2011;80(10):708-725. https://www.clinicalkey.es/playcontent/1-s2.0- S1386505611001481. doi: 10.1016/j.ijmedinf.2011.06.010. 51. Kaipio J, Kuusisto A, Hyppönen H, Heponiemi T, Lääveri T. Physicians’ and nurses’ experiences on EHR usability: Comparison between the professional groups by employment sector and system brand. International journal of medical informatics (Shannon, Ireland). 2020;134:104018. http://dx.doi.org/10.1016/j.ijmedinf.2019.104018. doi: 10.1016/j.ijmedinf.2019.104018. 52. Martikainen S, Viitanen J, Korpela M, Lääveri T. Physicians’ experiences of participation in healthcare IT development in finland: Willing but not able. International journal of medical informatics (Shannon, Ireland). 2011;81(2):98- 113. https://www.clinicalkey.es/playcontent/1-s2.0-S138650561100178X. doi: 10.1016/j.ijmedinf.2011.08.014. 53. Lyneborg AO, Damgaard MB. Knowledge lost – or gained? The changing knowledge base of Danish social work, Nordic Social Work Research, 2019;9:3,206-219, DOI: 10.1080/2156857X.2018.1494031 54. Kujala S. Effective user involvement in product development by improving the analysis of user needs. Behaviour & Information Technology. 2008. 27(6), 457- 473. 55. Wastell D, White S. Beyond bureaucracy: Emerging trends in social care informatics. Health informatics journal. 2014. 20(3), 213-219. 56. Martikainen S, Korpela M, Tiihonen T. User participation in healthcare IT development: A developers’ viewpoint in Finland. International Journal of Medical Informatics. 2013. 10.1016/j.ijmedinf.2013.12.003. 57. Rytkönen J, Kinnunen U-M, Martikainen S. Sosiaali- ja terveydenhuollon tietojärjestelmäkehittäjien kokemuksia ja näkemyksiä yhteistyöstä käyttäjien kanssa. Submitted 5/2021. 58. Gillingham P. Implementing electronic information systems in human service organisations: The challenge of categorisation. Practice (Birmingham, England). 2015;27(3):163-175. http://www.tandfonline.com/doi/abs/10.1080/09503153.2015.1014334. doi: 10.1080/09503153.2015.1014334. 59. Lazar J, Feng JH, Hochheiser H. Research methods in human-computer interaction. Morgan Kaufmann. 2017 60. Salovaara S. Considering the informational needs of social work in the information system Apotti. Finnish Journal of EHealth and EWelfare, 2021;13(2), 133–146. https://doi.org/10.23996/fjhw.100692 61. Otokiti A, Craven C, Shetreat-Klein A, Cohen S, Darrow B.Beyond Getting Rid of Stupid Stuff in the Electronic Health Record (Beyond-GROSS): Protocol for a User-Centered, Mixed-Method Intervention to Improve the Electronic Health Record System. JMIR Res Protoc 2021;10(3):e25148 URL: https://www.researchprotocols.org/2021/3/e25148. DOI: 10.2196/25148 62. Sosiaali- ja terveydenhuollon tietojärjestelmä-palveluiden seuranta ja arviointi (STePS 3.0). https://thl.fi/fi/tutkimus-ja-kehittaminen/tutkimukset-ja- hankkeet/sosiaali-ja-terveydenhuollon-tietojarjestelmapalveluiden-seuranta-ja- arviointi-steps-3.0. Accessed 13.7.2021.