How technology is empowering patients? A literature review Jorge Calvillo PhD,*§Isabel Rom an PhD*†§and Laura M. Roa PhD†‡§ *IT Health Scientist in Biomedical Engineering Group, †Professor, ‡Head of Biomedical Engineering Group, University of Seville, Seville and §CIBER de Bioingenierı ´a, Biomateriales y Nanomedicina (CIBER-BBN), Seville, Spain Correspondence Jorge Calvillo Escuela T ecnica Superior de Ingenierı ´a C. de los Descubrimientos s/n 41092 Sevilla Spain E-mail:
[email protected] Accepted for publication 1May 2013 Keywords: health information systems, medical informatics, patient education, patient empowerment, patient participation Abstract Background The term ‘Patient Empowerment’ (PE) is a growing concept –so in popularity as in application –covering situations where citizens are encouraged to take an active role in the management of their own health. This concept is serving as engine power for increasing the quality of health systems, but a question is still unanswered, ‘how PE will be effectively achieved?’ Beyond psychological implications, empowerment of patients in daily practice relies on technology and the way it is used. Unfortunately, the heterogeneity of approaches and technologies makes difficult to have a global vision of how PE is being performed. Objective To clarify how technology is being applied for enhancing patient empowerment as well as to identify current (and future) trends and milestones in this issue. Search strategy Searches for relevant English language articles using Medline, Scopus, ACM Digital Library, Springer Link, EBSCO host and ScienceDirect databases from the year 2000 until October 2012 were conducted. Among others, a selection criterion was to review articles including terms ‘patient’ and ‘empowerment’ in title, abstract or as keywords. Main results and conclusions Results state that practical approaches to empower patients vary in scope, aim and technology. Health literacy of patients, remote access to health services, and self-care mechanisms are the most valued ways to accomplish PE. Current technology already allows establishing the first steps in the road ahead, but a change of attitude by all stakeholders (i.e. professionals, patients, policy makers, etc.) is required. Introduction As the appearance of the first Internet-based applications supporting new methods of healthcare delivery, the potential of Information and Communication Technologies (ICT) for changing the role of users has been well-known. 1,2 Experts predicted a range of benefits from the efficient adoption of ICT in the healthcare domain. 3 For example, the possibility of providing citizens with mechanisms for accessing information and knowledge required to 643 ©2013 John Wiley & Sons Ltd Health Expectations,18, pp.643–652 doi: 10.1111/hex.12089
understand their health status and to make informed decisions. 4 ICT could also bring new ways of connectivity among patients, users and health providers to establish virtual communities in which end users adopt the role of information providers for their peers. 5,6 Furthermore, technology would ease the development of tools and solutions for maintenance of healthy habits, education in health, self-man- agement of chronic diseases and deployment and use of Personal Health Records (PHR) controlled by patients. 1 Finally, technology promised to reduce economic costs and promote a more sustainable health care. 7 Patient Empowerment (PE) is a growing concept –so in popularity as in application – that covers situations where citizens are encouraged to take an active role in the management of their own health, transforming the traditional patient–doctor relationship and providing citizens with real management capabilities. 1 Gibson, in a review about PE in health, redefined empowerment as a process of helping people to assert control over the factors which affect their health. 8 Another literature review defines PE as a continuous process through which patients (and patient groups) work in partnership with their health-care system. The objective of this collaboration is to enable patients to become more responsible for and involved in their treatment and health care. 9 Despite differences among definitions, the central idea is shared: an attempt for patients to take charge of their own health. Some common keys are as follows: 1. Health-care professionals should be the first promoters of PE. 10 2. An empowered patient should be educated to think critically, make informed decisions and then adjust to prescribed care plans (i.e. become a health literate). 11,12 3. Dependency on people should be partially transformed to dependency on systems. 6 4. Technology for PE may also bring problems due to digital divide existing in society between people with and without technology skills (what is called ‘digital literacy’). 13–16 At organizational level, it is assumed that PE is a cornerstone for the transformation and evolution of the health-care domain, becoming a philosophy inspiring policies and services. 17,18 As an example, the European Commission, the European Council and the World Health Organization (WHO) –Europe –are supporting actively the development of PE solutions by acting in several points such as bringing access to information and trust advice to people, promoting health literacy of patients or supporting new models of chronic care. 19 Thus, the concept PE is serving as engine power for increasing the quality of health systems by policy makers. 20 Beyond psychological implications, empowerment of patients in daily practice relies on technology and the way it is used. 16 As the application of ICT in the health-care domain has been performed in an uncontrolled way and with no formalization or guidelines, now empowerment of patients (as will be shown in this paper) is shared by a wide spectrum of different and separate research fields such as end-user applications, homecare, information systems and communications. This multidisciplinary divergence increases the complexity of the matter because it often requires a knowledge translation among areas. Moreover, PE (even the term) is performed through many different ways depending on the specific research field addressing it. These facts lead to a fragmented application of heterogeneous technologies for empowering patients. To our knowledge, several literature reviews focus on PE, but none has both a general purpose and an emphasis on technology. The scope of these reviews varies in different ways. Aujoulat et al. 21 examined how the term ‘empowerment’ has been used in relation to care and education of patients with chronic conditions over an 11-year period (1995–2006). This review bases on the theory of patient education as a mechanism for empowerment in chronic scenarios and its benefits for patients. Laugharne et al. 22 had mental health as domain for their review of trust, choice and PE from 1980 to 2005. This review stated that PE ©2013 John Wiley & Sons Ltd Health Expectations,18, pp.643–652 How technology is empowering patients? J Calvillo, I Rom an and L M Roa 644
was accomplished at organizational level (e.g. policies of patient involvement in health care), but there was no evidence of real performance of PE in daily practice of mental health care. Technology applications were not reported in this paper. Macq et al. 23 reviewed literature to extract findings of PE in tuberculosis control. They identified trends and barriers of empowering of tuberculosis patients but did not include any reference to technology applications. Another review focused on methodologies for PE was published by Virtanen et al. 24 . This paper made a literature review to describe the nature of empowering discourses between patient and nurse. The main result was a heterogeneous range of discourse methodologies to empower patients, but no technology was identified to do it. Finally, Lober et al. 25 published a review strongly oriented to find synergies between health care and ICT. Although its health focus was oncology nursing, it presented papers of general purpose. This review mainly placed emphasis on new models of interaction between patients, doctors and others thanks to the Internet and social media as well as known areas such as monitoring and administration systems. This was a broad review but with a research methodology covering rather than formal published papers (i.e. web references, tools and technical reports). Through a literature review, our paper aims at clarifying how technology is being applied to PE as well as future trends and milestones. The review focuses on identifying current trends of technology applied to PE, in particular, which technologies are being used, how they are applied and how empowerment is proposed and accomplished. Because approaches of technology applied to PE have recently appeared, this paper makes a review from the year 2000 to October 2012 considering that prior works are theoretical or related to social interaction. Finally, although there exist other concepts related to empowerment for nurses, doctors and others, 3 considering them is out of the scope of our review. Methods We searched for relevant English language articles using Medline, Scopus, ACM Digital Library, Springer Link, EBSCO host and ScienceDirect databases from the year 2000 until October 2012 (papers in press included). The first step was collecting articles that included ‘patient’ and ‘empowerment’ in title, abstract or as keywords. For the sake of thoroughness, several searches were performed by substituting ‘patient’ by ‘citizen’, ‘user’, ‘consumer’, ‘human’ and ‘subject of care’ but always with focus on the health domain. Another set of searches was performed to collect publications with the terms ‘empowering patients’, ‘empower patients’, ‘empower people’, ‘health informatics’, ‘health information systems’, ‘patient-centred’ or ‘patient-centric’ in title, abstract or keywords. Opinion papers, letters and reviews were excluded. Figure 1 shows the selection process for this article review. Identified abstracts and contents were screened by two peers in parallel to determine eligibility for further review. The following eligibility considerations were made: 1. The development, validation or assessment of technology for PE should be presented in the article. Those papers expressing opinions or conducting reviews were excluded; 2. The empowerment action should be performed for the direct benefit of patients. Technology applied to empowerment of medical staff, relatives or others were not considered, even although this had indirect benefits for patients; 3. Many technology applications in health lead to more efficient care but not to empowering individuals. For example, remote monitoring with no intervention by patients, or EHRs only for health professional use. This kind of applications means better health for patients but not increasing the trust, autonomy or safe sense of patients. Articles where patient empowerment is not properly justified or stated were excluded. ©2013 John Wiley & Sons Ltd Health Expectations,18, pp.643–652 How technology is empowering patients?, J Calvillo, I Rom an and L M Roa 645
From the eligible papers, the reviewers extracted data such as publication year, if they were focused on particular medical specialties, technology used or how PE was claimed to be achieved. Results Searching the online databases resulted in 12 389 articles. After combining results and excluding duplication, 9540 articles left. Later, reviews of abstract and content were performed. 9087 and 187 articles were excluded, respectively, according to eligibility criteria previously defined. Thus, only 266 papers satisfied all the requirements of the process. The set of references included in the review can be consulted via web. 26 As first point of this review, it is worthy to mention that the number of articles published about technology for PE has increased notably. However, the number of papers does not follow an exponential tendency, but it presents an increasing variability between consecutive years (Fig. 2). The articles selected have been classified according to the medical specialty or disease which the technology developed is planned to. Figure 3 shows the results. Technology in 45.9% (123) of articles is applied to all knowledge areas of health as in the case of medical information management or patient education. Both examples may cover the whole range of health issues and diseases. 11.5% (31) of articles focus on medical specialties (e.g. dermatology and paediatrics), and 13.6% (36) present solutions applied to some specific disease (e.g. HIV and depression). Diabetes (10.8%, 29), oncology (8.2%, 22) and chronic diseases Scopus Search strategy (n = 4.186) Medline Search strategy (n = 6.285) ScienceDirect Search strategy (n = 595) Combine results (n = 12.389) Abstract review (n = 9.540) Exclude duplicates (n = 2.849) Content review (n = 453) Excluded in abstract review (n = 9.087) Articles included (n = 266) Excluded in content review (n = 187) ACM Digital Library Search strategy (n = 213) Springer Link Search strategy (n = 613) EBSCO host Search strategy (n = 497) Figure 1Flow of article selection in the literature review. 0 5 10 15 20 25 30 35 40 45 50 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 Number of articles Publication year (until October 2012) Figure 2Number of articles published on technology for patient empowerment over the years. ©2013 John Wiley & Sons Ltd Health Expectations,18, pp.643–652 How technology is empowering patients? J Calvillo, I Rom an and L M Roa 646
(5.2%, 14) have been classified separately due to the relevant number of articles on these topics. Finally, technology in 4.8% (13) of reviewed papers is centred on other areas such as Pharmacy and Lactation, which cannot be grouped easily by the previous categories. Once determined the health issue in which PE is approached, the next point is to classify what technology is used to accomplish it (Fig. 4). Web services and communication networks are the most used technologies (74 and 51 articles, respectively). Both technologies are applied to different scenarios and purposes but always easing remote communication and access to health information and services. Besides them, both PHR and Electronic Health Record (EHR) approaches share outstanding positions. The reason could be that they have been considered cornerstones for the technological revolution of the health-care domain, and there are many efforts developing them. EHR and PHR are similar each other although address PE differently. EHR provides the patient with access and knowledge of his/her health information; meanwhile, PHR grants a patient with administration privileges too. 27,28 Another relevant set of approaches focuses on translating the methodology of patient support groups to virtual world using social media and online communities. In these scenarios, the patient receives advice from peers and he/she can be an information provider for others. Finally, other relevant technologies are as follows: Internet as source of information, software and mobile apps, security mechanisms, devices and communication media (such as traditional and IP telephony or e-mail). The last point of this review categorizes how the different approaches empower patients. The same technology may deploy two different General 45.9% Medical specialties 11.5% Diabetes 10.8% Chronic diseases 5.2% Oncology 8.2% Other 4.8% Specific disease 13.6% Figure 3Classification of selected articles according to the health area where they empower patients. 0 10 20 30 40 50 60 70 80 Number of articles Technology applied Figure 4Number of articles categorized by technology they use to empower patients. ©2013 John Wiley & Sons Ltd Health Expectations,18, pp.643–652 How technology is empowering patients?, J Calvillo, I Rom an and L M Roa 647
approaches for PE. For example, e-mail communication could be used for strengthening doctor–patient relation or alerting patient of modification of his/her health information record. Figure 5 summarizes the findings. Note that most articles adopt two or more mechanisms for PE. The most popular way for empowerment is patient education, shared for 40% of reviewed articles. It is widely argued that an educated patient can make more informed decisions, improve compliance, reduce anxiety levels and participate actively in the treatment of his/her diseases. This fact is more relevant in chronic scenarios where the patient must modify his/her life and adapt to permanent conditions. If healthy scenarios were considered, benefits of patient education could be translated to the maintenance of health and prevention tasks through citizen education. Enhancing commodity of patients is another important approach for empowerment. It is accomplished by reducing the complexity of daily tasks such as the patient–doctor communication (e.g. e-mail or instant messaging), online access to administrative services and telediagnosis. Self-care is a benefit of patient education with a significant number of articles proposing mechanisms to accomplish it. The access to own health information and to reliable advices improves awareness of patients in their condition and adherence to treatment plans. Another relevant paradigm to empower patients is to turn them into providers of support and advice for peers. As providers, patients feel useful, and as receivers, they obtain support and comprehension of peers that suffer (or suffered) similar conditions. Security also counts as a driving force for PE in different ways. Control of distribution and disclosure of personal information are the most relevant PE mechanisms followed by control over its edition, and privacy and confidentiality of communications. Due to the confidential content of health information, patients are very concerned with security requirements. Many reviewed articles consider scenarios with no mechanism to protect data and communications, but security is an essential requirement of technology applications in the health-care domain. Other PE ways are to strengthen the doctor–patient relation, to access general or personalized information and to promote behaviour modification, etc. Discussion In the introduction section, the objective of the review was stated: to identify how PE is being approached through technology and which milestones would be required to accomplish a real empowerment of patients. We selected and described the results of 266 papers on technology applications for PE. The number of such approaches shows a strong increase in recent years (Fig. 2). Two facts could be responsible of that tendency: the advancements in ICT (with the consequent wider application to health) and the currently rising awareness of 0 20 40 60 80 100 120 Number of articles Paradigms for empowering patients Figure 5Number of articles categorized by how they empower patients. ©2013 John Wiley & Sons Ltd Health Expectations,18, pp.643–652 How technology is empowering patients? J Calvillo, I Rom an and L M Roa 648
providing patients with an active role and capabilities of management. In a broad sense, the term ‘patient empowerment’ is often used in the literature and health policy to characterize an essential key for involving patients in their health care. In most approaches, PE is addressed from a psychological perspective (i.e. studies of how health-care professionals could make patients more confident and involved in their health-care processes), but also in some cases, technology is applied for empowering patients. Thus, implications on daily practice are restricted to modify attitudes of patients (what we classify as Milestone 1 of empowerment, as it will be further explained in the following) and not to involve them actively in processes. From the results of the review, some interesting points can be extracted for discussion. First, there is a wide spectrum of technologies empowering patients. There are initiatives that use promising technologies (such as games and virtual worlds 29 or textile monitoring 30 ), and others reuse common (and sometimes in disuse) technologies (e.g. audio call 31 or video recording 32 ). According to current and future technology trends for empowering patients, it can be stated that approaches in this domain follow the general tendencies of research and development in ICT. As revolutionary technology in every sector of society, Web services is also one of the most used technologies for empowering patients. Its application covers several approaches such as information web pages, interactive portals, infrastructure of distributed services and remote data access. Its versatility, popularity and development in other domains make it the first choice for developers of solutions in health domain. Booming technologies in other sectors (e.g. social media and mobile apps) are being steadily applied to empower patients. Forums, blogs and social networks are suitable vehicles to translate support groups from real life to electronic world, ease the communication among patients and professionals and strengthen the continuity of care beyond physical appointments. In addition, the wide adoption of smartphones in daily life brings many potential trends for patient empowerment such as ubiquitous access to health information for patients and professionals or smartphone applications for monitoring chronic conditions, disease prevention tasks and promotion of healthy habits. Finally, an interesting result is the slightly higher use of PHR (12.7% of articles) for empowering patients than of EHR (10%). According to public opinion, PHR is a natural evolution of EHR towards a real data management by patients. Thus, the common assumption a priori is that PHR would be a much more significant trend for empowering patients than EHR. But that is not right because although EHR does not delegate management capabilities to patients, it contributes to empower them through the access to their information. Often, the scenario of patients suffering burdens for accessing their own health data is underestimated or not considered. But our review reveals that is still a necessary action line without which other approaches (such as data management in PHR) cannot be addressed. From the literature review, we can conclude that different levels of empowerment exist (as has been stated in previous works 33 ). All the reviewed approaches have the same objective (i.e. to empower patients), but the grade of autonomy or involvement that the subject obtains varies from one solution to another. These levels of empowerment may serve as overview of milestones in the road of patient empowerment. 1. Milestone 1: Patient is aware of his/her health condition and properly informed by doctors. There is a first change of attitude from passive and ignorant to active and participant to face diseases. In this stage, patients are well informed about prognosis and treatment options, and this makes them to be more likely to make decisions, adhere to their treatment plan and have better outcomes. 2. Milestone 2: Individual (not necessarily patient) active not only in the treatment of ©2013 John Wiley & Sons Ltd Health Expectations,18, pp.643–652 How technology is empowering patients?, J Calvillo, I Rom an and L M Roa 649
diseases but also in the maintenance of his/ her health and prevention tasks. Now citizen is willing to perform healthy activities, monitor food and hygiene habits, modify damaging behaviours, etc., in order to prevent diseases or enhance his/her wellness. Doctors play the role of encouraging these attitudes which result in a healthier population. 3. Milestone 3: Citizens are educated (and not simply informed) in health. A democratization of knowledge is accomplished, that is, citizens can proactively access information, knowledge and advices not guarded by health professionals. Health knowledge must be understandable, reliable and accessible for any citizen according to different skills. Now doctor and patient are collaborators at the same level, and there is a mutual trust. Figure 5 shows how education of patients is the first concern (with 39.4% of selected articles). From their point of view, doctors plead for educated patients that adopt effectively their care plans and willing to know more about their health conditions. 4. Milestone 4: Citizens are not only health information sinks but also sources. Technology allows citizens to act as health information and advice providers for peers worldwide with little or no supervision by doctors. Citizens embrace a new dimension in empowerment when their health experiences can help others and when they are counselled by people who speak their own language. Finally, there may be some obstacles against patient empowerment. First, citizens must able to trust in technology empowering them in order to play their proactive roles. Thus, considering carefully, the application of technology to empower patients is required. For example, Web pages as information sources are promising for rising health literacy of patients, but they can lead to misunderstandings and wrong decisions without a precise assessment by trustworthy third party; or privacy and confidentiality of information flows from/to EHR and PHR should not be a secondary feature of systems. Indeed, security is an essential requirement on communication and health information systems, but only 6% of articles focus on (or include as complement) security mechanism. Another major obstacle is reluctance of doctors to lose their power. Health professionals encourage citizens to be informed and adhere to their treatment plans, but sometimes more knowledge is not desirable. This fact is consequence of people using Internet as first source of health information without considering the harm that unreliable and understandable information can make. But a well-educated patient through precisely assessed information sources by health professionals is a win–win scenario. Obviously, also citizens may desire to hold the status quo, that is, maintain a passive role on their health, as some studies pointed out previously. 34 Thus, involved actors’ attitudes towards PE will determine the real swift of health-care delivery models and the role of each actor. Conclusion In conclusion, practical approaches to empower patients vary in scope, aim and technology. The set of areas where empowerment may be accomplished is so wide (as was showed in Fig. 5) that almost any current initiative of ICT applied to health covers mechanisms for empowering patients. As has been reviewed, there exist different (in scope and autonomy grade of citizens) levels of empowerment that may be mapped to specific milestones. Current technology already allows establishing the first steps in the road ahead, but a change of attitude by all stakeholders (i.e. professionals, patients and policy makers) is required. Furthermore, despite motivation, PE strongly depends on accessibility of solutions and interfaces. For a real empowerment of patients, all citizens must be capable of accessing systems empowering them, no matter their digital literacy, economic level, education or disabilities. 35 Therefore, if obstacles and ©2013 John Wiley & Sons Ltd Health Expectations,18, pp.643–652 How technology is empowering patients? J Calvillo, I Rom an and L M Roa 650
gaps are successfully addressed, at mediumterm technology will ease the emergence of a new patient fully equipped for the health-care challenging scenarios of the 21st century. Source of funding This work has been partially supported by the CIBER-BBN (inside project PERSONA), the Biomedical Engineering Group at University of Seville, an Excellence Project of the Andalusian Council (TIC-6214) and a grant from the Fondo de Investigaci on Sanitaria inside project PI082023. CIBER-BBN is an initiative funded by the VI National R&D&i Plan 2008–2011, Iniciativa Ingenio 2010, Consolider Program, CIBER Actions and financed by the Instituto de Salud Carlos III with assistance from the European Regional Development Fund. Conflicts of interest No conflicts of interest occurred. References 1 Hasman A. Medical informatics: quo vadis? International Journal of Medical Informatics, 2002; 66: 113–120. 2 Gage EA, Panagakis C. The devil you know: parents seeking information online for paediatric cancer. Sociology of Health and Illness, 2012; 34: 444–458. 3 Haux R. Aims and tasks of medical informatics. International Journal of Medical Informatics, 1997; 44:9–20. 4 Haux R, Ammenwerth E, Herzog W, Knaup P. Health care in the information society. A prognosis for the year 2013. International Journal of Medical Informatics, 2002; 66:3–21. 5 Frank SR. Digital health care –The convergence of health care and the internet. Journal of Ambulatory Care Management, 2000; 23:8–17. 6 Bos L, Marsh A, Carroll D, Gupta S, Rees M. Patient 2.0 empowerment. Proceedings of the 2008 International Conference on Semantic Web and Web Services, SWWS 2008; 164-168. 7 Solovy A. The big payback: 2001 Survey shows a healthy return on investment for info tech. Hospitals and Health Networks, 2001; 75:40–50. 8 Gibson CH. A concept analysis of empowerment. Journal of Advanced Nursing, 1991; 16: 354–361. 9 Loukanova SN, Bridges JFP. Empowerment in medicine: an analysis of publication trends 1980– 2005. Central European Journal of Medicine, 2008; 3: 105–110. 10 Anderson RM, Funnell MM. Patient empowerment: myths and misconceptions. Patient Education and Counselling, 2010; 79: 277–282. 11 Health literacy. National network of libraries of medicine. Available at: http://nnlm.gov/outreach/ consumer/ hlthlit.html, accessed 17 May 2013. 12 Schulz PJ, Nakamoto K. Health literacy and patient empowerment in health communication: the importance of separating conjoined twins. Patient Education and Counseling, 2013; 90:4–11. 13 Parent F, Coppieters Y, Parent M. Information technologies, health, and “globalization”: anyone excluded? Journal of Medical Internet Research, 2001; 3: 134–139. 14 Brodie M, Flournoy RE, Altman DE, Blendon RJ, Benson JM, Rosenbaum MD. Health information, the internet, and the digital divide. Health Affairs, 2000; 19: 255–265. 15 Buckingham D. Defining digital literacy –What do young people need to know about digital media? Nordic Journal of Digital Literacy, 2006; 4: 263–276. 16 D’Alessandro DM, Dosa NP. Empowering children and families with information technology. Archives of Pediatrics and Adolescent Medicine, 2001; 155: 1131–1136. 17 Lenert L. Transforming healthcare through patient empowerment. Studies in Health Technology and Informatics, 2010; 153: 159–175. 18 Colombo C, Moja L, Gonzalez-Lorenzo M, Liberati A, Mosconi P. Patient empowerment as a component of health system reforms: rights, benefits and vested interests. Internal and Emergency Medicine, 2012; 7: 183–187. 19 Monteagudo JL, Moreno O. e-Health for patient empowerment in Europe. Report, 2007. 20 Chatzimarkakis J. Why patients should be more empowered: a European perspective on lessons learned in the management of diabetes. Journal of diabetes science and technology, 2010; 4: 1570–1573. 21 Aujoulat I, d’Hoore W, Deccache A. Patient empowerment in theory and practice: polysemy or cacophony? Patient Education and Counselling, 2007; 66:13–20. 22 Laugharne R, Priebe S. Trust, choice and power in mental health: a literature review. Social Psychiatry and Psychiatric Epidemiology, 2006; 41: 843–852. 23 Macq J, Torfoss T, Getahun H. Patient empowerment in tuberculosis control: reflecting on past documented experiences. Tropical Medicine and International Health, 2007; 12: 873–885. ©2013 John Wiley & Sons Ltd Health Expectations,18, pp.643–652 How technology is empowering patients?, J Calvillo, I Rom an and L M Roa 651