Dixon-Woods M ; Redwood S ; Leslie M; et al. Improving quality and safety of care using “technovigilance”: an ethnographic case study of secondary use of data from an electronic prescribing and decision support system. Milbank Q. 2013; 91: 424-454
This journal article is an ethnographic case study of a reasonable large National Health Service (NHS) acute care hospital situated in England, UK. The hospital distinguished itself from others by having a secondary use for electronic health records (EHS) from its electronic prescribing and decision-support system (ePDSS).
This is done with an intention to strategically improve quality and safety in the services provided by this organization. Their study was based on ethnographic observations of clinical setting (162 hours) and meetings (28 hours), informal observations with clinical staff, semi structured interviews with ten senior executives, and the collection of relevant documents. The authors analyzed their data based on constant comparative method.
Findings on their case study are analyzed on authors philosophical perspective being critical through out their analysis of the information system in place. These are looked at through principles proposed for critical research by Klein & Myers. Their theoretical framework is build on inductive reasoning through collection of data and building up the theory – often known as ‘bottom-up’ approach. Lastly, this review looks into the questions proposed, units of analysis, data collection, appropriateness of research method, and overall usefulness of the results.
Table of Contents
- Philosophical Perspective
- Research Method & Theoretical Framework
- Topic & Research Question Addressed
- Units of Analysis, Data Collection & Analysis
- Appropriateness of Research Method
- Validity, Plausibility & Overall Usefulness of Results
Critical Research in information systems is known to focus on “social issues such as freedom, power, social control, and values with respect to development, use, and impact of information technology” (Myers & Klein, 2011). In the journal article, the authors analyze some of the social issues faced within a hospital environment with use of information systems (ePDSS), the power associated with Hospital “blunt end” (board of the hospital, strategic and operational committees), and values with respect to professional development and the impact of information technology on all involved parties. Neuman defines critical research as “a critical process of inquiry that goes beyond surface illusions to uncover the real structure in the material world in order to help people change conditions and build a better world for themselves” (Neuman, 1994).
The purpose of this case study was also to reveal the underlying sources of inequality, and hidden agendas in using ePDSS system for secondary use. It was essential used as a method of monitoring staff and their performance in many quality standards that were in place. The ePDSS system allowed executives to see which staff members were generating high number of alerts and warnings. They were then invited to what they described as “unpleasant” meetings.
The inequalities were revealed in sense that the executive team weren’t involved in any parts of the operation. They had no clue on what was actually going on or what kind of situations different clinical wards faced. This happened due to extreme dependence on ePDSS system to raise alerts on quality aspects that were monitored. Other assumptions associated with critical research are also seen through out the journal. These include the fundamental nature of social reality that observed events and relations are on the surface are rooted in deep structures. The three elements of critical research being insight, critique, and transformation are also outlined all through out the journal (Myers & Klein, 2011).
This article is not purely written with a positivist or interpretive perspective. Not positivist, because positivists generally tests theory in an attempt to gain predictive understanding of a phenomena. The article presents what is being done in a hospital environment and tries to build theory after the collection of data. There are no predictors or hypothesis discussed for testing. It is not interpretive research because they generally attempt to study phenomena through meanings that people give them. Its best used when exploring a research topic or theory which is not the case as no theory has been formed nor a research topic explored.
The article clearly mentions the research method used to conduct the study – ethnographic case study of a hospital’s ePDSS system that has secondary use. A research method used generally gives idea on the strategy of inquiry from underlying philosophical assumptions to research design and data collection. The method chosen here influences the way in which data is collected and thereby analyzed. As mentioned earlier, the research method used is ethnography but a case study is what has been performed as part of the study. It is a case study research method design but an ethnography research method was mainly in focus. The article mentions this by saying their “study was based on ethnographic observations of clinical settings and meetings” (p.427).
Ethnography is often reputed for being the most in-depth research methods possible because an ethnographer sees what people are doing in comparison to what they are supposed to be doing or what they say they are doing (Myers, 1999). We can say this is more of an ethnography than a case study research by one prominent factor of the extent to which an ethnographer immerses himself or herself in observing the lives of the subject. The authors of the article have claimed to spent 162 hours in observing the clinical settings, 28 hours in meetings and everything was recorded and coded for drawing statements and concluding facts on how use of ePDSS increased “risk of focusing attention on aspects of patient safety made visible by the system at the expense of other, less measurable but nonetheless important, concerns” (p.425).
Use of Ethnography has its benefits in sense that its in depth. Owing to their extended period of research and immersing in lives of subject, they are able to understand everything from work ethics to other entities involvement and its impact. Its quite different to other forms of research methods in one specific area of having knowledge of what happens in the field and gaining comprehensive understanding of how different variables play part. It is also known that ethnography often leads the research to question what someone would take for granted (Myers, 1999). The information provided from this study also shows a deeper understanding of the problem which was never discovered by the top management. Its seen in example of how top management assumes by using ePDSS which covers majority of the quality factors they would like to keep up with, their staff were performing well compared to industry standards. This was proved not to be the case by this study as they looked into how the very aspect they were concentrating on (quality and care) was ignored in its most basic form (talking to patients) (p.448).
All research method come with their limitations and ethnography is seen as no different. One of the main disadvantages is the time it takes to study the subjects involved and systems they are interacting with. As seen by the hours mentioned above, the study took a long time and even more was the time consuming task of organizing data into categories in a coding scheme (p.430). Another biggest disadvantage is the lack of breadth which is openly acknowledged by the authors carrying out the study. The group said, “It is a single case study in an acute care hospital in a single country” (p.446). They also acknowledged how they didn’t conduct formal interviews with those at the ‘sharp end’ due to the staff’s busy schedule through out their shifts. This may have consequences since most of the views they constructed have been focused mainly on the executive team and their opinions. Although, these limitations exist, I would agree with Myer’s in his article, the lack of generalizability can be conquered by use of developing more meaningful contexts of various aspects of information systems development and application (Myers, 1999).
Lastly, the theoretical framework of this article is based on bottom-up approach. It is said that qualitative research can be used to build or test theory. I believe the article uses inductive reasoning where the data is collected before a theory is built. This can be seen in their findings where they set out to examine the existing use of ePDSS as secondary use to improve patient’s safety and quality of care he/she received. After their examination of data, they found that ePDSS system was use to provide almost real-time data on clinical activity. It was harnessed to be used as intelligence to know more about the performance of individuals, teams, and clinical services.
They concluded that although this secondary use of ePDSS has its benefits in monitoring various quality indicators, it also has a negative impact on the very aspect the management was trying to monitor (quality of care and services provided). They end theory and concluding facts included the heavy focus on “specific aspects of practice or performance of easily identifiable professional groups while neglecting other parts of sharp-end reality” (p.449). They also suggested on “using alternative and complementary forms of intelligence, including evidence of patients’ experience and executive visits to clinical areas” which would help in gaining fuller picture of the quality of care and safety of a patient.
For a reader to comprehend the focus of the research and have a degree of clarity, researchers suggest their research domain (Dube and Pare, 2003). According to the article under review, their research area lies in use of Information Systems in Health Care Sector. This can be seen through their research on secondary use for ePDSS in a hospital.
The authors explicitly try to address the reason for their reason and the questions they are trying to answer. The problems they observed is the burden of data collection and processing for improvements of safety and quality of care provided to patients. It was seen as a challenging task since timely analysis and decision making were almost non-existent as data sets were manually processed to generate intelligence. Their research was to focus on use of these EHRs to make timely information on the quality and safety aspects for a given patient.
Authors says that research on secondary use of ePDSS data mainly focused on research purposes, public health surveillance, and technical questions relating to data validity and interoperability but evaluations of specific models of meaningful use for purposes of improving quality and safety of care in health care organizations has remained relatively less (p.426). They seek to find out whether ePDSS is being effective in ensuring and improving quality and safety.
The units of analysis have been explicitly stated in the article under review. In describing their study design, they mention the unit of analysis as being the hospital (p.429). This unit of analysis allows the researchers to observe how the quality of care and safety provided plays with staff’s interaction with ePDSS. It helps understand how their ethnographic study relates to a broader body of knowledge, and helps to conclude their theoretical limits of their study case (Dube and Pare, 2003).
Their data collection included a 162 hours in ethnographic observations which were recorded as field notes on four clinical areas within the hospital they were studying. This helped them collect the data in both acute and complex care environments. One of the limitations of their study included lack of formal interviews with the front-end staff owing to their busy schedule and work hours. But they were able to compensate this by holding informal chats and conversions that were also recorded as part of their observations. The collaboration between the researchers is also mentioned as being very high which helps them sync towards a common goal of addressing their proposed question.
None of the staff refused any researcher’s requests to observe or talk informally which gave them more insight into how the system worked as staff were more casual in their answers without the need to hide facts from the researchers. They were open about what they answered in relation to questions asked. They also used semi structured interviews with hospital senior staff members which were listed in part of their analysis. I believe this was a good choice of data collection as first hand experiences on how things are dealt with is more likely to be shared.
The researchers seem to have taken part in internal meetings which included fiver care omission meetings and eight additional meetings (28 hours) and these were related to the system they were using – ePDSS. The data analysis was based on constant comparative method which included “revising, expanding, and collapsing the initial open codes through detailed analysis and eventually organizing them into categories in a coding scheme” (p.430). They used a well known software for coding giving them chance to extract key terms from interviews and field notes to help the present their findings.
Another good aspect of their data collection and research was that it was carried out by an independent team who were not related to the hospital. This would help trust the findings and present them in more credible form than if they were involved as part of the management. With that said, there was no lack of technical expertise as their team specifically affiliated themselves with a member who was expert in using the ePDSS system. This would be the more favorable outcome as data is analyzed by those outside the hospital but retrieved by an expert who has experienced and works within this hospital environment. They also gained explicit approval from NHS Research Ethics Committee, which gave their study a more favorable option and support (p.430).
The researchers explicitly mention that their study design was case study with use of ethnographic methods as it was relevant to what their research questions were. They were trying to figure out whether ePDSS had an effective in truly improving the quality of care and safety and whether it had an undesirable output that were ignored or omitted. This was a good method to choose as they needed to understand the situations nurses were facing, the environment they were in, the decisions they took and many other variables. They were only able to perform this type of study if it was ethnographic as there was a real need to observe the subject carefully over a period of time.
According to Klein and Myers (1999), a set of principles are written for conducting and evaluation critical case studies and ethnographies. One of the best ways to analyze an ethnography research is by looking at the report itself (Myers, 1999). Some of the things to consider includes if the study is a contribution to the field. In our case, although the case study focuses on a single hospital, it is likely to apply to all hospitals using ePDSS system with similar patients and case histories. It definitely contributes to the field of hospitals providing greater quality of care and safety through use of ePDSS for monitoring physicians and nurses involved. Its found that as a proven fact since nearly 7% drop was observed in missed doses due to modifications made to this information system.
Another way to measure the appropriateness of the research method is through seeing if the author/researchers offer rich insights. Looking through the journal, many aspects that were otherwise omitted were revealed in the study. Even the management weren’t aware of these issues. Some of them were revealed through interviews where staff explicitly stated that one could mark a dose as being given without actually administering it just to avoid being called up for “unpleasant” meetings with the top executives. It also revealed some organization flaws of how physicians ordered the drugs a day before which was subsequently marked as missed dose. Later it was released through meetings that it occurred due to delay in delivering required drugs from the pharmacy. This was completely outside the study but was revealed through material/data collected on site.
The research method used has collected significant amounts of material/data. It helps understand the comprehensiveness of the conclusions drawn from observing and analysis the data. They were able to find the hidden agendas of management purposely picking on cases which were incontestable in nature. The executive team did this to focus the discussion in their meetings on what had gone wrong than arguing about whether it had gone wrong. Another aspect that was revealed was on how management was intensely concentrating on monitoring data from ePDSS that it neglected the very basic form of what they were seeking to uphold – quality of care and safety. The staff were pressured into concentrating much more on what was being monitored that they neglected what wasn’t. In other words, they didn’t have time to talk to patients and ask them how they were doing. Instead they were focused constantly on what aspects the management was monitoring. All these above aspects helped me evaluate the quality of the ethnographic study that was performed.
The secondary use of ePDSS had five consequences which the research has identified. The first is their routine work – staff were more inclined towards directing their attention towards work that they knew was being monitored by the ePDSS for “which they could be “caught” because their compliance (or lack of it) could be easily discovered” (p.443). Second consequence was that staff spend more time justifying what they missed out on than what they really should be concentrating on. For example, they were found to write length explanations as to why they missed a dose. Third consequence was that it gave rise to “game” the system. Since the number of times they went to British National Formulary website was monitored, newer staff were constantly on it even though they were aware of what the dosage was. Fourth consequence was the increase in gap between staff and executive team owing to heavy reliance on ePDSS and RCA data. Lastly, the nursing staff in particular believed the way this system operated didn’t distribute blame fairly. Nurses were more likely to be caught out and punished than physicians who were less interacting with the system. These were not able to be found out if this ethnographic case study wasn’t performed effectively enough.
The researchers wanted to address the question of whether ePDSS has worked on improving the quality and safety. As per the research conducted, it has proved that it is indeed effective in improving the quality of care provided and the security of patients. In performing the research, and through use of ethnography as the study design, researchers found what was otherwise omitted. They revealed their findings by having as source various techniques. These included ethnographic observations in clinical settings, meetings, informal conversations, semi structured interviews, and collection of relevant documents.
I believe they have comprehensive analyzed their data to reveal and address the question that it is effective but does have some undesirable consequences. Owing to their extensive research, they also suggest ways of improvement by using additional intelligence in combination with ePDSS as best way of moving ahead. It offers new insights for all parties involved and helps other hospitals with similar environment to adapt these findings to benefit from it.
Dubé, L., & Paré, G. (2003). RIGOR IN INFORMATION SYSTEMS POSITIVIST CASE RESEARCH: CURRENT PRACTICES, TRENDS, AND RECOMMENDATIONS. MIS Quarterly, 27(4), 597-635.
Myers, M. D., & Klein, H. K. (2011). A SET OF PRINCIPLES FOR CONDUCTING CRITICAL RESEARCH IN INFORMATION SYSTEMS. MIS Quarterly, 35(1), 17-36.
Myers, M. D., & Young, L. W. (1997). Hidden agendas, power and managerial assumptions in information systems development. Information Technology & People Info Technology & People, 10(3), 224-240. doi:10.1108/09593849710178225
Myers, Michael D. (1999) “Investigating Information Systems with Ethnographic Research,” Communications of the Association for Information Systems: Vol. 2, Article 23. Available at: http://aisel.aisnet.org.ezproxy.auckland.ac.nz/cais/vol2/iss1/23
Neuman, W. L. (1994). Social research methods: Qualitative and quantitative approaches.
Boston: Allyn and Bacon.
Ravishankar, M. N., Pan, S. L., & Myers, M. D. (2012). Information technology offshoring in India: A postcolonial perspective. European Journal of Information Systems Eur J Inf Syst, 22(4), 387-402. doi:10.1057/ejis.2012.32
Schultze, U. (2000). A Confessional Account of an Ethnography about Knowledge Work. MIS Quarterly, 24(1), 3. doi:10.2307/3250978
Urquhart, C. (2013). Grounded theory for qualitative research: A practical guide. Los Angeles, CA: SAGE.
Walsham, G. (1993). Interpreting information systems in organizations. Chichester, West Sussex, England: Wiley.
Yin, R. K. (2009). Case study research: Design and methods. Los Angeles, CA: Sage Publications.
Roshan Roy Jonnalagadda – University of Auckland – ISOM Student