Sturt University, Sydney, Australia 2Hewlett Packard

Usability Evaluation of Electronic Health Records Using User Integrated Heuristic Walkthrough Method N. Wijesinghe1, P.W.C. Prasad1, Abeer Alsadoon1, L. Pham1, A. Elchouemi2 1School of Computing and Mathematics, Charles Sturt University, Sydney, Australia 2Hewlett Packard Enterprise Abstract—This research paper aims to evaluate to what extent users such as medical professionals can collaborate with other evaluators in the usability evaluation of Electronic Health Records (EHRs). The Heuristic Walkthrough (HW) method is a common technique in evaluating the usability of EHRs. It consists of two usability methods, Heuristic Evaluation (HE) and Cognitive Walkthrough (CW). After considering all participants, parameters, user interface elements and the usability characteristics in the evaluation process, issues have been identified in some areas of the HE and CW methods. As key findings, this study identifies lack of usability issues in EHRs negatively affecting user satisfaction with both techniques in the current HW method. The main aim of this paper is to evaluate and extend the existing model and propose a new method termed ‘User Integrated HW’ (UIHW) to eliminate the existing issues. Therefore a Likert scale is implemented together with an openended questionnaire for both, HE and CW methods, to enhance the user integration as well as to obtain user reviews about the subject matter to support usability expertise. Keywords—Electronic Health Records, Medical Information Systems, Usability Engineering, Heuristic Walkthrough, Usability Evaluation I. INTRODUCTION The Electronic Health Record (EHR) system is the most widely used application in Hospital Information Systems (HISs) to facilitate recording of patient information by medical professionals. Evaluation of the usability of EHRs helps to improve patient care and safety and assists physicians and other professionals to document their daily work with patients [1]. There are three types of usability evaluation methods, namely inquiry, inspection and testing modes based on gathered user data, user observation and user tests respectively [2]. There are two common inspection methods identified as Heuristic Evaluation (HE) and Cognitive Walkthrough (CW). The combination of the structure and techniques of both CW and HE methods implements the Heuristic Walkthrough (HW) method [3]. Jakob Nielsen, a well-known usability consultant has defined five traditional usability characteristics as learnability, efficiency, memorability, errors and satisfaction [4]. There are advantages as well as disadvantages arising from the usability assessment methods in terms of ease of learning, forecasting and generalizability [4]. This study identifies the issues in the current HW method, and, introduces the User Integrated Heuristic Walkthrough (UIHW) method together with justification of its strengths for an enhanced usability evaluation of EHRs. Ultimately, the result demonstrates improvements in usability that reduce the number of errors, lead to improved patient safety and efficiency and enable physicians to spend more time with patients [1]. This paper is arranged as follows. The literature review is explained in section II. Section III outlines the proposed UIHW method followed by a detailed evaluation. The last section contains the final conclusion. II. LITERATURE REVIEW A. Types of Usability Evaluation Methods for Electronic Health Records Generally an EHR is comprised of components of order entry, documentations of patient admissions and medication administration [5]. A usability evaluation method assists in identifying the usability issues in EHR embedded in these elements. The investigation is carried out by members of the evaluation team as shown in Fig. 1. Different evaluation methods are currently in use for various types of EHRs, for paediatric, cardiac, dental, mental and other health issues. These have been identified in the literature review of the EHR usability evaluation studies as shown in Fig. 2. According to the findings in expert analysis, cognitive walkthrough (CW), heuristic evaluation (HE) and expert-based evaluation are key inspection methods while focus groups are used as a method of inquiry. Technical consultation is an inspection method and interviews and surveys are inquiry methods in the category of user participation. Further, Empirical methods, monitoring physiological responses and techniques based on observation are methods of user testing. However there are numerous approaches available to measure the usability, regardless of its importance [2]. B. Assessment of Usability Evaluation Methods Similarities can be found between various usability evaluation methods. However, there are also unrelated sets of processes which provide benefits and drawbacks at different levels. While surveys are used with prototypes, site observations and interviews are conducted to collect user information in usability inquiry methods [2]. In HE inspection method, there is little input required from expert evaluators to explore the effects of the application. [6] However, when Multi-faceted Usability Evaluation (MUE) mechanisms including CW inspection methods and think aloud inquiry methods are utilized to find the interaction between users and the application [7] therefore in a multiple method strategy combination of inquiry methods such as surveys and interviews, model based testing methods and the HE inspection method are required [8]. Although it is evident from the foregoing that there are many usability evaluation methods available, they do not capture all the usability issues in EHRs. Fig. 1. EHR Entity Relationship C. Introduction of the Heuristic Walkthrough Method The HW method is not only a commonly used evaluation practice, but also a low-cost usability inspection method to evaluate electronic health records. It helps to identify the usability issues and the level of the usability. As the HW method is comprised of both CW and HE methods, it uses a two-pass approach – task-oriented and free-form evaluation. The current HW method, as shown in Fig. 3, is implemented by the evaluation team focusing on the parameters to identify the usability issues related to the interface features based on the usability characteristics. Inspection of the usability of a prototype is carried out with the participation of usability experts, software developers, the physician and nurses [2]. Usability inspection methods are recommended rather than user testing methods, because they are low cost and require less formal training [1]. D. Issues with the Heuristic Walkthrough Method Despite the advantages of the HW method, however, researchers point out limitations associated with the method. The combination of the HE and CW methods exposes a lack of usability, which is the primary aim of every usability evaluation study [9]. Therefore the HW method has noticeable drawbacks for each technique individually as well as for the method overall. Fig. 2. Systems & Usability Evaluation Method Issues in the Heuristic Evaluation Method: Several weaknesses have been identified in the HE technique. When considering the HE method, it is clear that the heuristic principles only consider user interface features and exclude external aspects [10]. Therefore it identifies the issues related to known characteristics [5]. Another notable drawback is unavailability of a proper source of user reviews despite the involvement of physicians and nurses in the evaluation process to assist other experts. Also, it is recommended that the evaluation possess both subject matter and usability expertise as there is increased complexity in terms of the usability evaluation [5]. It is evident that outcomes are influenced by expert reviewers as the predictive evaluation approach depends on expert judges to identify the usability problems [5]. As an organizational issue, heuristic help and documentation as defined by Nielsen are not available in most EHRs as almost every hospital has twenty-four hour help-desks [4]. Every healthcare provider expects maximum advantage from every usability eva
luation method. Issues in the Cognitive Walkthrough Method: Weaknesses also exist in terms of the CW method which identifies the issues based on user tasks within the system, and lacks a holistic view in the identification of other problems. Also important is the influence of evaluators on outcomes as the CW method does not gain other professional reviews [7]. Furthermore, due to an emphasis on low level information, evaluators may underestimate the complexity of system workflows in the CW method [7]. Additionally, there are some tasks which may be included in the CW technique which require users to use computers in a discontinuous manner that leads to difficulties with the evaluation of usability [5]. Furthermore the CW method costs more than other types of studies due to labour intensive processes [7]. III. THE USER INTEGRATED HEURISTIC WALKTHROUGH METHOD The proposed framework utilizes a Likert scale and an open-ended questionnaire in the HE and the CW methods respectively as shown in Fig. 4. They are introduced to increase user participation and to capture their opinions to strengthen certain areas in the evaluation process. Such additional support was not provided in previous models. A. Introduction of the Likert Scale into the Heuristic Evaluation Method EHR features and usability expertise are the known main parameters in the HE method. As it is currently problematic to assess user satisfaction with EHRs, the new model plans to obtain these data through the addition of a Likert scale. This consists of questions based on usability principles which are present in greater concentration in the HE method. The participants are able to rate questions along a scale which helps to identify usability issues related to user satisfaction. The main purpose is to collect user opinions by inviting users to participate actively in the evaluation process. Fig. 3. Heuristic Walkthrough Method B. Introduction of the Open-Ended Questionnaire into the Cognitive Walkthrough Method As domain expertise is more applicable in the CW method, user tasks as well as thought-focusing questions which comprised with standard questions are the main features existing in that method. It is recommended to use an openended questionnaire in CW method in the new framework. The main purpose of the open-ended questionnaire is to collect opinions from the users through their participation in the usability evaluation. This is also a great tool to catch the usability issues in EHRs associated with the user satisfaction. C. Framework of the User Integrated Heuristic Walkthrough Method Usability expert, software developer, physician and nurse participate in the HE method. Admin clerk and health informatics consultant join with aforementioned participants for the usability evaluation in CW method. Internal parameters such as EHR features and Likert scale together with an external parameter, usability expertise are the factors in HE method. On the other hand, CW method has elements known as user tasks, thought-focusing questions and open-ended questions. All these parameters assess all the user interface elements such as window, icons, menus, pointers, controls (widgets) and tabs in the EHRs. All the features identify issues related to the usability characteristics mainly learnability, efficiency, memorability and errors while introduced Likert scale and open-ended questionnaire specifically catch problems associated with satisfaction. IV. EVALUATION OF THE USER INTEGRATED HEURISTIC WALKTHROUGH METHOD As introduced above the Likert scale is implemented in the HE method and the open-ended questionnaire is implemented in CW method to foam the new User Integrated Heuristic Walkthrough (UIHW) method. The main purpose is to eliminate the issues discussed earlier in the HW method by integrating active user participation. A. Evaluation of the Likert Scale in the Heuristic Evaluation Method EHR features are compared against usability principles known as heuristics as shown in Table I [5]. Although it is the main task implementing in HE method, the modified Likert scale which comprised with eight questions based on heuristics as shown in Table II [2] is implemented to collect opinions from the user such as physicians and nurses. The users are able to rate those questions with a 7-point scale ranging 1 (=‘least satisfied’) to 7 (=‘most satisfied’) which help to identify the user satisfaction level followed by the usability issues in EHRs. Fig. 4. User Integrated Heuristic Walkthrough (UIHW) Method TABLE I. USABILITY HEURISTICS Usability Heuristics Description Visibility of system status The system should always keep users informed about what is going on, through appropriate feedback within reasonable time Match between system and the real world The system should speak the users’ language, with words, phrases and concepts familiar to the user, rather than system-oriented terms. Follow real-world conventions, making information appear in a natural and logical order User control and freedom Users often choose system functions by mistake and will need a clearly marked “emergency exit” to leave the unwanted state without having to go through an extended dialogue. Support undo and redo Consistency and standards Users should not have to wonder whether different words, situations or actions mean the same thing. Follow platform conventions Error prevention Even better than good error messages is a careful design, which prevents a problem from occurring in the first place Recognition rather than recall Make objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another. Instructions for use of the system should be visible or easily retrievable whenever appropriate Flexibility and efficiency of use Accelerators unseen by the novice user may often speed up the interaction for the expert user such that the system can cater to both inexperienced and experienced users. Allow users to tailor frequent actions Aesthetic and minimalist design Dialogues should not contain information that is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility Help users recognize, diagnose and recover from errors Error messages should be expressed in plain language (no codes), precisely indicate the problem and constructively suggest a solution Help and documentation Even though it is better if the system can be used without documentation, it may be necessary to provide help and documentation. Any such information should be easy to search, focused on the users task, list concrete steps to be carried out and not be too large TABLE II. ITEMS USED IN LIKERT SCALE 1. The interface sizes are helpful for information entry 2. The interfaces help during the information entry 3. The program flow helped me 4. Information entry is practical 5. The order of the menu system is helpful to the information entry 6. Using the software system is faster than filling the printed form 7. The order of the menu system is similar to the printed form 8. Reading and following the form information is faster than reading from papers B. Evaluation of the Open-Ended Questionnaire in the Cognitive Walkthrough Method Following user steps of users to use the system or to achieve a task is the main responsibility in CW method [2]. Therefore thought-focusing questions which comprised with four questions as shown in Table III [5] are used to assess the user tasks in EHRs. As Expert evaluators are the main participants in the evaluation team the modified open-ended questionnaire consisting of ten questions as shown in Table IV [4] is used to facilitate active user participation in the CW method. These questions are based on common user tasks through which users mostly interact with EHRs. This component is used to collect opinions not only from physicians and nurses but also from admin clerks and health informatics consultants to identify usability issues associated with user satisfaction. TABLE III. THOUGHT-FOCUSING QUESTIONS 1. Will users
know what they need to do next? 2. Will users notice that there is a control available that will allow them to accomplish the next part of their task? 3. Once users find the control, will they know how to use it? 4. If users perform the correct action, will they see that progress is being made toward completing the task? Does the system provide appropriate feedback? TABLE IV. OPEN-ENDED QUESTIONS 1. What do you think about the time taken to enter data? 2. What do you think about the time taken to visualize data? 3. What do you think about windows, titles and contents? 4. What do you think about the structure of the EHR? Is it simple and clear, avoiding unnecessary complications? 5. What do you think about the design of the EHR? Is it aesthetically pleasant? 6. What do you think about the language used? Is it simple and appropriated? 7. What do you think about the registration time? Is it possible to reduce the time by copying or modifying existing information? 8. What unnecessary data does the EHR contain? 9. What are the default values which appropriated? 10. What are the prompts, warnings and messages appear in prominent locations? Which locations? C. Advantages of the Likert Scale In the proposed UIHW method, the new HE technique is implemented at an early stage (in the design phase) which is more beneficial in the usability evaluation as well as in the design process. Most importantly, it allows identification not only of usability issues in the EHR, but also in terms of the user satisfaction level. The Likert scale allows experts to gain reviews from the users such as physicians and nurses through their active participation in the evaluation process. Also, the introduction of this feature into the HE method helps to clarify domain and usability expertise separately. Since usability experts dominate the HE method, active participation of the users reduces the influence by those expert reviewers. D. Advantages of the Open-Ended Questionnaire The new CW technique of the UIHE method is able to provide a holistic view by introducing the open-ended questionnaire to upgrade the investigation into ease of learning in EHRs. It allows the users such as physicians, nurses, admin clerks and health informatics consultants to express their ideas openly in response to the questions which are based on the user tasks in the interface. Therefore this feature allows the use of other professional reviews in the usability evaluation. Participation of different users assists to identify low-level details related to the user tasks precisely. Therefore it allows an estimate of the extent and the complexity of workflow in the EHR system. E. Benefits of the User Integrated Heuristic Walkthrough Method The main advantage of this new UIHW method is to gain user opinions to identify more usability issues connected with satisfaction characteristic in the usability of EHRs. The HE technique can be initiated in the beginning of the design phase and the CW technique can be implemented later to identify the design errors in the interface. The new method helps to avoid expensive user tests based on prototypes. Also by introducing health informatics consultants into the new technique will be helpful to maintain the EHR. Importantly, the UIHW method which is comprised of both usability and domain expertise assists in implementing a well-balanced usability evaluation of EHRs. V. CONCLUSION In this study, a new technique called User Integrated Heuristic Walkthrough (UIHW) has been introduced to address the identified main issues existing in the evaluation of usability of Electronic Health Records (EHRs) through the Heuristic Walkthrough (HW) method. The main purpose of this new method is to identify satisfaction characteristic relating to usability by collecting user opinions with active participation of real users. The evaluation team is comprised of experts and users who have usability and domain expertise respectively. The new framework is a hybrid method which is comprised of the Heuristic Evaluation (HE) method and Cognitive Walkthrough (CW) method. In the UIHW method, a Likert scale is embedded into the HE method and an open-ended questionnaire is added to the CW method to collect user opinions. Although evaluation through the new method eliminates issues with the HW method, drawbacks remain in terms of the UIHW model. Therefore it is recommended to increase user involvement through including it in both the HE and the CW methods. Also, to obtain positive results through the implementation of this new method, it is suggested to include relevant questions in both the Likert scale and the open-ended questionnaire, appropriate to the EHR system of the medical discipline. Even though cost and time factors are still excessive, the proposed method helps to improve the usability of EHR systems effectively and efficiently. REFERENCES [1] R. Pereira, J. Duarte, M. Salazar, M. Santos, J. Neves, A. Abelha, and J. Machado, “Usability evaluation of Electronic Health Record,” in Biomedical Engineering and Sciences (IECBES), 2012 IEEE EMBS Conference on , pp. 359-364, 17-19 Dec. 2012. [2] A. Karahoca, E. Bayraktar, E. Tatoglu, and D. Karahoca, “Information system design for a hospital emergency department: A usability analysis of software prototypes,” Journal of Biomedical Informatics, vol. 43, no. 2, 2010, pp. 224-232. [3] M. F. Walji, E. Kalenderian, M. Piotrowski, D. Tran, K. K. Kookal, O. Tokede, J. M. White, R. Vaderhobli, R. Ramoni, P. C. Stark, N. S. Kimmes, M. Lagerweij, and V. L. Patel, “Are three methods better than one? A comparative assessment of usability evaluation methods in an EHR,” International Journal of Medical Informatics, vol. 83, no. 5, 2014, pp. 361-367. [4] R. Pereira, J. Duarte, M. Salazar, M. Santos, A. Abelha, and J. Machado, “Usability of an electronic health record,” in Industrial Engineering and Engineering Management (IEEM), 2012 IEEE International Conference on , pp.1568-1572, 10-13 Dec. 2012. [5] P. J. Edwards, K. P. Moloney, J. A. Jacko, and F. Sainfort, “Evaluating usability of a commercial electronic health record: A case study,” International Journal of Human-Computer Studies, vol. 66, no. 10, 2008, pp. 718-728. [6] N. Staggers, S. Iribarren, J.-W. Guo, and C. Weir, “Evaluation of a BCMA’s electronic medication administration record,” Western Journal of Nursing Research, vol. 37, no. 7, 2015, pp. 899-921. [7] L.A. Grabenbauer, A.L. Fruhling, and J.R. Windle, “Towards a cardiology/EHR interaction workflow usability evaluation method,” 2014 47th Hawaii International Conference on System Sciences (HICSS), pp.2626-2635, 6-9 Jan. 2014. [8] T. C. Lacerda, C. G. von Wangenheim, A. von Wangenheim, and I. Giuliano, “Does the use of structured reporting improve usability? A comparative evaluation of the usability of two approaches for findings reporting in a large-scale telecardiology context,” Journal Of Biomedical Informatics, vol. 52, 2014, pp. 222-230. [9] R. Khajouei, L. Ahmadian, and M. W. M. Jaspers, “Methodological concerns in usability evaluation of software prototypes,” Journal of Biomedical Informatics, vol. 44, no. 4, 2011, pp. 700-701. [10] L. Longo, and B. Kane, “A novel methodology for evaluating user interfaces in health care,” 2011 24th International Symposium on Computer-Based Medical Systems (CBMS), pp.1-6, 27-30 June 2011.

Leave a Reply

Your email address will not be published. Required fields are marked *