Loading [Contrib]/a11y/accessibility-menu.js
Eeles E, Tronstad O, Flaws D, Teodorczuk A, Worthy P, Dissanayaka N. Adaptation of an electronic delirium screening tool for use in the medical setting based on a human centered design approach. Delirium Communications. Published online July 7, 2023. doi:10.56392/001c.77497
Download all (2)
  • Figure 1. Overview of the development process
  • Figure 2. An example of one of the questions taken from eDIS-MED after introduction of a skip function



Delirium is frequently missed in clinical practice. We used a user-centred redesign process to evaluate and adapt an existing electronic delirium screening tool (eDIS-ICU) for use in the medical setting.


In phase 1, we conducted a brainstorming session to establish context for delirium screening tools in medicine. In phase 2, a pluralistic walkthrough of eDIS-ICU was performed to identify prospective usability in the medical setting. We then extracted positive and negative qualities of eDIS-ICU. In phase 3, recommendations for change were made.


Pluralistic walkthrough highlighted that eDIS-ICU related to the key groups of functionality, diagnosis, links with management and potential integration with clinical information. Recommended changes to make eDIS-ICU suitable for use in a medical setting included the need for skip function, prior instructions and streamlined testing.


A human-centred redesign created a pilot electronic delirium screening tool for use in a general medical setting (eDIS-MED).


Delirium diagnosis is a common challenge in the general medical setting.1 Not only is delirium missed in most cases, but this omission also worsens outcomes.2 Criticisms of some of the commonly used delirium screening tools include the potential for re-test error and the requirement for extensive training.3 Many tools test single domains, such as attention, and fail to measure the full spectrum of DSM criteria.3 Technological solutions offer potential advantages in terms of the test construct. Examples include: (i) visual distractors to help evaluate sustained attention; (ii) randomised and rotating questions to help mitigate re-test error; and (iii) in app generated diagnosis to reduce user bias to final delirium status.4 Also, ergonomic design, adaptive features, and convenience can help lend user-friendly advantages to the uptake of mobile device-based screening tools.5

For these reasons, authors EE, OT, DF and and a multidisciplinary research group (Critical Care Research Group) have previously developed an electronic delirium screening tool [eDIS-ICU] that we successfully tested in the intensive care setting.6,7 eDIS-ICU incorporates user-centred, delirium-specific and technological advantages. Firstly, a large bank of potential questions is randomised after each test, thus avoiding re-test error. Secondly, eDIS-ICU is more dimensional than single attention tests as it targets all DSM criteria for delirium. Thirdly, adaptation to the delirium syndrome includes variable test frequency that reduces redundancy, test burden and potentially time to diagnosis.

Delirium presents very similarly, regardless of the place of onset. However, the characteristics of patients in the medical setting is distinct from that of the intensive care environment with more prevalent dementia and perceptual impairment.1,2 Here, we describe a process of redesigning eDIS-ICU, focusing on delirium screening within the medical setting.


We conducted a human-centred redesign, guided by: (i) the requirements of a screening tool and characteristics of a general medicine setting; (ii) the usability of DIS-ICU in its current format; (iii) implementing redesign based on user feedback (Figure 1). Human-centred design is a philosophy that directly involves users in designing a service or technology.8 Human-centred design approaches are often applied to redesign an existing service or technology.9 We conducted the redesign in February 2023 in a face-to-face setting at a metropolitan hospital, facilitated by one of the original eDIS-ICU co-designers (EE). An advanced trainee in medicine, physiotherapist, nurse, and nurse educator who all worked in a general medicine setting participated. A clinical researcher representative from the app development team (author OT), software developer and a patient consumer also participated. Four individuals were male, three were female. All clinicians had at least 5 years of postgraduate experience.

Figure 1
Figure 1.Overview of the development process


Phase 1: Defining the Context and needs assessment. We brainstormed to: (i) establish the testing environment; and (ii) define the requirements of a delirium screening tool, both within an internal medicine setting.8,10 Affinity mapping was conducted to group information generated from brainstorming into related clusters, and we ordered ideas into themes.11

Phase 2: Usability inspection. We performed a pluralistic walkthrough, which involves end-users, software developers, and researchers stepping through interactions with a technology, focusing on and discussing its usability and identifying potential issues.12 These walkthroughs support a clearer understanding of user needs when working with the technology. In a pluralistic walkthrough, the reviewers progress through a simulation or run-through of the test under question and are invited to comment. Responses are captured and used to inform usability and potential areas for further development.

First, we provided a brief orientation to delirium and a standard approach to screening, courtesy of the Western Australia Clinical Training Network. Second, we presented an eDIS-ICU video comprising a run-through of delirium screening against different possible outcome scenarios (delirium absent, delirium present, delirium possible). The facilitator (EE) ran the simulation, stopping regularly for comments and questions.13 The walkthrough was voice recorded and later transcribed by the facilitator. A copy of DSM-5 delirium diagnostic criteria was made available to attendees for reference.14 Third, participants were asked to write down positive and negative attributes of eDIS-ICU.15 The facilitator collected these responses and grouped them into themes after discussion.

Phase 3: Implementing redesign. We identified actionable items based on the feedback generated from the pluralistic walkthrough. The development group (EE, OT and PW) used an iterative approach to decide on potential areas for change. Corresponding adaptations were made to eDIS-ICU. This modified version represented the prototype eDIS-MED.

Ethics approval was granted [HREC/2021/QPCH/74715].


Phase 1 brainstorming generated eighty-seven statements. Context responses were clustered into patient factors (easier or more difficult to test), staff and ward characteristics (Appendix 1). Ideal features of a delirium screening tool were grouped into: diagnosis, links with management, functionality, and integration (Table 1).

Table 1.Results of affinity mapping in relation to the perceived ideal properties of a delirium screening tool in a medical setting with responses grouped into themes
Diagnosis Links with Management Functionality Integration
Accurate and generalisable Point of care tool Quick and Easy to use Easy to access
Reliable Track improvement Minimal education Link with electronic health records
Delirium severity Link with treatment options Special groups View responses, not just diagnosis
Distinguish from dementia Link with treatments undertaken No learning effect For use on tablet or mobile phone

Regarding functionality, participants identified that eDIS-ICU was short, portable, non-repetitive and adaptive in design, as highlighted by: [doctor] ‘eDIS-ICU asked fewer questions if the patient has delirium.’ By contrast, even though the number of questions was seen as an asset, the number of steps for each was noted as problematic: [doctor] ‘you have to click it three times before showing it to the patient.’ And [nurse] ‘yes, it’s very clicky.’

While there is an abort function embedded in eDIS-ICU if the patient is deemed too drowsy or agitated according to Richmond Agitation-Sedation Scale (RASS), this was still identified as an area of uncertain utility in medicine: [physiotherapist] ‘I have never seen RASS used in a medical ward.’16 Also, the functionality of eDIS-ICU was considered more research-focused than clinically useful: [physiotherapist] ‘in a practice partnership with nurses, how will we know that John in bed three hasn’t had the test ten times if it isn’t recorded somewhere?’. Only the delirium status was reported at the end of the screening; the lack of availability of the actual responses informing diagnosis was identified as a potential loss of valuable clinical data.

In terms of diagnosis, eDIS-ICU tested all the DSM criteria for delirium: [doctor] ‘it targets all the diagnostic domains affected in delirium.’ eDIS-ICU was felt to be appropriate for the target population: [doctor] ‘covers the majority of the general medical patient population.’

However, some groups were deemed at risk of being marginalised from testing with eDIS-ICU: [physiotherapist ‘what about visual problems and hemianopia?’. Suitability in the context of cultural diversity and limited education was another potential shortcoming, [consumer] ‘if you have someone who had no education or is from a rural background, then they will just stare at it.’

There was an impression that some of the questions were suggestible. In such an instance, an incorrect response may have arisen in the absence of knowledge rather than belief in the wrong response. For instance: [nurse educator] ‘if you had no idea where you are and saw a picture of an airport, then you would just answer yes.’ The importance of this distinction was debated: [physiotherapist] ‘it’s just an orientation question, so does it matter why it is answered incorrectly?’ Additionally, not all the questions were endorsed. Some were reported as being potentially [IT] ‘a bit confusing.’ It was suggested that the binary nature of submitted answers could be a possible confounder, [IT] ‘if they can’t answer the question then that’s it?’ Solutions to this potential problem were highlighted:[IT] ‘a skip function would work.’ (Figure 2).

Figure 2
Figure 2.An example of one of the questions taken from eDIS-MED after introduction of a skip function

A correct or incorrect response from the patient (no and yes, respectively) is registered, and the next question is presented. If an individual is unsure or unable to answer the revised question, registered in the unsure response in figure 2, then a skip function takes the clinician to the next question without scoring the omission.

Other future positive facets of eDIS-ICU included the potential ability for a simple translation into other languages for more general use. There were also general concerns around the feasibility of using an electronic device in the clinical setting. First, the availability of devices in the clinical setting was deemed problematic: [physiotherapist ‘there is a lack of access to iPads in the ward.’ Also, hygiene was expressed as a concern: [physiotherapist] 'personal phones in the clinical area are gross!’

The design team established the areas for actionable redevelopment and included and implemented focussed solutions in response to the findings of the pluralistic walkthrough and positive/ negative feedback.

Implementing Redesign

The following features were integrated into EDIS-ICU in response to the feedback, and the prototype eDIS-MED was created (Table 2).

  1. Diagnosis: a skip function has been integrated so that an alternative is presented if a patient cannot respond to a question.

  2. Functionality: The number of user steps per question was reduced.

  3. Management: An introductory slide to affirm intended use and raise awareness of the abort function in case of agitation/drowsiness.

  4. Integration: a process for making individual responses more readily available is underway.

Table 2.Main areas for eDIS-MED development and redesign
Attributes Diagnosis Links with Management Functionality Integration
Positive Targets DSM domains If abnormal RASS then provides advice Simple and uncluttered Potential digital repository
Negative Binary responses RASS not used in medicine Unnecessary steps Response to questions not easily available
Redesign Skip function Introductory advice slide Reduced steps Show item responses


The results showed that eDIS-ICU was easy to use, had adaptive design features, tested the pertinent domains affected in delirium and potentially provided point-of-care diagnostics. Actionable areas for change were identified and implemented. The redesign process created the next prototype, eDIS-MED, and further validation work is planned.

Compared with other electronic delirium screening tools, the proposed skip function has also been successfully introduced into the mobile app for delirium screening.17 Additionally, DelApp, has built-in instructions, scoring criteria, and is designed for use by non-experts.4 Unlike eDIS-MED, an inability to answer questions is determined as incorrect, according to DelApp.

There are several limitations to our study. Firstly, the individuals in our pluralistic walkthrough did not have the opportunity to interact with eDIS-ICU directly and could only view the functionalities remotely. Despite this, respondent feedback clearly identified functionality issues. Further, the nature of a pluralistic walkthrough is not based on direct and independent interaction as it is not a ‘test’ of usability. Instead, it is a collaborative inquiry into usability to inform further development/ improvement of a technology.

Before clinical studies, eDIS-MED will be subject to further content evaluation against an expert group to assess further the issue of suggestibility and possible confusion in some of the questions. It should be emphasised that eDIS-MED is intended for use as a screening tool prioritising brevity and high-performance sensitivity. As in other studies, concerns about the general use of mobile devices in the clinical setting were raised.17 In order to mitigate hygiene concerns, eDIS-ICU/eDIS-MED has been designed for contactless use with the patient.

A collaborative human-centred redesign process has created a mobile delirium electronic screening tool (eDIS-MED) for use in the general medical setting. Once adequately validated, this has the potential to bring a user-friendly, adaptive and dimensional delirium screening tool to assist frontline clinicians in making an accurate and timely diagnosis of delirium.

Conflicts of interest

All authors declare that they have no conflicts of interest.


We would like to thank research assistant Anne Bucetti, The Critical Care Research Group, Advancing Queensland Fellowship and the multidiciplinary panel of consumer/ experts

Accepted: July 20, 2022 CEST


Wilson JE, Mart MF, Cunningham C, et al. Delirium. Nat Rev Dis Primers. 2020;6(1):90. doi:10.1038/s41572-020-00223-4
Google ScholarPubMed CentralPubMed
Bellelli G, Nobili A, Annoni G, et al. Under-detection of delirium and impact of neurocognitive deficits on in-hospital mortality among acute geriatric and medical wards. Eur J Intern Med. 2015;26(9):696-704. doi:10.1016/j.ejim.2015.08.006
Google Scholar
Brefka S, Eschweiler GW, Dallmeier D, Denkinger M, Leinert C. Comparison of delirium detection tools in acute care: a rapid review. Z Gerontol Geriatr. 2022;55(2):105-115. doi:10.1007/s00391-021-02003-5
Google ScholarPubMed CentralPubMed
Rutter LM, Nouzova E, Stott DJ, et al. Diagnostic test accuracy of a novel smartphone application for the assessment of attention deficits in delirium in older hospitalised patients: a prospective cohort study protocol. BMC Geriatr. 2018;18(1):217. doi:10.1186/s12877-018-0901-5
Google ScholarPubMed CentralPubMed
Brouillette RM, Foil H, Fontenot S, et al. Feasibility, reliability, and validity of a smartphone based application for the assessment of cognitive function in the elderly. PLoS One. 2013;8(6):e65925. doi:10.1371/journal.pone.0065925
Google ScholarPubMed CentralPubMed
Eeles E, Gunn H, Sutt AL, et al. e-Screening revolution: A novel approach to developing a delirium screening tool in the intensive care unit. Australas J Ageing. 2018;37(2):147-150. doi:10.1111/ajag.12525
Google Scholar
Sutt AL, Flaws D, Gunn H, et al. Screening for delirium in the intensive care unit using eDIS-ICU – A purpose-designed app: A pilot study. Aust Crit Care. 2021;34(6):547-551. doi:10.1016/j.aucc.2020.12.008
Google Scholar
Giacomin J. What is human centred design? The Design Journal. 2014;17(4):606-623. doi:10.2752/175630614x14056185480186
Google Scholar
Merkel S, Kucharski A. Participatory design in gerontechnology: A systematic literature review. The Gerontologist. 2019;59(1):e16-e25. doi:10.1093/geront/gny034
Google Scholar
Paul BP, Kenworthy JB. Effective brainstorming. In: The Oxford Handbook of Group Creativity and Innovation. ; 2019:287-386.
Google Scholar
Holtzblatt K, Beyer H. The affinity diagram. In: Contextual Design. 2nd ed. ; 2017:127-146. doi:10.1016/b978-0-12-800894-2.00006-5
Google Scholar
Sharp H, et al. Interaction Design : Beyond Human-Computer Interaction. John Wiley & Sons, Incorporated; 2019.
Google Scholar
Martin B, Hanington B. Universal Methods of Design: 100 Ways to Research Complex Problems, Develop Innovative Ideas, and Design Effective Solutions. Digital ed. Rockport Publishers; 2019.
Google Scholar
American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders. 5th ed. American Psychiatric Publishing; 2013. doi:10.1176/appi.books.9780890425596
Google Scholar
Rödle W, Wimmer S, Zahn J, et al. User-Centered Development of an Online Platform for Drug Dosing Recommendations in Pediatrics. Appl Clin Inform. 2019;10(4):570-579. doi:10.1055/s-0039-1693714
Google ScholarPubMed CentralPubMed
Ely EW, Truman B, Shintani A, et al. Monitoring sedation status over time in ICU patients: reliability and validity of the Richmond Agitation-Sedation Scale (RASS). JAMA. 2003;289(22):2983-2991. doi:10.1001/jama.289.22.2983
Google Scholar
Armstrong B, Habtemariam D, Husser E, et al. A mobile app for delirium screening. JAMIA Open. 2021;4(2):ooab027. doi:10.1093/jamiaopen/ooab027
Google ScholarPubMed CentralPubMed


Appendix 1.Affinity mapping in relation to the perceived medical context
Patients easier to screen Patients more difficult to screen Ward characteristics Staff characteristics
Less illness severity Reduced nursing ratios More ward transfers Reduced nurse-to-patient ratios
Able to communicate Older, frailer with longer length of stay Outlying patients Attitudes of staff may be different
Aetiologies differ More agitation Technology availability Family engagement
More responsive Hygiene Greater patient freedom