Event-based knowledge elicitation of operating room management decision-making using scenarios adapted from information systems data

<p>Abstract</p> <p>Background</p> <p>No systematic process has previously been described for a needs assessment that identifies the operating room (OR) management decisions made by the anesthesiologists and nurse managers at a facility that do not maximize the efficiency of use of OR time. We evaluated whether event-based knowledge elicitation can be used practically for rapid assessment of OR management decision-making at facilities, whether scenarios can be adapted automatically from information systems data, and the usefulness of the approach.</p> <p>Methods</p> <p>A process of event-based knowledge elicitation was developed to assess OR management decision-making that may reduce the efficiency of use of OR time. Hypothetical scenarios addressing every OR management decision influencing OR efficiency were created from published examples. Scenarios are adapted, so that cues about conditions are accurate and appropriate for each facility (e.g., if OR 1 is used as an example in a scenario, the listed procedure is a type of procedure performed at the facility in OR 1). Adaptation is performed automatically using the facility's OR information system or anesthesia information management system (AIMS) data for most scenarios (43 of 45). Performing the needs assessment takes approximately 1 hour of local managers' time while they decide if their decisions are consistent with the described scenarios. A table of contents of the indexed scenarios is created automatically, providing a simple version of problem solving using case-based reasoning. For example, a new OR manager wanting to know the best way to decide whether to move a case can look in the chapter on "Moving Cases on the Day of Surgery" to find a scenario that describes the situation being encountered.</p> <p>Results</p> <p>Scenarios have been adapted and used at 22 hospitals. Few changes in decisions were needed to increase the efficiency of use of OR time. The few changes were heterogeneous among hospitals, showing the usefulness of individualized assessments.</p> <p>Conclusions</p> <p>Our technical advance is the development and use of automated event-based knowledge elicitation to identify suboptimal OR management decisions that decrease the efficiency of use of OR time. The adapted scenarios can be used in future decision-making.</p

Competency Assessment

Assessment is an essential feature of the competency-based educational model because only by means of evaluation can we verify achievement of specified learning outcomes. This is especially important in the context of health professions education, where the competencies of interest impact the well-being of patients. Therefore, just as with planning the instructional component of a curriculum, development of an assessment system must start with the specification of desired learning outcomes in the form of knowledge, skills, and attitudes expected of trainees or practitioners in order to provide safe and effective patient care. Issues to consider when judging the quality of evaluation methods include the reliability of data generated by the assessment, validity of decisions based on test results, educational impact on individuals undergoing evaluation and other stakeholders, and the feasibility of implementing the assessment system. In addition to these criteria and the particular competencies to be evaluated, the choice of testing methods from among numerous available techniques should consider multiple dimensions, such as appropriate level of assessment, stage of learner development, and, very importantly, overall purpose and context of the assessment. Ultimately, no one method can assess all aspects of professional competence, but familiarity with strengths and limitations of various modalities can guide the development of appropriate assessment systems. Strengths of simulation-based methods for evaluative purposes include the ability to assess actual performance of psychomotor skills and demonstration of nontechnical professional competencies in environments that safely and authentically mirror real practice settings. In addition, the programmability of simulations permits on-demand testing of rare but important clinical situations and consistent presentation of evaluation problems to multiple examinees; this reproducibility becomes especially important when high-stakes decisions are contingent upon such assessments