Domainbuilder: The knowledge authoring system for slide tutor intelligent tutoring system[version 1; peer review: 1 approved, 1 approved with reservations]

One of the major challenges in the development of medical Intelligent Tutoring Systems (ITS) is the development of authored content, a time-consuming process that requires participation of discipline experts. In this publication, we describe the development of software systems called DomainBuilder and TutorBuilder, designed to streamline and simplify the authoring process for general medical ITSs. The aim of these systems is to allow physicians without programming or ITSs background to create a domain knowledge base and author tutor cases in a time efficient manner. DomainBuilder combined knowledge authoring, case authoring, and validation tasks into a single work environment, enabling multiple authoring strategies. Natural Language Processing (NLP) methods were integrated for parsing existing clinical reports to speed case authoring. Similarly, TutorBuilder was designed to allow users to customize all aspects of ITSs, including user interface, pedagogic module, feedback module, etc. Both systems underwent formal usability studies with physicians specializing in dermatology. Open-ended questions assessed usability of the system and satisfaction with its features. Incorporating feedback from usability studies, DomainBuilder and TutorBuilder systems were deployed and used across multiple universities to create customized medical tutoring curriculum. Overall, both systems were well received by medical professionals participating in usability studies with participants highlighting ease of utilization and clarity of presentation. Usability study participants were able to successfully use the system for the authoring tasks. DomainBuilder and TutorBuilder are novel tools that combine comprehensive aspects of content creation, including creation of domain ontologies, case authoring, and validation

Number and allocation of citations per systematic review.

a<p>Galacto = Galactomannan.</p>b<p>Organ Trans = Organ Transplant.</p>c<p>% eligible = percentage provisionally eligible for inclusion in a review; judgments based on screening citations (titles and abstracts) by domain experts.</p

Effect of a limited-enforcement intelligent tutoring system in dermatopathology on student errors, goals and solution paths

OBJECTIVES: Determine effects of a limited-enforcement intelligent tutoring system in dermatopathology on student errors, goals and solution paths. Determine if limited enforcement in a medical tutoring system inhibits students from learning the optimal and most efficient solution path. Describe the type of deviations from the optimal solution path that occur during tutoring, and how these deviations change over time. Determine if the size of the problem-space (domain scope), has an effect on learning gains when using a tutor with limited enforcement.\ud \ud METHODS: Analyzed data mined from 44 pathology residents using SlideTutor-a Medical Intelligent Tutoring System in Dermatopathology that teaches histopathologic diagnosis and reporting skills based on commonly used diagnostic algorithms. Two subdomains were included in the study representing sub-algorithms of different sizes and complexities. Effects of the tutoring system on student errors, goal states and solution paths were determined.\ud \ud RESULTS: Students gradually increase the frequency of steps that match the tutoring system's expectation of expert performance. Frequency of errors gradually declines in all categories of error significance. Student performance frequently differs from the tutor-defined optimal path. However, as students continue to be tutored, they approach the optimal solution path. Performance in both subdomains was similar for both errors and goal differences. However, the rate at which students progress toward the optimal solution path differs between the two domains. Tutoring in superficial perivascular dermatitis, the larger and more complex domain was associated with a slower rate of approximation towards the optimal solution path.\ud \ud CONCLUSIONS: Students benefit from a limited-enforcement tutoring system that leverages diagnostic algorithms but does not prevent alternative strategies. Even with limited enforcement, students converge toward the optimal solution path.\ud \u

EDDA workflow.

<p>An overview of the project workflow. EDDA = Evidence in Documents, Discovery, and Analysis. Reference Filer = in-house Java program that sorts citations into folders; resultant datasets A and B are random halves of the citations stratified with respect to eligibility for provisional inclusion in a systematic review; citations include titles, abstracts, and metadata. RapidMiner is an open source, data mining suite. cNB = Weka complement naïve Bayes classifier available in Rapid Miner; suitable for imbalanced data typical of systematic reviews. Grid Parameter Optimization operator searches for best performance over a grid; dimensions based on combinations of parameter settings.</p

Mean performance of the cNB classifier by systematic review and feature set.

a<p>Baseline F3 (%): Ameloblastoma = 40.20; Influenza = 38.96; Galactomannan = 31.00; Malaria = 58.82; Organ transplant = 32.03. All mean F3 values surpassed the baseline values, one-tailed Z-tests, P<0.001.</p>b<p>Higher ranks associated with better performance.</p>c<p>Lower ranks associated with better performance.</p>d<p>Mean ranks significantly different for F3, precision, and classification error: Friedman's test of mean F3 ranks (4 df) = 9.760, <i>P</i> = .045; Friedman's test of mean precision ranks (4 df) = 16.480, <i>P</i> = .002; Friedman's test of mean classification error ranks (4 df) = 16.480, <i>P</i> = .002.</p>e<p>Friedman's test of mean recall ranks (4 df) = 1.980, <i>P</i> = .739, NS.</p

Feature set size by systematic review before and after filtering for information gain.

a<p>IG = information gain.</p>b<p>SR = systematic review.</p>c<p>A and B refer to random halves of the data.</p