Representing visual classification as a linear combination of words

Explainability is a longstanding challenge in deep learning, especially in high-stakes domains like healthcare. Common explainability methods highlight image regions that drive an AI model's decision. Humans, however, heavily rely on language to convey explanations of not only "where" but "what". Additionally, most explainability approaches focus on explaining individual AI predictions, rather than describing the features used by an AI model in general. The latter would be especially useful for model and dataset auditing, and potentially even knowledge generation as AI is increasingly being used in novel tasks. Here, we present an explainability strategy that uses a vision-language model to identify language-based descriptors of a visual classification task. By leveraging a pre-trained joint embedding space between images and text, our approach estimates a new classification task as a linear combination of words, resulting in a weight for each word that indicates its alignment with the vision-based classifier. We assess our approach using two medical imaging classification tasks, where we find that the resulting descriptors largely align with clinical knowledge despite a lack of domain-specific language training. However, our approach also identifies the potential for 'shortcut connections' in the public datasets used. Towards a functional measure of explainability, we perform a pilot reader study where we find that the AI-identified words can enable non-expert humans to perform a specialized medical task at a non-trivial level. Altogether, our results emphasize the potential of using multimodal foundational models to deliver intuitive, language-based explanations of visual tasks.Comment: To be published in the Proceedings of the 3rd Machine Learning for Health symposium, Proceedings of Machine Learning Research (PMLR

Diagnostic concordance of clinical diagnosis, tissue culture, and histopathology testing for skin and soft tissue infections: A single-center retrospective study

BACKGROUND: Tissue culture and histopathology are the conventional diagnostic modalities for skin and soft tissue infections (SSTIs), but few studies have investigated their concordance. OBJECTIVE: Determine concordance between histopathology and tissue culture in the diagnosis of suspected SSTIs. METHODS: Single-center retrospective study of 355 cases with suspected SSTIs identified from the dermatology inpatient consultation log January 2014-July 2017. RESULTS: Overall concordance between histopathology testing and tissue culture results was high (76.1%). Concordance was high for cases defined as no evidence of infection, fungal infection and mycobacterial infection by histopathology (77.8%, 74.2%, and 80.0%) and tissue culture (92.1%, 67.7%, and 83.3%). Concordance was lower for suspected SSTIs with bacterial infection by histopathology (61.9%) and tissue culture (28.4%). Concordance rates were not significantly affected by age, sex, race, antimicrobial agent use, immunologic status, or biopsy size. LIMITATIONS: Retrospective and single-institution nature of the study. CONCLUSION: This study demonstrated a high concordance between histopathology and tissue culture in SSTIs with no clinical evidence of infection and suspected fungal and mycobacterial SSTIs, though concordance was lower for suspected SSTIs with evidence of bacterial infection. Clinicians should not be deterred from relying on initial histopathological results based on patients\u27 immunosuppressed status, antimicrobial agent use, age, or biopsy tissue size

Equivalence between Graph Spectral Clustering and Column Subset Selection (Student Abstract)

The common criteria for evaluating spectral clustering are NCut and RatioCut. The seemingly unrelated column subset selection (CSS) problem aims to compute a column subset that linearly approximates the entire matrix. A common criterion is the approximation error in the Frobenius norm (ApproxErr). We show that any algorithm for CSS can be viewed as a clustering algorithm that minimizes NCut by applying it to a matrix formed from graph edges. Conversely, any clustering algorithm can be seen as identifying a column subset from that matrix. In both cases, ApproxErr and NCut have the same value. Analogous results hold for RatioCut with a slightly different matrix. Therefore, established results for CSS can be mapped to spectral clustering. We use this to obtain new clustering algorithms, including an optimal one that is similar to A*. This is the first nontrivial clustering algorithm with such an optimality guarantee. A variant of the weighted A* runs much faster and provides bounds on the accuracy. Finally, we use the results from spectral clustering to prove the NP-hardness of CSS from sparse matrices

Vestibular Function and Activities of Daily Living

Objective: Vestibular dysfunction increases with age and is associated with mobility difficulties and fall risk in older individuals. We evaluated whether vestibular function influences the ability to perform activities of daily living (ADLs). Method: We analyzed the 1999 to 2004 National Health and Nutrition Examination Survey of adults aged older than 40 years ( N = 5,017). Vestibular function was assessed with the Modified Romberg test. We evaluated the association between vestibular function and difficulty level in performing specific basic and instrumental ADLs, and total number of ADL impairments. Results: Vestibular dysfunction was associated with significantly higher odds of difficulty with nine ADLs, most strongly with difficulty managing finances (odds ratio [ OR ] = 2.64, 95% confidence interval [CI] = [1.18, 5.90]). In addition, vestibular dysfunction was associated with a significantly greater number of ADL impairments (β = .21, 95% CI = [0.09, 0.33]). This effect size was comparable with the influence of heavy smoking (β = .21, 95% CI = [0.06, 0.36]) and hypertension (β = .10, 95% CI = [0.02, 0.18]) on the number of ADL impairments. Conclusion: Vestibular dysfunction significantly influences ADL difficulty, most strongly with a cognitive rather than mobility-based task. These findings underscore the importance of vestibular inputs for both cognitive and physical daily activities

Cutaneous adverse events of immune checkpoint inhibitor therapy: incidence and types of reactive dermatoses

Background Dermatoses are common and potentially serious complications of programmed cell death receptor PD-1 immune checkpoint inhibitor (anti-PD-1 ICI) therapy. Understanding their incidence is necessary to support clinical awareness, diagnosis, and management. Objective To examine the incidence and odds of reported non-cancerous dermatoses in the setting of anti-PD-1 ICI therapy. Methods Cross-sectional study of anti-PD-1 (pembrolizumab or nivolumab) treated patients at a tertiary healthcare institution. Selected dermatologic events following immunotherapy were identified in the electronic medical record. Comparator arm were patients that developed these same dermatoses without receiving anti-PD-1 ICI therapy. Results There were 13.7% (254/1857) patients that developed one of 28 dermatoses. Compared with the general population, patients treated with anti-PD-1 had a greater risk for development of mucositis (OR 65.7, 95% CI 35.0–123.3), xerostomia (OR 11.9, 95% CI 8.4–16.8), pruritus (11.3, 95% CI 8.9–14.3), and lichen planus/lichenoid dermatitis (OR 10.7, 95% CI 5.6–20.7). Conclusions We report the frequency of dermatoses encountered in the setting of ICI therapy, both common (pruritus, rash, vitiligo) and uncommon (scleroderma, urticaria)