QUEM: An Achievement Test for Knowledge-Based Systems

Abstract-This paper describes the QUality and Experience Metric (QUEM), a method for estimating the skill level of a knowledgebased system based on the quality of the solutions it produces. It allows one to assess how many years of experience the system would be judged to have if it were a human by providing a quantitative measure of the system's overall competence. QUEM can be viewed as a type of achievement or job-placement test administered to knowledge-based systems to help system designers determine how the system should be used and by what level of user. To apply QUEM, a set of subjects, experienced judges, and problems must be identified. The subjects should have a broad range of experience levels. Subjects and the knowledge-based system are asked to solve the problems; and judges are asked to rank order all solutions}from worst quality to best. The data from the subjects is used to construct a skill-function relating experience to solution quality, and confidence bands showing the variability in performance. The system's quality ranking is then plugged into the skill function to produce an estimate of the system's experience level. QUEM can be used to gauge the experience level of an individual system, to compare two systems, or to compare a system to its intended users. This represents an important advance in providing quantitative measures of overall performance that can be applied to a broad range of systems

Ribosomal and Immune Transcripts Associate with Relapse in Acquired ADAMTS13-Deficient Thrombotic Thrombocytopenic Purpura.

Approximately 40% of patients who survive acute episodes of thrombotic thrombocytopenic purpura (TTP) associated with severe acquired ADAMTS13 deficiency experience one or more relapses. Risk factors for relapse other than severe ADAMTS13 deficiency and ADAMTS13 autoantibodies are unknown. ADAMTS13 autoantibodies, TTP episodes following infection or type I interferon treatment and reported ensuing systemic lupus erythematosus in some patients suggest immune dysregulation. This cross-sectional study asked whether autoantibodies against RNA-binding proteins or peripheral blood gene expression profiles measured during remission are associated with history of prior relapse in acquired ADAMTS13-deficient TTP. Peripheral blood from 38 well-characterized patients with autoimmune ADAMTS13-deficient TTP in remission was examined for autoantibodies and global gene expression. A subset of TTP patients (9 patients, 24%) exhibited a peripheral blood gene signature composed of elevated ribosomal transcripts that associated with prior relapse. A non-overlapping subset of TTP patients (9 patients, 24%) displayed a peripheral blood type I interferon gene signature that associated with autoantibodies to RNA-binding proteins but not with history of relapse. Patients who had relapsed bimodally expressed higher HLA transcript levels independently of ribosomal transcripts. Presence of any one potential risk factor (ribosomal gene signature, elevated HLA-DRB1, elevated HLA-DRB5) associated with relapse (OR = 38.4; p = 0.0002) more closely than any factor alone or all factors together. Levels of immune transcripts typical of natural killer (NK) and T lymphocytes positively correlated with ribosomal gene expression and number of prior episodes but not with time since the most recent episode. Flow cytometry confirmed elevated expression of cell surface markers encoded by these transcripts on T and/or NK cell subsets of patients who had relapsed. These data associate elevated ribosomal and immune transcripts with relapse history in acquired, ADAMTS13-deficient TTP