Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare delivery

Despite growing interest in using large language models (LLMs) in healthcare, current explorations do not assess the real-world utility and safety of LLMs in clinical settings. Our objective was to determine whether two LLMs can serve information needs submitted by physicians as questions to an informatics consultation service in a safe and concordant manner. Sixty six questions from an informatics consult service were submitted to GPT-3.5 and GPT-4 via simple prompts. 12 physicians assessed the LLM responses' possibility of patient harm and concordance with existing reports from an informatics consultation service. Physician assessments were summarized based on majority vote. For no questions did a majority of physicians deem either LLM response as harmful. For GPT-3.5, responses to 8 questions were concordant with the informatics consult report, 20 discordant, and 9 were unable to be assessed. There were 29 responses with no majority on "Agree", "Disagree", and "Unable to assess". For GPT-4, responses to 13 questions were concordant, 15 discordant, and 3 were unable to be assessed. There were 35 responses with no majority. Responses from both LLMs were largely devoid of overt harm, but less than 20% of the responses agreed with an answer from an informatics consultation service, responses contained hallucinated references, and physicians were divided on what constitutes harm. These results suggest that while general purpose LLMs are able to provide safe and credible responses, they often do not meet the specific information need of a given question. A definitive evaluation of the usefulness of LLMs in healthcare settings will likely require additional research on prompt engineering, calibration, and custom-tailoring of general purpose models.Comment: 27 pages including supplemental informatio

Power estimation for non-standardized multisite studies

AbstractA concern for researchers planning multisite studies is that scanner and T1-weighted sequence-related biases on regional volumes could overshadow true effects, especially for studies with a heterogeneous set of scanners and sequences. Current approaches attempt to harmonize data by standardizing hardware, pulse sequences, and protocols, or by calibrating across sites using phantom-based corrections to ensure the same raw image intensities. We propose to avoid harmonization and phantom-based correction entirely. We hypothesized that the bias of estimated regional volumes is scaled between sites due to the contrast and gradient distortion differences between scanners and sequences. Given this assumption, we provide a new statistical framework and derive a power equation to define inclusion criteria for a set of sites based on the variability of their scaling factors. We estimated the scaling factors of 20 scanners with heterogeneous hardware and sequence parameters by scanning a single set of 12 subjects at sites across the United States and Europe. Regional volumes and their scaling factors were estimated for each site using Freesurfer's segmentation algorithm and ordinary least squares, respectively. The scaling factors were validated by comparing the theoretical and simulated power curves, performing a leave-one-out calibration of regional volumes, and evaluating the absolute agreement of all regional volumes between sites before and after calibration. Using our derived power equation, we were able to define the conditions under which harmonization is not necessary to achieve 80% power. This approach can inform choice of processing pipelines and outcome metrics for multisite studies based on scaling factor variability across sites, enabling collaboration between clinical and research institutions

A comparison of two radioimmunoassay methods for the detection of carcinoembryonic antigen in patients with ovarian or cervical cancer

The carcinoembryonic antigen (CEA) was measured by radioimmunoassay (RIA) in the serum of patients with carcinoma of the ovary (100), cervix (100), and healthy controls (100). Two methods were used in the RIA for the precipitation of the CEA immune complex: (i) double antibody (DA), and (ii) ammonium sulfate (AS). The mean value of the controls was 14.67 ± 4.09 ng CEA/ml of serum using the DA technique. Based on the control's mean value +2 SD, there were 3% positive results in the control group, 16 and 19% positive results in the ovarian and cervical cancer groups, respectively. With the AS technique, all the control values were less than 1.5 ng CEA/ml of serum. This gave 20 and 3% positive results in the ovarian and cervical cancer groups, respectively. Although the assays (DA and AS) are comparable, the results show that only six ovarian and one cervical cancer patient had positive CEA levels, when determined by both the DA and AS techniques

A multiparametric approach to tumor markers detectable in serum in patients with carcinoma of the ovary or uterine cervix

A comparison of several serum tumor markers (lactate dehydrogenase (LDH), sialyltransferase (ST), carcinoembryonic antigen (CEA), β microglobulin (βM), γ-chain fetal hemoglobulin (HbF), immune complexes (ImCp), and spermine (Spm)) was made in patients with carcinoma of the ovary or cervix uteri and healthy control subjects. The greatest positive results were obtained with the markers LDH (40%) and βM (46%) for patients with carcinoma of the cervix and ovary, respectively. However, based on false positive results, the most suitable single marker for patients with carcinoma of the cervix was Spm (30%). When a multiparametric approach was taken, a combination of four of the seven markers resulted in an increase in the positive results, that is, the cancer patients were positive for one of the four markers. This was 76% for patients with carcinoma of the cervix using the markers LDH, ST, Spm, and βM and 79% for patients with carcinoma of the ovary using the markers ST, CEA, HbF, and βM. All markers in both groups of cancer patients were elevated above control levels with the exception of ST which was decreased in patients with carcinoma of the ovary. ST, CEA, and HbF were found to be associated by multilinear regression analysis in the ovarian cancer group. As the ST approached normal levels CEA and HbF became elevated. Similarly there was a linear relationship between βM and ImCp demonstrating positively at similar times

A comparison of the physicochemical properties of carcinoembryonic antigen in extracts of tumour tissue, ascitic and cyst fluid from ovarian cancer

In view of the reported disagreement in the physicochemical properties of ovarian carcinoembryonic antigen (CEA), this study was undertaken to compare the properties of CEA obtained from extracts of ovarian tumour tissue, ascitic fluid and cyst fluid. On the basis of molecular weight estimation and binding properties with Concanavalin A and wheat germ lectin, ovarian CEA from these three sources appeared similar, and also possessed similar properties to those of colonic CEA. On isoelectric focusing, however, it was found that the isoelectric point of CEA from tumour tissue and cyst fluid differed from that from ascitic fluid. It is most likely that this is due to a loss of sialic acid from the CEA released into ascitic fluid

Physical properties of new ordered bimetallic phases MCdPS (M = Zn, Ni, Co, Mn)

International audienceFour bimetallic phases of the thiophosphate family have been synthesized by the cationic exchange reaction using a freshly prepared K0.5Cd0.75PS3 precursor phase and methanolic solutions of nitrates of the divalent cations Zn II , Ni II , Co II , and Mn II. All the materials were characterized by FTIR, PXRD, SEM-EDXS and (in the case of the diamagnetic compounds) by solid state NMR. For the K0.5Cd0.75PS3 precursor, the X-ray powder diffraction data suggest a modification of the structure, while solid state NMR results confirm that this phase possesses an ordered arrangement of Cd vacancies. The cationic exchange reaction achieves a complete removal of potassium ions (no potassium detected by SEM-EDXS) and re-occupation of the vacancies by divalent cations. Therefore, the obtained compounds have an average composition of M0.25Cd0.75PS3 (M = Zn II , Ni II , Co II , Mn II) and possess an ordered distribution of the substituent cations. Even with the paramagnetic substitution level of 25%, antiferromagnetic behaviour is present in the phases with Mn II , Co II and Ni II , as evidenced by dc susceptibility and in the case of the Mn II substituted phase by EPR. The cooperative magnetic interactions confirm the conclusion that the paramagnetic ions adopt an ordered arrangement. The analysis by broad band impedance spectroscopy allows to attribute the conductivity in these materials to charge movements in the layers due to the difference in electronegativity of the metal ions. Zn0.25Cd0.75PS3 is the phase that shows the highest conductivity values. Finally, the band gap energies of the bimetallic phases tend to be lower than those of the single-metal phases, probably due to an overlap of the band structures

Effects of different endurance exercise modalities on migraine days and cerebrovascular health in episodic migraineurs: A randomized controlled trial

Aerobic exercise training is a promising complementary treatment option in migraine and can reduce migraine days and improve retinal microvascular function. Our aim was to elucidate whether different aerobic exercise programs at high vs moderate intensities distinctly affect migraine days as primary outcome and retinal vessel parameters as a secondary. In this randomized controlled trial, migraine days were recorded by a validated migraine diary in 45 migraineurs of which 36 (female: 28; age: 36 (SD:10)/BMI: 23.1 (5.3) completed the training period (dropout: 20%). Participants were assigned (Strata: age, gender, fitness and migraine symptomatology) to either high intensity interval training (HIT), moderate continuous training (MCT), or a control group (CON). Intervention groups trained twice a week over a 12-week intervention period. Static retinal vessel analysis, central retinal arteriolar (CRAE) and venular (CRVE) diameters, as well as the arteriolar-to-venular diameter ratio (AVR) were obtained for cerebrovascular health assessment. Incremental treadmill testing yielded maximal and submaximal fitness parameters. Overall, moderate migraine day reductions were observed (ηP2 = .12): HIT revealed 89% likely beneficial effects (SMD = 1.05) compared to MCT (SMD = 0.50) and CON (SMD = 0.59). Very large intervention effects on AVR improvement (ηP2 = 0.27), slightly favoring HIT (SMD=-0.43) over CON (SMD=0), were observed. HIT seems more effective for migraine day reduction and improvement of cerebrovascular health compared to MCT. Intermittent exercise programs of higher intensities may need to be considered as an additional treatment option in migraine patients