Performance of Large Language Models in a Computer Science Degree Program

Large language models such as ChatGPT-3.5 and GPT-4.0 are ubiquitous and dominate the current discourse. Their transformative capabilities have led to a paradigm shift in how we interact with and utilize (text-based) information. Each day, new possibilities to leverage the capabilities of these models emerge. This paper presents findings on the performance of different large language models in a university of applied sciences' undergraduate computer science degree program. Our primary objective is to assess the effectiveness of these models within the curriculum by employing them as educational aids. By prompting the models with lecture material, exercise tasks, and past exams, we aim to evaluate their proficiency across different computer science domains. We showcase the strong performance of current large language models while highlighting limitations and constraints within the context of such a degree program. We found that ChatGPT-3.5 averaged 79.9% of the total score in 10 tested modules, BingAI achieved 68.4%, and LLaMa, in the 65 billion parameter variant, 20%. Despite these convincing results, even GPT-4.0 would not pass the degree program - due to limitations in mathematical calculations.Comment: Submitted to AI4AI Workshop 202

FAA Designated Pilot Examiner System Insights

As part of the Reauthorization Act of 2018 the FAA was required to assign to the Aviation Rulemaking Advisory Committee (ARAC) a review of the current Designated Pilot Examiner (DPE) policies. The ARAC in turn assigned this task to the Designated Pilot Examiner Reforms Working Group (DPERWG). This Group delivered its recommendations to the FAA in June 2021, with an FAA response to the Group due by June 2022. The purpose of this research project is to provide more insight regarding the current DPE system from all stakeholders prior to that deadline. Survey data from both current DPE’s and flight schools nationwide will be shared. These surveys address stakeholder perceptions on components of the DPE system including: 1) wait times for check rides, 2) activity level of DPE’s, 3) the effect rescinding constraints on geographical regions and the ability to do up to three check rides per day has had, 4) the prevalence of applicants and/or examiners traveling to check ride sites other than their home airport, and 5) feedback on a number of specific recommendations made by the DPERWG. These items include changes to the DPE application process, the development of an applicant feedback system, changes to the number of events per day which can effectively be conducted, a national DPE oversight model versus the current FSDO oversight model, the treatment of oral and flight tests as separate events, and the effectiveness of the DPE locator on the FAA website

FAA Designated Pilot Examiner System Insights

As part of the Federal Aviation Administration (FAA) Reauthorization Act of 2018 the FAA was required by Congress to review Designated Pilot Examiner (DPE) policies and procedures. This task was delegated to the Designated Pilot Examiner Reforms Working Group (DPERWG). This Group delivered its recommendations to the FAA in June 2021, and this research study was conducted in late January of 2022 to attempt to provide additional insights to the agency prior to its required response to the DPERWG in June of 2022. This research project aimed to provide perceptions of the current DPE system from both DPEs and flight schools nationwide, as well as feedback on selected DPERWG recommendations. Surveys of these two populations were conducted seeking stakeholder perceptions on the current DPE system including: 1) wait times for scheduling check rides, 2) the level of activity of DPE’s, and 3) the prevalence of applicants and/or examiners traveling to check ride sites other than their home airport. Feedback on specific recommendations made by the DPERWG were also solicited including: 1) the implementation of a confidential survey applicant feedback system, 2) the possibility of moving to a national oversight model for the DPE system, 3) the perceptions of and improvements seen as necessary for the current FAA DPE locator website, 4) the possibility of treating oral and flight exams as separate events, and 5) changing medical certificate requirements for DPEs. There were significant differences in perceptions of DPEs and flight training providers regarding the wait times incurred when scheduling check rides, but there was general consensus regarding the travel of both applicants and DPEs for the conduct of those rides. There was also consensus between the two surveyed groups regarding most of the DPERWG recommendations which were examined by the surveys

Two-Staged Acoustic Modeling Adaption for Robust Speech Recognition by the Example of German Oral History Interviews

In automatic speech recognition, often little training data is available for specific challenging tasks, but training of state-of-the-art automatic speech recognition systems requires large amounts of annotated speech. To address this issue, we propose a two-staged approach to acoustic modeling that combines noise and reverberation data augmentation with transfer learning to robustly address challenges such as difficult acoustic recording conditions, spontaneous speech, and speech of elderly people. We evaluate our approach using the example of German oral history interviews, where a relative average reduction of the word error rate by 19.3% is achieved.Comment: Accepted for IEEE International Conference on Multimedia and Expo (ICME), Shanghai, China, July 201

Cognitive Trajectories in Preclinical and Prodromal Alzheimer's Disease Related to Amyloid Status and Brain Atrophy: A Bayesian Approach

Background: Cognitive decline is a key outcome of clinical studies in Alzheimer's disease (AD). Objective: To determine effects of global amyloid load as well as hippocampus and basal forebrain volumes on longitudinal rates and practice effects from repeated testing of domain specific cognitive change in the AD spectrum, considering non-linear effects and heterogeneity across cohorts. Methods: We included 1,514 cases from three cohorts, ADNI, AIBL, and DELCODE, spanning the range from cognitively normal people to people with subjective cognitive decline and mild cognitive impairment (MCI). We used generalized Bayesian mixed effects analysis of linear and polynomial models of amyloid and volume effects in time. Robustness of effects across cohorts was determined using Bayesian random effects meta-analysis. Results: We found a consistent effect of amyloid and hippocampus volume, but not of basal forebrain volume, on rates of memory change across the three cohorts in the meta-analysis. Effects for amyloid and volumetric markers on executive function were more heterogeneous. We found practice effects in memory and executive performance in amyloid negative cognitively normal controls and MCI cases, but only to a smaller degree in amyloid positive controls and not at all in amyloid positive MCI cases. Conclusions: We found heterogeneity between cohorts, particularly in effects on executive functions. Initial increases in cognitive performance in amyloid negative, but not in amyloid positive MCI cases and controls may reflect practice effects from repeated testing that are lost with higher levels of cerebral amyloid

Cognitive Trajectories in Preclinical and Prodromal Alzheimer's Disease Related to Amyloid Status and Brain Atrophy:A Bayesian Approach

Background: Cognitive decline is a key outcome of clinical studies in Alzheimer’s disease (AD). Objective: To determine effects of global amyloid load as well as hippocampus and basal forebrain volumes on longitudinal rates and practice effects from repeated testing of domain specific cognitive change in the AD spectrum, considering non-linear effects and heterogeneity across cohorts. Methods: We included 1,514 cases from three cohorts, ADNI, AIBL, and DELCODE, spanning the range from cognitively normal people to people with subjective cognitive decline and mild cognitive impairment (MCI). We used generalized Bayesian mixed effects analysis of linear and polynomial models of amyloid and volume effects in time. Robustness of effects across cohorts was determined using Bayesian random effects meta-analysis. Results: We found a consistent effect of amyloid and hippocampus volume, but not of basal forebrain volume, on rates of memory change across the three cohorts in the meta-analysis. Effects for amyloid and volumetric markers on executive function were more heterogeneous. We found practice effects in memory and executive performance in amyloid negative cognitively normal controls and MCI cases, but only to a smaller degree in amyloid positive controls and not at all in amyloid positive MCI cases. Conclusions: We found heterogeneity between cohorts, particularly in effects on executive functions. Initial increases in cognitive performance in amyloid negative, but not in amyloid positive MCI cases and controls may reflect practice effects from repeated testing that are lost with higher levels of cerebral amyloid

Relevance of Minor Neuropsychological Deficits in Patients With Subjective Cognitive Decline

peer reviewed[en] BACKGROUND AND OBJECTIVES: To determine the relevance of minor neuropsychological deficits (MNPD) in patients with subjective cognitive decline (SCD) with regard to CSF levels of Alzheimer disease (AD) biomarkers, cognitive decline, and clinical progression to mild cognitive impairment (MCI). METHODS: This study included patients with clinical SCD and SCD-free, healthy control (HC) participants with available baseline CSF and/or longitudinal cognitive data from the observational DZNE Longitudinal Cognitive Impairment and Dementia study. We defined MNPD as a performance of at least 0.5SD below the mean on a demographically adjusted total score derived from the Consortium to Establish a Registry for Alzheimer's Disease neuropsychological assessment battery. We compared SCD patients with MNPD and those without MNPD with regard to CSF amyloid-β (Aβ)42/Aβ40, phosphorylated tau (p-tau181), total tau and Aβ42/p-tau181 levels, longitudinal cognitive composite trajectories, and risk of clinical progression to incident MCI (follow-up M ± SD: 40.6 ± 23.7 months). In addition, we explored group differences between SCD and HC in those without MNPD. RESULTS: In our sample (N = 672, mean age: 70.7 ± 5.9 years, 50% female), SCD patients with MNPD (n = 55, 12.5% of SCD group) showed significantly more abnormal CSF biomarker levels, increased cognitive decline, and a higher risk of progression to incident MCI (HR: 4.07, 95% CI 2.46-6.74) compared with SCD patients without MNPD (n = 384). MNPD had a positive predictive value of 57.0% (95% CI 38.5-75.4) and a negative predictive value of 86.0% (95% CI 81.9-90.1) for the progression of SCD to MCI within 3 years. SCD patients without MNPD showed increased cognitive decline and a higher risk of incident MCI compared with HC participants without MNPD (n = 215; HR: 4.09, 95% CI 2.07-8.09), while AD biomarker levels did not differ significantly between these groups. DISCUSSION: Our results suggest that MNPD are a risk factor for AD-related clinical progression in cognitively normal patients seeking medical counseling because of SCD. As such, the assessment of MNPD could be useful for individual clinical prediction and for AD risk stratification in clinical trials. However, SCD remains a risk factor for future cognitive decline even in the absence of MNPD

Relevance of Minor Neuropsychological Deficits in Patients With Subjective Cognitive Decline

Background and ObjectivesTo determine the relevance of minor neuropsychological deficits (MNPD) in patients with subjective cognitive decline (SCD) with regard to CSF levels of Alzheimer disease (AD) biomarkers, cognitive decline, and clinical progression to mild cognitive impairment (MCI).MethodsThis study included patients with clinical SCD and SCD-free, healthy control (HC) participants with available baseline CSF and/or longitudinal cognitive data from the observational DZNE Longitudinal Cognitive Impairment and Dementia study. We defined MNPD as a performance of at least 0.5SD below the mean on a demographically adjusted total score derived from the Consortium to Establish a Registry for Alzheimer's Disease neuropsychological assessment battery. We compared SCD patients with MNPD and those without MNPD with regard to CSF amyloid-beta (A beta)42/A beta 40, phosphorylated tau (p-tau181), total tau and A beta 42/p-tau181 levels, longitudinal cognitive composite trajectories, and risk of clinical progression to incident MCI (follow-up M +/- SD: 40.6 +/- 23.7 months). In addition, we explored group differences between SCD and HC in those without MNPD.ResultsIn our sample (N = 672, mean age: 70.7 +/- 5.9 years, 50% female), SCD patients with MNPD (n = 55, 12.5% of SCD group) showed significantly more abnormal CSF biomarker levels, increased cognitive decline, and a higher risk of progression to incident MCI (HR: 4.07, 95% CI 2.46-6.74) compared with SCD patients without MNPD (n = 384). MNPD had a positive predictive value of 57.0% (95% CI 38.5-75.4) and a negative predictive value of 86.0% (95% CI 81.9-90.1) for the progression of SCD to MCI within 3 years. SCD patients without MNPD showed increased cognitive decline and a higher risk of incident MCI compared with HC participants without MNPD (n = 215;HR: 4.09, 95% CI 2.07-8.09), while AD biomarker levels did not differ significantly between these groups.DiscussionOur results suggest that MNPD are a risk factor for AD-related clinical progression in cognitively normal patients seeking medical counseling because of SCD. As such, the assessment of MNPD could be useful for individual clinical prediction and for AD risk stratification in clinical trials. However, SCD remains a risk factor for future cognitive decline even in the absence of MNPD