How Useful are Educational Questions Generated by Large Language Models?

Controllable text generation (CTG) by large language models has a huge potential to transform education for teachers and students alike. Specifically, high quality and diverse question generation can dramatically reduce the load on teachers and improve the quality of their educational content. Recent work in this domain has made progress with generation, but fails to show that real teachers judge the generated questions as sufficiently useful for the classroom setting; or if instead the questions have errors and/or pedagogically unhelpful content. We conduct a human evaluation with teachers to assess the quality and usefulness of outputs from combining CTG and question taxonomies (Bloom's and a difficulty taxonomy). The results demonstrate that the questions generated are high quality and sufficiently useful, showing their promise for widespread use in the classroom setting.Comment: Accepted to AIED Late Breaking Results 2023 - to be published in their proceeding

How Teachers Can Use Large Language Models and Bloom's Taxonomy to Create Educational Quizzes

Question generation (QG) is a natural language processing task with an abundance of potential benefits and use cases in the educational domain. In order for this potential to be realized, QG systems must be designed and validated with pedagogical needs in mind. However, little research has assessed or designed QG approaches with the input from real teachers or students. This paper applies a large language model-based QG approach where questions are generated with learning goals derived from Bloom's taxonomy. The automatically generated questions are used in multiple experiments designed to assess how teachers use them in practice. The results demonstrate that teachers prefer to write quizzes with automatically generated questions, and that such quizzes have no loss in quality compared to handwritten versions. Further, several metrics indicate that automatically generated questions can even improve the quality of the quizzes created, showing the promise for large scale use of QG in the classroom setting.Comment: 8 pages, 8 figures. Accepted to the main track of the EAAI-24: The 14th Symposium on Educational Advances in Artificial Intelligenc

The Stable Entropy Hypothesis and Entropy-Aware Decoding: An Analysis and Algorithm for Robust Natural Language Generation

State-of-the-art language generation models can degenerate when applied to open-ended generation problems such as text completion, story generation, or dialog modeling. This degeneration usually shows up in the form of incoherence, lack of vocabulary diversity, and self-repetition or copying from the context. In this paper, we postulate that ``human-like'' generations usually lie in a narrow and nearly flat entropy band, and violation of these entropy bounds correlates with degenerate behavior. Our experiments show that this stable narrow entropy zone exists across models, tasks, and domains and confirm the hypothesis that violations of this zone correlate with degeneration. We then use this insight to propose an entropy-aware decoding algorithm that respects these entropy bounds resulting in less degenerate, more contextual, and "human-like" language generation in open-ended text generation settings

Functional Analysis of the VirSR Phosphorelay from Clostridium perfringens

Toxin production in Clostridium perfringens is controlled by the VirSR two-component signal transduction system, which comprises the VirS sensor histidine kinase and the VirR response regulator. Other studies have concentrated on the elucidation of the genes controlled by this network; there is little information regarding the phosphorelay cascade that is the hallmark of such regulatory systems. In this study, we have examined each step in this cascade, beginning with autophosphorylation of VirS, followed by phosphotransfer from VirS to VirR. We also have studied the effects of gene dosage and phosphorylation in vivo. We have used random and site-directed mutagenesis to identify residues in VirS that are important for its function and have identified a region in the putative sensory domain of VirS that appeared to be essential for function. In vitro phosphorylation studies showed that VirSc, a truncated VirS protein that lacked the N-terminal sensory domain, was capable of autophosphorylation and could subsequently act as a phosphodonor for its cognate response regulator, VirR. Conserved residues of both VirS and VirR, including the D57 residue of VirR, were shown to be essential for this process. By use of Targetron technology, we were able to introduce a single copy of virR or virRD57N onto the chromosome of a virR mutant of C. perfringens. The results showed that in vivo, when virR was present in single copy, the production of wild-type levels of perfringolysin O was dependent on the presence of virS and an unaltered D57 residue in VirR. These results provide good evidence that phosphorylation is critical for VirR function

The Cysteine Protease α-Clostripain is Not Essential for the Pathogenesis of Clostridium perfringens-Mediated Myonecrosis

Clostridium perfringens is the causative agent of clostridial myonecrosis or gas gangrene and produces many different extracellular toxins and enzymes, including the cysteine protease α-clostripain. Mutation of the α-clostripain structural gene, ccp, alters the turnover of secreted extracellular proteins in C. perfringens, but the role of α-clostripain in disease pathogenesis is not known. We insertionally inactivated the ccp gene C. perfringens strain 13 using TargeTron technology, constructing a strain that was no longer proteolytic on skim milk agar. Quantitative protease assays confirmed the absence of extracellular protease activity, which was restored by complementation with the wild-type ccp gene. The role of α-clostripain in virulence was assessed by analysing the isogenic wild-type, mutant and complemented strains in a mouse myonecrosis model. The results showed that although α-clostripain was the major extracellular protease, mutation of the ccp gene did not alter either the progression or the development of disease. These results do not rule out the possibility that this extracellular enzyme may still have a role in the early stages of the disease process

Blocking Synthesis of the Variant Surface Glycoprotein Coat in Trypanosoma brucei Leads to an Increase in Macrophage Phagocytosis Due to Reduced Clearance of Surface Coat Antibodies

The extracellular bloodstream form parasite Trypanosoma brucei is supremely adapted to escape the host innate and adaptive immune system. Evasion is mediated through an antigenically variable Variant Surface Glycoprotein (VSG) coat, which is recycled at extraordinarily high rates. Blocking VSG synthesis triggers a precytokinesis arrest where stalled cells persist for days in vitro with superficially intact VSG coats, but are rapidly cleared within hours in mice. We therefore investigated the role of VSG synthesis in trypanosome phagocytosis by activated mouse macrophages. T. brucei normally effectively evades macrophages, and induction of VSG RNAi resulted in little change in phagocytosis of the arrested cells. Halting VSG synthesis resulted in stalled cells which swam directionally rather than tumbling, with a significant increase in swim velocity. This is possibly a consequence of increased rigidity of the cells due to a restricted surface coat in the absence of VSG synthesis. However if VSG RNAi was induced in the presence of anti-VSG221 antibodies, phagocytosis increased significantly. Blocking VSG synthesis resulted in reduced clearance of anti-VSG antibodies from the trypanosome surface, possibly as a consequence of the changed motility. This was particularly marked in cells in the G2/ M cell cycle stage, where the half-life of anti-VSG antibody increased from 39.3 ± 4.2 seconds to 99.2 ± 15.9 seconds after induction of VSG RNAi. The rates of internalisation of bulk surface VSG, or endocytic markers like transferrin, tomato lectin or dextran were not significantly affected by the VSG synthesis block. Efficient elimination of anti-VSG-antibody complexes from the trypanosome cell surface is therefore essential for trypanosome evasion of macrophages. These experiments highlight the essentiality of high rates of VSG recycling for the rapid removal of host opsonins from the parasite surface, and identify this process as a key parasite virulence factor during a chronic infection

PUBLICATIONS Journal Publications

of the Association for Computational Linguistics (ACL 2013), pages 392–401. Sofia