Effective and Efficient Transfer Learning in the Era of Large Language Models

Substantial progress has been made in the field of natural language processing (NLP) due to the advent of large language models (LLMs)—deep neural networks with millions or billions of parameters pre-trained on large amounts of unlabeled data. However, these models have common weaknesses, including degenerate performance in data-scarce scenarios, and substantial computational resource requirements. This thesis aims to develop methods to address these limitations for improved applicability and performance of LLMs in resource-constrained settings with limited data and/or computational resources. To address the need for labeled data in data-scarce scenarios, I present two methods, in Chapter 2 and Chapter 3, respectively. The first method leverages beneficial relationships between NLP tasks for transfer learning, while the second method combines data augmentation and self-training to boost few-shot learning performance—the ability to perform novel tasks from only a few labeled examples. Additionally, in Chapter 4, I introduce a novel parameter-efficient transfer learning approach that reuses a single frozen model for all tasks while only learning minimal task-specific parameters (soft/continuous prompts) to represent tasks and transfer knowledge. Our method can match or outperform fine-tuning task-specific models (training the whole model on each task). In Chapter 5, I demonstrate the benefits of parameter-efficient transfer learning in a cross-lingual transfer setting. Finally, I conclude the thesis in Chapter 6 by outlining potential avenues for future research that aim to advance NLP through large-scale multi-task learning using multilingual and multimodal data

On Improving Generalization of CNN-Based Image Classification with Delineation Maps Using the CORF Push-Pull Inhibition Operator

Deployed image classification pipelines are typically dependent on the images captured in real-world environments. This means that images might be affected by different sources of perturbations (e.g. sensor noise in low-light environments). The main challenge arises by the fact that image quality directly impacts the reliability and consistency of classification tasks. This challenge has, hence, attracted wide interest within the computer vision communities. We propose a transformation step that attempts to enhance the generalization ability of CNN models in the presence of unseen noise in the test set. Concretely, the delineation maps of given images are determined using the CORF push-pull inhibition operator. Such an operation transforms an input image into a space that is more robust to noise before being processed by a CNN. We evaluated our approach on the Fashion MNIST data set with an AlexNet model. It turned out that the proposed CORF-augmented pipeline achieved comparable results on noise-free images to those of a conventional AlexNet classification model without CORF delineation maps, but it consistently achieved significantly superior performance on test images perturbed with different levels of Gaussian and uniform noise

Identifying undergraduate pharmacy students’ achievement goals and their effects on academic achievement and teachers’ qualities

The work for this thesis started with generic questions about achievement motivation and its application in pharmacy education settings. Questions like: what are pharmacy students’ preferred achievement goals? Are there any relationships between these achievement goals and academic performance? Is there any relationship between exam types and adopted achievement goals? Is student motivation in any way related to the qualities they value in their teachers? Such questions have guided this doctoral work since August 2010. For more than three decades, achievement goal theory has been one of the most influential theories investigating students’ motivation to learn (Conley, 2012). Four types of achievement goals have been identified by scholars (Huang, 2012; Hulleman et al., 2010): (1) mastery-approach, where individuals strive to understand and learn the tasks and material at hand as thoroughly as possible; (2) mastery-avoidance, where the individual’s aim is to avoid not understanding and learning the task thoroughly; (3) performance-approach, where the individual’s aim is to demonstrate superior performance compared to one’s peers; and (4) performance-avoidance, where the individual strives to avoid the demonstration of a perceived lack of ability or avoid appearing less talented than others. In order to begin to answer the above questions, a qualitative study was undertaken to investigate first year students’ and teaching academics’ expectations and perceptions of the university learning environment, including students’ preferences for what they expect and value in their teachers. The findings of this study yielded some important preliminary insights regarding learning and teaching in the Faculty of Pharmacy at the University of Sydney, Australia. From this preliminary work emerged the chief aims of the program of doctoral work – 1) to investigate achievement goals in pharmacy students and their relationships to academic achievement both cross-sectionally and longitudinally; 2) to tease out the influence of ethnic background on goal orientation and academic achievement; and 3) to examine the relationships between achievement goals, assessment type and academic performance. A further aim was 4) to examine the effects of goal orientation on students’ preferences for teachers’ qualities. A two-step psychometric validation of two measures of achievement goal orientations was undertaken first of all. The first analysis was conducted by sampling pharmacy students from the Faculty of Pharmacy at the University of Sydney, Australia, and in the second analysis, pharmacy students from four countries (England, New Zealand, Wales and United States) were further sampled to confirm the replicability of the instrument in comparable pharmacy education settings. The outcome of this validation study was a robust instrument suited for research into pharmacy student achievement goals. This process paved the way for a further four studies. The first study sought to identify Australian undergraduate pharmacy students’ achievement goals and their relationship to both academic achievement and ethnicity, and to conduct a cross-sectional analysis of two cohorts. The second study followed these two cohorts (Cohort I from year one to year two and Cohort II from year three to year four) to assess the extent to which students’ goal orientations changed over time. The third study investigated the relationship between achievement goals, academic performance and assessment types in undergraduate pharmacy students, again with international participation by pharmacy cohorts from four countries; England, Wales, New Zealand and Australia. The fourth and final study aimed to examine how pharmacy students’ adopted achievement goals might influence their preferences regarding the qualities they would like to see in their teachers. The outcomes of these studies provide important and novel findings regarding students’ perceptions and preferences regarding their motivations for learning; the significance of validating apparently robust measuring instruments for local conditions; the importance of avoiding global measures of academic achievement when studying their relationship with achievement motivation ; the significant role that ethnicity plays in student achievement motivation; and how students’ achievement goals influence their preferred teaching styles of their teachers. This is the first project of its kind conducted into undergraduate pharmacy students’ achievement goal orientations and academic performance. The implications for pedagogical practices are discussed

Ontology model for zakat hadith knowledge based on causal relationship, semantic relatedness and suggestion extraction

Hadith is the second most important source used by all Muslims. However, semantic ambiguity in the hadith raises issues such as misinterpretation, misunderstanding, and misjudgement of the hadith’s content. How to tackle the semantic ambiguity will be focused on this research (RQ). The Zakat hadith data should be expressed semantically by changing the surface-level semantics to a deeper sense of the intended meaning. This can be achieved using an ontology model covering three main aspects (i.e., semantic relationship extraction, causal relationship representation, and suggestion extraction). This study aims to resolve the semantic ambiguity in hadith, particularly in the Zakat topic by proposing a semantic approach to resolve semantic ambiguity, representing causal relationships in the Zakat ontology model, proposing methods to extract suggestion polarity in hadith, and building the ontology model for Zakat topic. The selection of the Zakat topic is based on the survey findings that respondents still lack knowledge and understanding of the Zakat process. Four hadith book types (i.e., Sahih Bukhari, Sahih Muslim, Sunan Abu Dawud, and Sunan Ibn Majah) that was covering 334 concept words and 247 hadiths were analysed. The Zakat ontology modelling cover three phases which are Preliminary study, source selection and data collection, data pre-processing and analysis, and development and evaluation of ontology models. Domain experts in language, Zakat hadith, and ontology have evaluated the Zakat ontology and identified that 85% of Zakat concept was defined correctly. The Ontology Usability Scale was used to evaluate the final ontology model. An expert in ontology development evaluated the ontology that was developed in Protégé OWL, while 80 respondents evaluated the ontology concepts developed in PHP systems. The evaluation results show that the Zakat ontology has resolved the issue of ambiguity and misunderstanding of the Zakat process in the Zakat hadith. The Zakat ontology model also allows practitioners in Natural language processing (NLP), hadith, and ontology to extract Zakat hadith based on the representation of a reusable formal model, as well as causal relationships and the suggestion polarity of the Zakat hadith

Text mining and natural language processing for the early stages of space mission design

Final thesis submitted December 2021 - degree awarded in 2022A considerable amount of data related to space mission design has been accumulated since artificial satellites started to venture into space in the 1950s. This data has today become an overwhelming volume of information, triggering a significant knowledge reuse bottleneck at the early stages of space mission design. Meanwhile, virtual assistants, text mining and Natural Language Processing techniques have become pervasive to our daily life. The work presented in this thesis is one of the first attempts to bridge the gap between the worlds of space systems engineering and text mining. Several novel models are thus developed and implemented here, targeting the structuring of accumulated data through an ontology, but also tasks commonly performed by systems engineers such as requirement management and heritage analysis. A first collection of documents related to space systems is gathered for the training of these methods. Eventually, this work aims to pave the way towards the development of a Design Engineering Assistant (DEA) for the early stages of space mission design. It is also hoped that this work will actively contribute to the integration of text mining and Natural Language Processing methods in the field of space mission design, enhancing current design processes.A considerable amount of data related to space mission design has been accumulated since artificial satellites started to venture into space in the 1950s. This data has today become an overwhelming volume of information, triggering a significant knowledge reuse bottleneck at the early stages of space mission design. Meanwhile, virtual assistants, text mining and Natural Language Processing techniques have become pervasive to our daily life. The work presented in this thesis is one of the first attempts to bridge the gap between the worlds of space systems engineering and text mining. Several novel models are thus developed and implemented here, targeting the structuring of accumulated data through an ontology, but also tasks commonly performed by systems engineers such as requirement management and heritage analysis. A first collection of documents related to space systems is gathered for the training of these methods. Eventually, this work aims to pave the way towards the development of a Design Engineering Assistant (DEA) for the early stages of space mission design. It is also hoped that this work will actively contribute to the integration of text mining and Natural Language Processing methods in the field of space mission design, enhancing current design processes

From Texts to Prerequisites. Identifying and Annotating Propaedeutic Relations in Educational Textual Resources

openPrerequisite Relations (PRs) are dependency relations established between two distinct concepts expressing which piece(s) of information a student has to learn first in order to understand a certain target concept. Such relations are one of the most fundamental in Education, playing a crucial role not only for what concerns new knowledge acquisition, but also in the novel applications of Artificial Intelligence to distant and e-learning. Indeed, resources annotated with such information could be used to develop automatic systems able to acquire and organize the knowledge embodied in educational resources, possibly fostering educational applications personalized, e.g., on students' needs and prior knowledge. The present thesis discusses the issues and challenges of identifying PRs in educational textual materials with the purpose of building a shared understanding of the relation among the research community. To this aim, we present a methodology for dealing with prerequisite relations as established in educational textual resources which aims at providing a systematic approach for uncovering PRs in textual materials, both when manually annotating and automatically extracting the PRs. The fundamental principles of our methodology guided the development of a novel framework for PR identification which comprises three components, each tackling a different task: (i) an annotation protocol (PREAP), reporting the set of guidelines and recommendations for building PR-annotated resources; (ii) an annotation tool (PRET), supporting the creation of manually annotated datasets reflecting the principles of PREAP; (iii) an automatic PR learning method based on machine learning (PREL). The main novelty of our methodology and framework lies in the fact that we propose to uncover PRs from textual resources relying solely on the content of the instructional material: differently from other works, rather than creating de-contextualised PRs, we acknowledge the presence of a PR between two concepts only if emerging from the way they are presented in the text. By doing so, we anchor relations to the text while modelling the knowledge structure entailed in the resource. As an original contribution of this work, we explore whether linguistic complexity of the text influences the task of manual identification of PRs. To this aim, we investigate the interplay between text and content in educational texts through a crowd-sourcing experiment on concept sequencing. Our methodology values the content of educational materials as it incorporates the evidence acquired from such investigation which suggests that PR recognition is highly influenced by the way in which concepts are introduced in the resource and by the complexity of the texts. The thesis reports a case study dealing with every component of the PR framework which produced a novel manually-labelled PR-annotated dataset.openXXXIII CICLO - DIGITAL HUMANITIES. TECNOLOGIE DIGITALI, ARTI, LINGUE, CULTURE E COMUNICAZIONE - Lingue, culture e tecnologie digitaliAlzetta, Chiar

Leadership skills development in theological seminary: crucial factors in creating effective local church leadership

Leadership is a challenge for almost every organization. Like everyone else, the church is faced with similar challenges. Being leaders of the local congregation, pastors are the key element in church leadership. Church members are demanding more of their pastors than ever. Many pastors feel ill-prepared for these tasks. The theological seminary has been the primary place to prepare church leaders to do ministry. Previous researchers have stated that there is compelling evidence that many theological seminaries were not appropriately preparing men and women for the job the local congregation expects them to do for the church. The purpose of this study is to explore whether theological seminaries make leadership learning explicit through course offerings. The study is grounded in related literature on the foundation of leadership and attempts to determine the leadership components of a graduate education in Divinity. The curriculum of the Master\u27s of Divinity program from 10 theological seminaries are examined using catalog content analysis to determine the extent that theological seminaries offer leadership courses to their students to prepare them for future ministry. This is a qualitative study, semantically focused upon the vocabularies in the catalogs. Semantic feature analysis is used with quadrilateral instruments acting as data collection tools. The investigation revealed that graduate studies in Divinity do not universally offer leadership as a component of their curricula. The investigation concluded that the majority of theological seminaries have placed more emphasis on Biblical and theological education. This emphasis did not focus on leadership-skills development. Students were left to make the critical decision on taking leadership courses to prepare for their future ministry. For further study, the researcher recommends replicating this investigation with a larger and more representative sampling of theological seminaries from many more denominations

A Path to Visibility and Leadership: How Mentoring Relationships Impact Career Advancement in Student Affairs for Asian American Women

Asian American women who choose a career in student affairs within the field of higher education are severely underrepresented in both staff positions and top officer/administrator positions compared to White women and other women of color. As the Asian American college student population continues to rise and women make up the majority of undergraduate students in the United States, institutions of higher education must address the need for student affairs staff and administrators to proportionally represent their diverse student bodies. Unfortunately, with an average of 2% of the roles of higher education administrators filled by Asian Americans, few Asian American women are available to mentor and encourage other Asian American women through career advancement in student affairs. This dissertation study examines the mentoring relationships of Asian American women in student affairs and how those relationships impact career advancement within the context of multiple layers of culture. A qualitative biographical narrative inquiry approach and semistructured interviews provided story narratives as data, which were analyzed through open coding and categorization. The findings of this study revealed that mentoring relationships do in fact support Asian American women in student affairs as they navigate their careers, build professional networks, strengthen their skills, and pursue advanced degrees. This study also challenges the application of Kochan’s (2013) cultural framework for mentoring to Asian American women in student affairs, thus resulting in the creation of a multi-layered cultural framework for mentoring Asian American women in student affairs which expands Kochan’s framework

How School Library Media Specialists Support Reading and Information Literacy Skills Instruction for English Language Learners

HOW SCHOOL LIBRARY MEDIA SPECIALISTS SUPPORT READING AND INFORMATION LITERACY SKILLS INSTRUCTION FOR ENGLISH LANGUAGE LEARNERS by Melinda Morin This study explored the school library media programs in four schools. The percentage of English language learners (ELLs) enrolled in each of these schools was among the highest on their respective levels in their school districts. Moreover, the percentage of ELLs in these schools who met and exceeded the standard for reading and English/language arts on the Georgia Criterion-Referenced Competency Test (CRCT) in the spring of 2010 was more than the Annual Measureable Objective (AMO) of 73.3% or slightly less. The participants were the school library media specialists who administered the school library media programs in these schools. This was a qualitative study. During an inductive thematic analysis, the data coalesced into four themes that corresponded with the research questions: instruction, collaboration, media/technology, and interpersonal communication. These findings were derived from the data. 1. The participants used both conventional and technology-based instructional strategies to support reading and information literacy skills instruction for all of their students, including the ELLs. 2. The school library media collections included first language, bilingual, and multicultural literatures, picture books, nonfiction books written on a lower reading level, graphic materials, Hi-Lo reading materials and other digital resources; however, the materials varied in age, suitability, and condition. 3. The school library media specialists collaborated informally with the other members of the instructional team. 4. The school library media specialists undertook other practices that support reading and information literacy skills instruction for ELLs on a discretionary basis