Analyzing the Text Contents Produced by ChatGPT: Prompts, Feature-Components in Responses, and a Predictive Model

ChatGPT is a large language model that uses deep learning to produce natural language and generate intelligent and relevant responses to user prompts. It comes to the field of education as an inevitable wave. Educators have to deal with it and figure out appropriate ways to use it and produce positive learning. This study explores the use of ChatGPT from the perspective of front-end users, focusing on the text-content that ChatGPT can produce for learners to learn new knowledge (e.g., a concept, a theory, or an application). The sample of this study consists of 253 ChatGPT text responses derived from three types of initial prompts/questions: general questions, specific questions, and questions with interactive prompts. Six feature components of text-information that can help learners to understand new knowledge are analyzed (concept and definition, procedure, example, comparison or contrast, deductive or inductive argument, summary). The results from Chi-square tests indicate that the presence of each feature component in the responses differs by the types of prompts. The results from a logistic regression analysis reveal that the presence of five (out of the six) feature components are significant to the probability that a response providing accurate and reliable information. The integration of using ChatGPT into learning is discussed. Further research questions are suggested

Quantitative Content Analysis Methods in Instructional Technology Research: Defining, Coding, Analyzing and Modeling (DCAM)

Content analysis has been applied in the research of instructional design and technology to analyze (a) text-based contents, such as online discussions, social media communications, or published articles, and (b) other formats of contents such as videos, audios, or pictures. The purpose of this article is to introduce a method of DCAM (Defining, Coding, Analyzing and Modeling) for content analysis with practice examples. DCAM is a quantitative method generated from a series of studies in instructional design conducted by the author, and supported by the literature in the field. The variables defined from the text-content or other formats of contents can be design related variables, learning related variables, micro-activities in learning, or behavior-performance related learning outcome. In this article, first, nominal, ordinal and scaled coding methods on those variables are demonstrated. Second, reliability measures in content-variable coding are reviewed and explored. Third, parametric and nonparametric statistics methods to examine those variables for content analysis are presented. Finally, some cautions and suggestions to conduct content analysis is discussed

Characteristics of Initial Posts and Peer Engagement: Density Score Analyses for Social Presence in Online Discussions

A common activity in online courses that allows for student interaction is the asynchronous discussion; however, discussions do not inherently lead to meaningful engagement among students. This study explores how the moves that students make in their initial discussion posts influence the emotional engagement of their peers in response posts. 1500 asynchronous online discussion messages were collected from an undergraduate online course offered at a western state university. 608 online discussion threads were analyzed to determine how the characteristics of initial posts are associated with the engagement in peer responses. Six characteristic variables from initial posts were identified and analyzed. Density scores for social presence categories and indicators were calculated as the measure of the emotional engagement in the response posts. Results suggest that three characteristic variables in initial posts significantly influence the emotional engagement of peers in the response posts

Intrinsic Motivation Factors in Gamified Photography Learning: Direct and Indirect Effects

To help students with motivation to learn, gamification has been explored as a method of delivering content to students in an engaging and motivating way. This study explored the motivation components in gamified design that linked to learning outcomes in a gamified photography curriculum. Six intrinsically motivating factors (challenge, curiosity, control, cooperation, competition, and recognition) were carefully integrated into the game experience design. Participant test scores and survey data were used to develop a maximum likelihood structural equation model. The model showed that among the six intrinsic motivation factors curiosity and control were directly linked to learning outcomes while challenge had an indirect link to learning outcomes. These findings suggest that these three items need to be carefully integrated into gamified lesson design to promote engagement and learning

Methods to Analyze Likert-Type Data in Educational Technology Research

Likert-type items are commonly used in education and related fields to measure attitudes and opinions. Yet there is no consensus on how to analyze data collected from these items. In this paper, we first synthesized literature on strategies for analyzing Likert-type data and provided computing tools for these strategies. Secondly, to examine the use and analysis of Likert-type data in the field of educational technology, we reviewed 424 articles that were published in the journal Educational Technology Research and Development between 2016 and 2020. Our review showed that about 50% of the articles reported Likert-type data. A total of 139 articles used Likert-type data as a dependent variable, among which 86% employed parametric methods to analyze the data. In addition, less than 1% of the 139 articles used an ordered probit/ logit model, transformation, or strategy for rescaling Likert-type data to interval data to perform statistical analysis. Finally, to empower educational technology researchers to handle Likert-type data effectively, we concluded the paper with our suggestions and insight regarding alternative strategies and methods

Conducting Synchronous Assessment through Web Videoconference to Improve Online Learning: Case Outcomes with Nonparametric Analysis

Online assessment has always been a challenge to online teaching. Educators have been exploring a variety of methods to perform online assessment. However, it appears that there is not enough work in the field focusing on online synchronous assessment. This paper presents two cases that demonstrate the design and implementation of using web videoconference for synchronous assessment in an educational research methods online course and an instructional video production online course. The purpose of the two cases was to explore whether or with what methods student online learning could be improved through synchronous assessment. Case outcomes were analyzed with nonparametric methods, and the results did show students’ improvement in their learning, specifically in their understanding and mastering of factual, conceptual, procedural and metacognitive knowledge. Methods, procedures, tips and cautions of conducting such videoconference-based synchronous assessment in online courses are discussed

Flipped Learning and Influential Factors: Case Analysis

Flipped learning has been a focus of research to explore potential learning environments that may positively affect student learning. The key issue is whether or how educators design such a learning environment, and what might be the factors that educators need to consider when designing a flipped learning environment. The first part of this study presents a critical review and analysis on factors identified from the literature that may influence the success of a flipped-learning case. 216 cases selected from current literature were analyzed on seven factors (Overall Design, Design of Information, Design of Technology Use, Active Learning, Motivation, Special Guidance, and Self-Regulated Learning) regarding their influence on the success of flipped learning experiences. Among them the first five factors were found to be significant and included in a prediction model. The second part of this study demonstrates an on-going case of flipped learning that reflects and examines the prediction model

Social Media in Dynamic Learning: Logistics and Influential Factors

Educators have been using social media to enrich learning activities and promote interactive and collaborative learning. Under the context of dynamic learning – the way that 21st century’s learners learn, the new challenges are: how educators design such a setting to effectively integrate certain social media tools to improve learning, and what the influential factors might be that educators need to focus during the design. In this article, we employ the concept “logistics” to explain and redefine dynamic design, dynamic learning, and dynamic thinking, which furthermore formulate the framework of the study. This article presents a critical content review of current literature, and an analysis of 276 cases located from the literature on seven factors (Information Logistics, Technology Logistics, Overall Design Logistics, Collaborative Learning, Active Stimulation, Motivation, and Objective-Driven Activities) regarding their influence on the success of social medial supported learning experiences. All seven factors were found to be significant and included in a static predictive model. An in-depth comprehension of this static predictive model is provided, based on which a new dynamic model is proposed