Characterizing Interactive Communications in Computer-Supported Collaborative Problem-Solving Tasks: A Conditional Transition Profile Approach

Communication in a collaborative problem-solving activity plays a pivotal role in the success of the collaboration in both academia and the workplace. Computer-supported collaboration makes it possible to collect large-scale communication data to investigate the process at a finer granularity. In this paper, we introduce a conditional transition profile (CTP) to characterize aspects of each team member's communication. Based on the data from a large-scale empirical study, we found that participants in the same team tend to show similar CTP compared to participants from different teams. We also found that team members who showed more “negotiation” after the partner “shared” information tended to show more improvement after the collaboration while those who continued sharing ideas while their partners were negotiating tended to improve less

Using the Guttman Scale to Define and Estimate Measurement Error in Items over Time: The Case of Cognitive Decline and the Meaning of “Points Lost”

We used a Guttman model to represent responses to test items over time as an approximation of what is often referred to as “points lost” in studies of cognitive decline or interventions. To capture this meaning of “point loss”, over four successive assessments, we assumed that once an item is incorrect, it cannot be correct at a later visit. If the loss of a point represents actual decline, then failure of an item to fit the Guttman model over time can be considered measurement error. This representation and definition of measurement error also permits testing the hypotheses that measurement error is constant for items in a test, and that error is independent of “true score”, which are two key consequences of the definition of “measurement error” –and thereby, reliability- under Classical Test Theory. We tested the hypotheses by fitting our model to, and comparing our results from, four consecutive annual evaluations in three groups of elderly persons: a) cognitively normal (NC, N = 149); b) diagnosed with possible or probable AD (N = 78); and c) cognitively normal initially and a later diagnosis of AD (converters, N = 133). Of 16 items that converged, error-free measurement of “cognitive loss” was observed for 10 items in NC, eight in converters, and two in AD. We found that measurement error, as we defined it, was inconsistent over time and across cognitive functioning levels, violating the theory underlying reliability and other psychometric characteristics, and key regression assumptions

Principled development of Workplace English Communication part 1: a sociocognitive framework

Background: This study advances a sociocognitive approach to modeling complex communication tasks. Using an integrative perspective of linguistic, cultural, and substantive (LCS) patterns, we provide a framework for understanding the nature and acquisition of people’s adaptive capabilities in social/cognitive complex adaptive systems. We also illustrate the application of the framework to learning and assessment. As we will show, understanding the connection between measurement models and users’ needs is important to increase assessments’ educative usefulness. Questions Addressed: Our framework is designed to address questions regarding the following four areas: the nature of sociocognitive perspectives in educational measurement, the application of LCS patterns to complex communication tasks captured in an extended formative assessment of Workplace English Communication (WEC), the usefulness of psychometric models for instruction and assessment with such complex tasks, and considerations for measurement modeling. Conclusions: Our study concludes with reflections on the challenges of complex assessments such as WEC, the advantages of sociocognitive modeling for new assessment genres, and the roles of situated measurement models in meeting the challenges.Ye

Principled development of Workplace English Communication part 2: Expanded Evidence-Centered Design and Theory of Action frameworks

Open access article. Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International license (CC BY NC-ND 4.0) appliesBackground: In today’s rapidly evolving world, technological pressures coupled with changes in the nature of work increasingly require individuals to use advanced technologies to communicate and collaborate in geographically distributed multidisciplinary teams. These shifts present the need to teach and assess an expanded set of knowledge, skills, and attitudes, including how to communicate at work in collaborative environments using diverse forms of technology. They also present the opportunity to create novel forms of instructional materials and forms of assessment that extend the more traditionally used summative assessments to assessments used for learning and instruction. This design process can be facilitated through the use of conceptual frameworks employed to guide assessment design and development. Their use is important to support more expansive and complex design goals emerging in the design of assessments of 21st century skills such as Workplace English Communication (WEC). In this article, we reflect on an evolving WEC construct needed for today’s economy and discuss implications for expanding how we teach and assess it using formative assessments for learning. We then discuss the features of the expanded Evidence-Centered Design for learning and assessment systems (e-ECD) and Theory of Action (ToA) frameworks and illustrate their integrative application to inform the design and development of WEC training modules (or resources). We conclude with suggestions for next steps in this line of research. Questions Addressed: In reference to the e-ECD and ToA frameworks, our article addresses questions in two areas. We illustrate the benefits of using the ToA to explicitly identify the components of an assessment, its action mechanisms, stakeholders’ needs, score-based decisions and their impact, and the services designed for test takers and users. We illustrate the benefits of using the e-ECD framework to guide design efforts in principled ways to enable consideration of both key elements that relate evidentiary elements relevant to the construct, aspects of learning and assessment, and measurement models. Consideration of these frameworks is important to design assessments and make sense of the evidence for meaningful interpretation of students’ results. Conclusions: This article illustrates the application of conceptual (e.g., the e-ECD and ToA) frameworks that can be used to inform the design and development of similar modules for complex tasks of 21st century skills. This article contributes to the literature on WEC and complex assessments of hard-to-assess constructs more generally by offering a way of thinking about designing, assessing, and then evaluating the design and assessment of interactive educational modules for teaching complex communication knowledge and abilities while remaining attentive to (negative) consequences associated with the stakeholders designing, developing, and using the assessments.Ye

Principled development of Workplace English Communication part 3: an Integrated Design and Appraisal Framework

Open access article. Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC BY-NC-ND 4.0) appliesBackground: An expanded skillset is needed to meet today’s shifting workplace demands, which involve collaboration with geographically distributed multidisciplinary teams. As the nature of work changes due to increases in automation and the elevated need to work in multidisciplinary teams, enhanced visions of Workplace English Communication (WEC) are needed to communicate with diverse audiences and effectively use new technologies. Thus, WEC is ranked as one of the top five skills needed for employability. Even so, results of employer surveys report that incoming employees lack communication competency (National Association of Colleges and Employers [NACE], 2018). To address this issue, with a focus on WEC teaching and assessment, we describe a framework used to guide the design of WEC modules. We suggest that conceptual frameworks can be used to inform the design process of the module. In this article, we illustrate one such conceptual framework: the Integrated Design and Appraisal Framework (IDAF). IDAF holds consequences of testing as one of its central elements to guide test design and development. It emphasizes categorically identifying and ecologically modeling variables impacting WEC in general and the writing context in particular. It emphasizes the need for developing clearly articulated construct models to underpin the assessment, as well as incorporating a foundational focus on fairness and social consequences into the design process and use of assessments. Questions Addressed: In reference to the IDAF, our article addresses questions in the following three areas: the nature and benefits of an integrated design and appraisal approach to test design, development, and evaluation; the application of IDAF to complex communication tasks captured in formative assessment scenario-based modules of WEC; and the paramount importance of considering fairness and social consequences in the design and use of assessments administered to diverse populations. Thus, this article elaborates on the use of the IDAF to inform the design of WEC modules by explicitly articulating the needs of the test takers, the anticipated uses of the modules, and the contexts in which the modules would be used. Our questions are designed to address increasing complexities associated with the design of complex constructs such as WEC. This article describes considerations for the development of integrated learning and assessment modules for WEC. We start by reviewing principled assessment design frameworks, which have been used to inform the development of complex tasks across disciplines or fields. Following a description of WEC in terms of domain analysis and design patterns, we illustrate the application of the IDAF to inform the design of the modules. We conclude by providing an overview of our research questions and of how our article addressed them. We also discuss lessons learned with respect to the design of the prototype and the delicate balance of engaging in a principled design process that supports goals that empower students of diverse backgrounds to learn WEC. Conclusions: This article illustrates the application of the IDAF to inform the design and development of WEC modules. This article contributes to the literature on WEC and complex assessments of hard-to-assess constructs more generally by offering a way of thinking about designing, assessing, and then evaluating the design and assessment of interactive educational modules for teaching complex communication knowledge and approaches.Ye

Introduction: meeting the challenges of Workplace English Communication in the 21st century

Open access article. Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) applies[Abstract not available]Ye