Research Experiences Instrument Scoring Guide

This document is a scoring guide to assist higher-education administrators, faculty, and researchers who wish to use the Research Experiences Instrument (REI). There are five aspects, or factors, that the REI is intended to measure relative to engineering Ph.D. students’ opportunities to practice being a professional in their research experiences, and an overall REI score. Detailed scoring instructions are provided. The REI was developed to assess how the research experiences of engineering Ph.D. students are preparing them for practice. Utilizing a rigorous instrument development process, the REI was shown to be a tool that can reliably and validly be used to assess engineering Ph.D. students\u27 professional practice preparation

An Assessment Framework for First-Year Introduction to Engineering Courses

In this evidence-based practice paper, we describe an assessment framework that applies to first-year introductory engineering courses. First-year engineering courses cover a variety of learning objectives that address both technical and professional outcomes outlined in ABET. These courses also often involve open-ended design and modeling projects. The assessment of multiple competencies along with open-ended design can be a challenging task for educators. In this paper, we describe a framework that guides instructional processes for effective assessment for student learning. This assessment-centered teaching and learning framework helps connect specific learning objectives to broader learning goals or competencies and on-going formative feedback targeting student progression on specific learning objectives. Our plan is to refine the framework using a design-based research approach. Following the description of the model and its development, we present results from the first cycle of implementation. We conclude by discussing hybrid ways for combining traditional methods of assessment with the ability to highlight performance expectations and the appropriate uses of the framework in the classroom

Meaningful Learner Information for MOOC Instructors Examined Through a Contextualized Evaluation Framework

Improving STEM MOOC evaluation requires an understanding of the current state of STEM MOOC evaluation, as perceived by all stakeholders. To this end, we investigated what kinds of information STEM MOOC instructors currently use to evaluate their courses and what kinds of information they feel would be valuable for that purpose. We conducted semi-structured interviews with 14 faculty members from a variety of fields and research institutions who had taught STEM MOOCs on edX, Coursera, or Udacity. Four major themes emerged related to instructors’ desires: (1) to informally assess learners as an instructor might in a traditional classroom, (2) to assess learners’ attainment of personal learning goals, (3) to obtain in-depth qualitative feedback from learners, and (4) to access more detailed learner analytics regarding the use of course materials. These four themes contribute to a broader sentiment expressed by the instructors that they have access to a wide variety of quantitative data for use in evaluation but are largely missing the qualitative information that plays a significant role in traditional evaluation. Finally, we provide our recommendations for MOOC evaluation criteria, based on these findings

A Content Analysis of How Engineering is Assessed in Published Curricula

A Content Analysis of How Engineering is Assessed in Published Curricula (Fundamental) The purpose of this proposal is to present research findings concerning how and what about engineering is commonly assessed in well-known engineering or integrated STEM published curriculum. Two of the major shifts brought about by Next Generation Science Standards (NGSS) are an increased emphasis in students’ capabilities to perform higher-level reasoning skills and integrate content understanding into science practices. At the same time, NGSS has made engineering integration into science education a priority, and it is an exciting time of reform as schools are exploring curriculum resources and teachers are being trained in engineering design. When engineering is a part of science instruction, there must also be corresponding measurement of student learning, yet many teachers who are new to engineering are also unfamiliar with the process of assessing design practices. In addition, teachers must grapple with how to assess higher order skills, including how students use science to make design decisions. Evidence of higher levels of learning beyond memorizing content or identifying facts is the goal of engineering assessment, and NGSS provides general patterns of thought and behavior that students may exhibit at each grade level related to defining problems, developing solutions, and optimizing results which demonstrate learning. However, these guidelines are very broad and do not clearly recommend ways to go about assessing these practices in the classroom. It is imperative that teachers are provided with the means to properly assess student learning of both content and engineering practices. As part of a larger goal of developing an integrated STEM curriculum for grades 4 – 8, this content analysis addresses the need for questioning how engineering is commonly assessed in elementary engineering or integrated STEM curricula. By examining current STEM assessments with two frameworks, this study investigates the following research questions: (1) What aspects of engineering are being assessed in common engineering or integrated STEM curricular units? (2) What level of cognitive demand is being referenced by these assessments? Using a purposeful sampling strategy, the authors reviewed 15 engineering curricula units published by 3 different publishing companies. To address the research questions, assessment tasks were coded based on the Task Analysis Guide in Science (TAGS) framework and alignment to the engineering process of design (POD). Preliminary results show that the majority of integrated STEM assessment items are testing students on memorized engineering practice, and very few are referencing higher levels of cognitive demand. These results indicate that the more complex thinking that students use during engineering classroom learning is being overlooked by subsequent assessment. In addition, a large proportion of assessment items are dedicated to learning background information about the problem and planning solution ideas, compared with other steps in the design process. The full paper will discuss conclusions and implications of the study

Mitigating Biases in Student Performance Prediction via Attention-Based Personalized Federated Learning

Traditional learning-based approaches to student modeling generalize poorly to underrepresented student groups due to biases in data availability. In this paper, we propose a methodology for predicting student performance from their online learning activities that optimizes inference accuracy over different demographic groups such as race and gender. Building upon recent foundations in federated learning, in our approach, personalized models for individual student subgroups are derived from a global model aggregated across all student models via meta-gradient updates that account for subgroup heterogeneity. To learn better representations of student activity, we augment our approach with a self-supervised behavioral pretraining methodology that leverages multiple modalities of student behavior (e.g., visits to lecture videos and participation on forums), and include a neural network attention mechanism in the model aggregation stage. Through experiments on three real-world datasets from online courses, we demonstrate that our approach obtains substantial improvements over existing student modeling baselines in predicting student learning outcomes for all subgroups. Visual analysis of the resulting student embeddings confirm that our personalization methodology indeed identifies different activity patterns within different subgroups, consistent with its stronger inference ability compared with the baselines.Comment: 10 pages, CIKM 202

Video-related pedagogical strategies in massive open online courses: A systematic literature review

For engineers who work with rapidly changing technology in multi-disciplinary teams, massive open online courses (MOOCs) offer the unique ability to deliver free, convenient professional development by providing up-to-date information spanning a wide range of disciplines. However, the MOOC boom has not been without its criticisms; many question the effectiveness of MOOCs. In response, many research studies are being conducted across the world to explore the effectiveness of various pedagogical approaches in MOOCs for different stakeholders. As videos constitute one of the most prominent features of MOOCs, it is important to analyse the empirical evidence of best practices for MOOC videos. Through a systematic literature review, we identify a series of important considerations and actions for three groups: instructional teams, video production teams, and platform developers. Considerations include instructor actions, content design and navigation, video style and length, production quality, video annotation tools, viewing options, and embedded assessments

Sketchtivity, an Intelligent Tutoring Software: Broadening Applications and Impact

Freehand sketching is an essential skill for engineers. Sketching enables designers to represent ideas rapidly and offload working memory. Sketches in the design process also correlate with positive design outcomes. Teaching sketching to engineers, however, presents many challenges in engineering curriculums. Sketching is most often taught in large entry-level courses where individualized feedback, which is vital to learning sketching, is not possible. Sketchtivity is an intelligent tutoring software designed to aid in the practice and feedback on freehand sketching skills. Sketchtivity teaches the basics of two-point perspective sketching providing lessons, feedback, and tips on how to improve. The goal of this project is to implement Sketchtivity at a broader range of universities, to expand on the software functionality, and to understand more about the implications of improving sketching skill. Sketchtivity is currently being implemented in classrooms at three diverse universities across the United States. Our research has shown that the tablet interface does not negatively impact students’ sketching skill development. We are currently conducting further experiments to better understand how students are learning from the feedback the software provides. Beyond measuring current impact, this project seeks to expand the functionality of Sketchtivity to offer lessons on more complex sketching tasks and more poignant feedback to learners. The main research goal is focused on improving sketching skills. The remaining goals turn outward toward the implications for improved sketching skill for engineers. It is essential to understand how to measure sketching skill effectively, and what role sketching skill plays in engineering design. Sketching skill has been measured in many different ways in many different fields and there seems to be no consensus on measurement strategy or validity. As a part of this project, we are conducting a systematic literature review of sketching evaluation. This literature review aims to compile the different methods of measurement to remove some of the ambiguity around evaluating sketching skill in engineering research and education. There has been much research on the role of sketching in engineering design, but the research around sketching skill has been limited. Essential to improving skill in any task is understanding how confidence in the skill is affected. Engineers’ self-efficacy can enable or limit their application of skills. We are also looking at how sketching skill is related to drawing self-efficacy. Sketching has the largest impact in the early stages of design. The final research goal of this project is to investigate the relationship between sketching and creativity in the early stages of design. Through experiments of idea generation and sketching abilities, we hope to better understand to what degree sketching enables creativity in design. To summarize, this project seeks to improve sketching skill through Sketchtivity. We are accomplishing this through expanding reach at multiple universities and expanding the capabilities of the software. It is also essential to understand the implications of improving sketching skill. This is being investigated through improved evaluation of sketching and investigations of sketching’s relationship to drawing self-efficacy and creativity

PRL3-DDX21 transcriptional control of endolysosomal genes restricts melanocyte stem cell differentiation

Melanocytes, replenished throughout life by melanocyte stem cells (MSCs), play a critical role in pigmentation and melanoma. Here, we reveal a function for the metastasis-associated phosphatase of regenerating liver 3 (PRL3) in MSC regeneration. We show that PRL3 binds to the RNA helicase DDX21, thereby restricting productive transcription by RNAPII at master transcription factor (MITF)-regulated endolysosomal vesicle genes. In zebrafish, this mechanism controls premature melanoblast expansion and differentiation from MSCs. In melanoma patients, restricted transcription of this endolysosomal vesicle pathway is a hallmark of PRL3-high melanomas. Our work presents the conceptual advance that PRL3-mediated control of transcriptional elongation is a differentiation checkpoint mechanism for activated MSCs and has clinical relevance for the activity of PRL3 in regenerating tissue and cancer