Measuring Instruction in Higher Education: Summary of a Convening

What will it take to improve the quality of instruction in higher education? An important first step is the ability to measure quality. A variety of measurement systems exist, but how informative are they, and how can we bring greater coherence to instructional measurement in higher education?On November 17 -- 18, 2014, the William T. Grant Foundation, the Spencer Foundation, and the Bill & Melinda Gates Foundation sponsored a convening of experts on education and the learning sciences to address these questions and to guide possible future initiatives by the foundations.The report examines incentive structures in colleges and universities, looks at the goals toward which instructional measurement can be directed, describes past and current research on instructional measurement, and summarizes potential future initiatives

Utilization of Outdoor Resources to Enhance Understanding of Geosciences

Societal awareness and general understanding of the diversity of geosciences environments and phenomena across regions is vital. The primary method to obtain information about geosciences is through mass media. However, there is little coverage of geosciences information and the public may not trust it because of low geosciences awareness. Geosciences are rarely included as part of K-12 curricula and are optional at the collegiate level, contributing to the public deficiency regarding awareness and understanding of the geosciences. This research study investigated methods utilizing outdoor resources in an effort to increase region-specific understanding and awareness of geosciences among diverse groups. A Mississippi K-12 educator professional development session and two middle-high schools were assessed for impact on understanding and interest related to geosciences. All participants utilized outdoor resources to model erosional processes and potential natural hazard events. Both participating schools have high underrepresented minority populations. Pre assessment evaluated participant awareness of Mississippi natural hazards. Teachers and student participants all demonstrated a low awareness of erosion processes specific to the region. Teachers completed erosion models indoors or outdoors, with indoor participants having a significant increase in earth science interest. All student participants completed the erosion model outside, with pre-post erosion comprehension resulting in significant increases for both the middle and high schools. The middle school had significant gains in earth science interest while the high school had a significant decrease regarding careers in geosciences. Virtual field guides developed by online graduate students demonstrating personal understanding of broad geosciences concepts in their local region were evaluated pre-post for impact on geosciences awareness, understanding, and confidence. Awareness factors included geographical community size and locations included in the field guide to demonstrate participant understanding. Significant increases in awareness regarding geoscience resources, including those outdoors, occurred. Significant increases also occurred in confidence utilizing geosciences resources and communicating about geosciences. The majority agreed that the experience of creating the field guide enhanced understanding of geosciences and interest in outdoor activities. Geographical size of participants’ residential communities was significantly related to awareness of regional locations, with urban residents including fewer outdoor locations in the field guide

Engaging Undergraduates in Science Research: Not Just About Faculty Willingness.

Despite the many benefits of involving undergraduates in research and the growing number of undergraduate research programs, few scholars have investigated the factors that affect faculty members' decisions to involve undergraduates in their research projects. We investigated the individual factors and institutional contexts that predict faculty members' likelihood of engaging undergraduates in their research project(s). Using data from the Higher Education Research Institute's 2007-2008 Faculty Survey, we employ hierarchical generalized linear modeling to analyze data from 4,832 science, technology, engineering, and mathematics (STEM) faculty across 194 institutions to examine how organizational citizenship behavior theory and social exchange theory relate to mentoring students in research. Key findings show that faculty who work in the life sciences and those who receive government funding for their research are more likely to involve undergraduates in their research project(s). In addition, faculty at liberal arts or historically Black colleges are significantly more likely to involve undergraduate students in research. Implications for advancing undergraduate research opportunities are discussed

Science Identity Development Trajectories in a Gateway College Chemistry Course: Predictors and Relations to Achievement and STEM Pursuit

This investigation of undergraduates’ heterogeneous science identity trajectories within a gateway chemistry course identified three latent classes (High and Stable, Moderate and Slightly Increasing, Moderate and Declining) using growth mixture modeling. Underrepresented minorities were more likely to exhibit Moderate-and-Slightly-Increasing science identities versus High-and-Stable patterns. Students with higher perceived competence were more likely classified into the High-and-Stable class compared to the other classes. Students classified into the High-and-Stable class scored significantly higher on the final exam and appeared to be more likely to remain in a STEM major across fall and spring semesters compared to the other two classes. Results suggest that some students’ identities shift within a single semester and supporting science perceived competence before college may support students’ science identity development

Access to Online Formative Assessments in Introductory Biology Courses: Investigating Barriers to Student Engagement

Instructors use a variety of online formative assessment (FA) activities to support learning outside of class. Previous studies have revealed barriers for students in online courses, but little is known about the barriers students experience when completing online FA assignments. Understanding these barriers to access is critical to fostering more inclusive learning for all students. Using a framework from previous work in online learning, we examined student perceptions of online FA access with respect to five barrier categories: technical resources, instructor organization, social interactions, personal engagement, and learning environment. We developed and administered a survey to over 1200 undergraduate biology students at two-year and four-year institutions. Students responded to statements using Likert scales and open-ended prompts. Statistical models indicated differences in access across the barrier categories and revealed that demographic characteristics were associated with certain barrier categories. Furthermore, technical resources, instructor organization, and personal engagement barriers were associated with lower course performance. In open-ended responses, students most frequently suggested that changes to scheduling logistics, course delivery, and FA format would improve their online FA experience. We discuss how these findings and student suggestions can inform instruction, particularly how instructors can alter their FA characteristics to better suit their student population. Advisor: Brian A. Couc

Understanding the Relationship Between Organic Chemistry Misconceptions and Students’ Chemistry Self-efficacy in Higher Education Organic Chemistry Courses

Organic chemistry is accepted as a crucial part of science higher education funneling students into many career opportunities such as pharmaceuticals, biotechnology, and medical industries. Students attempting organic chemistry courses in higher education are among a plethora of majors including biology, chemistry, health science, and engineering. However, organic chemistry as a course has stayed fairly stagnant for the past 50 years. Students in this course typically resort to rote-memorization and often regard the course itself as insurmountable. To answer the decreasing retention rates seen across the United States, the research revolving around organic chemistry knowledge and teaching methodologies has increased in the past twenty years. Furthermore, self-efficacy in chemistry has been established as a pivotal aspect for students to be successful in chemistry; yet little effort has been made to understand if a relationship exists between foundational knowledge in organic chemistry and a student’s chemistry self-efficacy. In an effort to help fill this gap in the literature, this dissertation investigates the relationship between chemistry-oriented misconceptions held by university students and their organic chemistry self-efficacy during the first semester organic chemistry course. Specifically, 97 university students were surveyed using validated instruments regarding their foundational knowledge of chemistry based on NGSS standards, their chemistry selfefficacy, and demographic information. The results indicated that at the beginning of the semester, the more chemistry-oriented misconceptions students held, their chemistry selfefficacy was significantly lower. The students who had attempted organic chemistry at least once before were also highly correlated with more misconceptions. At the end of the semester, the number of misconceptions were again negatively correlated with student self-efficacy. These findings may have implications for organic chemistry instructors to set more foundational curriculum at the beginning of the semester to work through misconceptions as they could pose a roadblock to self-efficacy enhancement and ultimate success in the course

Effects of Extra-Curricular Project-Based Learning Experiences on Self-Efficacy and Interest in STEM Fields in High School

As our society and systems become more technologically advanced, increasing opportunities exist for students interested to pursue careers in Science, Technology, Engineering, and Mathematics (STEM) fields. However, pervasive inequities have led to differences in the extent to which women and underrepresented racial/ethnic groups choose to pursue study and career pathways in STEM. Project-Based Learning (PBL) is among the most widely researched strategies suggested to support student learning and motivation and has more recently been applied to school-based efforts to increase student interest in STEM related fields. Rooted in Social Cognitive Career Theory, this study examined changes in students’ self-reported general self-efficacy and interest in STEM fields following a four-week Project-Based Learning experience focused on career pathways in advanced manufacturing. Thirty students across four high schools participated in a month-long Project-Based Learning experience to introduce them to the field of advanced manufacturing. Findings from a 15-item online survey distributed at the beginning and end of a virtual four-week PBL workshop revealed significantly higher self-reported general self-efficacy scores following the PBL experience. While overall findings revealed a positive correlation between students’ self-reported general self-efficacy and STEM interest, the relationships varied by student demographic groups. Recommendations for further research and applications to practice are provided

Exploring Relations Between Motivation, Metacognition, and Academic Achievement Through Variable-Centered, Person-Centered and Learning Analytic Methodologies

The three studies that comprise this dissertation examine relations between student characteristics, motivations, metacognitive learning processes, and academic achievement. Methodologically, the dissertation demonstrated the potential of multiple types of approaches and data resource types. By employing multiple approaches including variable-centered, person-centered, and learning analytics, researchers can understand learning processes from various angles. In addition, through this triangulation by multiple types of methodological approaches, educational theories could be more thoroughly verified and supported by various empirical findings. Multiple types of data resources are related to analytical methods. The purpose of the first paper was to examine relations between achievement goals and metacognitive learning behaviors using a clustering analysis and visualization. A clustering analysis conducted with achievement goals produced three goal profiles; 1) mastery-approach, 2) performance-approach, and 3) performance-avoidance identified three goal profiles. The profiles include High Approach, High Mastery, and High Goal Endorsement groups. The finding demonstrated that students in the High Mastery group, who had greater use of the self-assessment tool, obtained higher final grades than other groups could be explained from the perspective of SRL. In addition, learners motivated by mastery approach goals engaged in the greater use of self-assessment quizzes. Students in the High Mastery group also used the tools earlier than other two groups for exam 2. As the most frequently used pattern, sequential pattern mining discovered the repeated use of self-assessment quizzes to monitor their learning. More students in the High Mastery group employ this pattern of metacognitive events than students in the High Performance and High-Goal endorsement groups, particularly during sessions in weeks before exams. A subsequent analysis revealed that for all exams, students who conducted a repeated behavior pattern indicative of metacognitive monitoring and control outperformed those who did not. From the research, it is confirmed that the person-centered analysis provided authentic and generalizable groups and afforded observation of the learning behaviors of learners with typical combinations of goals. In addition, sequential patterns provide instructor more interesting information on learning processes than the frequency of accesses. The purpose of the second research was to identify motivational profiles based on multiple types of motivations including self-efficacy, achievement goals, and expectancy-value from an integrative perspective. For this research, a LPA was conducted with ten types of motivational constructs and three kinds of metacognitive learning processes. The LPA identified four motivational profiles; 1) High Cost, 2) High Performance Goals, 3) High Goals and Values, and 4) Low Performance Goals, and three metacognitive profiles; 1) Infrequent metacognitive processing. 2) Checking performance and planning, and 3) Self-assessment. Student demographic information significantly influenced the membership of motivational profiles. Older students tend to have higher self-efficacy, mastery-approach, and values, but low cost than younger ones. In addition, compared to Caucasian and Asian students, underrepresented students tend to be more motivated by higher goals and values than high cost or high performance goals. Lastly, female students are more likely to be members of High performance goals and High goals and values than High cost oriented and Low performance goals and cost than males. In terms of the relations profiles with academic achievement, Low Performance Goals group showed the best performance. Among metacognitive profile groups, students in Checking performance and planning, and Self-assessment demonstrated similar academic performance. The investigation of relations between two profile groups demonstrated that students in the High cost group are more likely to be a member of self-assessment group than checking performance and planning as well as of a member of an infrequent metacognitive process than checking performance and planning. In addition, students in high performance and goals and high goals and values groups relative to the low performance goals group more likely to be a member of the infrequent metacognitive process than checking performance and planning. The findings of this research provide authentic motivation status and metacognition learning process as well as their relations. Addition, this research figured out specific motivational profiles through the multiple types of motivations from the integrative perspective. Therefore, instructors can provide more effective and specific interventions to students who have difficulty utilizing metacognitive learning processes, considering motivational status based on multiple motivations. In addition, instructors can understand motivational profiles by demographics so at the beginning of the semester in which the information on students is not enough to identify students learning processes, they intervene students based on demographic information. The purpose of the third paper was to consider the relative importance of capturing demographic, motivational and metacognitive processes as potential predictors of learning outcomes, and appraises them alongside both traditional prediction modeling approaches in higher education, and emergent methods, sequence pattern mining, arising from the field of educational data mining. The sequence pattern mining discovered the repeated use of self-assessment quizzes in Biology and repeated use of planning contents in Math. A regression model with combined resource types demonstrated the improved predictive power than models with individual resource types. Also, theory-aligned behaviors designed based on metacognitive learning processes better improved the accuracy of the model than non-theory-aligned behaviors automatically provided by the system. Lastly, when applying the same prediction model, the model better explained the variance of academic achievement in Biology in which metacognitive supporting tools designed based on an educational theory than that in Math that has few theory-aligned behavior variables. Therefore, this study emphasizes the importance of existing ambient data from university systems. Also, log data generated by systems such as LMS allows researchers to examine the same data in different ways with no need for additional data collection. Lastly, educational theory and contexts should be taken into consideration in designing courses and developing the prediction models. Therefore, instructors and researchers, in designing courses, the consideration of educational theories and contexts is the essential process. This dissertation provides insight regarding authentic relations between motivation, metacognition, and academic achievement. Specifically, instructors can understand how multiple types of motivations work together, and the motivational profiles influence metacognitive learning strategies. In courses, by examining motivational profiles, instructors can provide more effective intervention with which students change their resolve their weak learning easier. Practically, by investigating each type of predictor from data resources including demographic, motivation, and behavioral variables, findings from this dissertation can enable researchers to prioritize development of prediction models to identify students who are more likely to experience failure in courses. Additionally, instructors can figure out the importance of interpreting variables through educational theories and in context through the comparison of courses with differing instructional designs. Further, by appraising these results in light of theory, instructors can take action to improve student’s learning outcomes by adjusting the design of their courses

When Do Faculty inputs Matter? A Panel Study of Racial/Ethnic Differences in Engineering Bachelor\u27s Degree Production

Science, technology, engineering and mathematics (STEM) fields are widely credited as the primary drivers of economic growth through innovation, with engineering universally identified as especially critical. Yet as other nations have strengthened their engineering talent pools, the United States has struggled to cultivate an engineering workforce that reflects its diversity and takes full advantage of its human capital. Reflecting this dilemma, African Americans have consistently posted the weakest persistence and bachelor\u27s degree completion rates of all racial/ethnic groups in engineering, and by some indications, their postsecondary outcomes are worsening. The purpose of this study was to develop understanding about potential institutional levers for improving engineering bachelor\u27s degree attainment both for underrepresented minorities (URMs) broadly and Black students specifically. Drawing on the higher education production function, I used multiple sources of institutional panel data for 324 engineering schools/colleges from 2005 to 2011 to uncover differential relationships between faculty predictors and engineering bachelor\u27s degree production by student race/ethnicity and institutional context. I used multiple imputation to handle missing data and estimated fixed effects linear regression and dynamic panel models of engineering degree production, then I assessed institutions\u27 degree production efficiencies using stochastic frontier analysis. The findings indicate that from 2005 to 2011, the number of engineering bachelor\u27s degrees conferred to Black students declined 10%, with the smallest declines occurring at highly competitive institutions (2%) and the largest declines at HBCUs (30%). Results from the fixed effects models indicate that engineering faculty-to-student ratio was positively related and the proportion of research faculty negatively related to engineering bachelor\u27s degree production for every student subgroup in at least one institutional setting. The share of URM faculty was positively related to degree production for URMs and Blacks in some settings. However, no faculty measure was predictive of degree output for every student subgroup across every institution type. And in every instance where a faculty variable was related to degree output for multiple student subgroups, the magnitude of the estimated effect was greatest for Black students, then URMs, then all students. Ultimately, the study suggests that leveraging institutional resources to improve student outcomes in STEM calls for targeted analyses to develop strategies that reflect the heterogeneity of STEM disciplines, STEM students, and educational settings

Probing the National Geoscience Faculty Survey for Reported Use of Practices that Support Inclusive Learning Environments in Undergraduate Courses

What is the extent to which college and university geoscience faculty report using education practices that contribute to more inclusive learning environments and engage a diverse population of students? In the 2016 National Geoscience Faculty Survey, faculty answered questions about their practices in a specific introductory or major course they had taught in the previous two years, and about how they share and learn about the content and methods used in their teaching. Based on factor analysis, 22 of the survey questions divided into four categories associated with inclusive teaching practices: geoscientist representations, curricular choices, learning strategies, and career pathways. The self-reported use of practices across these four categories varies greatly, with some used by as many as 71% of faculty respondents whereas others by only 8%. These data provide new information on the current state of teaching practices in the geosciences with regard to inclusive practices, and establish a baseline to which responses from future surveys may be compared. Univariate general modeling combined with ANOVA tests on the responses to the questions shows that education practices differ based on variables such as teaching style, communication with colleagues, years of teaching experience, faculty type, institution type, class size, and course type (introductory or major). These differences suggest opportunities for focused geoscience faculty development around education practices that support the success of a diverse population of undergraduate students and the enhancement of inclusive learning environments in the geosciences