Effects of Metacognitive Monitoring on Academic Achievement in an Ill-Structured Problem-Solving Environment

Higher education courses are increasingly moving online while educational approaches are concurrently shifting their focus toward student-centered approaches to learning. These approaches promote critical thinking by asking students to solve a range of ill-structured problems that exist in the real world. Researchers have found that student-centered online learning environments require students to have self-regulated learning skills, including metacognitive skills to regulate their own learning processes. Much of the research suggests that externally supporting students while they are learning online, either directly or indirectly, helps them to succeed academically. However, few empirical studies have investigated what levels of support are most effective for promoting students\u27 self-regulated learning behaviors. Additionally, these studies reported conflicting results – some found maximum support to be most effective while others found no significant difference. The purpose of this study was to investigate the effectiveness of different levels of support for self-regulated learning during a complex learning activity to solve an ill-structured problem-solving situation in an online learning environment. In addition, the role of students\u27 self-efficacy on their academic achievement was examined. A total of 101 undergraduate students from three international studies courses offered at a large urban Southeastern public university in the United States participated in the study. The students were randomly assigned to treatment (minimum support, maximum support) and control groups. Students\u27 academic achievement scores were measured using a conceptual knowledge test created by the professor teaching the courses. O\u27Neil\u27s (1997) Trait Self-Regulation Questionnaire measured students\u27 self-efficacy. Analysis of Co-Variance (ANCOVA) was conducted to analyze the data. The ANCOVA results indicated significant improvement of the academic achievement of the minimum support group versus both the maximum support and control groups. Additionally, self-efficacy as a co-variable did not significantly impact students\u27 achievement scores in any of the groups. The overall results indicated that it is important to consider the level of self-regulated learning support when designing online learning environments promoting students\u27 critical thinking skills. Promoting students\u27 self-regulated learning skills is vital when designing online higher education courses

Scaffolding Novice Instructional Designers' Problem-Solving Processes Using Question Prompts in a Web-Based Learning Environment

The present study investigated the effects of question prompts in scaffolding novice instructional designers solving ill-structured, instructional design problems in a Web-based learning environment. The effects of question prompts were studied under different prompting conditions (Question-Elaboration vs. Question-Guidance), taking into consideration various levels of learners' prior knowledge and experience. The study employed a comparative, multiple-case study design using the technique of think-aloud protocols, which were followed by interviews. Eight graduate students from the program of Instructional Design and Technology participated in the study. While the qualitative findings supported the previous research on the advantages of question prompts in scaffolding ill-structured problem solving, they also shed light on the specific cognitive and metacognitive functions, as well as limitations, of question prompts in different conditions. The study has implications for designing instructional scaffolds for supporting ill-structured problem solving of various domains in a Web-based learning environment.Yeshttps://us.sagepub.com/en-us/nam/manuscript-submission-guideline

Scaffolding Reflection: Prompting Social Constructive Metacognitive Activity in Non-Formal Learning

The study explores the effects of three different types of non-adaptive, metacognitive scaffolding on social, constructive metacognitive activity and reflection in groups of non-formal learners. Six triads of non-formal learners were assigned randomly to one of the three scaffolding conditions: structuring, problematising or epistemological. The triads were then asked to collaboratively resolve an ill-structured problem and record their deliberations. Evidence from think-aloud protocols was analysed using conversational and discourse analysis. Findings indicate that epistemological scaffolds produced more social, constructive metacognitive activity than either of the two other scaffolding conditions in all metacognitive activities except for task orientation, as well as higher quality interactions during evaluation and reflection phases. However, participants appeared to be less aware of their activities as forming a strategic, self-regulatory response to the problem. This may indicate that for learning transfer, it may be necessary to employ an adaptive, facilitated reflection on learners' activities

The effects of problem-based learning scaffolds on cognitive load, problem-solving, and student performance within a multimedia-enhanced learning environment

textLearners who are novice problem solvers often encounter difficulty when solving complex problems. One explanation for this difficulty is that the cognitive requirements of problem-solving are sufficiently high that learners easily become overwhelmed and frustrated, leading to a state known as cognitive overload in which learning is obstructed. Cognitive Load Theory is concerned with the design of instructional approaches intended to manage the cognitive load required for thinking and problem-solving tasks. Scaffolds are any kind of support that facilitates the accomplishment of a difficult task that a learner would not be able to accomplish on their own. They are potential mechanisms to support students in negotiating the potentially high cognitive load required by complex problem-solving. The purpose of this study was to examine the effects of technology-based scaffolds within a problem-based learning environment known as Alien Rescue. The study investigated the impact of scaffolds on cognitive load, problem-solving behaviors, science knowledge, and student perceptions of the learning environment. Participants for this study included sixth grade students from a suburban middle school in the southwestern United States. Student classes were assigned to one of three treatment conditions: (a) a problem constraint condition in which students were guided through a problem-solving process similar to that of an expert, (b) a prompt condition in which students were provided with guiding messages during problem-solving, and (c) a control condition with no scaffolding. All conditions participated in the use of Alien Rescue for three weeks. Measures including a self-report measure of mental effort, calculated instructional efficiency scores, problem solution scores, student activity logs, and science knowledge test performance were used to evaluate students' cognitive load, problem-solving performance, problem-solving strategies, and learning gains. An open-ended questionnaire and student interviews were used to gather data on students' perceptions of the program. Results of the study indicate statistically significant differences between treatment conditions with respect to problem-solving efficiency, student problem-solving behaviors, and scientific knowledge gain. Additionally, qualitative findings highlight problematic aspects of the highly structured condition as implemented within the classroom context while also identifying components of the learning environment that were perceived as helpful and useful to participants. Teacher interviews also provided insight into classroom implementation of the program and opportunities to further enhance scaffolds to support student learning. Implications of the study from research, instructional design, and technology perspectives are discussed along with a treatment of study limitations and opportunities for further research in this area.Curriculum and Instructio

Scaffolding for Optimal Challenge in K–12 Problem-Based Learning

Establishing optimal challenge enhances intrinsic motivation, interest, and the probability of success in the learning activity. In K–12 problem-based learning (PBL), students may struggle to address associated tasks that are beyond their current ability levels. This paper suggested learner-centered scaffolding systems (LSS) to improve K–12 students’ perception of optimal challenge by addressing their learning issues in PBL. LSS enhances students’ experience in autonomy and competence by providing multiple types of scaffolding in accordance with students’ different needs and difficulties in PBL. Students can control the nature and frequency of scaffolding by themselves according to their needs and ability, and it plays a role in improving their self-directed learning skills. Last, peer scaffolding between students with similar abilities satisfies students’ needs for relatedness

Instructional Scaffolding in STEM Education: Strategies and Efficacy Evidence

science education; educational technology; learning and instructio

Solving Real World Problems With Alternate Reality Gaming: Student Experiences in the Global Village Playground Capstone Course Design

The Global Village Playground (GVP) was a capstone learning experience designed to address institutional assessment needs while providing an integrated and authentic learning experience for students aimed at fostering complex problem solving, as well as critical and creative thinking. In the GVP, students work on simulated and real-world problems as a design team tasked with developing an alternate reality game that makes an impact on the United Nations Millennium Development Goals. Researchers employed a qualitative case study approach to evaluate what aspects of this problem-based, hybrid, course design students found most and least beneficial to their learning. Findings suggest strategies for designing technology-based learning environments to support complex problem solving. Specific recommendations pertain to scaffolding team-based problem solving, particularly concept development processes, interdependence among team members, and group self-organization

Conceptualizing and supporting awareness of collaborative argumentation

In this thesis, we introduce “Argue(a)ware”. This is a concept for an instructional group awareness tool which aims at supporting social interactions in co-located computer-supported collaborative argumentation settings. Argue(a)ware is designed to support the social interactions in the content (i.e., task-related) and in the relational (i.e., social and interpersonal) space of co-located collaborative argumentation (Barron, 2003). The support for social interactions in the content space of collaboration is facilitated with the use of collaborative scripts for argumentation (i.e., instructions and scaffolds of argument construction) as well with the use of an argument mapping tool (i.e., visualization of argumentation outcomes in a form of diagrams) (Stegmann, Weinberger, & Fischer, 2007; van Gelder, 2013). The support for social interactions in the relational space of collaboration is facilitated with the use of different awareness mechanisms from the CSCL and the CSCW research fields (i.e., monitoring, mirroring and awareness notification tools). In this thesis, we examined how different awareness mechanisms facilitate the regulation of collaborative processes in the relational space of collaborative argumentation. Moreover, we studied how they affect the perceived team effectiveness (i.e., process outcome) and group performance (i.e., learning outcome) in the content space of collaboration. Thereby, we studied also the effects of the design of the awareness mechanisms on the application of the mechanisms and the user experience with them. In line with the design-based research paradigm, we attempted to simultaneously improve and study the effect of Argue(a)ware on collaborative argumentation (Herrington, McKenney, Reeves & Oliver, 2007). Through a series of design-based research studies we tested and refined the prototypes of the instructional group awareness tool. Moreover, we studied the ecological validity of dominant awareness and instructional theories in the context of co-located computer-supported collaborative argumentation. The underlying premise of the Argue(a)ware tool is that a combination of awareness and instructional support will result in increased awareness of collaboration, which will, in turn, mediate the regulation of collaborative processes. Moreover, we assume that successful regulation of collaboration will result in high perceived team effectiveness and the group performance in turn. In the first phase of development of the Argue(a)ware tool, we built support of the content space of collaborative argumentation with argument scaffold elements in a pedagogical face-to-face macro-script and an argument mapping tool. Furthermore, we extended the use of the script for supporting the relational space of collaboration by embedding awareness prompts for reflecting on collaboration during regular breaks in the script. Following, we designed two variations of the same pedagogical face-to-face macro-script which differ with respect to the type of group awareness prompts they used for supporting the relational space of collaboration i.e. behavioral and social. Upon designing the two script variations, we conducted a longitudinal, multiple-case study with ten groups of Media Informatics master students (n = 28, in groups of three or two, group=case, 4 sessions x70 min, Behavioural Awareness Script group= 5, Social Awareness Script group =5.) where each group was conceptualized as a case. Students collaborated every time for arguing to solve one different ill-structured problem and for transferring their arguments in the argument mapping tool Rationale. Thereby, we intended to investigate the effects of different awareness prompts on (a) collaborative metacognitive processes i.e., regulation, reflection, and evaluation (b) the relation between collaborative metacognitive processes and the quality of collaborative argumentation as well as (c) the impact of the two script variations on perceived team effectiveness and (d) what was experience with the different parts of the script variations in the two groups and how this fits into the design framework by Buder (2011). The quantitative analysis of argument outcomes from the groups yield no significant difference between the groups that worked with the BAS and the SAS variations. No significant difference between the script variations with respect to the results from the team effectiveness questionnaires was found either. Prompts for regulating collaboration processes were found to be the most successfully and consistently applied ones, especially in the most successful cases from both script variations and have influenced the argumentation outcomes. The awareness prompts afforded an explicit feedback display format (e.g. assessment of participation levels of self- and others) through discussion (Buder, 2011). The prompted explicit feedback display format (i.e., ratings of one’s self and of others) was criticized for running only on subjective awareness information on participation, contribution efforts and performance in the role. This resulted in evaluation apprehension phenomena (Cottrell, 1972) and evaluation bias (i.e., users may have not assessed themselves or others frankly) (Ghadirian et al., 2016). The awareness prompts for reflection and evaluation did reveal frictions in the plan making process (i.e., dropping out of the plan for collaboration) in the least successful groups. Problems with group dynamics (i.e., free-loading and presence of dominance) but were not powerful enough to trigger the desired changes in the behaviors of the students. The prompts for evaluating the collaboration in both script variations had no apparent connection to argumentation outcomes. The results indicated that dominant presence phenomena inhibited substantive argumentation in the least successful groups. They also indicated that the role-assignment influenced the group dynamics by helping student’s making clear the labor division in the group. In the second phase of development of the Argue(a)ware tool, the focus is on structuring and regulating social interactions in the relational space of collaborative argumentation by means of scripted roles and role-based awareness scaffolds. We designed support for mirroring participation in the role (i.e., a role-based awareness visualization) and support for monitoring participation, coordination and collaboration efforts in the role (i.e., self-assessment questionnaire). Moreover, we designed additional support for guiding participation in the role i.e., role-based reminders as notifications on smartwatches. In a between-subjects study, ten groups of three university students each (n = 30, Mage =22y, mixed educational backgrounds, 1x90min) worked with two variants of the Argue(a)ware for arguing to solve one ill-structured problem and transferring their arguments in the argument mapping tool Rationale. Next, to that, students should monitor their progress in their role with the role-based awareness visualization and the self-assessment questionnaire with the basic awareness support (role-based awareness visualization with the intermediate self-assessment) and the enhanced awareness support (additional role-based awareness reminders). Half of the groups worked only with the role-based awareness visualization and the self-assessment questionnaire (Basic Awareness Condition-BAC) while the other half groups received additional text-based awareness notifications via smartwatches that were sent to students privately (Enhanced Awareness Condition- EAC). Thereby, we tested the use of different degrees of awareness support in the two conditions with respect to their impact on a) self-perceived awareness of performance in the role and of collaboration and coordination efforts (measured with the same questionnaire at two time points), b) on perceive team effectiveness, c) group performance. We hypothesized that students in EAC will perform better thanks to the additional awareness reminders that increased the directivity and influenced their awareness in the role. The mixed methods analysis revealed that the awareness reminders, when perceived on time, succeeded in guiding collaboration (i.e., resulted in more role-specific behaviors). Students in the EAC condition improved their awareness over time (between the two measurements). These results indicated that enhanced awareness support in the form of additional guidance through awareness reminders can boost the awareness of students’ performance in the role as well as the awareness of their coordination and collaboration efforts over time by directing them back to the mirroring and monitoring tools. Moreover, students in EAC exhibited higher perceived team effectiveness than the students in BAC. However, no significant differences in building of shared mental models or performing in mutual performance monitoring were found between the groups. However, students in BAC and EAC did not differ significantly with respect to the formal correctness or evidence sufficiency of their group argumentation outcomes. Moreover, technical difficulties with the smartphones used as delivery devices for the awareness reminders (i.e., low vibration modus) hindered the timely perception of the reminders and thus their effect on participation. Finally, the questionnaire on the experience with the different parts of Argue(a)ware system indicated the need for exploring further media for supporting the awareness reminders to avoid the overwhelming effects of the multiple displays of the system and enhancing higher perceptiveness of the reminders with low interruption costs for other group members. The rather high satisfaction with the use of the role-based awareness visualization and the positive comments on the motivating aspects of monitoring how the personal success contributes to the group performance indicate that the group mirror succeeded in making group norms visible to group members in a non-obtrusive way. The high interpersonal comparability of performances without moderating the group ‘s interaction directly in the basic awareness condition was proven to be the favored design approach compared to the combination of group mirror and awareness reminders in the enhance awareness condition. In the third phase of development of Argue(a)ware, we focused on designing and testing different notification modes on different ubiquitous mobile devices for facilitating the next prototype of a notification system for role-based awareness reminders. Thereby, the aim of the system was again to guide students’ active participation in collaborative argumentation. More specifically, we focused on raising students’ attention to the reminders and triggering a prompter reaction to the contents of the reminders whilst avoiding a high interruption cost for the primary task (i.e., arguing for solving the problem at hand) in the group. These goals were translated into design challenges for the design of the role-based awareness notification system. The system should afford low interruptions, high reaction and high comprehension of notifications. Notification systems with this particular configuration of IRC values are known as "secondary display" systems (McCrickard et al., 2003). Next, we designed three low-fidelity prototypes for a role-based notification system for delivering awareness reminders: The first ran on a smartwatch and afforded text-based information with vibration and light notification modalities. The second ran on smartphone and afforded text-based information with vibrotactile and light-based notification modalities. Finally, the third prototype run on a smart-ring which afforded graphical- based (i.e. abstract light) information with and light and vibration notification modalities. To test the suitability of these prototypes for acting as “secondary display” systems, we conducted a within-subjects user study where three university students (n= 3, Mage=28, mixed educational background) argued for solving three different problem cases and producing an argument map in each of the three consecutive meetings (max 90min) in the Argue(a)ware instructional system. Students were assigned the roles of writer, corrector and devil`s advocate and were instructed to maintain the same role across the three meetings. In each meeting, students worked with a different role-based awareness notification prototype, where they received a notification indicating their balloon is not growing bigger after five minutes of not exhibiting any role-specific behaviors. The role-based awareness notification prototypes aimed at introducing timely interventions which would prompt students to check on their own progress in the role and the group progress as visualized by the role-based awareness visualization on the large display. Ultimately, this should prompt them to reflect on the awareness information from the visualization and adapt their behaviors to the desired behavior standards over time. Results showed that students perceived the notifications from all media mostly based on vibration cues. Thereby, the vibration cues on the wrist (smartwatch) were considered the least disruptive to the main task compared to the vibration cues on finger (smartwatch) and the vibration cues on the desk (smartphone). Students also declared that vibration cues on wrist prompted the fastest reaction i.e., attending to notification by interacting with the smartwatch. These results indicate that vibration cues on the wrist can be a suitable notification mechanism for increasing the perceived urgency of the message and prompting the reaction on it without causing great distraction to the main task, as studies previous studies showed before (Pielot, Church, & deOliveira, 2013; Hernández-Leo, Balestrini, Nieves & Blat, 2012). Based on very limited qualitative data on light as notification modality and awareness representation type no inferences could be made about its influence on the cost of interruption, reaction and comprehension parameters comprehensiveness. The qualitative and quantitative data on the experience with different media as awareness notification systems indicate that smartwatches may be the most suitable medium for acting as awareness notification medium with a “secondary display” IRC configuration (low-high-high). However, this inference needs to be tested in terms of a follow up study. In the next study, the great limitations of study (limited data due to low power and mal-structured measurement instruments) need to be repaired. Finally, the focus should be on comparing notification modalities of one medium (e.g., smartphone) based on a larger set of participants and with the use of objective measurements for the IRC parameter values (Chewar, McCrickard & Sutcliffe, 2004). Finally, we draw conclusions based on the findings from the three studies with respect to the role of awareness mechanisms for facilitating collaborative processes and outcomes and provide replicable and generalizable design principles. These principles are formed as heuristic statements and are subject to refinement by further research (Bell, Hoadley, & Linn, 2004; Van den Akker, 1999). We conclude with the limitations of the study and ideas for future work with Argue(a)ware

The effects of question prompt-based scaffolding and social presence enhancement on students' argumentation and ill-structured problem solving

Asynchronous online discussions have the potential to facilitate meaningful learning activities, such as ill-structured problem-solving, due to their asynchronicity and connectivity. However, the literature identifies two significant challenges in effectively supporting students' ill-structured problem-solving in these environments - argumentation and social interactions. Although argumentation is critical for ill-structured problem-solving, students tend to avoid engaging in critical argumentation activities, such as generating rebuttals and counterarguments, to construct meaning in discussion forums. Similarly, social interactions are essential to ill-structured problem-solving, but students tend to display low levels of engagement in asynchronous online discussions. This study aimed to explore how these two critical components of problem-solving can be supported in asynchronous online discussions using question prompt-based argumentation scaffolds in combination with social presence enhancement strategies. The primary focus was to compare the differential effects of question prompts alone and the combination of question prompts with social presence enhancement prompts on students' argumentation behaviors, problem-solving processes during discussions, and post-test problem-solving performance. This study randomly assigned 44 students to three conditions (control, question prompt-based, and question prompt with social presence enhancement) to analyze their argumentation behaviors, problem-solving processes, and problem-solving performance in asynchronous online discussions. The findings reveal that: (1) lower level interactive argumentation behaviors dominate over higher level interactive ones; (2) students focus more on identifying problem representations and generating solutions than on justifying and evaluating them; (3) there is no significant difference in argumentation behaviors and problem-solving processes in asynchronous online discussions across the conditions. However, students who received both the question prompt scaffolds and the social presence enhancement strategies consistently showed more evidence of engaging in argumentation, especially higher level interactive argumentation, and in problem-solving processes; and (4) there is a significant difference in post-test problem-solving performance across conditions, where students in the condition that received both the question prompt scaffolds and the social presence enhancement strategies performed significantly better than students in the control condition. The study's findings underscore the importance of supporting students' social presence to engage them in interactive argumentation and problem-solving processes, thereby enhancing their problem-solving performance. This study contributes to the understanding of argumentation scaffolding and social interactions in asynchronous online discussion environments and provides design recommendations for instructional designers and instructors seeking to support students' argumentation and social presence in these environments.Includes bibliographical references

Scaffolding online peer collaboration to enhance ill-structured problem solving with computer-based cognitive support.

The results reveal significant effects of procedure and metacognitive question prompts in ill-structured problem solving at both overall and univariate levels. However, there was no significant effect of online peer collaboration and no significant interaction. This study supported some previous research on using question prompts as a scaffolding strategy to support problem solving. Further, these findings support a redefined IDEAL problem solving model for solving ill-structured problems. The findings suggest many implications for instructional designers, educators in web-based learning environments, and educational researchers. These implications and the limitations of this study are discussed.The present study investigated the effects of question prompts and online peer collaborations on solving ill-structured problems. Sixty undergraduate students were randomly assigned to one of the four treatment groups: collaboration with question prompts, individual with question prompts, collaboration without question prompts, and individual without question prompts. Question prompts were designed to both facilitate problem solving procedure and promote students' metacognition. Students worked either individually or collaboratively with partners via MSN Messenger during the problem solving processes