The Pilot Proficiency Audit of Knowledge, Skills, and Abilities: Adding to the Air Carrier Safety Toolbox

The correlation between air carrier pilot performance and age, total flight time, total airline flight time, and time in current position has not been fully documented. This study observed 62 F/Os in line operations and graded 74 knowledge, skills, and abilities performance variables, utilizing a five-point Likert scale. Knowledge items scored slightly below the referent, with no improvement over time; skills improved with both flight time and years of service; and abilities decline markedly across all independent variables. Changes to pilot training syllabi and techniques, as well as hiring practices, may be indicated. Integrating Pilot Proficiency Audit data into existing LOSA, AQP, ASAP, and FOQA programs can provide a more robust air carrier safety program

The Effect Of Instrument-Specific Rater Training On Interrater Reliability And Counseling Skills Performance Differentiation

The purpose of this study was to explore the effect of instrument-specific rater training on interrater reliability (IRR) and counseling skills performance differentiation. Strong IRR is of primary concern to effective program evaluation (McCullough, Kuhn, Andrews, Valen, Hatch, & Osimo, 2003; Schanche, Nielsen, McCullough, Valen, & Mykletun, 2010) and counselor education (Baker, Daniels, & Greeley, 1990; Jennings, Goh, Skovholt, & Banerje-Steevens, 2003; Lepkowski, Packman, Smaby, & Maddux, 2009). The ability to differentiate between low and high performances of counseling skills is central to informing the classroom instruction of counseling students and the supervision of early clinical experiences (Byrne & Hartley, 2010; Fitch, Gillam, & Baltimore, 2004; Paladino, Barrio-Minton, & Kern, 2011). Participants were randomly assigned to one of four groups defined by whether they received instrument-specific training and the performance level of the counseling skills they assessed. Data was collected using the Universal Counseling Skills Assessment (UCSA) administered traditionally and through the Dynamic Scoring Interface (DSI). The researcher used a 2 X 2 factorial ANOVA, independent samples t-tests, intraclass correlation coefficients, and Fisher’s r to z transformations to analyze the data’s validity across the groups and reliability within the groups. Results that brief instrument-specific training and a structure scoring procedure can significantly strengthen IRR. The results of the analyses are discussed within the context of their implications for counselor education and future research possibilities

Clarity Precedes Competence: The Impact of a Revised Classroom Teacher Evaluation Instrument on Teaching and Learning

This study is a program evaluation to determine the impact of evaluators’ use of the revised Classroom Teacher Evaluation Instrument on instructional practice in a mid-sized, public school district serving approximately 43,000 Pre-Kindergarten through twelfth-grade students. The primary research question explored in this program evaluation is what theory of action for the evaluation system will establish reliable effectiveness measures to gauge teacher quality in the school district under study. Employing a mixed-methods approach, with data gathered from a principal survey, focus group transcriptions, and instructional practice and student achievement data, this study informed and improved current practices surrounding observation and evaluation processes. Policy implications and recommendations support a multidimensional view of teacher effectiveness through the inclusion of multiple measures of data, including deliberate practice and self-assessment, classroom observations, student voice, student achievement and growth, and school performance growth

Perceived fairness of and satisfaction with employee performance appraisal

Employee performance appraisal is one of the most commonly used management tools in the United States. Over 90 percent of large organizations including 75 percent of state employment systems require some type of annual performance appraisal (Seldon, Ingraham & Jacobson, 2001). Performance appraisal is one of the most widely researched areas in industrial/organizational psychology (Murphy & Cleveland, 1993). However, the traditional research agenda has done little to improve the usefulness of performance appraisal as a managerial tool. Recent research has moved away from studies of rater accuracy and psychometric measures to themes of employee reactions towards performance appraisal as indicators of system satisfaction and efficacy. Employee perception of fairness of performance appraisal has been studied as a significant factor in employee acceptance and satisfaction of performance appraisal. This study investigated employee reactions to fairness of and satisfaction with an existing performance appraisal system utilizing a hypothesized four-factor model (Greenberg, 1993) of organizational justice as the theoretical basis. The underlying hypothesis was that the conceptualized four-factor model, which differentiated between the constructs of interactional and procedural justice, would best represent the underlying factor structure of the data. Data were obtained via a survey questionnaire from 440 participants from two organizations that were part of a large public employment system. Ten multi-item scales representing four factors of organizational justice and performance appraisal fairness and three scales indicating satisfaction were included. The findings of the study indicated that respondents perceived the performance appraisal system was to be fair as indicated by their agreement with 9 of the 10 scales used to measure reactions to fairness. The respondents also indicated their relative satisfaction with their most recent performance appraisal rating and with their supervisor. Less satisfaction (although not dissatisfaction) was indicated with the performance appraisal system overall. The conceptualized four-factor model was not found to represent the underlying factor structure substantially better than alternative plausible three-factor models. The best fit three–factor model, however, provided some support for the differentiation between procedural and interactional organizational justice factors, which is a distinction that has been debated in the organizational justice literature

Elementary Principals\u27 Perceptions of Special Education Teachers\u27 Performance Evaluation Process

Evidence suggests that current special education teacher evaluation systems may not accurately reflect these teachers\u27 unique duties and responsibilities. In a Midwestern, urban school district, the teacher evaluation system was not adequately aligned with the performance expectations of special education teachers. Guided by Danielson\u27s framework for teaching, this qualitative case study explored elementary school principals\u27 perceptions of teacher evaluation approaches, perceived effectiveness of these approaches, applications of key indicators of teaching quality, and barriers preventing accurate evaluations of special education teachers\u27 performance. An online survey comprised of closed- and open-ended questions was distributed to 445 principals in the district with 97 responding. Descriptive analysis of closed-ended items indicated 70% of respondents perceived current evaluation methods to be insufficiently differentiated for special education teachers\u27 roles and 90% reported a need for additional measures of effectiveness to be used. Thematic analysis of open-ended survey responses confirmed the need for differentiated evaluation approaches to address a misalignment of key effectiveness indicators for special education teachers and revealed barriers to accurate evaluation including resource constraints. In response to these findings, a position paper with policy recommendation prescribed the revision of the current teacher evaluation practices to address the unique roles of special education teachers. Refining special education teacher evaluation practices may contribute to positive social change by aligning the evaluation process with special education teachers\u27 duties and responsibilities, thereby improving teacher performance and potentially increasing student achievement over time

How to Create an Innovation Accelerator

Too many policy failures are fundamentally failures of knowledge. This has become particularly apparent during the recent financial and economic crisis, which is questioning the validity of mainstream scholarly paradigms. We propose to pursue a multi-disciplinary approach and to establish new institutional settings which remove or reduce obstacles impeding efficient knowledge creation. We provided suggestions on (i) how to modernize and improve the academic publication system, and (ii) how to support scientific coordination, communication, and co-creation in large-scale multi-disciplinary projects. Both constitute important elements of what we envision to be a novel ICT infrastructure called "Innovation Accelerator" or "Knowledge Accelerator".Comment: 32 pages, Visioneer White Paper, see http://www.visioneer.ethz.c

Functional performance tests and return-to-sport decision-making::Focusing on translational research with special interest in fatigue and the brain.

The overall purpose of this dissertation was to contribute to clinical decision-making and functional performance testing across the sport injury spectrum. The three specific objectives encompassed: (1) establishing return to sport criteria following lateral ankle sprains, (2) mapping the reliability characteristics of the reactive balance test, (3) exploring electrophysiological brain changes induced by various types of fatigue when participants performed the Y-balance test and reactive balance test. The systematic review (Chapter 2) showed that currently no scientifically sound return to sport criteria following lateral ankle sprain injury can be determined. This was because not one original research study was performed on this topic. Therefore, we provided an overview of the relevant retrieved questionnaires, clinical assessment measures, functional and sport-specific performance tests within ankle sprain populations. Based upon this empirical research, return to sport variables were proposed. This chapter also encompasses rationales and considerations for return to sport decision-making following lateral ankle sprain injury. In Chapter 3, the reliability study is the first study to assess test-retest, intra- and inter-rater reliability of the reactive balance test within a recreationally trained population. Excellent intra- and inter-rater reliability for both outcomes (i.e. visuomotor response time and accuracy) were found. However, test-retest reliability showed good reliability for visuomotor response time and moderate reliability for accuracy. These results indicate that the reactive balance test is suited for performing analyses over time at the group level. The third study (Chapter 4) was designed to evaluate the impact of mental fatigue on electrophysiological brain measurements during Y-balance test and reactive balance test performance. Even though mental fatigue was successfully induced, it did not affect Y-balance test performance. However, an increase in prefrontal cortex theta activity was observed when performing the Y-balance test in a mentally fatigued state. Which means mental fatigue was successfully induced, and might suggest a lower availability of attentional resources and poorer decision-making. Regarding reactive balance test performance, only accuracy was compromised due to mental fatigue. No changes in visuomotor reaction time and electrophysiological brain outcome measures were found following the mental fatigue intervention. In Chapter 5, the results showed that acute physical fatigue impairs the accuracy of the reactive balance test, while Y-balance test performance and visuomotor response time of the reactive balance test remained unaffected. Nevertheless, the underlying changes at the peripheral and central physiological level completely differed from the previous study. During the execution of both tests, acute physical fatigue induced alterations in electrophysiological brain outcome measures in line with previous Wingate literature. Higher α power in the prefrontal cortex, motor cortex and posterior parietal cortex as well as higher β power in the prefrontal and posterior parietal cortex were observed during the execution of the Y-balance test in a fatigued state. Following acute physical fatigue, α and β power increments were found in the posterior parietal cortex and the prefrontal cortex during the execution of the reactive balance test, respectively. In summary, the research findings of this dissertation show that: (1) no scientifically sound return to sport criteria following lateral ankle sprain injury are currently available; (2) the reactive balance test has acceptable reliability characteristics and therefore is suited for performing analyses over time at the group level. (3) mental and acute physical fatigue do not impact YBT performance, even when underlying physiological and psychological changes are present; (4) the reactive balance test is capable of detecting fatigue changes at the group level culminating in decreased accuracy. When mentally fatigued, no changes at the electrophysiological brain level were measured. In a physically fatigued state the electrophysiological findings were partially in line with previous Wingate literature

Developing an Effective Instrument for Assessing the Performance of Public University Presidents

Conducting a worthwhile assessment of the performance of university presidents poses unique challenges for higher education institutions. One of the most difficult issues is determining the content and format of the assessment instrument. Due to the breadth and complexity of the job, the list of potential criteria that could serve as content for the assessment instrument is almost limitless. There are also many format options with respect to the arrangement of the assessment instrument and plan for administering it. In order to fill an apparent gap in research on university president assessment, this study proposes a model for determining the content and format of an assessment instrument — the Assessment Instrument Development Approach (AIDA). The AIDA model incorporates a mixed-methods research design aimed at identifying the content and format of an assessment instrument that will produce reliable and valid results. Addressing the question, What approach can UNM and other public universities use to develop an effective performance assessment instrument for their president?\u27 this study critiqued the initial AIDA model and suggests a revised model based on study findings. Addressing the question, \u27What is the preferred content and format for a president performance assessment instrument?\u27 this study proposed an initial framework for the relationships among variables derived from pertinent theory and concepts pertaining to university president assessment. Considering the study results, a revised framework was developed along with example hypotheses that should be tested in future research to gain additional insights into performance assessment for university presidents. The results of this study reveal there are many candidate criteria and formats for assessing performance. What appears to be lacking in literature and in practice is a means to identify the best criteria and formats that will produce reliable, valid, and useful results for assessment of university presidents. The methods and findings described in this study provide additional insight into the \u27means\u27 for developing an assessment instrument and the \u27ends\u27 which are fair, equitable, and productive assessments of university president performance

Teacher Evaluation in Practice: Implementing Chicago's REACH Students

Historically, teacher evaluation in Chicago has fallen short on two crucial fronts: It has not provided administrators with measures that differentiated among strong and weak teachers -- in fact, 93 percent of teachers were rated as Excellent or Superior -- and it has not provided teachers with useful feedback they could use to improve their instruction. Chicago is not unique -- teacher evaluation systems across the country have experienced the exact same problems.Recent national policy has emphasized overhauling these systems to include multiple measures of teacher performance, such as student outcomes, and structuring the evaluations so they are useful from both talent management and teacher professional development perspectives. Principals and teachers need an evaluation system that provides teachers with specific, practice-oriented feedback they can use to improve their instruction and school leaders need to be able to identify strong and weak teachers. Required to act by a new state law and building off lessons learned from an earlier pilot of an evidence-based observation tool, Chicago Public Schools (CPS) rolled out its new teacher evaluation system -- Recognizing Educators Advancing Chicago's Students (REACH Students) -- in the 2012-13 school year. The REACH system seeks to provide a measure of individual teacher effectiveness that can simultaneously support instructional improvement. It incorporates teacher performance ratings based on multiple classroom observations together with student growth measured on two different types of assessments. While the practice of using classroom observations as an evaluation tool is not completely new, REACH requires teachers and administrators to conceptualize classroom observations more broadly as being part of instructional improvement efforts as well as evaluation; evaluating teachers based on student test score growth has never happened before in the district. REACH implementation was a massive undertaking. It required a large-scale investment of time and energy from teachers, administrators, CPS central office staff, and the teachers union. District context played an important role and provided additional challenges as the district was introducing other major initiatives at the same time as REACH. Furthermore, the school year began with the first teacher strike in CPS in over 25 years. Teacher evaluation was one of several contentious points in the protracted negotiation, and the specific issue of using student growth on assessments to evaluate teachers received considerable coverage in the media. This report focuses on the perceptions and experiences of teachers and administrators during the first year of REACH implementation, which was in many ways a particularly demanding year. These experiences can be helpful to CPS and to other districts across the country as they work to restructure and transform teacher evaluation