Fast and Robust Rank Aggregation against Model Misspecification

In rank aggregation, preferences from different users are summarized into a total order under the homogeneous data assumption. Thus, model misspecification arises and rank aggregation methods take some noise models into account. However, they all rely on certain noise model assumptions and cannot handle agnostic noises in the real world. In this paper, we propose CoarsenRank, which rectifies the underlying data distribution directly and aligns it to the homogeneous data assumption without involving any noise model. To this end, we define a neighborhood of the data distribution over which Bayesian inference of CoarsenRank is performed, and therefore the resultant posterior enjoys robustness against model misspecification. Further, we derive a tractable closed-form solution for CoarsenRank making it computationally efficient. Experiments on real-world datasets show that CoarsenRank is fast and robust, achieving consistent improvement over baseline methods

Spam elimination and bias correction : ensuring label quality in crowdsourced tasks.

Crowdsourcing is proposed as a powerful mechanism for accomplishing large scale tasks via anonymous workers online. It has been demonstrated as an effective and important approach for collecting labeled data in application domains which require human intelligence, such as image labeling, video annotation, natural language processing, etc. Despite the promises, one big challenge still exists in crowdsourcing systems: the difficulty of controlling the quality of crowds. The workers usually have diverse education levels, personal preferences, and motivations, leading to unknown work performance while completing a crowdsourced task. Among them, some are reliable, and some might provide noisy feedback. It is intrinsic to apply worker filtering approach to crowdsourcing applications, which recognizes and tackles noisy workers, in order to obtain high-quality labels. The presented work in this dissertation provides discussions in this area of research, and proposes efficient probabilistic based worker filtering models to distinguish varied types of poor quality workers. Most of the existing work in literature in the field of worker filtering either only concentrates on binary labeling tasks, or fails to separate the low quality workers whose label errors can be corrected from the other spam workers (with label errors which cannot be corrected). As such, we first propose a Spam Removing and De-biasing Framework (SRDF), to deal with the worker filtering procedure in labeling tasks with numerical label scales. The developed framework can detect spam workers and biased workers separately. The biased workers are defined as those who show tendencies of providing higher (or lower) labels than truths, and their errors are able to be corrected. To tackle the biasing problem, an iterative bias detection approach is introduced to recognize the biased workers. The spam filtering algorithm proposes to eliminate three types of spam workers, including random spammers who provide random labels, uniform spammers who give same labels for most of the items, and sloppy workers who offer low accuracy labels. Integrating the spam filtering and bias detection approaches into aggregating algorithms, which infer truths from labels obtained from crowds, can lead to high quality consensus results. The common characteristic of random spammers and uniform spammers is that they provide useless feedback without making efforts for a labeling task. Thus, it is not necessary to distinguish them separately. In addition, the removal of sloppy workers has great impact on the detection of biased workers, with the SRDF framework. To combat these problems, a different way of worker classification is presented in this dissertation. In particular, the biased workers are classified as a subcategory of sloppy workers. Finally, an ITerative Self Correcting - Truth Discovery (ITSC-TD) framework is then proposed, which can reliably recognize biased workers in ordinal labeling tasks, based on a probabilistic based bias detection model. ITSC-TD estimates true labels through applying an optimization based truth discovery method, which minimizes overall label errors by assigning different weights to workers. The typical tasks posted on popular crowdsourcing platforms, such as MTurk, are simple tasks, which are low in complexity, independent, and require little time to complete. Complex tasks, however, in many cases require the crowd workers to possess specialized skills in task domains. As a result, this type of task is more inclined to have the problem of poor quality of feedback from crowds, compared to simple tasks. As such, we propose a multiple views approach, for the purpose of obtaining high quality consensus labels in complex labeling tasks. In this approach, each view is defined as a labeling critique or rubric, which aims to guide the workers to become aware of the desirable work characteristics or goals. Combining the view labels results in the overall estimated labels for each item. The multiple views approach is developed under the hypothesis that workers\u27 performance might differ from one view to another. Varied weights are then assigned to different views for each worker. Additionally, the ITSC-TD framework is integrated into the multiple views model to achieve high quality estimated truths for each view. Next, we propose a Semi-supervised Worker Filtering (SWF) model to eliminate spam workers, who assign random labels for each item. The SWF approach conducts worker filtering with a limited set of gold truths available as priori. Each worker is associated with a spammer score, which is estimated via the developed semi-supervised model, and low quality workers are efficiently detected by comparing the spammer score with a predefined threshold value. The efficiency of all the developed frameworks and models are demonstrated on simulated and real-world data sets. By comparing the proposed frameworks to a set of state-of-art methodologies, such as expectation maximization based aggregating algorithm, GLAD and optimization based truth discovery approach, in the domain of crowdsourcing, up to 28.0% improvement can be obtained for the accuracy of true label estimation

Scaling Up Student Assessment: Issues and Solutions

Online courses permit the enrollment of large numbers of students, which forces instructors to address the problem of providing valid and reliable assessments of student performance on a large scale. This paper examines two broad approaches for scaling up student assessment and feedback in higher education: automated assessment techniques and distributed assessment methods

Multi-Target Prediction: A Unifying View on Problems and Methods

Multi-target prediction (MTP) is concerned with the simultaneous prediction of multiple target variables of diverse type. Due to its enormous application potential, it has developed into an active and rapidly expanding research field that combines several subfields of machine learning, including multivariate regression, multi-label classification, multi-task learning, dyadic prediction, zero-shot learning, network inference, and matrix completion. In this paper, we present a unifying view on MTP problems and methods. First, we formally discuss commonalities and differences between existing MTP problems. To this end, we introduce a general framework that covers the above subfields as special cases. As a second contribution, we provide a structured overview of MTP methods. This is accomplished by identifying a number of key properties, which distinguish such methods and determine their suitability for different types of problems. Finally, we also discuss a few challenges for future research

Innovation in Pedagogy and Technology Symposium: University of Nebraska, May 8, 2018

Selected Conference Proceedings, Presented by University of Nebraska Online and University of Nebraska Information Technology Services. University of Nebraska Information Technology Services (NU ITS) and University of Nebraska Online (NU Online) present an education and technology symposium each spring. The Innovation in Pedagogy and Technology Symposium provides University of Nebraska (NU) faculty and staff the opportunity to learn from nationally recognized experts, share their experiences and learn from the initiatives of colleagues from across the system. This event is offered free to NU administrators, faculty and staff free of charge. Tuesday, May 8, 2018 The Cornhusker Marriott, Lincoln, NE Technology has forever changed the landscape of higher education and continues to do so—often at a rapid pace. At the University of Nebraska, we strive to embrace technology to enhance both teaching and learning, to provide key support systems and meet institutional goals. The Innovation in Pedagogy and Technology Symposium is designed for any NU administrator, faculty or staff member who is involved in the use of technology in education at all levels. Past events have drawn over 500 NU faculty, staff and IT professionals from across the four campuses for a day of discovery and networking. The 2018 event was held in downtown Lincoln. The schedule included: • Presentations by University of Nebraska faculty, staff and administrators • Concurrent sessions focused on pedagogy/instructional design, support and administrative strategies and emerging technologies • Panel discussions • Roundtable discussions and networking time • Sponsor exhibits • Continental breakfast and lunch Keynote Presentation: Learning How to Learn: Powerful Mental Tools to Help You Master Tough Subjects • Barbara Oakley, Ph.D., Oakland University Fostering Quality by Identifying & Evaluating Effective Practices through Rigorous Research • Tanya Joosten, University of Wisconsin-Milwaukee Synchronous Online & In Person Classrooms: Challenges & Rewards Five Years Into Practice • Elsbeth Magilton We Nudge and You Can Too: Improving Outcomes with an Emailed Nudge • Ben Smith It Takes a System to Build an Affordable Content Program • Brad Severa, Jane Petersen, Kimberly Carlson, Betty Jacques, Brian Moore, Andrew Cano, Michael Jolley Five Generations: Preparing Multiple Generations of Learners for a Multi-Generational Workforce • Olimpia Leite-Trambly, Sharon Obasi., Toni Hill Schedule NU! Schedule SC! • Cheri Polenske, Jean Padrnos, Corrie Svehla See It & Believe It (Assessing Professional Behaviors & Clinical Reasoning with Video Assignments) • Grace Johnson, Megan Frazee Group Portfolios as a Gateway to Creativity, Collaboration & Synergy in an Environment Course • Katherine Nashleanas Learning to Learn Online: Helping Online Students Navigate Online Learning • Suzanne Withem Beyond Closed Captioning: The Other ADA Accessibility Requirements • Analisa McMillan, Peggy Moore (UNMC) Using Interactive Digital Wall (iWall) Technology to Promote Active Learning • Cheryl Thompson, Suhasini Kotcherlakota, Patrick Rejda, Paul Dye Cybersecurity Threats & Challenges • JR Noble Digital Badges: A Focus on Skill Acquisition • Benjamin Malczyk Creating a Student Success Center Transitioning Graduate Students to an Online Community • Brian Wilson, Christina Yao, Erica DeFrain, Andrew Cano Male Allies: Supporting an Inclusive Environment in ITS • Heath Tuttle (, Wes Juranek Featured Extended Presentation: Broaden Your Passion! Encouraging Women in STEM • Barbara Oakley, Oakland University in Rochester, Michigan Students as Creative Forces to Enhance Curriculum via E-Learning • Betsy Becker, Peggy Moore, Dele Davies Rethinking Visual Communication Curriculum: The Success of Emporium Style • Adam Wagler (UNL), Katie Krcmarik, Alan Eno A Course Delivery Evolution: Moving from Lecture to Online to a Flipped Classroom • Kim Michael, Tanya Custer Enhancing the Quality of Online Teaching via Collaborative Course Development • B. Jean Mandernach, Steve McGahan Collaborating Across NU for Accessible Video • Heath Tuttle, Jane Petersen, Jaci Lindburg Structuring Security for Success • Matt Morton, Rick Haugerud Future Directions for University of Nebraska Wireless Networking • Brian Cox, Jay Wilmes Using Learning Analytics in Canvas to Improve Online Learning • Martonia Gaskill,, Phu Vu, Broaden Your Passion! Encouraging Women in STEM • Featured Speaker: Barbara Oakley, Oakland University in Rochester, MI Translating Studio Courses Online • Claire Amy Schultz Hidden Treasures: Lesser Known Secrets of Canvas • Julie Gregg, Melissa Diers, Analisa McMillan Your Learners, Their Devices & You: Incorporating BYOD Technology into Your Didactics • Tedd Welniak Extending the Conversation about Teaching with Technology • Marlina Davidson, Timi Barone, Dana Richter-Egger, Schuetzler, Jaci Lindburg Scaling up Student Assessment: Issues and Solutions • Paul van Vliet Closing Keynote: Navigating Change: It’s a Whitewater Adventure • Marjorie J. Kostelnik, Professor and Senior Associate to the President doi 10.13014/K2Q23XFDhttps://digitalcommons.unl.edu/zeabook/1068/thumbnail.jp