Capturing Ambiguity in Crowdsourcing Frame Disambiguation

FrameNet is a computational linguistics resource composed of semantic frames, high-level concepts that represent the meanings of words. In this paper, we present an approach to gather frame disambiguation annotations in sentences using a crowdsourcing approach with multiple workers per sentence to capture inter-annotator disagreement. We perform an experiment over a set of 433 sentences annotated with frames from the FrameNet corpus, and show that the aggregated crowd annotations achieve an F1 score greater than 0.67 as compared to expert linguists. We highlight cases where the crowd annotation was correct even though the expert is in disagreement, arguing for the need to have multiple annotators per sentence. Most importantly, we examine cases in which crowd workers could not agree, and demonstrate that these cases exhibit ambiguity, either in the sentence, frame, or the task itself, and argue that collapsing such cases to a single, discrete truth value (i.e. correct or incorrect) is inappropriate, creating arbitrary targets for machine learning.Comment: in publication at the sixth AAAI Conference on Human Computation and Crowdsourcing (HCOMP) 201

Peer Harassment: A Weapon in the Struggle for Popularity and Normative Hegemony in American Secondary Schools

This paper addresses two of secondary education’s most serious problems—peer abuse of weaker socially unskilled students and a peer culture that in most schools discourages many students from trying to be all that they can be academically. We have documented the two problems by reviewing ethnographies of secondary schools, by interviewing students in eight suburban high schools and by analyzing data from questionnaires completed by nearly 100,000 students at Educational Excellence Alliance schools. Grounded in these observations, we built a simple mathematical model of peer harassment and popularity and of the pressures for conformity that are created by the struggle for popularity and then tested it in data from the Educational Excellence Alliance. Students entering middle school learn its norms by trying to copy the traits and behaviors of students who are respected and by avoiding contact with those who are frequently harassed. Peer norms are enforced by encouraging ‘wannabes,’ aspirants for admission to popular crowds, to harass those who visibly violate them. Consequently, one can infer the norms by noting who gets harassed and who doesn’t. Traits that in EEA data led to higher risks of being bullied and harassed were: being in a special education, being in gifted programs, taking accelerated courses in middle school, tutoring other students, enjoying school assignments, taking a theatre course, not liking rap-hip hop music and liking instead musicals, heavy metal, country, or classical music. The relationship between harassment and academic effort was curvilinear; both the nerds and the slackers were harassed. To some degree these norms are, as Kenneth Arrow suggests, trying to internalize externalities. But why are music preferences such good predictors of harassment? Why are the student tutors victimized? We propose that norms also have a “We’re cool, Honor us” function of legitimating the high status that the leading crowds claim for themselves. As a result the traits and interests that members of leading crowds have in common tend to become normative for everyone. The norms that prevailed were: “Spend your time socializing, do not “study too hard.” Value classmates for their athletic prowess and their attractiveness, not their interest in history or their accomplishments in science.

The Size Conundrum: Why Online Knowledge Markets Can Fail at Scale

In this paper, we interpret the community question answering websites on the StackExchange platform as knowledge markets, and analyze how and why these markets can fail at scale. A knowledge market framing allows site operators to reason about market failures, and to design policies to prevent them. Our goal is to provide insights on large-scale knowledge market failures through an interpretable model. We explore a set of interpretable economic production models on a large empirical dataset to analyze the dynamics of content generation in knowledge markets. Amongst these, the Cobb-Douglas model best explains empirical data and provides an intuitive explanation for content generation through concepts of elasticity and diminishing returns. Content generation depends on user participation and also on how specific types of content (e.g. answers) depends on other types (e.g. questions). We show that these factors of content generation have constant elasticity---a percentage increase in any of the inputs leads to a constant percentage increase in the output. Furthermore, markets exhibit diminishing returns---the marginal output decreases as the input is incrementally increased. Knowledge markets also vary on their returns to scale---the increase in output resulting from a proportionate increase in all inputs. Importantly, many knowledge markets exhibit diseconomies of scale---measures of market health (e.g., the percentage of questions with an accepted answer) decrease as a function of number of participants. The implications of our work are two-fold: site operators ought to design incentives as a function of system size (number of participants); the market lens should shed insight into complex dependencies amongst different content types and participant actions in general social networks.Comment: The 27th International Conference on World Wide Web (WWW), 201

Challenges in Complex Systems Science

FuturICT foundations are social science, complex systems science, and ICT. The main concerns and challenges in the science of complex systems in the context of FuturICT are laid out in this paper with special emphasis on the Complex Systems route to Social Sciences. This include complex systems having: many heterogeneous interacting parts; multiple scales; complicated transition laws; unexpected or unpredicted emergence; sensitive dependence on initial conditions; path-dependent dynamics; networked hierarchical connectivities; interaction of autonomous agents; self-organisation; non-equilibrium dynamics; combinatorial explosion; adaptivity to changing environments; co-evolving subsystems; ill-defined boundaries; and multilevel dynamics. In this context, science is seen as the process of abstracting the dynamics of systems from data. This presents many challenges including: data gathering by large-scale experiment, participatory sensing and social computation, managing huge distributed dynamic and heterogeneous databases; moving from data to dynamical models, going beyond correlations to cause-effect relationships, understanding the relationship between simple and comprehensive models with appropriate choices of variables, ensemble modeling and data assimilation, modeling systems of systems of systems with many levels between micro and macro; and formulating new approaches to prediction, forecasting, and risk, especially in systems that can reflect on and change their behaviour in response to predictions, and systems whose apparently predictable behaviour is disrupted by apparently unpredictable rare or extreme events. These challenges are part of the FuturICT agenda

An Economic Theory of Academic Engagement Norms: The Struggle for Popularity and Normative Hegemony in Secondary Schools

[Excerpt] Why and how do groups create norms? Kenneth Arrow proposed that “norms of social behavior, including ethical and moral codes, ….are reactions of society to compensate for market failure”. This internalize the real externalities explanation for norms is also standard among rational choice theorists in sociology. The situation becomes more complex when we recognize some actions create positive externalities for some individuals and negative externalities for others. Often this results in no norm being established. However, sometimes one segment of a social system has normative hegemony and enforces norms that enhance their power and prestige at the expense of other groups. Norms regarding caste in India, for example, were functional for Brahmins but humiliating for Harijans. Caste and status norms of this type will also be referred to as “Honor us; Not them” norms. Such norms arise when one group is much more powerful (has greater ability to enforce their preferred social norm) than other groups and it imposes its will on others. An additional requirement is that the people who oppose the norm established by the dominant group must be unable or unwilling to leave the social system in which the norm operates