Evaluating Design Solutions Using Crowds

Crowds can be used to generate and evaluate design solutions. To increase a crowdsourcing system’s effectiveness, we propose and compare two evaluation methods, one using five-point Likert scale rating and the other prediction voting. Our results indicate that although the two evaluation methods correlate, they have different goals: whereas prediction voting focuses evaluators on identifying the very best solutions, the rating focuses evaluators on the entire range of solutions. Thus, prediction voting is appropriate when there are many poor quality solutions that need to be filtered out, and rating is suited when all ideas are reasonable and distinctions need to be made across all solutions. The crowd prefers participating in prediction voting. The results have pragmatic implications, suggesting that evaluation methods should be assigned in relation to the distribution of quality present at each stage of crowdsourcing

A Prototype to Support Business Model Innovation through Crowdsourcing and Artificial Intelligence

The development of new and innovative business models is a central challenge for many companies, particularly for small and medium-sized companies. Information systems could support these companies by actively guiding them through a business model development process. However, the existing business model development tools only provide passive support for their users (e.g., digital whiteboards). Therefore, we set out to develop a prototype that actively supports its users by generating business model ideas. Informed by an existing design theory, we built a prototype relying on hybrid intelligence (i.e., the combination of human and artificial knowledge). The prototype iteratively generates new business model ideas by recombining existing knowledge, posts the ideas to a crowdsourcing platform for evaluation, and learns from the crowds’ evaluation. This demonstration paper presents the prototype, the challenges we faced during its implementation, and directions for future research on machine-supported business model development

PEER RATINGS AND ASSESSMENT QUALITY IN CROWD-BASED INNOVATION PROCESSES

Social networks – whether public or in enterprises – regularly ask users to rate their peers’ content using different voting techniques. When employed in innovation challenges, these rating procedures are part of an open, interactive, and continuous engagement among customers, employees, or citizens. In this regard, assessment accuracy (i.e., correctly identifying good and bad ideas) in crowdsourced eval-uation processes may be influenced by the display of peer ratings. While it could sometimes be useful for users to follow their peers, it is not entirely clear under which circumstances this actually holds true. Thus, in this research-in-progress article, we propose a study design to systematically investigate the effect of peer ratings on assessment accuracy in crowdsourced idea evaluation processes. Based on the elaboration likelihood model and social psychology, we develop a research model that incorporates the mediating factors extraversion, locus of control, as well as peer rating quality (i.e., the ratings’ corre-lation with the evaluated content’s actual quality). We suggest that the availability of peer ratings de-creases assessment accuracy and that rating quality, extraversion, as well as an internal locus of control mitigate this effect

When Life Gives You Lemons: How rating scales affect user activity and frustration in collaborative evaluation processes

Initiators of open innovation processes involving customers or employees often face vast amounts of idea proposals. These proposals vary greatly in terms of quality, which is why organizers often engage the users themselves in the evaluation process. Building on the concept of information overload, we evaluate the effects of three distinct rating scales on users’ activity and frustration measures. On the basis of an open innovation campaign for employees of a public-private institution in Germany, we systematically compare the novel “bag of lemons” method with conventional Likert scales and up-down-voting schemes. Our results demonstrate that the “bag of lemons”-approach yields higher levels of user activity, but is also perceived as significantly more frustrating. We find this effect to be fully mediated by perceived information overload, which points to potential avenues for the design of stimulating yet tolerably complex Information Systems for open innovation and rating techniques

Ideate. Collaborate. Repeat. A Research Agenda for Idea Generation, Collaboration and Evaluation in Open Innovation

Open innovation has been and remains to be a rapidly changing field of research in Information Systems and various other disciplines. With the rise of professional open innovation platforms and the emergence of crowdsourcing as well as employee-driven innovation, studies on the front-end of open innovation – namely idea generation, collaboration and evaluation – are facing new challenges. In this structured literature review, we analyze a large body of prior research in order to derive a framework, which is able to classify and reflect the lively debate on open innovation. In addition, we identify important implications for practitioners with advise on the design of open innovation systems. Moreover, our study identifies several promising areas for future research

The analysis and presentation of patents to support engineering design

This paper explores the role of patents in engineering design, and how the extraction and presentation of patent data could be improved for designers. We propose the use of crowdsourcing as a means to post tasks online for a crowd of people to participate and complete. The is-sues of assessment, searching, clustering and knowledge transfer are evaluated with respect to the literature. Opportunities for potential crowd intervention are then discussed, before the presentation of two initial studies. These related to the categorization and interpretation of patents respectively using an online platform. The initial results establish basic crowd capabilities in understanding patent text and interpreting patent drawings. This has shown that reasonable results can be achieved if tasks of appropriate duration and complexity are set, and if test questions are incorporated to ensure a basic level of understanding exists in the workers

Generating creative ideas through crowds: An experimental study of combination

The crowd is emerging as a new source of innovation, and here a new way of organizing the crowd to produce new ideas is discussed: an idea generation system using combination in which participants synthesize new designs from the efforts of their peers. A crowd generates designs; then another crowd combines the designs of the previous crowd. In an experiment with 540 participants, the combined designs are compared to the initial designs, and to a control condition in which fresh idea generation rather than combination is used. The results show that designs become more creative in later generations of the combination system, and the combination produces more creative ideas than the fresh idea generation. The model of crowdsourced idea generation discussed here may be used to instantiate systems that can be applied to a wide range of design problems. The work has pragmatic implications, and also theoretical implications: new forms of coordination are now possible, and, using the crowd, it is possible to build and test existing and emerging theories of coordination and design

The Impact of Crowdsourcing on Organisational Practices:The Case of Crowdmapping

In this paper, we investigate the possible impact of crowdsourcing on organisational practices. We answer the research question of whether and to what extent the practices of crowdmapping impact humanitarian organisations. To answer this question, we examine a crowdmapping initiative during a natural disaster. The data collection is based on forty interviews with different actors including crowdmappers, humanitarian organisations, government specialists and technology providers. Concepts from structuration theory are applied to conceptualise and make sense of the data. The findings reveal the process of change that took place in the practices of a humanitarian organisation. They also show that these changes recursively impacted the practices of crowdmapping. We then argue that there is a duality of change between the micro-practices of crowdmapping and the macro-practices of a humanitarian organisation. The implications of the study on research and practice are then discussed

Medical Crowdsourcing: Harnessing the “Wisdom of the Crowd” to Solve Medical Mysteries

Medical crowdsourcing offers hope to patients who suffer from complex health conditions that are difficult to diagnose. Such crowdsourcing platforms empower patients to harness the “wisdom of the crowd” by providing access to a vast pool of diverse medical knowledge. Greater participation in crowdsourcing increases the likelihood of encountering a correct solution. However, more participation also leads to increased “noise,” which makes identifying the most likely solution from a broader pool of recommendations (i.e., diagnostic suggestions) difficult. The challenge for medical crowdsourcing platforms is to increase participation of both patients and solution providers, while simultaneously increasing the efficacy and accuracy of solutions. The primary objectives of this study are: (1) to investigate means to enhance the solution pool by increasing participation of solution providers referred to as “medical detectives” or “detectives,” and (2) to explore ways of selecting the most likely diagnosis from a set of alternative possibilities recommended by medical detectives. Our results suggest that our strategy of using multiple methods for evaluating recommendations by detectives leads to better predictions. Furthermore, cases with higher perceived quality and more negative emotional tones (e.g., sadness, fear, and anger) attract more detectives. Our findings have strong implications for research and practice