Gamification has risen meteorically in popularity since the beginning of the decade, both in practitioner circles and among researchers. We show that empirical results of gamificationa s effects do not match the hype around it as studies have largely failed to prove any effects. We posit that a proper evaluation of gamification requires an understanding of how gamification can be expressed in real-world applications and employ Wittgensteinian family resemblances as a basis for such a definition. We have collected a set of properties that gamified applications can have through the analysis of goals and means of gamification mentioned in the literature and through an expert survey. We then used those results to create the Gamification Inventory, an instrument for the qualitative assessment of gamification in a given system. We have tested the instrument with a set of evaluators in the field of learning management systems (LMSs), informing both a refinement of the instrument and the preparation of an experiment with the intent of testing the effectiveness of common forms of gamification. The analysis of these LMSs led to results very similar to what our analysis of previous empirical studies in gamification, and especially gamification in education, have shown: most gamification is concentrated on using points, badges, levels and leaderboards as game design elements. We argue that a large-scale, long-term experiment with a proper factorial design is needed to evaluate the effectiveness of gamification and have prepared such a study. Having identified points and badges as two major elements to be tested, we developed an extension to a competency grid add-on for the LMS Moodle that allows for a 2x2 factorial design of using points and badges. The system is designed for large-scale distribution among schools using the competency grid in Moodle, with minimal invasiveness in mind. We briefly discuss the challenges that come with such large-scale experiments, especially in German schools. As a result, we present a new, tested, and refined instrument for the qualitative assessment of gamification in a given system, an overview over gamification as it is being used in the most popular LMSs, and an experimental setup to test the effectiveness of points and badges in schools, using custom add-ons to the competency grid for Moodle and to the corresponding mobile application
