Measuring Researcher-Production in Information Systems

While many studies have assessed IS researcher-production, most have focused on either ranking IS journals or assessing prolific researchers using a restricted time frame and a small inbasketld of journals (i.e., those journals selected for sampling). We found no research that has assessed the IS specificity of journals (i.e., the suitability of journals for publishing IS research) nor any that evaluated IS researcher-production measures. Based on a coding of over 26,000 articles and more than 1,900 authors, this study attempts such an evaluation by (1) determining the rate of publication of IS researchers in 58 journals perceived by at least one IS institution as IS specific, (2) profiling prolific and typical IS researchers using descriptive statistics, (3) evaluating the convergent validity of various researcher-production measures, (4) assessing the reliability of these researcher-production measures by varying baskets of Measuring Researcher-Production in Information Systems by C. Chua, L. Cao, K. Cousins, and D. W. Straub journals and time periods, and (5) comparing the sensitivity of measures across prolific and typical researchers. The study demonstrates that many journals perceived to be of high quality by IS researchers are not specifically targeted to information systems. Changing the evaluation procedure has a significant impact on measures of typical and prolific IS researchers. For typical IS researchers, measures of production are strongly convergent and are not sensitive to changes in journal baskets. However, for prolific researchers, measures of production are not convergent and highly sensitive to changes in journal baskets. The evaluation of both prolific and typical IS researchers is also highly sensitive to temporal effects. The differences in convergent validity and reliability demonstrate that prolific researchers are more sensitive to minor variations in the assessment procedure. Based on the empirical findings, the study closes with recommendations both for the evaluation of researcher-production and for developing institutional target journal lists, i.e., lists of journals viewed favorably by an institution

A Survey of Scholarly Data: From Big Data Perspective

Recently, there has been a shifting focus of organizations and governments towards digitization of academic and technical documents, adding a new facet to the concept of digital libraries. The volume, variety and velocity of this generated data, satisfies the big data definition, as a result of which, this scholarly reserve is popularly referred to as big scholarly data. In order to facilitate data analytics for big scholarly data, architectures and services for the same need to be developed. The evolving nature of research problems has made them essentially interdisciplinary. As a result, there is a growing demand for scholarly applications like collaborator discovery, expert finding and research recommendation systems, in addition to several others. This research paper investigates the current trends and identifies the existing challenges in development of a big scholarly data platform, with specific focus on directions for future research and maps them to the different phases of the big data lifecycle

Experimental Studies in Learning Technology and Child–Computer Interaction

This book is about the ways in which experiments can be employed in the context of research on learning technologies and child–computer interaction (CCI). It is directed at researchers, supporting them to employ experimental studies while increasing their quality and rigor. The book provides a complete and comprehensive description on how to design, implement, and report experiments, with a focus on and examples from CCI and learning technology research. The topics covered include an introduction to CCI and learning technologies as interdisciplinary fields of research, how to design educational interfaces and visualizations that support experimental studies, the advantages and disadvantages of a variety of experiments, methodological decisions in designing and conducting experiments (e.g. devising hypotheses and selecting measures), and the reporting of results. As well, a brief introduction on how contemporary advances in data science, artificial intelligence, and sensor data have impacted learning technology and CCI research is presented. The book details three important issues that a learning technology and CCI researcher needs to be aware of: the importance of the context, ethical considerations, and working with children. The motivation behind and emphasis of this book is helping prospective CCI and learning technology researchers (a) to evaluate the circumstances that favor (or do not favor) the use of experiments, (b) to make the necessary methodological decisions about the type and features of the experiment, (c) to design the necessary “artifacts” (e.g., prototype systems, interfaces, materials, and procedures), (d) to operationalize and conduct experimental procedures to minimize potential bias, and (e) to report the results of their studies for successful dissemination in top-tier venues (such as journals and conferences). This book is an open access publication

Experimental Studies in Learning Technology and Child–Computer Interaction

This book is about the ways in which experiments can be employed in the context of research on learning technologies and child–computer interaction (CCI). It is directed at researchers, supporting them to employ experimental studies while increasing their quality and rigor. The book provides a complete and comprehensive description on how to design, implement, and report experiments, with a focus on and examples from CCI and learning technology research. The topics covered include an introduction to CCI and learning technologies as interdisciplinary fields of research, how to design educational interfaces and visualizations that support experimental studies, the advantages and disadvantages of a variety of experiments, methodological decisions in designing and conducting experiments (e.g. devising hypotheses and selecting measures), and the reporting of results. As well, a brief introduction on how contemporary advances in data science, artificial intelligence, and sensor data have impacted learning technology and CCI research is presented. The book details three important issues that a learning technology and CCI researcher needs to be aware of: the importance of the context, ethical considerations, and working with children. The motivation behind and emphasis of this book is helping prospective CCI and learning technology researchers (a) to evaluate the circumstances that favor (or do not favor) the use of experiments, (b) to make the necessary methodological decisions about the type and features of the experiment, (c) to design the necessary “artifacts” (e.g., prototype systems, interfaces, materials, and procedures), (d) to operationalize and conduct experimental procedures to minimize potential bias, and (e) to report the results of their studies for successful dissemination in top-tier venues (such as journals and conferences). This book is an open access publication

Rigorous, transparent, and eye-catching: Exploring the universalistic parameters of impactful theory building in management

In the management discipline, scholarly impact is most commonly measured using a researcher perspective, by counting the number of times a particular article is mentioned in the references section of other articles (Aguinis, Shapiro, Antonacopoulou, and Cummings, 2014). This approach conceptualizes scholarly impact using a measurable indicator, the citation count an article receives. Several studies have been conducted to examine what drives scholarly impact in the field of management. The originality of the idea, rigor of the study, and clarity of writing have been identified as the most significant universalistic parameters of scholarly impact (Judge, Colbert, Cable, and Rynes, 2007). This dissertation sets out to do a detailed examination of these parameters. The six articles included in the thesis do so in two ways: either by offering recommendations for improving these universalistic parameters of scholarly impact or by further exploring the relationship between the universalistic parameters and scholarly impact. Our first empirical article, here relayed in Chapter II, focuses on case studies, and analyzes the methodological rigor of all case studies published during the period 1996-2006. We point out different types of replication logic, and illustrate how their individual research actions have differential effects on the internal and external validity (in that order of priority) of the emerging theory. Chapter III follows up on the previous chapter, extending the investigation to quantitative as well as qualitative research, and offers replication logic as a tool for analyzing deviant cases identified during the course of a qualitative or quantitative study. We call this technique the \u2018Deviant Case Method\u2019 (\u2018DCM\u2019). Through this study, we explain the theoretical consequentiality (Aguinis et al., 2013; Cortina, 2002) of analyzing three different kinds of outliers (construct, model fit, and prediction outliers/ deviant cases) and offer DCM for analyzing prediction outliers/deviant cases. In Chapter IV, we extend this method to have a look at medium-N studies. Here we focus on inconsistent or deviant cases which turn up during a fuzzy set Qualitative Comparative Analysis (fsQCA). We offer a method called \u2018Comparative Outlier Analysis\u2019 (\u2018COA\u2019) which combines DCM and Mill\u2019s canons (1875) to examine these multitude of inconsistent cases. We explicate this using exemplars from fields like politics, marketing, and education. Unlike in other disciplines or methods, it is far from clear what the label \u2018transparent research procedures\u2019 constitutes in management field studies, with adverse effects during write-up, revision, and even after publication. To rectify this, in Chapter V, we review field studies across seven major management journals (1997- 2006) in order to develop a transparency index, and link it to article impact. Chapter VI is a sequel to the previous chapter. We propose a new method for assessing the methodological rigor of grounded theory procedures ex-post using an audit trail perspective. While existing research on the methodological sophistication of grounded theory was typically done from the perspective of the author or producer of the research, our perspective is customer-centric, both in terms of the end-customer (i.e. the reader or other author), as well as the intermediate customer (i.e. reviewers and editors). The last empirical article in the thesis, Chapter VII, focuses on yet another parameter influencing impact: the style of academic writing. Specifically, we focus on the attributes of article titles and their subsequent influence on the citation count. At this early stage of theory development on article titles, we do this in the specific application context of management science. We conclude with Chapter VIII where we sum up the findings and implications of all preceding studies and put forth suggestions for future research

The impact of National Science Foundation investments in undergraduate engineering education research: A comparative, mixed methods study

The U.S. invests billions of taxpayers\u27 dollars in research tied to the national priorities that contribute to its competitiveness in a global economy. As the federal funding agency with an explicit focus on engineering education, the National Science Foundation (NSF) contains a portfolio of projects focused on improving the quantity of engineering graduates and the quality of engineering programs. Within the agency, the Division of Undergraduate Education invests approximately $190 million (FY 2012) annually on science, technology, engineering and mathematics (STEM) education projects. Although the DUE portfolio includes a suite a projects with different foci supporting national initiatives and Principal Investigators (PIs) report their results in annual reports and conferences, there is little consistency on how impact is defined, evaluated, and measured. ^ While many agree on the importance of investing in research, the stiff economic climate necessitates that the research that demonstrates impact is what will continue to be supported. However, the dearth of scholarship on impact contributes to the lack of understanding around this topic. This study links the fragmented literature on impact to form a unified starting point for continuing the conversation. While existing literature includes three dimensions of research impact (i.e., scientific, societal, and domain-specific impact), this study focuses on the domain-specific impacts of engineering education research using two guiding frameworks, namely, Toulmin\u27s Model (1958) and the Common Guidelines for Education Research and Development (Earle et al., 2013), and a multiphase mixed methods research design (Creswell & Plano Clark, 2011).^ The qualitative phase of this study explores how researchers on NSF-funded STEM education R&D projects talk about the impact of their work; the findings reveal eight themes that are commonly discussed when PIs articulate the impact of their research, and two themes related to how they support their claims. The findings also indicate that the STEM discipline associated with the study and the project focus have more to do with the types of impact PIs claim than the amount of funding awarded to the project. As a result of identifying the points of alignment between PIs\u27 perspectives on impact and existing literature, a preliminary description of what impact looks like in this context is proposed—using the three dimensions of research impact as an organizing framework. Although this study puts forth a preliminary description of the impact of STEM education research, extensions of this work are necessary before providing practitioners and policymakers with a valid, comprehensive framework characterizing what impact means in this context.^ Ideas supporting the types of claims PIs make when discussing the impact of their work were used to develop a survey that was distributed to a small sample of current and former NSF Program Officers (POs) in the second phase of this study. The survey results provide preliminary evidence on how PIs and NSF PO\u27 perspectives on research impact compare, and affirm that additional studies are needed. Consequently, implications for policy and practice and potential research directions are also discussed

An Assessment of Undergraduate Students’ Research Literacy

Research literacy refers to the knowledge and application of statistics and research methods knowledge. Research literacy is important because it enables individuals to become autonomous lifelong learners and informed research consumers. Compared to other types of literacies (e.g., informational, statistical, scientific, etc.), research literacy in the social sciences has received limited attention in psychological theory and research. As a result, assessments of research literacy have notable limitations. Some assessments place undue emphasis on content knowledge of statistics and research methods neglecting the application of knowledge, others present items in a de-contextualized manner, exploring conceptions or attitudes toward research itself rather than research literacy; and some ask respondents to report subjective assessments of their own research literacy as a means of assessment. The aim of the current research was to assess research literacy in undergraduate students in a reliable and valid way by developing the Critical Research Literacy Assessment (CRLA), an assessment that is more comprehensive (tapping diverse sub-domains believed to be part of research literacy) and uses contextually valid testing formats that tap both knowledge and application domains of research literacy. Results demonstrated that the CRLA was a reliable assessment. Evidence for concurrent, divergent, and criterion validity was also found

Predicting And Characterizing The Health Of Individuals And Communities Through Language Analysis Of Social Media

A large and growing fraction of the global population uses social media, through which users share their thoughts, feelings, and behaviors, predominantly through text. To quantify the expression of psychological constructs in language, psychology has evolved a set of “closed-vocabulary” methods using pre-determined dictionaries. Advances in natural language processing have made possible the development of “open-vocabulary” methods to analyze text in data-driven ways, and machine learning algorithms have substantially improved prediction performances. The first chapter introduces these methods, comparing traditional methods of text analysis with newer methods from natural language processing in terms of their relative ability to predict and elucidate the language correlates of age, gender and the personality of Facebook users (N = 65,896). The second and third chapters discuss the use of social media to predict depression in individuals (the most prevalent mental illness). The second chapter reviews the literature on detection of depression through social media and concludes that no study to date has yet demonstrated the efficacy of this approach to screen for clinician-reported depression. In the third chapter, Facebook data was collected and connected to patients’ medical records (N = 683), and prediction models based on Facebook data were able to forecast the occurrence of depression with fair accuracy–about as well as self-report screening surveys. The fourth chapter applies both sets of methods to geotagged Tweets to predict county-level mortality rates of atherosclerotic heart disease mortality (the leading cause of death in the U.S.) across 1,347 counties, capturing 88% of the U.S. population. In this study, a Twitter model outperformed a model combining ten other leading demographic, socioeconomic and health risk factors. Across both depression and heart disease, associated language profiles identified fine-grained psychological determinants (e.g., loneliness emerged as a risk factor for depression, and optimism showed a protective association with heart disease). In sum, these studies demonstrate that large-scale text analysis is a valuable tool for psychology with implications for public health, as it allows for the unobtrusive and cost-effective monitoring of disease risk and psychological states of individuals and large populations