External Validity in IS Survey Research

This study focuses on the issues of external validity, coverage error and nonresponse error in IS survey research. Data from the empirical papers in 5 years of issues from three A rated IS journals are presented and analyzed. Recommendations are made based on the analyses, basic scientific principles and the authors\u27 experience and judgment

Innovative Work Behaviour: Measurement and Validation

Although both scientists and practitioners emphasize the importance of innovative work behavior (IWB) of individual employees for organizational success, the measurement of employees' IWB is still in evolution. Here, we present two multi-source studies that aimed to develop and validate a measure of IWB. Four related dimensions of IWB are distinguished: opportunity exploration, idea generation, championing and application. We derived a tenitem measure of these IWB dimensions from a pilot survey among matched dyads of 81 professionals in a research institute and their supervisors. Next, a survey among a matching sample of 703 knowledge workers and their supervisors from 94 different firms was done. We used confirmatory factor analyses to examine convergent and discriminant validity, and hierarchical multilevel regression to test hypothesized relationships of IWB with participative leadership, external work contacts and innovative output (proposed as an initial nomological network). Results demonstrate strong convergent validity of the IWB measure as all four dimensions contribute to an overall measure of IWB. Support for discriminant validity is weaker as correlations between some dimensions are relatively high. Finally, IWB is positively related with participative leadership, external work contacts and innovative output, providing first evidence for nomological validity.

Exploratory study on performance measures as indicators of IS effectiveness

The Information Systems Directorate at JSC/NASA has undertaken the reevaluation of its performance measures process and measures. Under the direction of a quality approach it is essential to identify an external perspective of how well an organization is performing. This study was conducted with two major objectives: (1) survey and summarize the academic literature on performance measures as indicators of information systems (IS) effectiveness; and (2) survey organizations for their experience in measuring for IS effectiveness. Four approaches to measuring the effectiveness of IS performance were identified: (1) listen to the customer for the things they need; (2) align with corporate goals; (3) benchmark against well-respected organizations; and (4) ask yourself what critical factors lead to success. The list of known methods for soliciting customer feedback are as follows: (1) executive visit; (2) survey, interview, and focus group; (3) complaints and compliments; and (4) service level agreements. A common set of characteristics that satisfy customers was identified from the literature. The list includes elements such as the following: accuracy, timeliness, relevance, understandability, reliability, and completeness. Future research in this topic area should prove beneficial to determine the metrics for external validity

Worse Than Spam: Issues In Sampling Software Developers

Background: Reaching out to professional software developers is a crucial part of empirical software engineering research. One important method to investigate the state of practice is survey research. As drawing a random sample of professional software developers for a survey is rarely possible, researchers rely on various sampling strategies. Objective: In this paper, we report on our experience with different sampling strategies we employed, highlight ethical issues, and motivate the need to maintain a collection of key demographics about software developers to ease the assessment of the external validity of studies. Method: Our report is based on data from two studies we conducted in the past. Results: Contacting developers over public media proved to be the most effective and efficient sampling strategy. However, we not only describe the perspective of researchers who are interested in reaching goals like a large number of participants or a high response rate, but we also shed light onto ethical implications of different sampling strategies. We present one specific ethical guideline and point to debates in other research communities to start a discussion in the software engineering research community about which sampling strategies should be considered ethical.Comment: 6 pages, 2 figures, Proceedings of the 2016 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM 2016), ACM, 201

Are there gambling effects in incentive-compatible elicitations of reservation prices? An empirical analysis of the BDM-mechanism

Pricing research suggests incentive compatible evaluations of separate products in so-called monadic designs when consumers\u27 situation-specific WTP is to be elicited in a monopolistic purchase setting. In our study, the lottery-based BDM-mechanism is applied for measuring subjects\u27 WTP for a fast moving consumer good in binding one-on-one interviews at the point of purchase. In previous studies, the validity of elicited WTP measures is commonly checked within subjects with respect to indicators of face and criterion validity (such as interest in buying, preference ratings, compliance rates). In addition, we observed real purchases of a separate validation sample at the point of purchase, thus checking external validity between subjects. As a result, the BDM-based WTPs reveal a sufficient degree of internal face validity. However, the external validity in terms of a goodness of fit between WTP-based predictions and purchases of the validation sample is significantly reduced. Specifically, we observed a substantial underestimation of shares of non-buyers. Hence, a potential bias is indicated, leading to an overrating of consumers\u27 true WTP in the lottery-based BDM-mechanism in the setting of our survey

Methods for Evaluating Respondent Attrition in Web-Based Surveys

Background: Electronic surveys are convenient, cost effective, and increasingly popular tools for collecting information. While the online platform allows researchers to recruit and enroll more participants, there is an increased risk of participant dropout in Web-based research. Often, these dropout trends are simply reported, adjusted for, or ignored altogether. Objective: To propose a conceptual framework that analyzes respondent attrition and demonstrates the utility of these methods with existing survey data. Methods: First, we suggest visualization of attrition trends using bar charts and survival curves. Next, we propose a generalized linear mixed model (GLMM) to detect or confirm significant attrition points. Finally, we suggest applications of existing statistical methods to investigate the effect of internal survey characteristics and patient characteristics on dropout. In order to apply this framework, we conducted a case study; a seventeen-item Informed Decision-Making (IDM) module addressing how and why patients make decisions about cancer screening. Results: Using the framework, we were able to find significant attrition points at Questions 4, 6, 7, and 9, and were also able to identify participant responses and characteristics associated with dropout at these points and overall. Conclusions: When these methods were applied to survey data, significant attrition trends were revealed, both visually and empirically, that can inspire researchers to investigate the factors associated with survey dropout, address whether survey completion is associated with health outcomes, and compare attrition patterns between groups. The framework can be used to extract information beyond simple responses, can be useful during survey development, and can help determine the external validity of survey results

Validity concerns in software engineering research

Empirical studies that use software repository artifacts have become popular in the last decade due to the ready avail-ability of open source project archives. In this paper, we survey empirical studies in the last three years of ICSE and FSE proceedings, and categorize these studies in terms of open source projects vs. proprietary source projects and the diversity of subject programs used in these studies. Our survey has shown that almost half (49%) of recent empirical studies used solely open source projects. Existing studies either draw general conclusions from these results or explic-itly disclaim any conclusions that can extend beyond specific subject software. We conclude that researchers in empirical software engi-neering must consider the external validity concerns that arise from using only several well-known open source soft-ware projects, and that discussion of data source selection is an important discussion topic in software engineering re-search. Furthermore, we propose a community research in-frastructure for software repository benchmarks and sharing the empirical analysis results, in order to address external validity concerns and to raise the bar for empirical software engineering research that analyzes software artifacts

The Effect of External Pressure, Management Commitment and Accessibility towards Transparency of Financial Reporting

Research Aims: This study aims to examine and empirically prove the Effects of External Pressure, Management Commitment, and Accessibility of Financial Statements on Transparency of Financial Reporting in Regional Work Units in Yogyakarta City.Design/Methodology/Approach: The methodology used is survey with purposive sampling. The respondents of this study consists of 74 people. The analytical tool used in this study is multiple linear regression using the SPSS.Research Findings: The test results indicate that external pressures, management commitment, and accessibility of financial statements have a positive effect on financial reporting transparency.Theoretical contribution/Originality:  this research filled the research gap related to the issue of determinants of transparency of financial reporting in local governments which has got minimal attention from researchers.Practitioner/Policy implication:  departing from the results of this study, it is recommended to increase the transparency of financial reporting, external pressure, management commitment and accessibility must be encouraged.Research limitation/Implication:  This research was only conducted in the city of Yogyakarta, so it has weak external validity. The generalization of this study results does not represent a broader scope