An Algorithm to Determine Peer-Reviewers

The peer-review process is the most widely accepted certification mechanism for officially accepting the written results of researchers within the scientific community. An essential component of peer-review is the identification of competent referees to review a submitted manuscript. This article presents an algorithm to automatically determine the most appropriate reviewers for a manuscript by way of a co-authorship network data structure and a relative-rank particle-swarm algorithm. This approach is novel in that it is not limited to a pre-selected set of referees, is computationally efficient, requires no human-intervention, and, in some instances, can automatically identify conflict of interest situations. A useful application of this algorithm would be to open commentary peer-review systems because it provides a weighting for each referee with respects to their expertise in the domain of a manuscript. The algorithm is validated using referee bid data from the 2005 Joint Conference on Digital Libraries.Comment: Rodriguez, M.A., Bollen, J., "An Algorithm to Determine Peer-Reviewers", Conference on Information and Knowledge Management, in press, ACM, LA-UR-06-2261, October 2008; ISBN:978-1-59593-991-

Bayesian Model Averaging for Model Implied Instrumental Variable Two Stage Least Squares Estimators

Model-Implied Instrumental Variable Two-Stage Least Squares (MIIV-2SLS) is a limited information, equation-by-equation, non-iterative estimator for latent variable models. Associated with this estimator are equation specific tests of model misspecification. We propose an extension to the existing MIIV-2SLS estimator that utilizes Bayesian model averaging which we term Model-Implied Instrumental Variable Two-Stage Bayesian Model Averaging (MIIV-2SBMA). MIIV-2SBMA accounts for uncertainty in optimal instrument set selection, and provides powerful instrument specific tests of model misspecification and instrument strength. We evaluate the performance of MIIV-2SBMA against MIIV-2SLS in a simulation study and show that it has comparable performance in terms of parameter estimation. Additionally, our instrument specific overidentification tests developed within the MIIV-2SBMA framework show increased power to detect model misspecification over the traditional equation level tests of model misspecification. Finally, we demonstrate the use of MIIV-2SBMA using an empirical example.Comment: 31 pages, 8 figures, supplementary materials available upon reques

Mapping the Bid Behavior of Conference Referees

The peer-review process, in its present form, has been repeatedly criticized. Of the many critiques ranging from publication delays to referee bias, this paper will focus specifically on the issue of how submitted manuscripts are distributed to qualified referees. Unqualified referees, without the proper knowledge of a manuscript's domain, may reject a perfectly valid study or potentially more damaging, unknowingly accept a faulty or fraudulent result. In this paper, referee competence is analyzed with respect to referee bid data collected from the 2005 Joint Conference on Digital Libraries (JCDL). The analysis of the referee bid behavior provides a validation of the intuition that referees are bidding on conference submissions with regards to the subject domain of the submission. Unfortunately, this relationship is not strong and therefore suggests that there exists other factors beyond subject domain that may be influencing referees to bid for particular submissions

Using RDF to Model the Structure and Process of Systems

Many systems can be described in terms of networks of discrete elements and their various relationships to one another. A semantic network, or multi-relational network, is a directed labeled graph consisting of a heterogeneous set of entities connected by a heterogeneous set of relationships. Semantic networks serve as a promising general-purpose modeling substrate for complex systems. Various standardized formats and tools are now available to support practical, large-scale semantic network models. First, the Resource Description Framework (RDF) offers a standardized semantic network data model that can be further formalized by ontology modeling languages such as RDF Schema (RDFS) and the Web Ontology Language (OWL). Second, the recent introduction of highly performant triple-stores (i.e. semantic network databases) allows semantic network models on the order of $10^9$ edges to be efficiently stored and manipulated. RDF and its related technologies are currently used extensively in the domains of computer science, digital library science, and the biological sciences. This article will provide an introduction to RDF/RDFS/OWL and an examination of its suitability to model discrete element complex systems.Comment: International Conference on Complex Systems, Boston MA, October 200

When Good Loadings Go Bad: Robustness in Factor Analysis

Structural misspecifications in factor analysis include using the wrong number of factors and omitting cross loadings or correlated errors. The impact of these errors on factor loading estimates is understudied. Factor loadings underlie our assessments of the validity and reliability of indicators. Thus knowing how structural misspecifications affect a factor loading is a key issue. This paper develops analytic conditions of when misspecifications affect Bollen's (1996) model implied instrumental variable, two stage least squares (MIIV-2SLS) estimator of a factor loading. It shows that if an indicator equation is correctly specified, then correlated errors among other measures, mixing up causal indicators with reflective, omitting cross loadings, and omitting direct effects between indicators leave the MIIV-2SLS estimator of the factor loading unchanged. Alternatively, if the indicator or the scaling indicator equation is misspecified, then the loading is unlikely to be robust. The results are illustrated with hypothetical and empirical examples