Multidimensional signals and analytic flexibility: Estimating degrees of freedom in human speech analyses

Recent empirical studies have highlighted the large degree of analytic flexibility in data analysis which can lead to substantially different conclusions based on the same data set. Thus, researchers have expressed their concerns that these researcher degrees of freedom might facilitate bias and can lead to claims that do not stand the test of time. Even greater flexibility is to be expected in fields in which the primary data lend themselves to a variety of possible operationalizations. The multidimensional, temporally extended nature of speech constitutes an ideal testing ground for assessing the variability in analytic approaches, which derives not only from aspects of statistical modeling, but also from decisions regarding the quantification of the measured behavior. In the present study, we gave the same speech production data set to 46 teams of researchers and asked them to answer the same research question, resulting insubstantial variability in reported effect sizes and their interpretation. Using Bayesian meta-analytic tools, we further find little to no evidence that the observed variability can be explained by analysts’ prior beliefs, expertise or the perceived quality of their analyses. In light of this idiosyncratic variability, we recommend that researchers more transparently share details of their analysis, strengthen the link between theoretical construct and quantitative system and calibrate their (un)certainty in their conclusions

Banner blindness: What searching users notice and do not notice on the World Wide Web

Web designers attempt to draw attention to important links by making them distinctive. However, when users are asked to find specific items, they often overlook these distinctive banners. The irony of this phenomenon I call "banner blindness" is that the user who really wants to find the information the designer has highlighted is not likely to do so. In the experiments reported here, banner blindness was investigated under controlled conditions. Banners located higher on the page and therefore farther from other links were missed more often than banners located lower on the page and closer to the other links. Banners were missed more often when located on pages containing links to categories than when located on pages with links to specific items. Users rarely noticed banners when clicking the banner was not required to accomplish a task. Banner blindness occurred with several types of distinctive links---graphical banners that resembled advertisements, large plain-text banners and small plain-text banners that were very unlike advertisements. Increasing the perceptual grouping between the banner and the "menu" of hyperlinks helped users notice the banners only slightly more often. Adding animation to graphical banners did not help mitigate the effect. Users searching for specific information seem to focus exclusively on the link-rich areas of the page and do not notice distinctive items outside of that area. The last two experiments in this research focused on emphasizing one item within a menu of search-engine "hits." Three types of emphasis were used. Very large text caused a slight banner-blindness effect. Subtly large text had no effect at all. Highlighting one menu item by giving it a brightly-colored background did not cause banner blindness. In fact, it attracted the attention of users: users were more likely to select the highlighted item and did so more quickly. This type of color highlighting was most effective when it emphasized the first item in the menu. It was slightly less effective when it emphasized items in the middle of the menu