"Pyramiding: Efficient search for rare subjects"

Abstract

The need to economically identify rare subjects within large, poorly mapped search spaces is a frequently encountered problem for social scientists and managers. It is notoriously difficult, for example, to identify "the best new CEO for our company," or the "best three lead users to participate in our product development project." Mass screening of entire populations or samples becomes steadily more expensive as the number of acceptable solutions within the search space becomes rarer. The search strategy of "pyramiding" is a potential solution to this problem under many conditions. Pyramiding is a search process based upon the idea that people with a strong interest in a topic or field tend to know people more expert than themselves. In this paper we report upon four experiments empirically exploring the efficiency of pyramiding searches relative to mass screening. We find that pyramiding on average identified the most expert individual in a group on a specific topic with only 28.4% of the group interviewed - a great efficiency gain relative to mass screening. Further, pyramiding identified one of the top 3 experts in a population after interviewing only 15.9% of the group on average. We discuss conditions under which the pyramiding search method is likely to be efficient relative to screening.Sequential search Research method Lead user innovation Serial experimentation Network theory