One of the advantages cited for case study research is its uniqueness, its capacity for understanding complexity in particular contexts. A corresponding disadvantage often cited is the difficulty of generalising from the single case. Such an observation assumes a polarity and stems from a particular view of research. Looked at differently, from within a holistic perspective and direct perception, there is no disjunction. What we have is a paradox, which if acknowledged and explored in depth, yields both unique and universal understanding. This paper revisits this problem in case study research. It argues that, with the pressure for quantification and the growth of multi?site case study design in policy research, the original vision and utility of case study for understanding complex educational phenomena has been clouded and thereby an opportunity diminished for new ways of knowing

Bundy, Alan

English

Alan Bundy

CiteSeerX

The Paradox of the Case Study

Simons, Helen

Southampton (e-Prints Soton)

The paradox of case study

One of the advantages cited for case study research is its uniqueness, its capacity for understanding complexity in particular contexts. A corresponding disadvantage often cited is the difficulty of generalising from the single case. Such an observation assumes a polarity and stems from a particular view of research. Looked at differently, from within a holistic perspective and direct perception, there is no disjunction. What we have is a paradox, which if acknowledged and explored in depth, yields both unique and universal understanding. This paper revisits this problem in case study research. It argues that, with the pressure for quantification and the growth of multi‐site case study design in policy research, the original vision and utility of case study for understanding complex educational phenomena has been clouded and thereby an opportunity diminished for new ways of knowin

Edinburgh Research Explorer

     Edinburgh Research Explorer                                      The Paradox of the Case StudyCitation for published version:Bundy, A 2004, 'The Paradox of the Case Study' Cambridge Journal of Education, vol 26, no. 2, pp. 225-240., 10.1080/0305764960260206Digital Object Identifier (DOI):10.1080/0305764960260206Link:Link to publication record in Edinburgh Research ExplorerDocument Version:Author final version (often known as postprint)Published In:Cambridge Journal of EducationGeneral rightsCopyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)and / or other copyright owners and it is a condition of accessing these publications that users recognise andabide by the legal requirements associated with these rights.Take down policyThe University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorercontent complies with UK legislation. If you believe that the public display of this file breaches copyright pleasecontact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately andinvestigate your claim.Download date: 20. Feb. 2015The Paradox of the Case StudyAlan BundySeptember 20041 IntroductionIn order to demonstrate the scalability of automated reasoning techniques, it isimportant to embark on large-scale case studies. As a field, we need to presenta reward system for the completion of such case studies. However, there is aparadox at the heart of the case study. It is this paradox, and its resolution,that I intend to explore in this note.The classic example of the theorem proving case study is Shankar’s proof ofGodel’s incompleteness proof using the Nqthm theorem prover [Shankar, 1994].Shankar’s reward for this case study was that he received a PhD. Note thatthe theorem prover was developed by third parties: Shankar’s supervisors BobBoyer and J Moore.2 The ParadoxIt is my contention that any good piece of scientific or engineering researchdevelops a hypothesis and then provides evidence to support (or refute) thathypothesis. What is the hypothesis in the case of a theorem-proving case study?The obvious hypothesis, is that the theorem prover is capable of proving hardtheorems. The evidence supporting this hypothesis is the automated proof of ahard theorem.What would be the ideal such evidence? Presumably, that using the mostobvious and direct representation of the theory and the theorem, that the the-orem prover could prove the theorem totally automatically and within minutesif not seconds. Herein lies the paradox. If it was that easy, then this workwould scarcely be worthy of a PhD. However, if the work was so hard that itwas worthy of a PhD, then the evidence supporting the hypothesis would berather thin. If the student had had to work really hard to find an appropriaterepresentation of the theory and the theorem; if s/he had had to work evenharder to search a huge search space, guiding the prover at nearly every step,then the prover would have been shown to be barely capable of proving thehard theorem, and then only under the direction of an expert in the field. Itseems that the student’s success is inversely proportional to the success of thetheorem prover. This is the paradox of the case study. What industry wants1is push-button and fast technology, but it seems that we are unable to rewardcase studies that demonstrate such technology1.Of course, this analysis is superficial. Any theorem prover has a range ofapplication: from simple theorems that it can prove totally automatically, tohard ones at the limit of its capacity, requiring expert guidance over multiplesteps. The case study typically explores the outer limits of this range: thosetheorems that are barely reachable and then only under expert guidance. So amore for sophisticated hypothesis is that the outer limits of the theorem proverunder test exceeds that of rival theorem provers. But then, of course, the evi-dence should include unsuccessful experiments to prove the same theorem usingeach of these rival provers. This is seldom demonstrated. Indeed, although theexperimenter will usually not attempt any case studies using rival provers, thechampions of many rival provers frequently take up the challenge, subsequentlyshowing that their provers are equally, if not more, capable of completing thecase study. This refutation of the implicit hypothesis often occurs after theoriginal student has been rewarded with a PhD.A further problem with this kind of case study is it becomes an awful slog.The student has to put his/her head down and push on relentlessly for weeks andmonths, even years, overcoming one difficulty after another, until they finallyproduce a huge proof, which no one else may ever study in detail. It is easyto get disheartened, to wonder whether it is all worth it, or what the point is.The attrition rate for case study students is much higher than for other kindsof project.3 Can we Find Another Hypothesis?One way out of this paradox, is to seek an alternative hypothesis to thosepresented in the last section. I propose that the case study be seen as ananalysis of the system under test. We are asking not just whether the systemcan prove this hard problem, but how easy the prover makes producing thisproof: what kind of tools does it provide to support the representation of thetheory or the theorem; what kind of search-control tools help the user guide theproof; what kind of visualisation tools help the user to understand the structureof proof; what kind of analysis tools help the user to identify problems and solvethem. The student might also try to classify the types of intervention that wererequired to guide the proof. For instance, was it enough to set the parametersof existing heuristics or tactics, was it necessary to write new kind of tactics, orwas a major overhaul of the theorem prover necessary? In such an analysis, alarge part of the work will be in the development of an appropriate hypothesis.As an exemplar of what I have in mind, I will describe the kind of analysisthat we did in Francisco Cantu’s PhD project, in which he used our Claminductive proof planner to verify various hardware algorithms and systems1Unless the case studies are carried out by the system’s developers. A short evaluationsection within a system description would work.2[Cantu et al, 1996]. He kept a record of the kind of intervention that was neces-sary to guide the proofs. He classified these interventions by the kind of personwho would be needed to make such interventions. For instance, choices be-tween existing tactics might be made by the hardware developer within a usercompany without any deep knowledge of the theorem prover. However, if newtactics were required, then it might be necessary to call in the company’s localexpert. Finally, if a major overhaul of the prover was needed, then it wouldprobably be necessary to ask the prover’s supplier to issue a new version. Fran-cisco tried to demonstrate that most interventions were of a simple nature thatcould be handled by the user or, at least, within the user’s company, and did notrequire supplier intervention. He further demonstrated that the interventionswere front-loaded and lumpy. That is to say, most interventions were required atthe beginning of a series of experiments on a new class of algorithms or systems.After tuning the prover to this new class, most subsequent proofs required littleor no human guidance. In this way, he was able to develop a usage scenario forthe prover, showing that it could be a practical tool within a company doinghardware development.This kind of analysis is both more sophisticated and more interesting thantesting the simple hypotheses that this prover can prove hard theorems, orharder ones than its rivals. It avoids the case study paradox since the effortrequired from the student is now directly proportional to the value of the analysisthat is produced from the case study. Moreover, it is much more interestingwork, since the student must continually pay attention not only to the courseof the proof, but to the experience of proving and to the assistance provided (ornot provided) by the theorem prover. It is this analysis, rather than the proof,that is the product of the case study. This is much more likely to be usefulto the developers of the theorem prover and to the user community, so, unlikethe gory details of the proof itself, the analysis will be widely disseminated andappreciated.4 ConclusionIn this note, we have argued for a more sophisticated hypothesis to underliecase study research. The simplistic hypothesis that the prover can prove hardtheorems — or harder ones than its rivals — leads to a paradox in which thesuccess of the student is inversely proportional to the success of the prover, andwhere the proof process becomes a relentless and unrewarding slog. The moresophisticated hypothesis consists of an analysis of the strengths and weaknessesof the prover. It consists of a series of small hypotheses: the prover providesgood visualisation or analysis tools, the prover provides good automatic searchcontrol guidance, etc. It leads to more interesting research because the studentmust demonstrate awareness at multiple levels simultaneously: at the object-level of guiding the search for a proof and at the meta-level of the experience ofthe proof process. It also leads to research that will be more useful to both thedevelopers of the prover and their user community.3Superficially, this discussion seems relevant only to the interactive theorem-proving community. This is not correct. Case studies are also carried outwithin the totally automated theorem proving community, for instance, EQP’sproof of the Robbins Algebra Conjecture [McCune, 1997]. Note that this proofwas not produced from a single run of the prover on the problem. Rather,it took place over a decade in which: the proof was divided up into lemmas,the theorem prover was developed and refined, and a series experiments wastried with a wide variety different parameter settings. The kind of analysisproposed above is just as relevant to this kind of case study as to those usingexplicitly interactive provers. Indeed, we badly need such analytical case studiesof automated provers, to better understand and improve their usability.References[Cantu et al, 1996] Cantu, Francisco, Bundy, Alan, Smaill, Alan and Basin,David. (1996). Experiments in automating hardware veri-fication using inductive proof planning. In Srivas, M. andCamilleri, A., (eds.), Proceedings of the Formal Methods forComputer-Aided Design Conference, number 1166 in Lec-ture Notes in Computer Science, pages 94–108. Springer-Verlag.[McCune, 1997] McCune, W. (1997). Solution of the Robbins problem. J.Automated Reasoning, 19(3):263–276.[Shankar, 1994] Shankar, N. (1994). Metamathematics, Machines, andGo¨del’s proof. Cambridge University Press.4

Experiments in automating hardware veriﬁcation using inductive proof planning.

Metamathematics, Machines, and G¨ odel’s proof.

Solution of the Robbins problem.

The Paradox of the Case Study

Abstract

Similar works

Full text

Available Versions

CiteSeerX

Southampton (e-Prints Soton)

Edinburgh Research Explorer