An Experimental Methodology to Evaluate Concept Generation Procedures Based on Quantitative Lifecycle Performance

This study presents an experimental methodology to measure how concept generation procedures can affect the anticipated lifecycle performance of engineering systems design concepts. The methodology is based on objective and quantitative measurements of anticipated lifecycle performance of the design concepts. It merges cognitive and computer-aided techniques from the fields of collaboration engineering, creativity, and engineering design. It complements the body of existing techniques relying on subjective expert assessments, and other objective metrics not explicitly measuring anticipated lifecycle performance (e.g. development time and cost). Application of the methodology is demonstrated through evaluation of design procedures generating flexibility in engineering systems design. Experiments had ninety participants generate creative design alternatives to a simplified real estate development design problem. Thirty-two teams of two to three participants performed the collaborative design exercise. An online Group-Support System interface enabled efficient data collection and analysis. A computationally efficient mid-fidelity model was used to evaluate flexible design concepts quantitatively based on real options analysis techniques.Massachusetts Institute of Technology. Center for Real EstateNatural Sciences and Engineering Research Council of CanadaMassachusetts Institute of Technology. Engineering Systems DivisionSingapore University of Technology and Design. International Design Cente

Quantitative performance-based evaluation of a procedure for flexible design concept generation

Thesis (Ph. D.)--Massachusetts Institute of Technology, Engineering Systems Division, 2011.Cataloged from PDF version of thesis.Includes bibliographical references (p. 152-163).This thesis presents an experimental methodology for objective and quantitative design procedure evaluation based on anticipated lifecycle performance of design concepts, and a procedure for flexible design concept generation. The methodology complements existing evaluation methodologies by measuring anticipated performance via efficient computer modeling techniques. The procedure, in contrast to others, stimulates flexible design concept generation by packaging a short lecture on flexibility, and a prompting ideation mechanism. Controlled collaborative experiments had participants suggest alternative solutions to a design problem under different treatment conditions. Experimental conditions used the procedure for flexibility, while control conditions relied on prior training in science and engineering only, and free undirected ideation. Measures included the quantity of flexible design concepts generated, anticipated economic performance improvements compared to a benchmark design, participants' subjective impressions of satisfaction with the process and results, and results quality assessments. Seventy-one designers divided among twenty-six teams performed the experiments involving a simplified real estate infrastructure design problem. Application of the methodology demonstrated effective and efficient evaluation of the design procedure based on anticipated performance of design concepts. The lecture and prompting mechanism significantly improved anticipated performance compared to the benchmark design, by nearly thirty-six percent. The prompting mechanism significantly improved generation of valuable flexible design concepts. Lecturing improved significantly user satisfaction with the process and results, as well as results quality assessments. Even though prompting demonstrably improved anticipated performance and concept generation, it had no effect on participants' satisfaction with the process and results - unless combined with the lecture. Also, prompting did not lead participants to expect better results quality. This demonstrates the need for thorough and rigorous procedure evaluations based both on subjective user impressions and objective quantitative measurements. A preliminary analysis suggests that the proposed experimental platform can be used to study the influence of uncertainty and flexibility related words on discussion content, although more work is necessary to fully validate the approach.by Michel-Alexandre Cardin.Ph.D