The act of ‘design’ has been a rich and difficult subject of study in the context of information systems for many years (see, for example, [Adelson and Soloway 1985, Guindon et al 1986, Sen 1997]). A number of design studies have been conducted, relying on techniques such as verbal protocols or videotapes to gain insight into the many facets of system design. Typically, such studies (a) have been planned around a simple design task, (b) have involved a single design session, and (c) have focused the data analysis on identification of repeated design behaviors or problems

Bush, Ashley

Purao, Sandeep

English

AIS Electronic Library (AISeL)

Association for Information SystemsAIS Electronic Library (AISeL)AMCIS 1998 Proceedings Americas Conference on Information Systems(AMCIS)December 1998Characterizing Process Variables in a Study ofDesign BehaviorsAshley BushGeorgia State UniversitySandeep PuraoGeorgia State UniversityFollow this and additional works at: http://aisel.aisnet.org/amcis1998This material is brought to you by the Americas Conference on Information Systems (AMCIS) at AIS Electronic Library (AISeL). It has been acceptedfor inclusion in AMCIS 1998 Proceedings by an authorized administrator of AIS Electronic Library (AISeL). For more information, please contactelibrary@aisnet.org.Recommended CitationBush, Ashley and Purao, Sandeep, "Characterizing Process Variables in a Study of Design Behaviors" (1998). AMCIS 1998 Proceedings.227.http://aisel.aisnet.org/amcis1998/227-676-Characterizing Process Variables in a Study of Design BehaviorsAshley BushSandeep PuraoDepartment of Computer Information SystemsGeorgia State University IntroductionThe act of ‘design’ has been a rich and difficult subject of study in the context of information systems for many years (see,for example, [Adelson and Soloway 1985, Guindon et al 1986, Sen 1997]).  A number of design studies have been conducted,relying on techniques such as verbal protocols or videotapes to gain insight into the many facets of system design.  Typically,such studies (a) have been planned around a simple design task, (b) have involved a single design session, and (c) have focusedthe data analysis on identification of repeated design behaviors or problems.These studies thus have not (a) accounted for behaviors that may be manifested only in complex design tasks, (b) allowedinterpretations that may be possible only by observing behaviors in multiple sessions, and (c) provided attention to duration andthe sequencing of tasks.  In so far as they have ignored the above, they have lacked a process theoretic perspective.A process theoretic perspective allows analysis of variables over time to understand how they behave individually and relateto each other [Monge 1990].  Though processual studies of system development have been reported [Sabherwal and Robey1995], they have focused on project management issues rather than detailed design tasksThis is the key distinguishing feature of the present study.  The purpose of the study is to examine the behaviors of novicedesigners over the duration of a small project emphasizing the notion of process variables – based on Monge’s [1990] framework.In the research stream on the study of design behaviors, our approach thus represents a shift in focus from a task-based view ofdesign to a process-based one.Related WorkWe examine below a representative sample of efforts in this stream with a view to highlighting the distinguishing featuresof our research approach. Adelson and Soloway [1985] examine the design behaviors of expert and novice designers for an assigned design task (infamiliar and unfamiliar domains) by videotaping the designers in action during a single session.  The videotapes are analyzedto extract repeatedly occurring, common behaviors, such as formation of mental models, simulation etc.Guindon et al [1986] identify breakdowns that affect design processes of professional software designers.  They, too, analyzea single videotaped design session to identify categories of breakdowns: knowledge-related, cognitive limitations, and lack ofknowledge.More recently, Sen [1997] proposes a cognitive model for demand-side reuse that illustrates how design reuse activities areclustered within different panels of a blackboard model such as control, global design, specific design etc.  To verify the model,he relies on verbal protocols of designers engaged in constructing a data model collected during a single two-hour design session.These studies rely on data collected from single sessions, and simple design tasks.  Further, they focus on extracting repeatedbehaviors [Adelson and Soloway 1985], identifying categories of breakdowns [Guindon et al 1986], and clustering reuse tasks[Sen 1997].  To address the shortcomings (discussed in the Introduction) of these and similar studies, our research designincludes a slightly more complex design task that naturally requires multiple design sessions.  Accordingly, we also performmultiple observations that are spread across the entire duration of the design project.A Process-oriented PerspectiveTo structure our research design, we draw on the framework presented by Monge [1990] for understanding process variables.He suggests collection of data at multiple points in time and applying verbal or mathematical formulations to this data forinterpretation.  The interpretations he suggests include: continuity, magnitude, rate of change, trend, periodicity, and duration(see Table 1).  In addition to these, he also suggests a number of ways of ‘relating’ these variables, such as lag, concomitant cause etc[Monge 1990, p. 417]. This framework forms the basis of our data analysis.-677-D1D3D5D70100200300Time (minutes)DaysFigure 1.  Participant 10D1 D2 D3 D4D5D6D7050100150200250300DaysFigure 2.  Participant 03Continuity presence of consistent nonzerovalue Magnitude amount of the variable at eachpoint Rate ofChangehow fast the magnitude changesTrend long term change in themagnitude Periodicity amount of time between values Duration length of time with a nonzerovalueTable 1.  Dimensions of Process Variables Research Setting and Data CollectionTo reflect the process perspective, we adapted aresearch setting that consisted of a week-long design project.Voluntary participation for the study was sought fromstudents completing a graduate level Object-OrientedSystem Design course at a major university.  The casedescription asked the participants to design an ElectronicJournal Publishing and Subscription System.  They wereprovided the description of system requirements[http://cis.gsu.edu/~spurao/research/ BPCase.html] alongwith a partial list of use cases [Jacobson et al 1992].  Thedeliverables for the project were a complete class diagram[Rational 1998] and interaction diagrams [Rational 1998] fortwo use cases.  During the project, the students could consultthe UML notation [Rational 1998], analysis patterns [Coad1996], books [Booch 1994] and other material[http://cis.gsu.edu/~spurao/cis813].The research design was considered to be non-controlled, following Kerlinger [1986].  Data wascollected in the form of self-reported daily activitysummaries in response to electronic questionnaires[http://cis.gsu.edu/ ~spurao/research/BPds-template.html],and daily copies of the working papers (copies of handdrawn diagrams or printouts from CASE tool).  The dailyactivity summaries included two parts.  The first wasquantitative data: (a) time spent on reviewingrequirements, (b) consulting UML summary, (c)consulting analysis patterns, (d) consulting other sources,(e) drawing class diagrams, and (f) drawing interactiondiagrams.  The second part included qualitative data suchas the specific portion of the requirements documentvisited, for what purpose etc.  The first subset, quantitativedata, forms the basis of the current paper.  As we proceedwith analysis of the qualitative data, we expect that it willfurther inform tentative hypotheses that we may form (orinferences we may draw) from the former.As may be expected, the longer design task precludesconstant observations such as videotaped design sessionsor verbal protocols.  Due to the complexity and length oftime introduced by multiple design sessions, our researchplan needs to sacrifice some of the richness seen in priorstudies of design.  The gains, on the other hand, are in thearea of examining how different design behaviors(variables) are manifested across multiple design sessions.As we learn more about these design behaviors from theanalysis of the data we gather, we expect that additionalexperiments can be conducted to further probe some or allof these behaviors.ResultsSince this paper primarily focuses on the quantitativedata, the analysis reported below primarily revolvesaround the quantitative variables.  The six variablesreported by the participants (a through f above), werecollapsed into four for analysis, representing time spent on four activities: (a) reading/reviewing the case description, (b)consulting other resources, (c) drawing class diagrams, and (d) drawing interaction diagrams.  Graphs for these variables werecreated for all twelve participants.  Figures 1 through 3 show these for three participants1.  In these figures, the vertical axisshows time spent; and the time-points (days 1-7) are shown on the horizontal axis. -678-D1D3D5D7050100150200250300Time (minutes)DaysFigure 3.  Participant 15Participant 10 (P10, Figure 1)1 is a late starter as seen by zero magnitude for all variables on the first three days.  He,however, does not take a break once work begins.  On the fourth day, he begins work by looking for "general information" inthe case description.  On day six, he again reviews the case description looking for classes and use cases, and consulting analysispatterns.  For the seventh day, he plans: "I have to finish everything tomorrow, so I guess the answer should be everything thatI haven’t finished."   From day four, all activities show an increasing trend.  The activity ‘read/ review case description’ (firstshaded line) has a positive magnitude and a fast rate of change; the third activity ‘draw class diagrams’ shows the most dramaticincrease in magnitude and rapid rate of change.  The activities ‘consult other resources’ (second shaded line) and ‘drawinteraction diagrams’ (fourth shaded line) both have an increasing trend but the rate of change is slower than that of the othervariables. Similar interpretations can be made for participant 03 (P03, Figure 2)1, and participant 15 (P15, Figure 3) 1.  P03 starts early.The rate of change for the activity ‘read/review case description’ (first line) is rapidly increasing and the trend is positive.  Theactivity ‘consult other resources’ (second shaded line) has a positive magnitude on the first day, but it quickly falls to and staysat zero.  The activity ‘draw class diagrams’ (third shaded line) appears to have a steady trend.  The last activity ‘draw interactiondiagrams’ has only one non-zero magnitude late in the week. Participant 15 is another early starter.  She begins onday two by engaging in all activities except ‘drawinteraction diagrams.’ Work on day three shows most timespent on ‘drawing class diagram.’  The activities‘read/review case’ and ‘consult other resources’ both showa downward trend and a slow rate of change after thesecond day.  On the other hand, the activities ‘draw classdiagrams’ and ‘draw interaction diagrams’ have a rapidrate of change and an increasing trend.As can be seen from the three examples shown above,it is possible to start interpreting the design process usingthe dimensions suggested by Monge [1990] (see table 1).For our purpose, the applicable dimensions are: continuity,magnitude, rate of change, and trend. Periodicity andduration are not relevant since the observations occur atfixed time intervals.DiscussionIn general, the participants fell into four broad categories: early starters, late starters, laggards, and extreme laggards, whichmay be considered typical of the way people work.  Some general patterns suggested by Monge's framework could also bediscerned for participants from the different categories.  The four dimensions discussed in the previous section form the basisfor identifying these patterns.  Specifically, we use three indicators: discontinuity (variables having both zero and nonzero valuesat different points in time), concomitant cause (a possible cause and effect relationship), and lag (elapsed time between thechange in one variable and the resulting change in another).For the early starters, some discontinuity was observed.  Most early starters ‘took a break’ and ‘planed to revisit’ the designtask.  This is indicated by the zero magnitude across all variables on a given day.  For this group, the activities ‘Read/review casedescription’ and ‘Consult other resources’ appeared to be concomitant causes for the activity ‘Draw class diagrams’; which inturn appeared to be a concomitant cause for the activity ‘Draw interaction diagrams.’  Inferences about concomitant causes canbe drawn from observing the magnitude and trend dimensions.  A cause has positive magnitude that drops immediately precedinga positive magnitude in the effect.  The trend of a cause is typically downward across the period; the trend of an effect is typicallyupward or steady across the same period.  There was also a lag between the activities ‘Draw class diagram’ and ‘Draw interactiondiagrams’ as evidenced by the initial increase in magnitude occurring on different days.Late starters typically began their design task around day three.  No discontinuity was evident in their activity pattern asindicated by the positive magnitude of one or more variables once an upward trend is observed in the variables.  The activities‘Read/review case description’ and ‘Consult other resources’ still appeared to be concomitant causes not only for the activity‘Draw class diagram’ but also for the activity ‘Draw interaction diagram.’  No lag was observed between the activities ‘Drawclass diagram’ and ‘Draw interaction diagram’ which is illustrated by the simultaneous positive magnitude in these variables.The laggards did not begin work until day five or six and extreme laggards did a majority of their work on the last day.  Forboth of these groups, variables are not discontinuous, no concomitant causes are apparent and no lag is evident.  All variablesshow positive magnitudes, increasing trends, and continuity once work has begun late in the design period.-679-Discontinuity Taking a Break & Revisiting theDesign Process indicateDiscontinuity in the Design TaskConcomitantCauseReviewing Requirements is aConcomitant Cause for DrawClass DiagramLag A Lag occurs between Draw ClassDiagram & Draw InteractionDiagramTable 2.  Possible Hypotheses Based on Monge [1990] ConclusionsThis paper has presented the first segment of our project,resulting in a plausible characterization of some processvariables that occur during design.  Further analysis of thequalitative data is necessary to support/refute or further interpretthe data and/or hypotheses presented above.To assess the impact of the interpretations we have madeabove on the eventual design quality, we plan to juxtapose thebehaviors of these variables against expert assessments ofdesigns generated by participants in the four categories. Thisanalysis will also benefit from interpre-tations of textualdescriptions of the design tasks gathered along with thequantitative variables introduced in this paper. ReferencesReferences are available at http://cis.gsu/edu/~spurao/AIS98BushPuraoRefs.html.1Legend:  Read/Review Requirements,     Consult Other Resources,   Draw Class Diagrams,     Draw  InteractionDiagrams 

Characterizing Process Variables in a Study of Design Behaviors

https://aisel.aisnet.org/cgi/viewcontent.cgi?article=1654&amp;context=amcis1998

Characterizing Process Variables in a Study of Design Behaviors

Abstract

Similar works

Full text

Available Versions

AIS Electronic Library (AISeL)