Building Agent-Based Models requires assumptions (as does building any model).The advantage of ABM is the assumptions can be realistic, rather than made in order to solve a calculus-based problem. But what does "realistic" mean? I shall outline the issues of verification of a model (is it working as the modeller wants?) and validation of a model (how close is the model behaviour to some measures of reality?), after talking about the general issue of choosing models.Australian Research Council DP0881799 IF Wilkinson, RE Marks and LC Young; Discipline of Marketing, The University of Sydney Business School; The University of Sydney Business School; Australia New Zealand Marketing Academy; School of Marketing, University of Western Sydney; Discipline of Marketing, The University of Western Australia Business Schoo

Marks, Robert

English

Sydney eScholarship

ABM Colloquium R.E. Mar ks © 2011 Page 1Choosing the Right Model — Rober t Mark sModelling(from March & Lave (19 75))A. What is a model?B. Why model?C. What is a good model?(Any model, not jus t an ABM.)A. A model:• a simplified picture of a par t of the real world.• has some of the real world’s attr ibutes, but not all.• a picture simpler than reality.We cons truct models in order to explain and underst and.>ABM Colloquium R.E. Mar ks © 2011 Page 2Three Rules of Thumb for Model Building:1. Think “process”.2. Develop interes ting implications.3. Look for gener ality/robus tness.Judge models using: trut h, beauty, jus tice.That is, an inter play between the real world (trut h), world ofæs t hetics (beauty), world of ethics (justice), and the modelworld.< >ABM Colloquium R.E. Mar ks © 2011 Page 3Rule: “stop having kids when your sons outnumber yourdaught ers”“Ques tion: (Are you ready?) What will be the ratio of boys(wit h) to girls (without) in such a society?”A Sur pr ise —→ for mos t couples: more sons than daughter s.but —for socie ty: more girls than boys, almos t alw ays.Let ’s simulat e this using NetLogo.http://www.agsm.edu.au/bobm/teaching/SimSS/NetLogo41-models/boysngirls2.html< >ABM Colloquium R.E. Mar ks © 2011 Page 4All Models Require AssumptionsClosed-for m models often require assumptions for tract ability(so that the modeller can solve the mathematical problem).Simulation models also require assumptions (to simplify reality).But the simulation model can be made much closer to realitythan many closed-for m models.The trade-of f: the exact answer to the wrong ques tion (closed-form), or an approximat e answer to the right ques tion(simulation models)— John Tukey (19 15−2000)Need measures of closeness to reality, for sets of time series.< >ABM Colloquium R.E. Mar ks © 2011 Page 5Verification + Validation ≡ Assur anceVerification (or inter nal validity): is the simulation working asyou want it to:— is it “doing the thing right?”Validation: is the model used in the simulation correct?— is it “doing the right thing?”To Ver ify: use a suite of tes ts, and run them every time youchange the simulation code — to ver ify the changes have notintroduced extr a bugs.Perhaps code using a different platfor m, or dock.< >ABM Colloquium R.E. Mar ks © 2011 Page 6ValidationFor whom?With reg ard to what?A good simulation is one that achieves its goals:• to explore• to predict• to explainOr• what is? (i.e. description, positive)• what could be? (i.e. exis t ence, plausibility)• what should be? (i.e. prescr iption, normative)< >ABM Colloquium R.E. Mar ks © 2011 Page 7ValidationIdeall y: compare the simulation output with the real world.But :1. stoc hastic ∴ complet e accord is unlikel y, and the distr ibution ofdif ferences is usually unknown2. pat h-dependence: output is sensitive to initialconditions/par ameter s3. tes t for “retrodiction”: reversing time in the simulation; or: tes tfrom a past dat e to the present : calibr ate wit h his t ory4. what if the model is correct, but the input data are bad?Use Sensitivity Analysis, to ask:• robus tness of the model to assumptions made• which are the crucial initial conditions/paramet ers?use: randomised Monte Carlo, with many runs.< >ABM Colloquium R.E. Mar ks © 2011 Page 8Consider the following historical market data:Pr ice$/lbQuantitylb/week200040002.003.0020 40 60 80Figure 1: Weekl y Pr ices and Sales (Source: Midgley et al. 1997)(Coloured lines: Folger s, Maxwell House, Hills Bros, CFON)< >ABM Colloquium R.E. Mar ks © 2011 Page 9Stylised Facts of the Market Behaviour• Much movement in prices and quantities of four brands —a riv alrous dance.• Patt ern: high price (and low quantity) punctuated by lowpr ice (and high quantity).• Another four brands: stable prices and quantitiesQues tions:What is the cause of these patter ns?— shif ts in brand demand?— reactions by brands?— actions by the supermarket chain?— unobser ved marketing actions?< >ABM Colloquium R.E. Mar ks © 2011 Page 10Explanations?Int eractions of profit-maximising agents, plus ext ernal orint ernal fact ors → via a model → behaviourSimilar (qualit ativel y or quantit ativel y) to the brands ’behaviour s of pricing and sales.No te: assuming profit-maximising (or purposeful) agents meansthat we are not simpl y cur ve-fitting or description using D.E.s.Going beyond the riv alrous dance.< >ABM Colloquium R.E. Mar ks © 2011 Page 11Validation and MeasurementQ: how can we measure the degree of similar ity of two sets oftime-ser ies?One: the historical record of the riv alrous dance among theseller s in an oligopoly, whileThe other: the output from a (agent-based) simulation model ofthe market, where each seller agent prices this week as afunction of the state of the market last week (or earlier).Q: how can we output validate our model agains t his t ory?Or : how can we der ive a deg ree of confidence in the modeloutput?< >ABM Colloquium R.E. Mar ks © 2011 Page 12The Issue: Heterogenous Agents and Time-ser ies Pr icesTwo reasons to compare such model output agains t his t ory:1. To choose better paramet er values, to “calibr ate” or(more for mally) “estimat e” the model agains t thehis t orical record.2. To measure how closel y the output reflects history, tovalidat e the model.We are int eres t ed in the second, having used machine learning(t he Genetic Algorit hm) to der ive the model paramet ers in orderto improve each agent ’s weekl y profits (instead of fitting tohis t ory) in our agent-based model.Figure 1 shows his t orical data from a U.S. supermarket chain’ssales of (heterogeneous) brands of sealed, ground coffee, byweek in one city (Midgley et al. 1997).< >ABM Colloquium R.E. Mar ks © 2011 Page 13His t orical Data: Market Prices and Volumes$/lblb/week200040002.003.0020 40 60 80Figure 1: Weekl y Sales and Prices (Source: Midgley et al. 1997)< >ABM Colloquium R.E. Mar ks © 2011 Page 14Dichotomous Price Par titioning of the Historical DataTo handle the curse of dimensionality.$/lb2.003.0020 40 60 80Figure 2: Partitioned Historical Weekl y Pr ices of the Four Brands< >ABM Colloquium R.E. Mar ks © 2011 Page 15A Model of Str ategic Inter actionWe assume that the price Pbw of brand b in weekw is afunction of the state of the market Mw at weekw , where Mwin turn is the product of the weekl y pr ices Sw of all brands overseveral week s:Pbw = fb(Mw ) = fb(Sw ×Sw−1 ×Sw−2 . . .)Earlier in the research prog ram undertaken wit h David Midgleyet al., we used the Genetic Algorit hm to search for “better”(i.e. more profit able) br and-specific mappings, fb , from marketst ate to pricing action.And derived the paramet ers of the model, and derived itssimulat ed behaviour, as time-ser ies patt erns (below).< >ABM Colloquium R.E. Mar ks © 2011 Page 16The State Similar ity Measure (SSM)The SSM derives the distance between two sets of time-series,by calculating the sum of absolute dif ferences in observedwindow states between the two set, so what?First, the great er the sum, the more dis tant the two sets oftime-ser ies.Second, we can calculate the maximum size of the summeddif ference: zero int ersection between the two sets (no states incommon) implies a measure of 2 ×S where S is the number ofpossible window states, from the data.Third, we can derive some statis tics to show that any pair ofsets in not likel y to include random series (below).< >ABM Colloquium R.E. Mar ks © 2011 Page 17Example of a Simulated Oligopoly (Mark s et al. 1995)Simulating riv alry between the three asymmetric brands: 1, 2,and 5, Folger s, Maxwell House, and Chock Full O Nuts.Week$/lb10 20 30 40 501.522.53Figure 3: Example of a Simulated Oligopoly (Mark s et al. 1995)< >ABM Colloquium R.E. Mar ks © 2011 Page 18Dis tances Between History and Three Runs (Brands 1, 2, 5)His t ory Run 11 Run 26a Run 26bHis t ory 0  82* 68 68Run 11 82* 0 66 60Run 26a 68 66 0 30Run 26b 68 60 30 0Table: Dis tances Between History and Three Runs (Brands 1, 2, 5)(* : cannot reject the null at the 5% level)Here, S , the maximum number of states = 48, so the maximumdis tance apart is 96. The three Runs are closer to each otherthan to the Historical Data; Runs 26a and 26b are ver y close,onl y 30/96 = 31.25% apart.< >ABM Colloquium R.E. Mar ks © 2011 Page 19Testing for RandomnessSSMsCMF40 60 8000.250.50.751 H,11H,a11 ,a11 ,ba,b• • • • • • • ••••••••• •The red lines are the CMF of pairs of sets of random series (3ser ies, 48 observations) from 100,000 Monte Carlo paramet erboots traps.The one-sided c.i. at 1% corresponds to a SSM of 76, and at 5%80.Cannot reject the null hypothesis (random sets) for Historicaldat a and Run 11; reject the null (random) hypothesis for allot her pairs.< >ABM Colloquium R.E. Mar ks © 2011 Page 20Conclusions — the State Similar ity MeasureThis measure, the State Similarity Measure (SSM), is sufficient toallow us to put a number on the degree of similar ity betweentwo sets of time-series which embody dynamic responses.Such a metric is necessary for scoring the distance between anytwo such sets, which previousl y was unavailable.Here, the SSM has been developed to allow us to measure theextent to which a simulation model that has been chosen onsome other crit erion (e.g. weekl y profit ability) is similar tohis t orical sets of time-series.The SSM will also allow us to measure the distance betweenany two sets of time-series and so to estimat e the par ameter s,or to help calibrat e a model agains t his t ory.< >ABM Colloquium R.E. Mar ks © 2011 Page 21References[1] March J. and Lave C. (19 75), Introduction to Models in the SocialSciences, (New York: HarperCollins).[2] Marks R.E. (2010), “Compar ing two sets of time-series: the StateSimilar ity Measure,” In 2010 Joint Statis tical Mee tings Proceedings —St atis tics: A Key to Inno vation in a Data-centric World, Statis ticalComputing Section. (Alexandr ia, VA: Amer ican St atis tical Association),pp. 539−551.http://www.agsm.edu.au/bobm/papers/asap.pdf[3] Marks R.E. (2012) “Mont e Carlo,” in The Palg rave Encyclopædia ofStrategic Management, edit ed by David Teece and Mie Augier, (London:Palg rav e), for thcoming.http://www.agsm.edu.au/bobm/papers/MonteCarloPESM-longer.pdf[4] Marks R.E., Midgley D.F., and Cooper L.G. (1995) “Adaptive behaviorin an oligopoly,” Evolutionar y Algorit hms in Management Applications,ed. by J. Biet hahn and V. Nissen, (Berlin: Springer-Verlag), pp.225−239.[5] Midgley, D.F., Marks, R.E., and Cooper, L.G. (1997) “Breedingcompetitive str ategies,” Management Science, 43(3): 257−275.http://www.santafe.edu/media/workingpapers/95-06-052.pdf<

Choosing the Right Model

https://ses.library.usyd.edu.au/bitstream/2123/7732/5/BobMarksPresentation.pdf

Choosing the Right Model

Abstract

Similar works

Full text

Available Versions

Sydney eScholarship