Session: Open methods in archaeology: how to encourage reproducible research as the default practic

Lipo, Carl P.

Madsen, Mark E.

English

The Open Repository @Binghamton (The ORB)

Binghamton UniversityThe Open Repository @ Binghamton (The ORB)Anthropology Faculty Scholarship Anthropology4-2015Tools for Transparency and Replicability ofSimulation in ArchaeologyMark E. MadsenUniversity of Washington - Seattle CampusCarl P. LipoBinghamton University--SUNY, clipo@binghamton.eduFollow this and additional works at: https://orb.binghamton.edu/anthropology_facPart of the Anthropology CommonsThis Presentation is brought to you for free and open access by the Anthropology at The Open Repository @ Binghamton (The ORB). It has beenaccepted for inclusion in Anthropology Faculty Scholarship by an authorized administrator of The Open Repository @ Binghamton (The ORB). Formore information, please contact ORB@binghamton.edu.Recommended CitationMadsen, Mark E. and Lipo, Carl P., "Tools for Transparency and Replicability of Simulation in Archaeology" (2015). AnthropologyFaculty Scholarship. 20.https://orb.binghamton.edu/anthropology_fac/20Tools for Transparency andReplicability of Simulation inArchaeologyMark E. Madsen and Carl P. Lipo University of Washington, SeattleCalifornia State University at Long BeachSession: Open methods in archaeology: how to encourage reproducible research as the default practiceWhy We SimulateExpress models of social and evolutionary dynamicsUnderstand model outcomesPredict archaeologically relevant patternsCompare archaeological data to the patternsDiﬃcult to demonstrate correctnessHard to manage data, software, parametersHard to separate exploration from rigorousexperimentationWhy Simulation Is HardOur Toolset  Anaconda Scientiﬁc PythonsimuPOP MongoDBGithubGraphvizR and R Studiohttp://continuum.iohttp://simupop.sourceforge.nethttp://www.mongodb.comhttps://github.comhttp://graphviz.orghttp://www.r-project.orgOpen Source ToolsCommercial ResourcesAmazon EC2:  compute clusterAmazon S3:  long-term bulk storagehttps://github.com/mmadsen/seriationcthttps://github.com/mmadsen/experiment-seriationct (-2)Best PracticesEverything lives in a revision control system(Git/Github, Subversion, Mercurial)Experiments and data in separate repositoryfrom code.Production work is templated and scriptedEvery simulation run gets a UniversallyUnique Identiﬁer (UUID)Random seeds are generated beforehand,and stored with all resultsAll components take command lineparameters for ease of scripting and scalingfrom laptop to cloud compute clusters.$ create-experiment-directory.sh \seriationct-9                                                                                                                                                                                                                                                                                                     demo-experiment├── README.md├── bin│   ├── annotate-seriation-output.sh│   ├── build-networkmodel.sh│   ├── build-simulations.sh│   ├── run-seriations.sh│   └── simulation-postprocess.sh├── exported-data│   └── README├── jobs│   └── README├── networks├── rawnetworkmodels├── run-experiment-steps.sh├── sampled-data│   └── README├── seriation-results│   └── README├── seriationct-priors.json├── temporal│   └── README└── xyfiles    └── README9 directories, 14 filesCreating New Experiment├── README.md├── bin│ ├── annotate-seriation-output.sh│   ├── build-networkmodel.sh│   ├── build-simulations.sh│   ├── run-seriations.sh│   └── simulation-postprocess.sh├── jobs│   └── job-seriationct-9-simulations.sh├── rawnetworkmodels│   ├── seriationct-9-full-network.zip│   └── seriationct-9-networkmodel│       ├── build-networkmodel.sh│       ├── seriationct-9-001.gml│       ├── seriationct-9-002.gml│       ├── seriationct-9-003.gml│       ├── seriationct-9-004.gml│       ├── seriationct-9-005.gml├── run-experiment-steps.sh├── sampled-data│   ├── 36acbc00-d441-11e4-b725-b8f6b1154c9b-0-sampled-0.07│   ├── 6aa72822-d443-11e4-bed5-b8f6b1154c9b-0-sampled-0.07├── seriation-results|├── 36acbc00-d441-11e4-b725-b8f6b1154c9b-0-sampled-0.07.txt│   ├── 6aa72822-d443-11e4-bed5-b8f6b1154c9b-0-sampled-0.07│   └── README├── seriationct-priors.jsonExperiment in Progress...Universally Unique IdentifiersInternet RFC 4122: https://www.ietf.org/rfc/rfc4122.txtimport uuid # uuid1 incorporates hardware address and timeunique_id = uuid.uuid1() print unique_id ba3a318a-d4cb-11e4-b4f9-b8f6b1154c9bComponent of all ﬁle namesField in all database recordsPrimary means of tying data elements together{    "simulation_run_id" : "urn:uuid:eaf71706-ce8c-11e4-a9ac-b8f6b1154c9b",    "random_seed" : 2127774500,    "elapsed_time" : 257.4463579654694,    "experiment_name" : "seriationct-1",    "full_command_line" : "sim-seriationct-networkmodel.py -mf 0.0938     --popsize 250 --nm hier-1.zip"}{    "_id" : ObjectId("5514e910544bd6744cae8aec"),    "simulation_run_id" : "urn:uuid:36acbc00-d441-11e4-b725-b8f6b1154c9b",    "random_seed" : 1601673696,    "replication" : 0,    "class_freq" : {        "0-3-4" : 0.6857142857142857,        "2-4-1" : 0.1428571428571428,        "0-4-4" : 0.1714285714285714    },    "simulation_time" : 3000,    "subpop" : "assemblage-33-6",    "mutation_rate" : 0.00668494110834,    "population_size" : 250,    "class_richness" : 3}Simulation MetadataSimulation Output DataIssues with Large ProjectsGithub repositories soft limited to 1G or lessGithub hard limit on ﬁle size 100MBFigshare limits ﬁles to 250MB with free planCurrently compressing some intermediate ﬁlesafter processingMoving some raw DB ﬁles to S3 buckets for longterm storage after extracting analysis dataset"Continuation" repositories with additional analysisWorkaroundshttps://github.com/mmadsen/experiment-seriationct-2SumatraLancethttp://neuralensemble.org/sumatra/ Numerical analysis or simulation projecttracking and replicability toolhttp://ioam.github.io/lancet/ Strong parameter management andexperiment execution libraryOther ToolsWhere We're HeadedSumatra needs ﬁles as "data" capture, extend tohandle database as data store, requires archivalschemeLancet replacing our simple execution scripts andparameter JSON ﬁlesCombination of Sumatra for object management andLancet for simulation control, with UUIDs and randomseeds scripted as in our current exampleRaw data archiving is still a problem -- exploringAmazon Glacier for post-analysis storageThank YouFor more information, templates, etc: mark@madsenlab.orghttp://notebook.madsenlab.org

Tools for Transparency and Replicability of Simulation in Archaeology

https://orb.binghamton.edu/cgi/viewcontent.cgi?article=1016&amp;context=anthropology_fac

Tools for Transparency and Replicability of Simulation in Archaeology

Abstract

Similar works

Full text

Available Versions

The Open Repository @Binghamton (The ORB)