Appavoo, Jonathan

Eldridge, Schuyler

Homer, Steve

Joshi, Ajay

Seltzer, Margo I.

Waterland, Amos

Zhao, Katherine

Harvard University - DASH 

Programmable Smart Machines:A Hybrid Neuromorphic approachto General Purpose ComputationThe Harvard community has made thisarticle openly available.  Please share  howthis access benefits you. Your story mattersCitation Appavoo, Jonathan, Amos Waterland, Schuyler Eldridge, KatherineZhao, Ajay Joshi, Steve Homer, Margo Seltzer. 2014. "ProgrammableSmart Machines: A Hybrid Neuromorphic approach to GeneralPurpose Computation." In Proceedings of NeuromorphicArchitectures (NeuroArch) Workshop at 41th InternationalSymposium on Computer Architecture (ISCA-41), Minneapolis, MN,June 14-18, 2014.Citable link http://nrs.harvard.edu/urn-3:HUL.InstRepos:22719512Terms of Use This article was downloaded from Harvard University’s DASHrepository, and is made available under the terms and conditionsapplicable to Open Access Policy Articles, as set forth at http://nrs.harvard.edu/urn-3:HUL.InstRepos:dash.current.terms-of-use#OAPProgrammable Smart Machines:A Hybrid Neuromorphic approach to General Purpose ComputationJonathan Appavoo1, Amos Waterland2, Katherine Zhao1, Schuyler Eldridge1, Ajay Joshi1, Margo Seltzer2,and Steve Homer11Boston University {jappavoo,kzhao,schuye,joshi,homer}@bu.edu2Harvard University {apw,margo}@seas.harvard.eduWe describe our research on integrating neuromorphic de-vices into a new computing system, where the system is notapproximate by default and its operation automatically im-proves as a function of its resources and “experience”. Wefocus on communicating key insights and observations andthen sketch our research agenda. We use the term neuromor-phic broadly to mean any hardware specifically designed formachine learning or inspired by biological neural networks.There appear to be two basic modes of computing: (1) bylogic, and (2) by pattern recognition. To date, there has gener-ally been a gulf between these two modes, with the exceptionof work on approximate computing [2, 4, 1]. We seek to de-fine and exploit a closer bridge between the two. We defineProgrammable Smart Machines (PSMs) as hybrid computingsystems that behave as programmed but transparently learnand automatically improve their operation.The PSM research agenda explores the use of neuromorphicdevices in addressing the challenges set forth by Michie in theNature ’68 paper that introduced memoization [3]:“It would be useful if computers could learn fromexperience and thus automatically improve the ef-ficiency of their own programs during execution...When I write a clumsy program for a contempo-rary computer a thousand runs on the machine donot re-educate my handiwork. On every execution,each time-wasting blemish and crudity, each need-less test and redundant evaluation, is meticulouslyreproduced.”Michie observed that computing the output of any functiongiven its input can be done either by calculation (rule) ormemory recall (rote), and concluded that:“on each given occasion proceed either by rule, orby rote, or by a blend of the two, solely as dictatedby the expediency of the moment ... rule versus rotedecisions shall be handled by the machine behindthe scenes.”To address these challenges, we integrate pattern recognitioninto the execution model of a general purpose computer. Ourgoal is a computing system that (1) automatically learns and re-calls patterns in its operation to achieve correctness-conservingspeedups by eliminating large fractions of its conventional in-struction execution stream, and (2) recognizes and exploitsincreasingly complex patterns as a function of the size of itsneuromorphic hardware resources.Since the PSM research agenda hybridizes conventionalexecution with learning, prediction, and auto-associative mem-ory recall, our work is well-positioned to exploit efficienthigh-capacity neuromorphic hardware.Execution Model Two key observations give rise to thePSM research agenda: (1) execution of a von Neumann com-puter naturally produces a vector-valued binary signal fromthe dynamics of the architected state (all memory and regis-ters), and (2) a forward update of the architected state can bedone either by rule (execution) or rote (lookup in a cache ofpreviously executed complete state traces).These observations link our research agenda to neural net-works in specific and machine learning and approximate com-puting in general. Learning, prediction and recall are at theheart of our design for a computing system that automaticallyimproves as a function of its hardware resources. Neural net-works are increasingly pervasive in domains such as imageand video processing. Both machine learning researchers andbiologists have shown that large neural networks can automati-cally find and recall complex structure from patterns in binarydata.By coding zeros and ones as e.g. black and white, the com-plete n bits of instantaneous architected state of a computercan be represented as a 1×n black and white image, illustratedas a single row of pixels in Figure 1. Execution can then berepresented as a video of the architected state as it evolveswith each instruction execution. This gives a natural way tointerface computer execution to neuromorphic devices whosevalue has already been proven in the domains of binary-valuedimage and video processing. Neural networks implemented inneuromorphic hardware can be used to learn, recognize andrecall patterns in execution as predictions over future frames.Recognizable clips in “execution movies” can be interpretedas repeated computation that maps the state of the systemrepresented by the first frame of the clip to the system stateassociated with the last frame. As such, if one is able torecognize, learn and predict reoccurring clips in executionmovies then one has captured and compressed patterns inexecution over all aspects of the program and its data.In this fashion a neuromorphic mechanism can store knowl-edge of execution in the form of recognized patterns. Themechanism can provide feedback on the current executionin the form of predicted frames (states of the system). Fora PSM to be effective, it must use this feedback to generatefuture states of the system that are known to ensue from the10Execution as n bit "video" signal of system state...... n-1...... ...One row of pixels is a frame of execution and describes entire system state at a particular step of instruction execution...One column of pixels for each bit of system state (register, ram and memory-mapped I/O devices)initial statefuture statecurrent stateExecution "video" proceeds from top to bottomFigure 1: Execution as n-bit “video” signal of system state.current state. One approach is to view the neuromorphic mech-anisms as part of a two stage process. In aggregate the neuro-morphic devices encapsulate a massive, first stage, long-termknowledge-base of execution patterns of various scales andcomplexity. When fed information from the current execution,they generate predicted start states of a clip of computationthat it believes will follow based on the patterns it knows. Thisfeedback must be converted into an exact match to a statethat will deterministically follow from the current state of thecomputer or it must be discarded.1For this conversion a second "resolution" stage can be em-ployed. The idea is to use a smaller set of traditional resourcesto maintain a short term cache of start to end state pairs thatis populated based on the neuromorphic predictions. Ratherthan directly updating the state of the computer with the neuro-morphic prediction the prediction is used to generate a cacheentry that maps a set of precise start states to a set of resultingend states. To do this the resolution stage allocates traditionalcores and memory to determine future states. Given that theprediction defines a complete system state a traditional paral-lel thread of computation can be used to generate the clip ofcomputation that it represents simply by executing forward.This allows the prediction to be converted into a start and endstate pair. This pair is then stored in a cache that is consultedas the computation proceeds. If a match is found the systemcan directly be advanced to the associated end state. In currentand future work we explore more subtle resolution and cachematching procedures that allow us to generalize and reusecache entries thus eliminating cases in which execution is re-quired. Details of our current resolution process are beyondthe scope of this paper (see Waterland et al. [5]).We see the PSM approach as providing a means for integrat-ing neuromorphic devices into a general purpose computingsystem that maintains a simple precise and deterministic pro-gramming model. We also see PSMs as a way of allowingarbitrary programs to transparently benefit from learning whenthere is sufficient redundancy in behavior either due to the com-1PSM computing systems also enable an error-bounded form of approxi-mate computing, but we leave this for future work.putation being well structured, or the data that it is operatingon, or both. Finally we believe that the PSM model has theadvantage that it can allow for “experience” from one systemto speed up other systems via a shared hierarchical approach.That is to say while some of the neuromorphic devices couldbe local to a system, others could be remotely hosted andshared across several machines.To date we have been using a combination of softwaremachine learning agents to learn and predict each value ofthe system’s state. In particular, we combine the results oftwo widely-used learning algorithms: logistic regression andlinear regression. These two approaches are closely related tosingle layer neural networks. In the case of logistic regressionpredictors, each predicted state bit is produced as a linearcombination of all bits values plus a bias weight. The weightsare updated using a standard gradient descent online updatemethod. Our linear regression predictors also operate on non-overlapping 32-bit receptive fields interpreted as real values.These two approaches are chosen to minimize the softwareoverheads and to permit an online learning model.Next Steps We will evaluate the use of hardware neuralnetwork accelerators in place of software implementations.Our goal is to quantify the implications of using more complexmulti-layer networks within the PSM model. Speedup in aPSM is a function of the prediction accuracy and the scalesand complexities of the patterns recognized. As has beenobserved in vision based applications, deeper networks permitimprovements on all of these fronts.To proceed we will first break down our PSM model intoa version that is friendly to offline training given that mostcurrent hardware neural network accelerators follow this ap-proach. To do this we will assume a model where a hardwareaccelerator is synthesized to a particular multi-layer percep-tron network configuration and then its weights can be loadedor reloaded as needed. As such we will conduct training foreach application that we intend to run and store the attendantweights with the application binaries. When we launch anapplication we will load the neural network accelerators withthe associated weights. Given this setup we will evaluate arange of applications, application-specific workloads and arange of network complexities as dictated by a range of hard-ware resource specifications. To do this we will build uponcomponents from our independent prior work into the designand implementation of a multi-layer preceptron acceleratorarchitecture [1]. We envision an architecture that incorporatesnetwork accelerators operating alongside traditional compu-tation units and provides a path to deliver More-than-Mooreperformance improvements in spite of the harsh landscape ofnanoscale CMOS. Interestingly, such an architecture has thepotential to leverage both precise and approximate regimes ofcomputation for performance improvements. We will migrateportions of the online learning model from our prior softwareimplementation to hardware and investigate increasingly ad-vanced architectures (e.g. HMAX) for the ability to discernmore complex patterns.2References[1] S. Eldridge, F. Raudies, D. Zou, and A. Joshi, “Neural network-basedaccelerators for transcendental function approximation,” in Proceedingsof the Great Lakes Symposium on VLSI (GLSVLSI), 2014.[2] H. Esmaeilzadeh, A. Sampson, L. Ceze, and D. Burger, “Neural acceler-ation for general-purpose approximate programs,” in Microarchitecture(MICRO), 2012 45th Annual IEEE/ACM International Symposium on,Dec 2012, pp. 449–460.[3] D. Michie, “"Memo" Functions and Machine Learning,” Nature,vol. 218, no. 5136, pp. 19–22, Apr. 1968. [Online]. Available:http://dx.doi.org/10.1038/218019a0[4] M. Samadi, D. A. Jamshidi, J. Lee, and S. Mahlke, “Paraprox: Pattern-based approximation for data parallel applications,” in Proceedingsof the 19th International Conference on Architectural Support forProgramming Languages and Operating Systems, ser. ASPLOS ’14.New York, NY, USA: ACM, 2014, pp. 35–50. [Online]. Available:http://doi.acm.org/10.1145/2541940.2541948[5] A. Waterland, E. Angelino, R. P. Adams, J. Appavoo, and M. Seltzer,“ASC: Automatically Scalable Computation,” in Proceedings ofthe 19th International Conference on Architectural Support forProgramming Languages and Operating Systems, ser. ASPLOS ’14.New York, NY, USA: ACM, 2014, pp. 575–590. [Online]. Available:http://doi.acm.org/10.1145/2541940.25419853

Programmable Smart Machines: A Hybrid Neuromorphic approach to General Purpose Computation

https://dash.harvard.edu/bitstream/1/22719512/1/88829472.pdf

Programmable Smart Machines: A Hybrid Neuromorphic approach to General Purpose Computation

Abstract

Similar works

Full text

Available Versions

Harvard University - DASH