Brock-Nannestad, Laust

DelSignore, John

Karlsson, Sven

Mohror, Kathryn

Squyres, Jeffrey M.

Online Research Database In Technology

                       General rights Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.  • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal   If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim.     Downloaded from orbit.dtu.dk on: Dec 20, 2017Exposing MPI Objects for DebuggingBrock-Nannestad, Laust; DelSignore, John; Squyres, Jeffrey M.; Karlsson, Sven ; Mohror, KathrynPublication date:2014Document VersionPublisher's PDF, also known as Version of recordLink back to DTU OrbitCitation (APA):Brock-Nannestad, L., DelSignore, J., Squyres, J. M., Karlsson, S., & Mohror, K. (2014). Exposing MPI Objectsfor Debugging. Poster session presented at International Conference for High Performance Computing,Networking, Storage and Analysis, SC14, New Orleans, United States.Lawrence LivermoreNational LaboratoryLawrence LivermoreNational LaboratoryExposing MPI Objects for DebuggingLaust Brock-Nannestad 1 John DelSignore 2 Jeffrey M. Squyres 3 Sven Karlsson 1 Kathryn Mohror 41Technical University of Denmark 2Rogue Wave Software, Inc. 3Cisco Systems, Inc. 4Lawrence Livermore National LaboratoryLawrence LivermoreNational LaboratoryLawrence LivermoreNational LaboratoryMotivation•Debugging MPI applications is difficult as developerscannot inspect the state of the MPI runtime•The MPI Tools Working Group has proposed astandard interface between the debugger and MPIruntime [1]•With this interface, debuggers can easily present MPIstate to the developerContributions•Support for MPI handle introspection in theTotalView debugger•A reference introspection implementation inOpen MPI•A demonstration of simplified MPI debuggingDebugger access to MPI state•MPI implementation provides functions forintrospection as a library• Introspection functions extract data from MPIhandles•Debugger and library interact through mutualcallback functions [2]Figure 1 : Example use case to fetch communicator information.1 Debugger queries a communicator2 Library requests raw data from process3 Debugger extracts raw data from process4 Library parses raw data and returns standardizedcommunicator information5 Debugger presents data to developerComplex MPI Debugging Session with a DebuggerBrowse'internal'Handle'data'structure'Name'of'communicator'Rank'Flags'need'to'be''decoded'manually'Locate'ﬁrst'process''in'communicator'Here'we'ﬁnd'name'of'host'A?ach'debugger''to'applica@on'Con@nue'to'internal''process'group'data''structure'And'so'forth…'Figure 2 : Best case scenario for existing MPI debugging. Developer must understand internal datastructures. For closed source MPI implementations, even this is not possible.Simplified Debugging with MPI Handle Introspection in TotalViewDebugger uses thishandle for furtherqueries on this communicator.Introspection library allocates work memory using callbacks indebugger.Function callData read or event Interaction between Debugger, Introspection library, and Application ProcessRepeated for all relevant information about communicator Foo in Process BarFig. 3. Sequence diagram for querying a communicator.1 d1.<> . mpidbgLoaded MPI s u p p o r t l i b r a r y / g / g90 / l a u s t b n / l o c a l / l i b /3 openmpi / l i bomp i dbg mp ihand l e s . so :Open MPI hand l e i n t e r p r e t a t i o n s u p p o r t f o r p a r a l l e l5 d ebugge r s compi l ed on Sep 5 20147 F i n i s h e d l o a d i n g MPI i n t r o s p e c t i o n s u p p o r t .9 d1.<> d focu s p2p2.<11 p2.<> . mpidbgdumpName Handle13 MPI COMM WORLD 0x6028a0MPI COMM SELF 0 x2aaaab01aa0015 MPI COMM PARENT 0 x2aaaab01a9e0MPI COMM NULL 0 x2aaaab01a3e017p2.<> . mpidbgquery b a s i c 0 x6028a019 Query ing communica tor 0 x6028a0 i n p r o c e s s 0x4878780Communicator : MPI COMM WORLD21 Rank : 0S i z e : 423 F l ag ValueMPIDBG COMM INFO PREDEFINED True25 MPIDBG COMM INFO CARTESIAN Fa l s eMPIDBG COMM INFO GRAPH Fa l s e27 MPIDBG COMM INFO TOPO REORDERED Fa l s eMPIDBG COMM INFO INTERCOMM Fa l s e29 { . . . }Query was s u c c e s s f u lFig. 4. A TotalView command line debugging session. The implementation iswork in progress and can currently fetch and decode MPI_Comm information.Figure 4 shows a debugging session. It starts by loadingthe debug library and then performs two queries. The firstquery shows Open MPI’s mapping of internal communicators.The second query provides basic information on the built-inMPI_COMM_WORLD communicator.IV. CONCLUSIONS AND FUTURE WORKWe have successfully implemented the proposed HandleIntrospection interface in TotalView and Open MPI.The benefit for MPI developers is clear. Introspectionenables high level debugging of MPI applications withoutdeep knowledge of the MPI library implementation. For MPIimplementation vendors, it decouples the internals from thedebugging and even allows flexible debugging of closed sourceMPI implementations. Further work will include integrationinto the graphical interface of TotalView and support foradditional MPI handle types.ACKNOWLEDGMENTSThis work (LLNL-CONF-660001) was performed underthe auspices of the U.S. Department of Energy by LawrenceLivermore National Laboratory under Contract DE-AC52-07NA27344.REFERENCES[1] “MPI Forum Tools Working Group,” https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Tools, September 2014.Only'show'relevant'data,'not'raw'data'structures'Implementa@on'deﬁned''structures'such'as'bit''ﬁelds'and'types'are''automa@cally'decoded'Figure 3 : Debugging with introspection support thorugh the TotalView command line. TotalView displays the status of acommunicator.•MPI implementation traverses data structures – assisted by the Debugger•Developer sees MPI API level informationFuture Work• Integration into TotalView’s graphical user interface•Validation against other MPI implementations•Visualization of MPI communication and processes•Querying MPI handle state from the debugger;filtering and showing handles matching certaincriteriaAcknowledgementsThe authors would like to thank Adam Moody atLawrence Livermore National Laboratory for useful in-sights on MPI debugging.This work (LLNL-POST-658417) was performed un-der the auspices of the U.S. Department of Energyby Lawrence Livermore National Laboratory under Con-tract DE-AC52-07NA27344, and co-funded by the Euro-pean Artemis PaPP Project nr. 295440 and COPCAMSproject nr. 332913.References[1] MPI Tools Working Group.https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/MPI3Tools, October 2014.[2] James Cownie and William Gropp.A Standard Interface for Debugger Access to Message QueueInformation in MPI.In Recent Advances in Parallel Virtual Machine and MessagePassing Interface, pages 51–58. Springer, 1999.Contact InformationLaust Brock-NannestadTechnical University of DenmarkEmail: laub@dtu.dkPhone: +45 4525 9223http://www.compute.dtu.dk/~laub/

Exposing MPI Objects for Debugging

http://orbit.dtu.dk/files/104320135/laub_sc_14_precropped.pdf

Exposing MPI Objects for Debugging

Abstract

Similar works

Full text

Available Versions

Online Research Database In Technology