Paganelli, Florido

English

ZENODO

EMI is partially funded by the European Commission under Grant Agreement RI-261611GLUE2 in the the EMI-ES job management interface and in the ARCLIB compute libraryFlorido Paganelli, Lund UniversityEGI Technical Forum, Prague, 17-21 Sept 2012Towards an integrated information system3rd workshop●EMI INFSO-RI-261611Outline● The EMI-ES information model● The Resource document● The Activity document● An LDAP overlook● The arclib client library● A modular plugin system● Retrievers and GLUE2● Sample code●EMI INFSO-RI-261611EMI-ES Functionalities● Five port-types:● ActivityCreation● ActivityManagement● Delegation● ActivityInfo● ResourceInfo●EMI INFSO-RI-261611EMI-ES Information Model● Five port-types:● ActivityCreation● ActivityManagement● Delegation● ActivityInfo● ResourceInfoInformationport typesNeed both for full info!Single ActivityinformationOnly URLs toAccess ResourceInfoand ActivityManagement●EMI INFSO-RI-261611EMI-ES Information Model● Five port-types:● ActivityCreation● ActivityManagement● Delegation● ActivityInfo● ResourceInfoInformationport typesNeed both for full info!Single ActivityinformationOnly a URL toAccess ResourceInfo●EMI INFSO-RI-261611EMI-ES Info Model facts● WS based● Uses the GLUE2 jargon whenever possible● Each PortType represented as aGLUE2 Endpoint● ResourceInfo → Resource Document: Based on GLUE2 XML schema● ActivityInfo → Activity Document: A GLUE2-like document●EMI INFSO-RI-261611Resource Document Structure<GetResourceInfoResponse>  <Services>    <ComputingService>        ID, Type, HealthState, Capability, QualityLevel         CreationTime, Validity, Name, OtherInfo, StatusInfo, Complexity, TotalJobs, RunningJobs, WaitingJobs, StagingJobs, SuspendedJobs,          PreLRMSWaitingJobs, Associations<ComputingEndpoint>      InterfaceName = org.ogf.glue.emies.activitycreation      Capability, URL, ID, ImplementationName, ImplementationVersion, QualityLevel, HealthState,         ServingState, Staging, JobDescription                  CreationTime, Validity, Name, OtherInfo, Technology,InterfaceVersion, InterfaceExtension, WSDL, SupportedProfile, Semantics,      Implementor, HealthStateInfo, StartTime, IssuerCA, TrustedCA, DowntimeAnnounce, DowntimeStart, DowntimeEnd, DowntimeInfo,       TotalJobs, RunningJobs,  WaitingJobs, StagingJobs, SuspendedJobs, PreLRMSWaitingJobs, Associations<AccessPolicy></AccessPolicy></ComputingEndpoint><ComputingEndpoint>   InterfaceName = org.ogf.glue.emies.activitymanagement , Capability, ... </ComputingEndpoint><ComputingEndpoint>   InterfaceName = org.ogf.glue.emies.activityinfo , Capability, …</ComputingEndpoint><Endpoint>   InterfaceName = org.ogf.glue.emies.resourceinfo , Capability, …</Endpoint><Endpoint>   InterfaceName = org.ogf.glue.emies.activityinfo , Capability, …</Endpoint><ComputingManager></ComputingManager><ComputingShare></ComputingShare><Location></Location><Contacts></Contacts><ComputingEndpoint>   InterfaceName = org.nordugrid.gridftpjob , Capability, …</ComputingEndpoint><Endpoint>   InterfaceName = org.nordugrid.ldapng ,Capability, …</Endpoint></ComputingService><Services></GetResourceInfoResponse>●EMI INFSO-RI-261611Resource Document Structure<GetResourceInfoResponse>  <Services>    <ComputingService>        ID, Type, HealthState, Capability, QualityLevel         CreationTime, Validity, Name, OtherInfo, StatusInfo, Complexity, TotalJobs, RunningJobs, WaitingJobs, StagingJobs, SuspendedJobs,          PreLRMSWaitingJobs, Associations<ComputingEndpoint>      InterfaceName = org.ogf.glue.emies.activitycreation      Capability, URL, ID, ImplementationName, ImplementationVersion, QualityLevel, HealthState,         ServingState, Staging, JobDescription                  CreationTime, Validity, Name, OtherInfo, Technology,InterfaceVersion, InterfaceExtension, WSDL, SupportedProfile, Semantics,      Implementor, HealthStateInfo, StartTime, IssuerCA, TrustedCA, DowntimeAnnounce, DowntimeStart, DowntimeEnd, DowntimeInfo,       TotalJobs, RunningJobs,  WaitingJobs, StagingJobs, SuspendedJobs, PreLRMSWaitingJobs, Associations<AccessPolicy></AccessPolicy></ComputingEndpoint><ComputingEndpoint>   InterfaceName = org.ogf.glue.emies.activitymanagement , Capability, ... </ComputingEndpoint><ComputingEndpoint>   InterfaceName = org.ogf.glue.emies.activityinfo , Capability, …</ComputingEndpoint><Endpoint>   InterfaceName = org.ogf.glue.emies.resourceinfo , Capability, …</Endpoint><Endpoint>   InterfaceName = org.ogf.glue.emies.activityinfo , Capability, …</Endpoint><ComputingManager></ComputingManager><ComputingShare></ComputingShare><Location></Location><Contacts></Contacts><ComputingEndpoint>   InterfaceName = org.nordugrid.gridftpjob , Capability, …</ComputingEndpoint><Endpoint>   InterfaceName = org.nordugrid.ldapng ,Capability, …</Endpoint></ComputingService><Services></GetResourceInfoResponse>Might changewrtofficial GLUE2XML schema●EMI INFSO-RI-261611Activity Document Structure<ActivityInfoDocument>        ID    State    IDFromEndpoint    Owner          CreationTime,    Validity,   LocalIDFromManager   JobDescription,    RestartState,   ExitCode,        ComputingManagerExitCode,   Error,   WaitingPosition,         UserDomain,  LocalOwner,   RequestedTotalWallTime,  RequestedTotalCPUTime,         RequestedSlots,      RequestedApplicationEnvironment,   Sdtin,   Stdout,   StdErr,      LogDir,   ExecutionNode,   Queue,   UsedTotalWallTime,   UsedTotalCPUTime,       UsedMainMemory, SubmissionTime,   ComputingManagerSubmissionTime         StartTime,   ComputingManagerEndTime,   EndTime,         WorkingAreaEraseTime,   ProxyExpirationTime,   SubmissionHost,         SubmissionClient,   OtherMessages        StageInDirectory    StageOutDirectory    SessionDirectory    ComputingActivityHistory</ActivityInfoDocument>●EMI INFSO-RI-261611Activity Document Structure<ActivityInfoDocument>        ID    State    IDFromEndpoint    Owner          CreationTime,    Validity,   LocalIDFromManager   JobDescription,    RestartState,   ExitCode,        ComputingManagerExitCode,   Error,   WaitingPosition,         UserDomain,  LocalOwner,   RequestedTotalWallTime,  RequestedTotalCPUTime,         RequestedSlots,      RequestedApplicationEnvironment,   Sdtin,   Stdout,   StdErr,      LogDir,   ExecutionNode,   Queue,   UsedTotalWallTime,   UsedTotalCPUTime,       UsedMainMemory, SubmissionTime,   ComputingManagerSubmissionTime         StartTime,   ComputingManagerEndTime,   EndTime,         WorkingAreaEraseTime,   ProxyExpirationTime,   SubmissionHost,         SubmissionClient,   OtherMessages        StageInDirectory    StageOutDirectory    SessionDirectory    ComputingActivityHistory</ActivityInfoDocument>Adoption of David Meredith's flat GLUE2 XML schema might make this document  a GLUE2 document.●EMI INFSO-RI-261611ARC EMIES LDAP rendering●EMI INFSO-RI-261611GLUE2 arclib● A client library for accessing grid resources.● Provides a simple but powerful API to:● discover services● create and manage jobs● handle data transfers● handle user credentials● Features a modular architecture: it uses plugins to be able to talk to different services.On behalf of Z. Nagy,M. Skou Andersen●EMI INFSO-RI-261611Arclib Retrievers● ServiceEndpointRetrieverclass to query registry services, and receive endpoints of computing elements, storage elements or even other registries.(queries information system index-levels)● TargetInformationRetrieverclass to get the information about a computing element from the computing element itself, about the state of the service, queues, runtime environments, etc.This information is used to do matchmaking of job descriptions and resources.(queries information system local-levels)●EMI INFSO-RI-261611ServiceInformationRetrieversCurrent plugins for querying:● EGIIS (NorduGrid Index)● EMIR (EMI Registry)Planned:● top-level BDII●EMI INFSO-RI-261611TargetInformationRetrieversCurrently it has plugins to query:● LDAP with NorduGrid schema● LDAP with GLUE1.x schema● LDAP with a rendering of GLUE2● WSRF with an XML rendering of GLUE2● EMIES ResourceInfo with an XML rendering of GLUE2●EMI INFSO-RI-261611A sample chunk of code// Create a UserConfig object, and setting the credentialsUserConfig uc;uc.ProxyPath("/path/to/proxy");uc.CACertificatesDirectory("/");// Create some endpoint objects from known computing elementsstd::list<Endpoint> endpoints;endpoints.push_back(Endpoint("hostname.of.endpoint", Endpoint::COMPUTINGINFO));endpoints.push_back(Endpoint("hostname.of.other.endpoint", Endpoint::COMPUTINGINFO));endpoints.push_back(Endpoint("hostname.of.third.endpoint", Endpoint::COMPUTINGINFO));// Parse the job description(s) from a filestd::list<Arc::JobDescription> jobdescriptions;JobDescription::ParseFromFile("filename.jsdl", jobdescriptions);// Query the information about the endpoints, do matchmaking of the job descriptions and submit the jobsArc::SubmissionStatus status = s.BrokeredSubmit(endpoints, jobdescriptions);●EMI INFSO-RI-261611The workflow1) BrokeredSubmit runs all Retrievers on a set of known Endpoints;2) Each Endpoint URL is passed to all kind of enabled retriever plugin3) Every Target and Service retriever, in parallel, tries to contact each Endpoint and tries to communicate; only returns data if communication is successful4) Currently, the data used is that returned by the first retriever to finish. More elaborate techniques can be added.●EMI INFSO-RI-261611Summary● EMI-ES enforces the use of GLUE2● ResourceInfo GLUE2 document● ActivityInfo GLUE2-like document● Documents might change depending on the GLUE2 group decisions● arclib is a versatile plugin-based library● Enables interaction with the current registries and direct interaction with CEs● May be used as the foundation for new clients and gateways●EMI INFSO-RI-261611Thanks!CREDIT 0Insert coin to play GRID!

GLUE2 in the the EMI-ES job management interface and in the ARCLIB compute library

https://zenodo.org/record/6580/files/EMIESarclibGLUE2.pdf

GLUE2 in the the EMI-ES job management interface and in the ARCLIB compute library

Abstract

Similar works

Full text

Available Versions

ZENODO