LOGGING AND BOOKKEEPING – TROUBLESHOOTING GUID

Sustr, Zdenek

English

ZENODO

EUROPEAN MIDDLEWARE INITIATIVELOGGING AND BOOKKEEPING –TROUBLESHOOTING GUIDEDocument version: 1.4.12EMI Component Version: 4.xDate: March 29, 20131/9TITLE:Logging and Bookkeeping – Troubleshooting GuideDate: March 29, 2013This work is co-funded by the European Commission as part of the EMI project under Grant AgreementINFSO-RI-261611.Copyright c© Members of the EGEE Collaboration. 2004. See http://www.eu-egee.org/partners/ fordetails on the copyright holders.Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file exceptin compliance with the License. You may obtain a copy of the License athttp://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, software distributed under the Licenseis distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, ei-ther express or implied. See the License for the specific language governing permissions andlimitations under the License.2/9TITLE:Logging and Bookkeeping – Troubleshooting GuideDate: March 29, 2013CONTENTSL&B DOCUMENTATION AND VERSIONS OVERVIEW 41 TROUBLESHOOTING 62 FAQ—FREQUENTLY ASKED QUESTIONS 72.1 JOB IN STATE ‘RUNNING’ DESPITE HAVING RECEIVED THE ‘DONE’ EVENT FROMLRMS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72.2 WMS CANNOT PURGE JOBS OR PERFORM OTHER PRIVILEGED TASKS . . . . . . . 72.2.1 FOR L&B VERSION 3.0.11 OR HIGHER, USING YAIM . . . . . . . . . . . . . . . 72.2.2 FOR ALL VERSIONS OF L&B, USING YAIM . . . . . . . . . . . . . . . . . . . . 72.2.3 FOR L&B VERSION 2.1 OR HIGHER, WITHOUT YAIM . . . . . . . . . . . . . . . 72.3 L&B SERVER THROWS “DUPLICATE ENTRY . . . FOR KEY 1” ERRORS . . . . . . . . . 83/9TITLE:Logging and Bookkeeping – Troubleshooting GuideDate: March 29, 2013L&B DOCUMENTATION AND VERSIONS OVERVIEWThe Logging and Bookkeeping service (L&B for short) was initially developed in the EU DataGrid project1as a part of the Workload Management System (WMS). The development continued in the EGEE, EGEE-II and EGEE-III projects,2 where L&B became an independent part of the gLite3 middleware [1], and thenin the EMI Project.4The complete L&B Documentation consists of the following parts:• L&B User’s Guide – this document. The User’s Guide explains how to use the Logging andBookkeeping (L&B) service from the user’s point of view. The service architecture is describedthoroughly. Examples on using L&B’s event logging commands to log user tags and change jobACLs are given, as well as L&B query and notification use cases.• L&B Administrator’s Guide [2]. The Administrator’s Guide explains how to administer the Loggingand Bookkeeping (L&B) service. Several deployment scenarios are described together with theinstallation, configuration, running and troubleshooting steps.• L&B Developer’s Guide [3]. The Developer’s Guide explains how to use the Logging and Book-keeping (L&B) service API. Logging (producer), querying (consumer) and notification API as wellas the Web Services Interface is described in details together with programing examples.• L&B Test Plan [4]. The Test Plan document explains how to test the Logging and Bookkeeping(L&B) service. Two major categories of tests are described: integration tests (include installation,configuration and basic service ping tests) and system tests (basic functionality tests, performanceand stress tests, interoperability tests and security tests).The following versions of L&B service are covered by these documents:• L&B version 4.0: included in the EMI-3 Monte Bianco release• L&B version 3.2: included in the EMI-2 Matterhorn release• L&B version 3.1: an update for the EMI-1 Kebnekaise release• L&B version 3.0: included in the EMI-1 Kebnekaise release• L&B version 2.1: replacement for L&B version 2.0 in gLite 3.2• L&B version 2.0: included in gLite 3.2 release• L&B version 1.x : included in gLite 3.1 releaseL&B packages can be obtained from two distinguished sources:• gLite releases: gLite node-type repositories, offering a specific repository for each node type suchas glite-LB. Only binary RPM packages are available from that source.1http://eu-datagrid.web.cern.ch/eu-datagrid/2http://www.eu-egee.org/3http://www.glite.org4http://www.eu-emi.eu/4/9TITLE:Logging and Bookkeeping – Troubleshooting GuideDate: March 29, 2013• emi releases: EMI repository5 or EGI’s UMD repository,6 offering all EMI middleware packagesfrom a single repository. There are RPM packages, both source and binary, the latter relying onEPEL for dependencies. There are also DEB packages (starting with EMI-2) and tar.gz archives.Note: Despite offering the same functionality, binary packages obtained from different repositories differand switching from one to the other for upgrades may not be altogether straightforward.Updated information about L&B service (including the L&B service roadmap) is available at the L&Bhomepage: http://egee.cesnet.cz/en/JRA1/LB5http://emisoft.web.cern.ch/emisoft/6http://repository.egi.eu/5/9TITLE:Logging and Bookkeeping – Troubleshooting GuideDate: March 29, 20131 TROUBLESHOOTINGPlease, report usage problems via the GGUS support system athttps://gus.fzk.de/index.htmlApparent software bugs are tracked in Savannah athttps://savannah.cern.ch/bugs/?func=additem&group=jra1mdwWhen submitting a L&B specific problem/bug, the following information might be useful:• version of software used (all glite-lb-* packages installed at your site)• description of the problem, the jobId, addresses of all relevant machines (L&B server, ...), envi-ronment variables set, etc.• output from the following commands:glite-wms-job-status <jobId>glite-wms-job-logging-info -v 3 <jobID>• information on your proxy:voms-proxy-info -debug -allor, if VOMS client is not installed,grid-proxy-info -debug• sometimes additional information can be found in the output from the commandsglite-lb-job_status <jobId>glite-lb-job_log <jobId>that should be available in the $GLITE_LOCATION/examples directory,• appropriate excerpts from the logs on the server side are also highly appreciated, please tell youradministrator to look at the Troubleshooting section in LB Administrator’s Guide [2] to follow thesteps there and provide you the necessary information.Users are encouradged to send developers all non-bugs comments and questions by email to addressegee-jra1@lindir.ics.muni.cz.6/9TITLE:Logging and Bookkeeping – Troubleshooting GuideDate: March 29, 20132 FAQ—FREQUENTLY ASKED QUESTIONS2.1 JOB IN STATE ‘RUNNING’ DESPITE HAVING RECEIVED THE ‘DONE’ EVENT FROM LRMSJobs stay in state Running until a Done event is received from the workload management system. Doneevents from local resource managers are not enough since the job in question may have been resubmittedin the meantime.2.2 WMS CANNOT PURGE JOBS OR PERFORM OTHER PRIVILEGED TASKSIn short, WMS has not been given adequate permissions when configuring the L&B server. You need tomodify your configuration and restart the server:2.2.1 FOR L&B version 3.0.11 or higher, USING YAIMModify your siteinfo.def, specifying the DN of your WMS server in YAIM parameter GLITE_LB_WMS_DN;for instance:GLITE_LB_WMS_DN=/DC=cz/DC=cesnet-ca/O=CESNET/CN=wms01.cesnet.czThen rerun YAIM: /opt/glite/yaim/bin/yaim -c -s site-info.def -n glite-LBThis will give your WMS exactly the right permissions to carry out all required operations.2.2.2 FOR ALL VERSIONS OF L&B, USING YAIMModify your siteinfo.def, specifying the DN of your WMS server in YAIM parameter GLITE_LB_SUPER_USERS;for instance:GLITE_LB_SUPER_USERS=/DC=cz/DC=cesnet-ca/O=CESNET/CN=wms01.cesnet.czThen rerun YAIM: /opt/glite/yaim/bin/yaim -c -s site-info.def -n glite-LBThis will give your WMS adequate rights to perform its operations and requests (running purge, queryingfor statistics, etc.) but it will also grant it additional administrator rights (such as granting job ownership).On newer installations, the method explained in section 2.2.1 is preferable.2.2.3 FOR L&B version 2.1 or higher, WITHOUT YAIML&B’s authorization settings can be found in file [/opt/glite]/etc/glite-lb/glite-lb-authz.confPermit actions PURGE, READ_ALL and GET_STATISTICS for your WMS and restart the L&B server. This willlead to results equivalent to 2.2.1. For instance, change the adequate sections in glite-lb-authz.confto:action "READ_ALL" {rule permit {subject = "/DC=cz/DC=cesnet-ca/O=CESNET/CN=wms01.cesnet.cz"7/9TITLE:Logging and Bookkeeping – Troubleshooting GuideDate: March 29, 2013}}action "PURGE" {rule permit {subject = "/DC=cz/DC=cesnet-ca/O=CESNET/CN=wms01.cesnet.cz"}}action "GET_STATISTICS" {rule permit {subject = "/DC=cz/DC=cesnet-ca/O=CESNET/CN=wms01.cesnet.cz"}}2.3 L&B SERVER THROWS “DUPLICATE ENTRY . . . FOR KEY 1” ERRORSThe L&B server will occasionally report errors through syslog saying,ERROR CONTROL - ... : File exists (Duplicate entry ’...’ for key 1)These error messages are caused by certain portions of code that take care of storing database recordsfor keys, which may or may not already exist in the database, and do so by trying to insert the record first(hence the key violation) and modify the record if the insert fails. This has the unfortunate side effect ofthe unsuccessful insert being reported as an ERROR in the logging output.Unless you are experiencing trouble with the specific data entity7 referenced in the error message, it issafe to disregard.7Usually a Job ID8/9TITLE:Logging and Bookkeeping – Troubleshooting GuideDate: March 29, 2013REFERENCES[1] E. Laure, F. Hemmer, F. Prelz, S. Beco, S. Fisher, M. Livny, L. Guy, M. Barroso, P. Buncic, P. Kunszt,A. Di Meglio, A. Aimar, A. Edlund, D. Groep, F. Pacini, M. Sgaravatto, and O. Mulmo. Middleware forthe next generation grid infrastructure. In Computing in High Energy Physics and Nuclear Physics(CHEP 2004), 2004.[2] A. Krˇenek et al. L&B Administrator’s Guide. http://egee.cesnet.cz/en/JRA1/LB/.[3] A. Krˇenek et al. L&B Developer’s Guide. http://egee.cesnet.cz/en/JRA1/LB/.[4] A. Krˇenek et al. L&B Test Plan. http://egee.cesnet.cz/en/JRA1/LB/.9/9

Middleware for the next generation grid infrastructure.

LOGGING AND BOOKKEEPING, TROUBLESHOOTING GUIDE

https://zenodo.org/record/6843/files/LBTG.pdf

LOGGING AND BOOKKEEPING, TROUBLESHOOTING GUIDE

Abstract

Similar works

Full text

Available Versions

ZENODO