The Cern Virtual Machine File System is most well known as a distribution mechanism for the WLCG VOs@@ experiment software; as a result, almost all the existing expertise is in installing clients mount the central Cern repositories. We report the results of an initial experiment in using the cvmfs server packages to provide Glasgow-based repository aimed at software provisioning for small UK-local VOs. In general, although the documentation is sparse, server configuration is reasonably easy, with some experimentation. We discuss the advantages of local CVMFS repositories for sites, with some examples from our test VOs, vo.optics.ac.uk and neiss.org.uk

Doherty, Thomas

Skipsey, Samuel

Turner, A G D

Journal of Physics : Conference Series

English

Enlighten: Publications

  
 
 
 
Skipsey, Samuel, Doherty, Thomas, and Turner, A G D (2012) Taking the 
C out of CVMFS. In: International Conference on Computing in High 
Energy and Nuclear Physics (CHEP 2012), 21-25 May 2012, New York, 
NY, USA. 
 
 Copyright © 2012 The Authors 
 
A copy can be downloaded for personal non-commercial research or 
study, without prior permission or charge 
 
Content must not be changed in any way or reproduced in any 
format or medium without the formal permission of the copyright 
holder(s) 
 
 
When referring to this work, full bibliographic details must be given 
 
 
 
http://eprints.gla.ac.uk/95109/ 
 
 
 
  Deposited on: 17 July 2014 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
Enlighten – Research publications by members of the University of Glasgow 
http://eprints.gla.ac.uk 
Taking the C out of CVMFS
Samuel Skipsey1, Thomas Doherty1, A G D Turner2
1 Department of Physics and Astronomy, University of Glasgow, G12 8QQ
2 University of Leeds, East Building, School of Geography, Leeds, LS2 9JT
E-mail: Samuel.Skipsey@glasgow.ac.uk
Abstract. The Cern Virtual Machine File System is most well known as a distribution
mechanism for the WLCG VOs' experiment software; as a result, almost all the existing expertise
is in installing clients mount the central Cern repositories. We report the results of an initial
experiment in using the cvmfs server packages to provide Glasgow-based repository aimed at
software provisioning for small UK-local VOs. In general, although the documentation is sparse,
server conguration is reasonably easy, with some experimentation. We discuss the advantages
of local CVMFS repositories for sites, with some examples from our test VOs, vo.optics.ac.uk
and neiss.org.uk.
1. Introduction
The Cern Virtual Machine File System (CVMFS)[1] was originally developed as an underlying
lesystem for the Cern Virtual Machine project. However, it has become vastly more successful
as the current medium for software distribution on the grid, used by ATLAS[2] and LHCb, and
shortly by CMS. The properties that make CVMFS useful as a lesystem for LHC software
distribution also lend themselves to distribution of common software in general. In order to
do this, however, the software in question must be hosted in a properly congured CVMFS
repository. While there is plentiful documentation on the conguration of CVMFS clients,
provided by the developers[3], regions[4] and the experiments [5], information on the server-side
is much sparser, being limited to the nal section of one chapter of the technical report[6].
This paper discusses the experience of installing and conguring a central CVMFS server for
small local VOs at the UKI-SCOTGRID-GLASGOW site. For information on CVMFS itself,
we direct the reader to Jakob Blomer's talk in this proceedings.
2. Use case and Virtual Organisations
For the purposes of our test installation, we chose two smaller VOs supported locally by UKI-
SCOTGRID-GLASGOW (and a small number of other UK sites). The rst VO, vo.optics.ac.uk,
exists to support the pool of UK academics making common use of the Lumerical software
package for microoptical simulation. Software distribution for this VO consists mainly of the
Lumerical software installation itself, but also the distribution of licence les providing individual
sessions with the site's installed copy. Licence le distribution clearly cannot be solved by
CVMFS, as licences are necessarily local to the individual site.
The second VO is neiss.org.uk, the virtual organisation for the National E-Infrastructure for
Social Simulation. NEISS manages several projects, but the application supported on the grid is
a population demographics simulation called GENESIS. Traditionally, the project has shipped
International Conference on Computing in High Energy and Nuclear Physics 2012 (CHEP2012) IOP Publishing
Journal of Physics: Conference Series 396 (2012) 042054 doi:10.1088/1742-6596/396/4/042054
Published under licence by IOP Publishing Ltd 1
the java bytecode for the simulation with the jobs themselves, but the payload is close to the
size of the maximum input sandbox, and may become larger with time. As with vo.optics.ac.uk,
there is very little free eort to manage structured software distribution, which has been a factor
in hampering the growth of the VO.
3. Server Conguration
The CVMFS server packages are mutually incompatible with the CVMFS client, and are
currently dependant on the (unmaintained) redirfs[10] kernel module. This latter dependency
will go away in the next major release, but at present it prevents installation of the CVMFS server
on RHEL6/SL6 systems, as the module is incompatible with newer kernels. It is possible to
recompile the redirfs source against the stock RHEL6 kernel, but issues with the cvmfs modules
which interface with it prevent the solution from working. We installed our CVMFS server
on a 64-bit SL5 virtual machine, hosted on a VMware instance. The host being a member
of a pool of virtual machines maintained at Glasgow has the semi-anonymous hostname of
vm004.gla.scotgrid.ac.uk. As the documentation is sparse on CVMFS server conguration,
we will take some time here to go over the installation process for the benet of others.
The CVMFS server packages are mutually incompatible with the CVMFS client, but are
hosted in the same repository, available at http://cvmrepo.web.cern.ch/cvmrepo/yum/. With
the CVMFS repositories set up, the initial installation is managed by yum
yum install cvmfs-server
which installs cvmfs-server, and should also pull in the redirfs and cvmfst packages as a
dependancy. Conguration of the server begins by creating the working repository (each cvmfs
server can only manage a single repository for release).
cvmfs_server mkfs local.gla.scotgrid.ac.uk
Two directory structures are created in the process of making the repository lesystem:
/srv/cvmfs/local.gla.scotgrid.ac.uk/
which contains the "release" repository, as a tree of les arranged by le hash, and various
other support les which represent the repository state and conguration, and
/cvmfs/local.gla.scotgrid.ac.uk
which is a virtual lesystem managed by the cvmfst module, via redirfs. Writes to this
lesystem are redirected and used to build a set of dis against the current repository when
updates are committed. Essentially, this the inverse process to that performed by the clients in
creating the mounted fs of the same path from the published repository.
This process also generates the signing credentials for the repository, which are placed in
/etc/cvmfs/keys.
The public part of the credential needs to be distributed to all clients wishing to mount the
repository.
The process for installing (or updating) the contents of the repository is straightforward.
Simply install the precise directory structure required for your software into the
/cvmfs/local.gla.scotgrid.ac.uk lesystem. In our case, as we wished to support multiple
VOs, but had only one test server, we installed the software side-by-side in the same repository.
For a more robust solution, multiple repository servers collated by a single publishing web server
or top tier squid cache would be superior.
When the distribution is updated to your satisfaction, the nal step is
cvmfs_server publish
International Conference on Computing in High Energy and Nuclear Physics 2012 (CHEP2012) IOP Publishing
Journal of Physics: Conference Series 396 (2012) 042054 doi:10.1088/1742-6596/396/4/042054
2
The publish command commits the changes made to the /cvmfs/ directory structure since the
last publication. On a new server, there will be nothing available to clients until the rst publish
invocation pushes the initial conguration into the underlying lesystem. Errors are not always
explicitly reported by the publishing tool, but can be inferred by a failure of the published
packages to update. In general, we found that permissions on the congured shadow lesystem
at /srv/cvmfs/were an initial cause of failures; making sure that the entire directory structure
is owned by the cvmfs user and group prevents this.
4. Client Conguration
In order to mount the new repository on clients, a few changes are necessary to the client
conguration. All of the following changes are described relative to the cvmfs conguration root
/etc/cvmfs. Firstly, the repository must be added to the $CVMFS_REPOSITORIES environment
variable by suitable conguration in default.local. eg.
CVMFS_REPOSITORIES=atlas .cern.ch , atlascondb.cern.ch, lhcb.cern.ch,
cms.cern.ch, geant4.cern.ch, sft.cern.ch, local.gla.scotgrid.ac.uk
Secondly, and most signicantly, a new le is required in the domain.d/ directory to
congure the domain that the repository is based in. This will probably be the only new
domain conguration le in a "standard" CVMFS client install, as the standard repositories
are all hosted in the default cern.ch domain. The le must be named for the repository,
and congure the $CVMFS_SERVER_URL to point at the http endpoint for the repository, and
$CVMFS_PUBLIC_KEYpointing at the public key for the repository.
For example, in our case:
cat /etc/cvmfs/domain.d/local.gla.scotgrid.ac.uk.conf
CVMFS_SERVER_URL=${CERNVM_SERVER_URL:=
http://vm004.gla.scotgrid.ac.uk/cvmfs/local.gla.scotgrid.ac.uk/}
CVMFS_PUBLIC_KEY=/etc/cvmfs/keys/local.gla.scotgrid.ac.uk.pub
Finally, as mentioned before, the relevant public key must be made available to the clients,
installed in the /etc/cvmfs/keys directory on each.
5. Testing
Limited time and free resources at the VOs in question prevented a large scale test of the
implementation for either entity. However, some representative tests were performed to validate
the functionality of the software on the new mount points, and the use cases considered.
5.1. neiss.org.uk
For the NEISS VO, a number of standard jobs were run against worker nodes mounting the
CVMFS copy of the Java libraries comprising the GENESIS software. The jobs completed
successfully, in a time similar to that required for the "standard" data staging mode of operation,
within margin of error.
5.2. vo.optics.ac.uk
A simple test job utilising the Lumerical software was run at both the Glasgow and Durham
grid sites, using the distribution of Lumerical hosted in the CVMFS lespace. For the purpose
of this test, the licence keys were also distributed via CVMFS; in a production system, another
mechanism would be needed to provide individual copies.
All jobs, albeit short, were successful. The test node at the Durham site was subsequently
decongured to prevent the licenses from being used by others.
International Conference on Computing in High Energy and Nuclear Physics 2012 (CHEP2012) IOP Publishing
Journal of Physics: Conference Series 396 (2012) 042054 doi:10.1088/1742-6596/396/4/042054
3
6. Conclusion and Comments
Although the documentation is sparse, conguring a cvmfs server is not particularly arduous,
with some experimentation. We have demonstrated and tested the use of the standard cvmfs
server package in software provisioning for two small VOs, across two sites. What remains to
be demonstrated is a secure scheme for enabling VO members to update their own software
releases on the repository provided for them. The authors consider that this problem is soluble
by existing standard policies for allowing remote access and authentication, however (in the
worst case, gsissh and VOBOX-like functionality on the repository servers).
The use of proprietary software requiring licence les or other les that are site specic is
problematic with CVMFS. Arguably, this is not a use-case for which the software framework
was intended, and we do not consider it's inability to address the problem to be a deciency.
Nonetheless, we cannot recommend VOs depending on proprietary software to use CVMFS as
their distribution platform.
References
[1] P Buncic et al 2010 J. Phys.: Conf. Ser. 219 042003
[2] De Salvo, A, et al Software installation and condition data distribution via CernVM FileSystem in ATLAS
J. Phys.: Conf. Ser. THIS PROCEEDINGS CHEP 2012
[3] http://cernvm.cern.ch/portal/cvmfs/examples
[4] https://www.gridpp.ac.uk/wiki/UK CVMFS Deployment
[5] https://twiki.cern.ch/twiki/bin/view/Atlas/CernVMFS
[6] https://cernvm.cern.ch/project/trac/downloads/cernvm/cvmfstech-2.0-6.pdf
[7] Lumerical Inc. http://www.lumerical.com
[8] Birkin, M., et al (2010): Phil. Trans. R. Soc. A, 368:3797-3812; doi:10.1098/rsta.2010.0145.
http://www.neiss.org.uk
[9] Turner, A.G.D. 2010 Data Driven e-Science: Use Cases and Successful Applications of Distributed Computing
Infrastructures ed Simon C. Lin and Eric Yen (to be published, Springer) chapter 1
[10] http://www.redirfs.org/tiki-index.php
International Conference on Computing in High Energy and Nuclear Physics 2012 (CHEP2012) IOP Publishing
Journal of Physics: Conference Series 396 (2012) 042054 doi:10.1088/1742-6596/396/4/042054
4


Taking the C out of CVMFS

Samuel Skipsey

Thomas Doherty

A G D Turner

Crossref

Journal of Physics Conference Series

Enlighten

     Skipsey, Samuel, Doherty, Thomas, and Turner, A G D (2012) Taking the C out of CVMFS. In: International Conference on Computing in High Energy and Nuclear Physics (CHEP 2012), 21-25 May 2012, New York, NY, USA.   Copyright © 2012 The Authors  A copy can be downloaded for personal non-commercial research or study, without prior permission or charge  Content must not be changed in any way or reproduced in any format or medium without the formal permission of the copyright holder(s)   When referring to this work, full bibliographic details must be given    http://eprints.gla.ac.uk/95109/      Deposited on: 17 July 2014                Enlighten – Research publications by members of the University of Glasgow http://eprints.gla.ac.uk Taking the C out of CVMFSSamuel Skipsey1, Thomas Doherty1, A G D Turner21 Department of Physics and Astronomy, University of Glasgow, G12 8QQ2 University of Leeds, East Building, School of Geography, Leeds, LS2 9JTE-mail: Samuel.Skipsey@glasgow.ac.ukAbstract. The Cern Virtual Machine File System is most well known as a distributionmechanism for the WLCG VOs' experiment software; as a result, almost all the existing expertiseis in installing clients mount the central Cern repositories. We report the results of an initialexperiment in using the cvmfs server packages to provide Glasgow-based repository aimed atsoftware provisioning for small UK-local VOs. In general, although the documentation is sparse,server conguration is reasonably easy, with some experimentation. We discuss the advantagesof local CVMFS repositories for sites, with some examples from our test VOs, vo.optics.ac.ukand neiss.org.uk.1. IntroductionThe Cern Virtual Machine File System (CVMFS)[1] was originally developed as an underlyinglesystem for the Cern Virtual Machine project. However, it has become vastly more successfulas the current medium for software distribution on the grid, used by ATLAS[2] and LHCb, andshortly by CMS. The properties that make CVMFS useful as a lesystem for LHC softwaredistribution also lend themselves to distribution of common software in general. In order todo this, however, the software in question must be hosted in a properly congured CVMFSrepository. While there is plentiful documentation on the conguration of CVMFS clients,provided by the developers[3], regions[4] and the experiments [5], information on the server-sideis much sparser, being limited to the nal section of one chapter of the technical report[6].This paper discusses the experience of installing and conguring a central CVMFS server forsmall local VOs at the UKI-SCOTGRID-GLASGOW site. For information on CVMFS itself,we direct the reader to Jakob Blomer's talk in this proceedings.2. Use case and Virtual OrganisationsFor the purposes of our test installation, we chose two smaller VOs supported locally by UKI-SCOTGRID-GLASGOW (and a small number of other UK sites). The rst VO, vo.optics.ac.uk,exists to support the pool of UK academics making common use of the Lumerical softwarepackage for microoptical simulation. Software distribution for this VO consists mainly of theLumerical software installation itself, but also the distribution of licence les providing individualsessions with the site's installed copy. Licence le distribution clearly cannot be solved byCVMFS, as licences are necessarily local to the individual site.The second VO is neiss.org.uk, the virtual organisation for the National E-Infrastructure forSocial Simulation. NEISS manages several projects, but the application supported on the grid isa population demographics simulation called GENESIS. Traditionally, the project has shippedInternational Conference on Computing in High Energy and Nuclear Physics 2012 (CHEP2012) IOP PublishingJournal of Physics: Conference Series 396 (2012) 042054 doi:10.1088/1742-6596/396/4/042054Published under licence by IOP Publishing Ltd 1the java bytecode for the simulation with the jobs themselves, but the payload is close to thesize of the maximum input sandbox, and may become larger with time. As with vo.optics.ac.uk,there is very little free eort to manage structured software distribution, which has been a factorin hampering the growth of the VO.3. Server CongurationThe CVMFS server packages are mutually incompatible with the CVMFS client, and arecurrently dependant on the (unmaintained) redirfs[10] kernel module. This latter dependencywill go away in the next major release, but at present it prevents installation of the CVMFS serveron RHEL6/SL6 systems, as the module is incompatible with newer kernels. It is possible torecompile the redirfs source against the stock RHEL6 kernel, but issues with the cvmfs moduleswhich interface with it prevent the solution from working. We installed our CVMFS serveron a 64-bit SL5 virtual machine, hosted on a VMware instance. The host being a memberof a pool of virtual machines maintained at Glasgow has the semi-anonymous hostname ofvm004.gla.scotgrid.ac.uk. As the documentation is sparse on CVMFS server conguration,we will take some time here to go over the installation process for the benet of others.The CVMFS server packages are mutually incompatible with the CVMFS client, but arehosted in the same repository, available at http://cvmrepo.web.cern.ch/cvmrepo/yum/. Withthe CVMFS repositories set up, the initial installation is managed by yumyum install cvmfs-serverwhich installs cvmfs-server, and should also pull in the redirfs and cvmfst packages as adependancy. Conguration of the server begins by creating the working repository (each cvmfsserver can only manage a single repository for release).cvmfs_server mkfs local.gla.scotgrid.ac.ukTwo directory structures are created in the process of making the repository lesystem:/srv/cvmfs/local.gla.scotgrid.ac.uk/which contains the "release" repository, as a tree of les arranged by le hash, and variousother support les which represent the repository state and conguration, and/cvmfs/local.gla.scotgrid.ac.ukwhich is a virtual lesystem managed by the cvmfst module, via redirfs. Writes to thislesystem are redirected and used to build a set of dis against the current repository whenupdates are committed. Essentially, this the inverse process to that performed by the clients increating the mounted fs of the same path from the published repository.This process also generates the signing credentials for the repository, which are placed in/etc/cvmfs/keys.The public part of the credential needs to be distributed to all clients wishing to mount therepository.The process for installing (or updating) the contents of the repository is straightforward.Simply install the precise directory structure required for your software into the/cvmfs/local.gla.scotgrid.ac.uk lesystem. In our case, as we wished to support multipleVOs, but had only one test server, we installed the software side-by-side in the same repository.For a more robust solution, multiple repository servers collated by a single publishing web serveror top tier squid cache would be superior.When the distribution is updated to your satisfaction, the nal step iscvmfs_server publishInternational Conference on Computing in High Energy and Nuclear Physics 2012 (CHEP2012) IOP PublishingJournal of Physics: Conference Series 396 (2012) 042054 doi:10.1088/1742-6596/396/4/0420542The publish command commits the changes made to the /cvmfs/ directory structure since thelast publication. On a new server, there will be nothing available to clients until the rst publishinvocation pushes the initial conguration into the underlying lesystem. Errors are not alwaysexplicitly reported by the publishing tool, but can be inferred by a failure of the publishedpackages to update. In general, we found that permissions on the congured shadow lesystemat /srv/cvmfs/were an initial cause of failures; making sure that the entire directory structureis owned by the cvmfs user and group prevents this.4. Client CongurationIn order to mount the new repository on clients, a few changes are necessary to the clientconguration. All of the following changes are described relative to the cvmfs conguration root/etc/cvmfs. Firstly, the repository must be added to the $CVMFS_REPOSITORIES environmentvariable by suitable conguration in default.local. eg.CVMFS_REPOSITORIES=atlas .cern.ch , atlascondb.cern.ch, lhcb.cern.ch,cms.cern.ch, geant4.cern.ch, sft.cern.ch, local.gla.scotgrid.ac.ukSecondly, and most signicantly, a new le is required in the domain.d/ directory tocongure the domain that the repository is based in. This will probably be the only newdomain conguration le in a "standard" CVMFS client install, as the standard repositoriesare all hosted in the default cern.ch domain. The le must be named for the repository,and congure the $CVMFS_SERVER_URL to point at the http endpoint for the repository, and$CVMFS_PUBLIC_KEYpointing at the public key for the repository.For example, in our case:cat /etc/cvmfs/domain.d/local.gla.scotgrid.ac.uk.confCVMFS_SERVER_URL=${CERNVM_SERVER_URL:=http://vm004.gla.scotgrid.ac.uk/cvmfs/local.gla.scotgrid.ac.uk/}CVMFS_PUBLIC_KEY=/etc/cvmfs/keys/local.gla.scotgrid.ac.uk.pubFinally, as mentioned before, the relevant public key must be made available to the clients,installed in the /etc/cvmfs/keys directory on each.5. TestingLimited time and free resources at the VOs in question prevented a large scale test of theimplementation for either entity. However, some representative tests were performed to validatethe functionality of the software on the new mount points, and the use cases considered.5.1. neiss.org.ukFor the NEISS VO, a number of standard jobs were run against worker nodes mounting theCVMFS copy of the Java libraries comprising the GENESIS software. The jobs completedsuccessfully, in a time similar to that required for the "standard" data staging mode of operation,within margin of error.5.2. vo.optics.ac.ukA simple test job utilising the Lumerical software was run at both the Glasgow and Durhamgrid sites, using the distribution of Lumerical hosted in the CVMFS lespace. For the purposeof this test, the licence keys were also distributed via CVMFS; in a production system, anothermechanism would be needed to provide individual copies.All jobs, albeit short, were successful. The test node at the Durham site was subsequentlydecongured to prevent the licenses from being used by others.International Conference on Computing in High Energy and Nuclear Physics 2012 (CHEP2012) IOP PublishingJournal of Physics: Conference Series 396 (2012) 042054 doi:10.1088/1742-6596/396/4/04205436. Conclusion and CommentsAlthough the documentation is sparse, conguring a cvmfs server is not particularly arduous,with some experimentation. We have demonstrated and tested the use of the standard cvmfsserver package in software provisioning for two small VOs, across two sites. What remains tobe demonstrated is a secure scheme for enabling VO members to update their own softwarereleases on the repository provided for them. The authors consider that this problem is solubleby existing standard policies for allowing remote access and authentication, however (in theworst case, gsissh and VOBOX-like functionality on the repository servers).The use of proprietary software requiring licence les or other les that are site specic isproblematic with CVMFS. Arguably, this is not a use-case for which the software frameworkwas intended, and we do not consider it's inability to address the problem to be a deciency.Nonetheless, we cannot recommend VOs depending on proprietary software to use CVMFS astheir distribution platform.References[1] P Buncic et al 2010 J. Phys.: Conf. Ser. 219 042003[2] De Salvo, A, et al Software installation and condition data distribution via CernVM FileSystem in ATLASJ. Phys.: Conf. Ser. THIS PROCEEDINGS CHEP 2012[3] http://cernvm.cern.ch/portal/cvmfs/examples[4] https://www.gridpp.ac.uk/wiki/UK CVMFS Deployment[5] https://twiki.cern.ch/twiki/bin/view/Atlas/CernVMFS[6] https://cernvm.cern.ch/project/trac/downloads/cernvm/cvmfstech-2.0-6.pdf[7] Lumerical Inc. http://www.lumerical.com[8] Birkin, M., et al (2010): Phil. Trans. R. Soc. A, 368:3797-3812; doi:10.1098/rsta.2010.0145.http://www.neiss.org.uk[9] Turner, A.G.D. 2010 Data Driven e-Science: Use Cases and Successful Applications of Distributed ComputingInfrastructures ed Simon C. Lin and Eric Yen (to be published, Springer) chapter 1[10] http://www.redirfs.org/tiki-index.phpInternational Conference on Computing in High Energy and Nuclear Physics 2012 (CHEP2012) IOP PublishingJournal of Physics: Conference Series 396 (2012) 042054 doi:10.1088/1742-6596/396/4/0420544

http://eprints.gla.ac.uk/95109/1/95109.pdf

Taking the C out of CVMFS

Abstract

Similar works

Full text

Available Versions

Enlighten: Publications

Crossref

Enlighten