Brett Zimmerman,	
  University of Oklahoma	
  Senior Systems Analyst,	
  OU	 Supercomputing Center	for Education and Research (OSCER)Linux Cluster Institute (LCI) Workshop

May 18-22, 2015
University of Oklahoma, Thurman J. White Forum Building (OU Forum)
1704 Asp Avenue
Norman, Oklahoma 73072

This workshop was focused toward Linux system administrators who were new to HPC.University of Oklahoma

 OU Supercomputing Center for Education and Research (OSCER)

Linux Cluster InstituteN

Zimmerman, Brett

SHAREOK repository

Linux  Clusters  Ins.tute:  Cluster  Stack  BasicsBre$	  Zimmerman,	  University	  of	  Oklahoma	  Senior	  Systems	  Analyst,	  OU	  Supercompu<ng	  Center	  for	  Educa<on	  and	  Research	  (OSCER)	  A  Bunch  of  Computers• Users	  can	  login	  to	  any	  node	  • Filesystems	  aren’t	  shared	  between	  nodes	  • Work	  is	  run	  wherever	  you	  can	  ﬁnd	  space	  • Nodes	  maintained	  individually	  4-8 August 2014 2 What’s  wrong  with  a  bunch  of  nodes?• Compe<<on	  for	  resources	  • Size	  and	  type	  of	  problem	  is	  limited	  • Nodes	  get	  out	  of	  sync	  • Problems	  for	  users	  • Diﬃculty	  in	  management	  	  4-8 August 2014 3 Cluster  Approach• Shared	  ﬁlesystems	  • Job	  management	  • Nodes	  dedicated	  to	  compute	  • Consistent	  environment	  • Interconnect	  	  4-8 August 2014 4 What’s  right  about  the  cluster  approach?• Easier	  to	  use	  • Maximize	  eﬃciency	  • Can	  do	  bigger	  and	  be$er	  problems	  • Nodes	  can	  be	  used	  coopera<vely	  4-8 August 2014 5 The  Types  of  Nodes• Login	  • Users	  login	  here	  • Compiling	  • Edi<ng	  • SubmiTng	  and	  Monitoring	  jobs	  • Compute	  • Users	  might	  login	  here	  • Run	  jobs	  as	  directed	  by	  the	  scheduler	  • Support	  • Users	  don’t	  login	  here	  • Do	  all	  the	  other	  stuﬀ	  4-8 August 2014 6 What  a  cluster  needs  –  the  mundane• Network	  services	  –	  NTP,	  DNS,	  DHCP	  • Shared	  Storage	  -­‐-­‐	  NFS	  • Logging	  –	  Consolidated	  Syslog	  as	  a	  star<ng	  point	  • Licensing	  –	  FlexLM	  and	  the	  like	  • Database	  –	  User	  and	  Administra<ve	  Data	  • Boot/Provisioning	  –	  PXE,	  build	  system	  • Authen<ca<on	  –	  LDAP	  	  4-8 August 2014 7 What  a  cluster  needs  -­‐-­‐  Specialized• Interconnect	  –	  An	  ideally	  low-­‐latency	  network	  • Job	  manager	  –	  Resource	  manager/	  scheduler	  • Parallel	  Storage	  –	  Get	  around	  the	  limita<ons	  of	  NFS	  4-8 August 2014 8 Network  Services • NTP	  –	  Network	  Time	  Protocol,	  provides	  clock	  synchroniza<on	  across	  all	  nodes	  in	  the	  cluster	  • DHCP	  –	  Dynamic	  Host	  Conﬁgura<on	  Protocol,	  allows	  central	  conﬁgura<on	  of	  host	  networking	  • DNS	  –	  Provides	  name	  to	  address	  transla<on	  for	  the	  cluster	  • NFS	  –	  Basic	  UNIX	  network	  ﬁlesystem	  4-8 August 2014 9 Logging• Syslog	  	  • The	  classic	  system	  for	  UNIX	  logging	  • Applica<on	  has	  to	  opt	  to	  emit	  messages	  • Monitoring	  	  • Ac<ve	  monitoring	  to	  catch	  condi<ons	  elec<ve	  monitoring	  doesn’t	  catch	  • Resource	  manager	  • Nagios/cac</zabbix/ganglia	  • IDS	  • Intrusion	  detec<on	  • Monitoring	  targe<ng	  misuse/a$acks	  on	  the	  cluster	  4-8 August 2014 10 Basic  services,  con.nued• Licensing	  –	  FlexNet/FlexLM	  or	  equivalent,	  mediates	  access	  to	  a	  pool	  of	  shared	  licenses.	  • Database	  –	  Administra<ve	  use	  for	  logging/monitoring,	  dynamic	  conﬁgura<on.	  	  Requirements	  of	  user	  so`ware.	  • Boot/Provisioning	  –	  For	  example	  PXE/Cobbler,	  PXE/Image	  or	  part	  of	  a	  cluster	  management	  suite	  4-8 August 2014 11 Authen.ca.on• Flat	  ﬁles	  	  -­‐-­‐	  passwd,	  group,	  shadow	  entries	  • NIS	  -­‐-­‐	  network	  access	  to	  central	  ﬂat	  ﬁles	  • LDAP	  -­‐-­‐	  Read/Write	  access	  to	  a	  dynamic	  tree	  structure	  of	  account	  and	  other	  informa<on	  • Host	  equivalency	  4-8 August 2014 12 Cluster  Networking• Hardware	  Management	  –	  Lights	  out	  management	  	  • External	  –	  Public	  interfaces	  to	  the	  cluster	  • Internal	  –	  General	  node	  to	  node	  communica<on	  • Storage	  –	  Access	  to	  network	  ﬁlesystems	  • Interconnect	  –	  high-­‐speed,	  low-­‐latency	  for	  mul<-­‐node	  jobs	  Some	  of	  these	  can	  share	  a	  medium	  4-8 August 2014 13 InterconnectIn	  the	  most	  recent	  Top	  500	  list	  (h$p://top500.org)	  there	  were	  224	  installa<ons	  relying	  on	  Inﬁniband,	  100	  using	  Gigabit	  Ethernet,	  and	  88	  using	  10	  Gigabit	  Ethernet	  	  • Ethernet	  –	  Latency	  of	  50-­‐125	  µs	  (GbE),	  5-­‐50	  µs	  	  (10GbE),	  ~5	  µs	  RoCEE	  • Inﬁniband	  –	  Latency	  of	  1.3	  µs	  (QDR)	  .7	  µs	  (FDR-­‐10/FDR),	  .5	  µs	  (EDR)	  	  4-8 August 2014 14 Parallel  FilesystemLustre	  -­‐	  h$p://lustre.org/	  PanFS	  -­‐	  h$p://www.panasas.com/	  GPFS	  -­‐	  h$p://www-­‐03.ibm.com/so`ware/products/en/so`ware	  	  Parallel	  ﬁlesystems	  take	  the	  general	  approach	  of	  separa<ng	  ﬁlesystem	  metadata	  from	  the	  storage.	  	  Lustre	  and	  PanFS	  have	  dedicated	  nodes	  for	  metadata	  (MDS	  or	  director	  blades).	  	  GPFS	  distributes	  metadata	  throughout	  the	  cluster	  	  4-8 August 2014 15 Cluster  Management• Automates	  the	  building	  of	  a	  cluster	  • Some	  way	  to	  easily	  maintain	  cluster	  system	  consistency	  • The	  ability	  to	  automate	  cluster	  maintenance	  tasks	  • Oﬀer	  some	  way	  to	  monitor	  cluster	  health	  and	  performance	  4-8 August 2014 16 Cluster  Managemement  SoNware The	  resource	  manager	  knows	  the	  state	  of	  the	  various	  resources	  on	  the	  cluster	  and	  maintains	  a	  list	  of	  the	  jobs	  that	  are	  reques<ng	  resources	  	  The	  scheduler,	  using	  the	  informa<on	  from	  the	  resource	  manager	  selects	  jobs	  from	  the	  queue	  for	  execu<on	  	  • Rocks	  (h$p://www.rocksclusters.org/wordpress/)	  • Bright	  Cluster	  Manager	  (h$p://www.brightcompu<ng.com/Bright-­‐Cluster-­‐Manager)	  •  xCAT	  (Extreme	  Cluster/Cloud	  Administra<on	  Toolkit)	  (h$p://sourceforge.net/p/xcat/wiki/Main_Page/	  	  4-8 August 2014 17 Conﬁgura.on  ManagementWhile	  it	  is	  true	  that	  boo<ng	  with	  a	  central	  boot	  server	  can	  make	  it	  easier	  to	  make	  sure	  the	  OS	  on	  each	  compute	  node	  (or,	  at	  least,	  each	  type	  of	  compute	  node)	  has	  an	  iden<cal	  setup/install,	  there	  are	  s<ll	  ﬁles	  which	  wind	  up	  being	  more	  dynamic.	  	  Some	  such	  ﬁles	  are	  password/group/shadow	  and	  hosts	  ﬁles.	  	  • Rsync	  • Cfengine	  • Chef	  • Puppet	  •  Salt	  4-8 August 2014 18 SoNware  Installa.on  and  ManagementAll	  linux	  distros	  have	  some	  sort	  of	  package	  management	  tool.	  For	  Redhat/CentOS/Scien<ﬁc	  based	  clusters,	  this	  is	  rpm	  and	  yum.	  	  Debian	  has	  dpkg	  and	  apt	  	  In	  any	  case	  pre-­‐packaged	  so`ware	  tends	  to	  assume	  that	  it	  is	  going	  to	  be	  installed	  in	  a	  speciﬁc	  place	  on	  the	  machine	  and	  that	  it	  will	  be	  the	  only	  version	  of	  that	  so`ware	  on	  the	  machine.	  	  One	  a	  cluster,	  it	  may	  be	  necessary	  to	  look	  at	  so`ware	  installa<on	  diﬀerently	  from	  a	  standard	  linux	  machine	  	  •  Install	  to	  global	  ﬁlesystem	  • Keep	  boot	  image	  as	  small	  as	  possible	  • Maintain	  mul<ple	  versions	  	  4-8 August 2014 19 SoNware  installa.on  and  managementThere	  are	  a	  couple	  of	  tools	  useful	  for	  naviga<ng	  the	  diﬃcul<es	  of	  maintaining	  user	  environments	  when	  dealing	  with	  mul<ple	  versions	  of	  so`ware	  or	  so`ware	  in	  non-­‐standard	  loca<ons.	  •  So`Env	  (h$p://h$p://www.lcrc.anl.gov/info/So`ware/So`env)	  	  	  	  	  Useful	  for	  packaging	  sta<c	  user	  environment	  required	  by	  packages	  • Modules	  	  (h$p://modules.sourceforge.net/)	  	  	  Can	  be	  used	  to	  make	  dynamic	  changes	  to	  a	  users	  environment.	  4-8 August 2014 20 Resource  Manager/Scheduler• Accepts	  job	  submissions,	  maintains	  a	  queue	  of	  jobs	  • Allocates	  nodes/resources	  and	  starts	  jobs	  on	  compute	  nodes	  • Schedules	  wai<ng	  jobs	  	  • Available	  op<ons	  • SGE	  (Sun	  Grid	  Engine)	  • LSF	  /	  Openlava	  (Load	  Sharing	  Facility)	  • PBS	  (Portable	  Batch	  System)	  • OpenPBS	  • Torque	  • SLURM	  4-8 August 2014 21 Best  Prac.cesHere	  is	  a	  quick	  overview	  of	  the	  general	  func<ons	  to	  secure	  a	  cluster	  • Risk	  Avoidance	  • Deterrence	  • Preven<on	  • Detec<on	  • Recovery	  	  The	  priority	  of	  these	  will	  depend	  on	  your	  security	  approach	  4-8 August 2014 22 Risk  Avoidance• Provide	  the	  minimum	  of	  services	  necessary	  • Grant	  the	  least	  privileges	  necessary	  • Install	  the	  minimum	  so`ware	  necessary	  The	  simpler	  the	  environment,	  the	  fewer	  the	  vectors	  available	  for	  a$ack.	  	  	  	  4-8 August 2014 23 Deterrence• Limit	  the	  discoverability	  of	  the	  cluster	  • Publish	  acceptable	  use	  policies	  4-8 August 2014 24 Preven.on• Fix	  known	  issues	  (patching)	  • Conﬁgure	  services	  for	  minimal	  func<onality	  • Restrict	  user	  access	  and	  authority	  • Document	  ac<ons	  and	  changes	  	  Detec.on• Monitor	  the	  cluster	  • Integrate	  feedback	  from	  the	  users	  • Set	  alerts	  and	  automated	  response	  4-8 August 2014 25 Recovery• Backups	  • Documenta<on	  • Deﬁne	  acceptable	  loss	  

Cluster Stack Basics

https://shareok.org/bitstream/11244/15467/1/cluster_stack_basics.pdf

Cluster Stack Basics

Abstract

Similar works

Full text

Available Versions

SHAREOK repository