The author presents NSF, program to move science, math, engineering education in the Unites States to digital age, focused effort to develop and model infrastructure for science education on the web. Over 80 independent grants exploring NSDL goals

Lagoze, Carl

English

E-LIS repository

NSDL: OAI and a large-
s c ale di gi t al li b rary
Carl Lagoze, Cornell University
N S D L D irec tor of  T ec h nology
lagoze@ c s. c ornell. ed u
What is the NSDL?
 NSF program to move science, math, engineering 
ed u cation in the U S to d igital  age
 http://www.ehr.nsf.gov/ehr/due/programs/nsdl/
 O ver 8 0  ind epend ent grants ex pl oring NSD L  goal s
 http://c omm.nsdli b .org
 Focu sed  ef f ort to d evel op and  mod el  inf rastru ctu re 
f or science ed u cation on the w eb .
 http://c i news.c omm.nsdli b .org/c gi -b i n/wi k i .pl
 A  prod u ction d igital  l ib rary
 http://www.nsdl.org
1996  V i s i o n  a r t i c u l a t e d  b y  N S F ' s  D i v i s i o n  o f  U n d e r g r a d u a t e  
E d u c a t i o n
1997   N a t i o n a l  R e s e a r c h  C o u n c i l  w o r k s h o p
1998   P r e l i m i n a r y  g r a n t s  t h r o u g h  D i g i t a l  L i b r a r i e s  I n i t i a t i v e  2
1998   S M E T E -L i b  w o r k s h o p  
1999  N S D L  S o l i c i t a t i o n   
2 0 0 0   6 C o r e  I n t e g r a t i o n  d e m o n s t r a t i o n  p r o j e c t s  +  2 3  o t h e r s  
f u n d e d
2 0 0 1  1 l a r g e  C o r e  I n t e g r a t i o n  S y s t e m  p r o j e c t  f u n d e d
2 0 0 2 M o r e  t h a n  8 0  i n d e p e n d e n t  p r o j e c t s  f u n d e d
2 0 0 3 C o r e  I n t e g r a t i o n  f u n d i n g  f i x e d  u n t i l  2 0 0 6
Short History of the NSDL
NSF Grant Structure
http://www.nsf.gov/pubs/2002/nsf02054/nsf02054.html
 Collections
 Develop and maintain content
 S er v ices
 F or  u s er s ,  collection pr ovider s ,  cor e integ r ation
 T a r g eted  r esea r ch
 Cor e I nteg r a tion
 O r g aniz ational,  economic,  tech nical
 $ U S 5 M  of  total $ U S 2 5 M  total b u dg et
 A collaborative project
University Corporation for 
A tm osph eric  R esearc h - Dave Fulker
Cornel l  University - W i lli am  A rm s
Col u m b ia University - K at e W i t t en b erg
 W ith  ad d ition al partn ers
E astern M ic h ig an University
S yrac u se University
U M ass-A m h erst
UC-S anta B arb ara
UC-S an D ieg o ( S u perc om pu ter Center)
 D irector of  T ech n olog y - C arl L ag o z e
NSDL CI Technical Organization
browsing
 







 annotating
c u rric u l u m  bu il d ing
	
















ff
fi



fl
ffi
Building service and knowledge layers 
over a variet y of  resources f or a variet y of  
users
Open Access 
W eb
Open Access 
W eb
P u b l i sh er sP u b l i sh er s
N S F -f u nd ed  
C o l l ect i o ns
N S F -f u nd ed  
C o l l ect i o ns
All branches of science, all levels of education, very 
broadly defined:
Five year targets
1 ,0 0 0 ,0 0 0  different users
1 0 ,0 0 0 ,0 0 0  dig ital obj ects
1 0 ,0 0 0  to 1 0 0 ,0 0 0  indep endent sites
How Big might the NSDL be?
It is possible to build a very large dig ital libr ar y  w ith  a 
sm all staf f .
B u t  . . .
E v er y  aspec t of  th e libr ar y  m ust be plan n ed w ith  sc alability  in  m in d.
S om e c om pr om ises w ill be m ade.
Core Integration Philosophy
Perspective on the Budget
	

Resources for Core Integration
Core Integration
Budget $ 4 -6  m i l l i o n
S ta f f 2 5  - 3 0
M a n a gem en t D i f f us e
H o w  c a n  a  s m a l l  tea m ,  w i th o ut di r ec t m a n a gem en t c o n tr o l ,  
c r ea te a  v er y  l a r ge-s c a l e di gi ta l  l i b r a r y ?
		
 Aggregation rather than collection
 Core integration team will not manage any collections
 S p ectru m  of  interop erab ility
 A ccommod ate d iv ersity of  p articip ation mod els
 O p en interf aces and  stand ard s p ermitting p lu g in of  array of  v alu e-ad d ed  serv ices
 O ne lib rary  m any  p ortals
 A ccommod ate mu ltip le q u ality and  selection metrics
 T ailor p resentation of  content and  natu re of  serv ices to au d ience need s
 O p en toolk it of  s of tw are and  s erv ices  f or lib rary  
b u ild ing
NSDL technical mantras
	
Level A g r eem en t s E x a m p le
Federation Strict use of standards A A C R ,  M A R C
( sy ntax ,  sem antic,  Z  3 9 . 5 0
and b usiness)
H arv es ting D ig ital  l ib raries ex p ose O p en A rch iv es
m etadata;  sim p l e m etadata h arv esting  
p rotocol  and reg istry
G ath ering D ig ital  l ib raries do not W eb  craw l ers
coop erate;  serv ices m ust and search  eng ines
seek  out inform ation
Spectrum of interoperability
	
 This is a big task that no one has done before!
 W ork on the p riorities
 Focus on one point on spectrum of interoperability
 Metadata harvesting
 I nc o rp o rate N S F  f u nded c o l l ec tio ns and sel ec ted o ther c o l l ec tio ns
 L ev erag e ex isting  ( or at least emerg ing )  tech nolog ies and  protocols
 O A I ,  u P o rtal ,  S hib b o l eth,  S D L I P ,  I nQ u ery
 P rov id e reliable base lev el serv ices
 S earc h and D isc o very ,  A c c ess Managem ent,  U ser P ro f il es,  E x em p l ary  
P o rtal s,  P ersistenc e
 P l ant som e seeds for the fu tu re
 M ach ine-assisted  metad ata g eneration
 A utomated  collection ag g reg ation
 W eb g ath ering  strateg ies
Translating to first release goals
	
 Central storage of all metadata about all resources in the 
N S D L
 Defines the extent of NSDL collection
 M eta d a ta  inclu d es collections,  item s,  a nnota tions,  etc.
 MR main functions
 A g g r eg a tion
 Nor m a liz a tion
 r ed istr ib u tion
 I ngest of metadata by  v arious means
 H a r v esting ,  m a nu a l,  a u tom a tic,  cr oss-w a lk ing
 O p en access to M R  contents for serv ice builders v ia O A I -
P M H
Metadata Repository
	
Metadata Strategy
 Collect and redistribute any native (XML) 
m etadata f orm at
 P rovide crossw alk s to D ublin Core f rom  eig h t 
standard f orm ats 
 Dublin Core, DC-G E M , L T S C ( I M S ) , A DL  
( S CO R M ) , M A R C, F G CD, E A D
 Concentrate on collection-level m etadata
 U se autom atic g eneration to aug m ent item -
level m etadata
	
Importing metadata into the MR
Collections
H a r v est
S ta g ing  a r ea
Clea nu p  
a nd  
cr ossw a lk s
D a ta b a se loa d
M eta d a ta  R ep ositor y
	
Exporting metadata from the MR
NSDL 
s e r v i c e s
C r e a t e  O A I  
s e r v e r  t a b l e s
M e t a d a t a  R e p o s i t o r y
SQ L q u e r i e s O A I  s e r v e r H a r v e s t s e r v i c e s
C r e a t e O A I
s e r v e r t a b l e s
M e t a d a t a R e p o s i t o r y
Q q u e r i e s O A I s e r v e r H a r v e s t
	
Simple Metadata-B as ed Ser v ic es :
The recognition of common elements among a set of core Library 
serv ices ( initially E x hibits N ew s,  A nnotation,  E q u iv alence,  and  My 
Site) ,  led  the N SD L Team to create a mod el for the d ev elop ment 
and  imp lementation of serv ices that cou ld  be based  on simp le 
ex tensions to stand ard  Metad ata R ecord s. Serv ices that fit this 
mod el are k now n as Simp le Metad ata-B ased  Serv ices,  or SiMB aS.
	
SIMBaS C h ar ac t e r i s t i c s
 Services provide metadata records for 
h arvestin g  b y  M R
 M etadata records may  in cl u de ty ped 
rel ation sh ip l in k s to each  oth er or to pre-
ex istin g  M etadata R ecords in  th e M R .
 E x ampl e rel ation sh ip l in k s
 Collections-> item s.
 A nnota tion m eta d a ta  r ecor d -> item -lev el 
m eta d a ta  r ecor d .


	
What to Index?
When possible, full text indexing is excellent, but full text indexing 
is not possible for  a ll m a ter ia ls ( non-textua l, no a ccess for  
indexing) .
C om pr ehensiv e m eta da ta  is a n a lter na tiv e, but a v a ila ble for  v er y
few  of the m a ter ia ls.
What A r c hi tec tu r e to U s e?
F ew  collections suppor t a n esta blished sea r ch pr otocol ( e. g. , 
Z 3 9 . 5 0 )
Searching

 Implement a query language that includes most 
f eatures that are common in commercial and W eb  
search engines.
 P eriodically harv est the M R  ( v ia O A I-P M H )  to 
incorporate the latest changes in the lib rary.
 A llow  search on resources’  metadata as w ell as 
tex tual content,  w hen av ailab le.
 C ommunication w ith portals is done v ia the S imple 
D igital L ib rary Interoperab ility P rotocol ( S D L IP ) .
Search system general features

Search Architecture
Metadata
R ep o s i to r y
C o n ten t
P o r tal
P o r tal
P o r tal
S ear c h
E n g i n e
S D L I P
W r ap p erS D L I P
O A I
H ar v es ter
O A IS ear c h  an d D i s c o v er y  S er v er
h ttp / f tp
H ar v es ter h ttp / f tp
“ D o c u m en t”
g en er ato r

Persistent Archive for the NSDL
 Provide a persistent copy of the 
resou rces identified in the N S D L  
repository
 Provide a mechanism to retrieve prior 
versions of  resou rces
 V erify avail ab il ity of on-l ine dig ital  
resou rces that have presence in M R

Persistent Archive Approach
 Use data grid technology to:
 Implement a persistent logical name space for registering 
resou rces
 M anage arch iv ing of mod u les on d istrib u ted  storage sy stems
 Use O A I  harv esting to ex tract m etadata f rom  the 
N S D L  rep ository
 C raw l the w eb  to retriev e resou rces
 P rov ide O A I  interf ace f or rep orting v alidation resu lts
 M anage the p ersistent archiv e throu gh a sep arate 
inf orm ation rep ository

Experience thus far
 OAI – l o w  b a r r i e r ?
 Sets
 I d en ti f i er s
 X M L  f l a k i n e s s
 L i m i t a t i o n s  o f  b a s i c  D u b l i n  C o r e
 M e t a d a t a  q u a l i t y  a n d  t r u s t
 R e s o u r c e  g r a n u l a r i t y

Closing Thoughts
 We have only just begun!
 A utom ati on i s k ey to sc alabi li ty
 Metadata generation
 L ongev ity / p res erv ation
 Q u al ity  and s el ec tion
 C ol l ec tion dev el op m ent
 T he N S D L  need s to be m or e that data
 K now l edge
 C u rric u l a
 C om m u nity
 c ol l ab oration


NSDL: OAI and a large-scale digital library

http://eprints.rclis.org/4556/2/CERN_NSDL_October_2002.pdf

NSDL: OAI and a large-scale digital library

Abstract

Similar works

Full text

Available Versions

E-LIS repository