The Data Storage and Retrieval System (DSRS) consists of off-the-shelf system components integrated as a file server supporting very large files. These files are on the order of one gigabyte of data per file, although smaller files on the order of one megabyte can be accommodated as well. For instance, one gigabyte of data occupies approximately six 9 track tape reels (recorded at 6250 bpi). Due to this large volume of media, it was desirable to shrink the size of the proposed media to a single portable cassette. In addition to large size, a key requirement was that the data needs to be transferred to a (VME based) workstation at very high data rates. One gigabyte (GB) of data needed to be transferred from an archiveable media on a file server to a workstation in less than 5 minutes. Equivalent size, on-line data needed to be transferred in less than 3 minutes. These requirements imply effective transfer rates on the order of four to eight megabytes per second (4-8 MB/s). The DSRS also needed to be able to send and receive data from a variety of other sources accessible from an Ethernet local area network

Nakamoto, Glen

English

NASA Technical Reports Server

_y-_ 2....
/_/ _'_
_93-15034
Data Storage
and
Retrieval System
24 July 1991
Glen Nakamoto
MITRE Corporation
Bedford, MA 01730
(6 ! 7) 271-3032
https://ntrs.nasa.gov/search.jsp?R=19930005845 2020-03-17T08:34:38+00:00Z
Data Storage and Retrieval System
The Data Storage and Retrieval System (DSRS) consists of off-the-shelf system
components integrated as a file server supporting very large files. These fries are on the
order of one gigabyte of data per file, although smaller files on the order of one megabyte
can be accommodated as well. For instance, one gigabyte of data occupies approximately
six 9 track tape reels (recorded at 6250 bpi). Due to this large volume of media, it was
desirable to "shrink" the size of the proposed media to a single portable cassette. In
addition to large size, a key requirement was that the data needs to be transferred to a (VME
based) workstation at very high data rates. One gigabyte (GB) of data needed to be
transferred from an archiveable media on a file server to a workstation in less than 5
minutes. Equivalent size, on-line data needed to be transferred in less than 3 minutes.
These requirements imply effective transfer rates on the order of four to eight megabytes
per second (4-8 MB/s). The DSRS also needed to be able to send and receive data from a
variety of other sources accessible from an Ethernet local area network.
System Configuration
In order to meet these requirements, a system was configured using Aptec's
Input/Output Computer (IOC-24) with Storage Concepts C51 disk array and Honeywell's
Very Large Data Store (VLDS) tape drive (dual channel unit) as the basic components for
this file server. The I(3(2-24 has eight megabytes of shared memory and was hosted on a
VAX 11/750 which, in turn, was connected to an Ethernet local area network (LAN). The
interface to the VME based workstation was accomplished via Aptec's VME Gateway
Controller. The specific (and initial) VME based workstation that needed to be interfaced
for this project was the Sun 3/260 workstation containing a Vicom II-9 image computer and
the Vicom Fast Disk (Maximum Strategy's parallel disk array). The Sun workstation also
contained a 16 megabyte high speed (32 MB/s) memory card from Micro Memory. This
memory card was used as a high speed receiver (or _s_tter) of data during the initial
period (prior to the Vicom Fast Disk becoming available) for debugging purposes. Data
needed to be transferred to/from the DSRS file server (C51 disk or VLDS tape) to the
parallel disk array under the control of the Sun workstation operating at the data rates
previously discussed. The specific configuration is illustrated in Figure 1.
94
°°l
Ii
VLDS
Tape
Drive
[]
9-T_k
Tape
Drive
Disk
Drives
C51
Disk
Array
Ethemet
I I
VAX
Host
I
I/O
Computer H I
I
V'_V[E I
GC I
I
I
Vicom
Fast
Disk
[3-
Sun
Workstation
1
Vicom 1]-9
Image
Computer
Figure 1. Data Storage and Retrieval System
_Qmpgnent Performance
From a performance standpoint, the C51 disk is the fastest peripheral on the DSRS.
It has been measured as transferring (across the VMEbus to high speed memory) in excess
of 9 MB/s using transfer block size of one megabyte. The transfer block size is critical in
determining the effective transfer rate. Figure 2 shows how the effective transfer rate
varies with the block size used when transferring data from the C51 to the Vicom Fast Disk
(C512FD) or vice versa (FD2CS1). These transfers were done using files on the order of
500 M bytes of data. For the DSRS application, a block size of two megabytes (512 K
words) was chosen. This final size was dictated more by the Vicom Fast Disk rather than
the C51 disk. For this configuration, the C51 disk array has 2.5 gigabytes of formatted
disk space.
7M 6
B 5
e 4
S
!
s 3
e
c 2
i ii!11 il illl ilill
m ! in n me
16K
I i I I
32K 64K 128K 256K
Transfer Block Size in 4 Bytes Words
C512FD -O- 4:i- FD2C51
Figure 2. Transfer Rate as a Function of Block Size
!
512K
The VLDS is a streaming tape drive and streams data at approximately 4 MB/s.
Since it cannot start/stop like a conventional tape drive, it is imperative that it operate at its
full transfer rate. In theread/playback mode, the VLDS can stop/restart taking
approximately eight seconds to restart. Any significant _smatch in transfer rates between
the VLDS and another device could slow the overall transfer rate down to eight kilobytes
per second (KB/s). On the write/record side of the transfer, the VLDS will write "padding
blocks" while continuing to stream. However, there is a maximum number of padding
blocks that is (user) specified prior to the system halting (i.e., no restart). VLDS tapes that
have padding blocks will have a natural degradation in transfer rate as well as in tape
capacity. With no padding blocks, one VLDS cassette (super VHS T- 120 cassette) will
store approximately 5.2 gigabytes.
98
TheVME gateway controUcr can effectively move data from the IOC's shared
memory to the VMEbus on the workstation at rates in excess of I I MB/s. The transfer rate
also varies as a function of block size as shown in Figure 3. These transfer rates were
12
10
IV[
8
B
e 6
S
/
S
e 4
C
2
0
.25K
i I I I I I I i
.5K 1K 2K 4K 8K 16K 32K 64K
Transfer Block Size in 32-bit Words
Figure 3. VME Transfer Rate as a Function of Block Size
I
128K
obtained using the VME gateway controller in the Master mode. In the final configuration
of the DSRS, the VME gateway controller is used in the Slave mode. Experience to date
indicates that Slave mode transfer performance is similar to Master mode operation. For
Sun 3/2XX workstations, the CPU exhibits a 190 microsecond bus timeout. This means
that the VME GC cannot hold the bus longer than 190 microseconds after the CPU asks for
it else it times out and the Sun operating system (SunOS) crashes. Since the VME GC
operates in a release-when-done (RWD) mode versus release-on-request (ROR), it holds
97
the bus for the entire transfer. This practically limits the maximum block size to under one
kilowords in Master mode operation (in order to stay under the 190 microsecond time
limit). This limitation was overcome by using the VME GC in Slave mode and using a fast
master controller (Maximum Strategy disk continUer) operating in a ROR mode to effect the
high speed transfer.
While not part of the DSRS, the Vicom Fast Disk is the other key device when
examining end-to-end file data transfer. The Vicom Fast Disk is actually a disk array made
by Maximum Strategy, Inc. (Strategy One Controller). In this configuration, the disk army
contained approximately 6 GB of formatted space. The hardware is unchanged from the
original product. However, Vicom has written a device driver and a file management
system for it. The Vicom Fast Disk was rated at running at eight MB/s. While Vicom
literature indicates 12 MB/s burst and eight MB/s sustained, in reality, 12 MB/s burst only
occurs during the first megabyte of data (since it comes from memory and not the disk
array). The eight MB/s sustained rate applies to the transfer once the disk heads are in
position and is "streaming" data. In reality, large block sizes were needed to maintain
transfer rates near eight MB/s.
While all the hardware components were available off-the-shelf, no software was
available to allow the components to function as a system. The key integration task was to
develop the software and ensure that device performance would not be impeded by this
software. A key challenge was to ensure that the VLDS tape unit operated at its full 4 MB/s
capacity since it functions as a streaming tape drive. As indicated earlier, running the
VLDS at slower than 4 MB/s would significantly slow down the total transfer time.
Paramount to the development effort was to keep all overhead to an absolute minimum.
Another challenge was to keep all software development at a high level - not develop any
assembly language or micr_ode. Efficient use of library routines was essential. In order
to promote a high level of portability, all routines developed on the Sun workstation had to
be written in the C language and interfac,'xt to existing Vicom device drivers. No kernel
modifications were allowed as well. Since the software effectively controlled key hardware
components directly, it was extremely difficult to debug since the typical error message was
"bus error" (followed by a system crash). The use of a bus logic analyzer was essential to
do problem identification and debugging. During this development effort, several key
problems were discovered and t"kxed on the VME gateway controller. All problems dealt
08
with the use of the VME GC in slave mode operating at high (-.12 MB/sec) speeds.
Software modifications at the microcode level (in Aptec's software) were also made to get
the system operating properly. These changes have now been incorporated as part of
Aptec's baseline.
VLDS Ta_ Unit
The VLDS was the first peripheral to be interfaced to the IOC-24. By using a dual
buffering scheme to keep data transferring between the VLDS and the shared memory, the
effective transfer rate of 4 MB/s was easily maintained when sending data to the IOC's
memory. When the VME gateway controller (VME GC) was added to the IOC, data was
then made to flow from the VLDS, to the shared memory, to the VME GC Input/Output
Processor (IOP), to the VME GC, and then on to a high speed memory card on Sun 3/260
workstation with no delays. Later when the Vicom Fast Disk was added to the Sun
workstation, it became apparent that the Fast Disk needed large (one megabyte or larger)
transfer blocks in order to maintain high throughput. Since the VLDS reads and writes in
principal block increments of 65,536 bytes (64 KB), a buffer size mismatch needed to be
fixed. The dual buffering scheme had to be modified to accommodate 64 KB buffers on
the VLD$ side and 2 MB buffer size on the VMF/Vicom side. This was accomplished by
using multiple VLDS buffers adding up to the (two MB) VME side buffer, then taking into
account partial buffers and last buffer anomalies. With this approach, it became possible to
transfer files from slow devices such as the Sun local disk to/from the VLDS at high data
rates (as fast as the slowest device) for files up to 2 megabytes in size with no degradation
in performance. Larger files could also be sent but the VLDS start/stop action would cause
a degradation in performance.
Interfacing the C51 disk array into the DSRS involved making a key decision
regarding its use. The C.51 could be used in a dedicated manner, i.e., used by a single
process until completed, or shared like a disk server. Used in a dedicated manner, the
performance would be optimized and the software would be easier to develop. The major
drawbacks were that the disk array would not be shareable between processes and only
contiguous files would be supported. This last condition was quite restrictive when there is
2.5 GB of disk space and file sizes of one to two GB could be expected. The contiguous
requirement would prevent files from being written even though the space may exist due to
99
disk fragmentationor possiblybaddisk blocks/sectors. The DSRS configuration uses the
more complex VAX/VlVIS file management system (QIO) to create and manipulate files on
the disk array. This allows files to be written out with multiple extents ff needed. Another
advantage of this approach is that the disk subsystem is shareable by different processes.
Thus, a lengthy transfer that may take 3 minutes can function at the same time that another
user is accessing a small file on the same disk, without having to wait for the first transfer
to complete. From a VAX user viewpoint, the (251 appears as a standard disk (though
non-system bootable) and functions, such as file transfer (ftp) and copy, can be used to
transfer data from VAX based peripherals or other peripherals attached via the Ethemet
LAN. Transfers involving the (251 also use a similar dual buffering scheme to maximize
the transfer to/from the IOC's shared memory then on to the destination device.
VME Gatew_ Controller
The VME GC was, by far,themost challengingpieceof equipment to understand
and integrateintothe system. A seriesof protocolswere developed to allow the DSRS to
communicate with thetargetworkstation.The firstprotocol(command protocol)involved
sending a command and appropriateparametersfrom the Sun workstationindicatingwhat
transferneeded tobc done with what files.The second protocol(informationprotocol)
involvedcommunicating informationregardingfilesize,transferblock size,and sizeof the
lasttransfer.This informationwas criticalto ensureboth sides(DSRS and workstation)
knew exactlywhat and how much datawas being sent.Finally,the transferprotocol
involved the low level "handshaking" needed to keep the data transferred in proper
synchronization, i.e., ensure that source and destination devices were ready to receive the
appropriate blocks of data. From an ISO networking viewpoint, these protocols fall into
the application layer. They effectively establish a means of communications at a high level
between a workstation and the file server (DSRS) for the transfer of a file. From a logical
viewpoint, the VME GC is set up like a data structure featuring a first-in-fi_t-out (FIFO)
location, a mailbox area for small messages, and a set of 16 registers. The base address to
thislogicalstructureisuserpmgranm_ble and can existanywhere withinthe workstation's
memory space. Physically, the VME GC is also user programmable and can exist
anywhere within the VMEbus 32-bit address space (barring colLflicts with existing
devices). The FIFO is used to wansfer large blocks of data between the workstation and
the DSRS. The mailbox region is used to pass file names, file sizes, and other
miscellaneous pieces of information. The registers perform all control functions including
setting the direction of transfer, number of bits per word, FIFO full/empty indication,
IO0
semaphore acknowledgements, etc. The DSRS is set up with a predefined data structure.
The Sun workstation also sees this same data strucune and communicates with the DSRS
by reading and writing to these memory locations as if the DSRS was local to it. By using
this approach, different VME based workstations can be integrated with the DSRS with
minimal difficulty. All that is needed is an understanding of how to memory map to a
specific memory location and a VME memory device driver (both commonly standard with
any workstation/operating system). An interface guideline document describing both the
hardware interface requirements (for the VME GC board set) as well as a detailed
description of the above protocols (for a software interface) has been published.
Typical Data Flow (C51 to Vicom Fast Disk)
Once a user on the Sun workstation "launches" a transfer function, the VME IOP
spawns a request to the C51 IOP to initiate the file transfer. Four megabytes of shared
memory (organizedas two 2 MB buffers)are allocatedinthe IOC. The C51 IOP's taskis
to fill each 2 bib buffer and mark the empty semaphore flags as "full" upon completion of
writing a buffer. At the same, time, the VME IOP reads the same buffers (when the
semaphore indicates that the buffer is "full") and flags the buffer "empty" upon completion
of theread function.The two processesrun _ switchingbuffersas necessary to
keep a steadyflow of datamoving. When theVME IOP readsthe 2 MB buffer,thedatais
simultaneouslytransferredtotheVME GC which has previouslyinitiated"handshaking"
with the Sun workstationCPU. In the meantime, theSun CPU has transferredcontrolto
theVicom FastDisk controllerwhich masterscontrolof the VMEbus and extractsthe2 MB
of datafrom the (FIFO addresslocationof the)VME GC and transfersthedata toitsinput
bufferand subsequentlyto thedisk array.
This single cycle which involves transferring data across three busses (DIB,
OPENbus, and VMEbus) with "handshaking" and resource arbitration is executed 500
times ( to transfer a gigabyte of data) and runs at 96% of the maximum burst speed of the
slowest link.
Software Architecture
The software on the VAX was set up as a single program running as a server
listening to commands (via a semaphore register) that it might receive from the Sun
workstation. In actuality, the VME IOP has software constantly checking one of the
101
registerstodetermineif a workstation wants to deposit a command into the mailbox. Once
a command is received, the VME IOP is allocated and cannot be used by another device
until the command has been completed. The VME IOP "downloads" the appropriate code
and executes all transfer routines or spawns appropriate routines to accomplish the
commands. Upon completion, the semaphore register is setup for the next request. During
the transfer of information, ff any substantial delays occur, either side will tirneout and
revert back to listening mode. This allows the seTver software to recover from errors that
may occur on the workstation. Using this approach with multiple VME IOPs and VME
GCs would allow multiple VME based workstations to access the DSRS resources in
parallel, limited only by the device s_, the amount of shared memory, or bus
bandwidth within the IOC-24. Over 60 STAPLE routines/procedures were written to
support this server software.
The software on the Sun workstation was written as a series of short C routines that
effect a transfer from one device to another. A typical routine name was vlds2fd implying
(in this case) transfer of data from the VLDS to the Vicom Fast Disk CFD). Following the
= === L
command, the source and destination file names would be included as parameters. Ten of
these routines were developed to transfer data in all conceivable directions. Over 30
routines/procedures were written to support the Sun based software.
Due to the development environn'_nt and the tools that were, available, the entire
system integration and software development effort for the DSRS took less than four
months (with a staff of two). Interfacing the Sun/Vicom system took an additional two
months (although hardware problems precluded use of the system for almost half of that
time).
In the end, the DSRS successfully transferred one gigabyte of data from the VLDS
to the Sun/Vicom Fast Disk in 4 rain. 13 sec. + 25 sec. for tape setup and rewind. This
translates to an effective transfer rate of 4 MB/s during the transfer (which is the streaming
rate for the VLDS). The file transfer from the C51 _-_ array to the SunNicom F-ast]3isk
took 2 rain. 15 sec. using 2 MB transfer blocks. This translates to an effective transfer rate
of 7.7 MB/s (out of a theoretical maximum rate of 8 MB/s as constrained by the Vicom Fast
Disk). File transfer was also perforn_ between the C51 disk array _ a device (a local
disk attached to another Sun workstation) over Ethemet while, at the same time,
transferringa filefrom theVicom FastDisk to theC51. Although theperformance was
slightlydegraded due tothe sharingof the C51 disk,itwas not noticeableusing small files
(lessthan 10 megabytes).
All requirements were fulfilled using commercially available off-the-shelf
components with a relatively small software development effort. The system is now
operational and is being used to store and retrieve large files on VHS cassette tapes and can
load the Sun workstation (Vicom Fast Disk) in minutes versus the many hours it used to
take when using 9 track tapes.
108
0_
ca
I.-
_ c
m
c_
|mC_ in
CD '_
10dL
"(3
£:
0
Im
[3')
0
m
im
o
Q.
x
a)
a)
105
EL_
13.
<C
@Q.
Ca
I"
t_
Q
e,-
@
qP
t=
dkl
Q.
m
qP
4rim
lil
O
_u
?
u
{o
I,--
41P|
qp
e"
mum
Z
0
a=
Q,
C
Hal
QO
liB
0
"1"
'WE
c_
leo
'r"
i
tt_
106
• oI
I i _ I I 'I::
• _
_ w e_
I
L_
108
0
I
2
iil
C
0
ibm
a3
tot
¢::
0
¢,.)
i
I
[]
110

14rap
112
a}
N
llm
U}
O
m
N.,.=
a3
f_
I.I=
|
I
I
I
I
I I
I I
I i
IZ3
(I)
I....
4)
I
114
c_
i
=-
115
0
0
116


Data storage and retrieval system

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server