Three experiments on fault tolerant multiprocessors (FTMP) were begun. They are: (1) measurement of fault latency in FTMP; (2) validation and analysis of FTMP synchronization protocols; and investigation of error propagation in FTMP

Shin, K. G.

English

NASA Technical Reports Server

General Disclaimer 
One or more of the Following Statements may affect this Document 
 
 This document has been reproduced from the best copy furnished by the 
organizational source. It is being released in the interest of making available as 
much information as possible. 
 
 This document may contain data, which exceeds the sheet parameters. It was 
furnished in this condition by the organizational source and is the best copy 
available. 
 
 This document may contain tone-on-tone or color graphs, charts and/or pictures, 
which have been reproduced in black and white. 
 
 This document is paginated as submitted by the original source. 
 
 Portions of this document are not fully legible due to the historical nature of some 
of the material. However, it is the best reproduction available from the original 
submission. 
 
 
 
 
 
 
 
Produced by the NASA Center for Aerospace Information (CASI) 
https://ntrs.nasa.gov/search.jsp?R=19850024508 2020-03-20T18:18:55+00:00Z
^}	 ;  ^; alt• ^. ^-. ^	
'±.^—.t
I .^
STATUS REPORT
on the Projects under Support of
NASA Grants NAG-1-298, NAG-1-492, and NGT-23- 00r- -801
(NASA-Ch-176098) FEBFCENINCF IND E9rALUAT10N
OF HEAL-LIME HU121CCMEDIEB CCNI ECL SYSTEMS
Status Report, Sep. 1584 - Aug. 1985
(dicbigan unio.)	 1 1 F HC !02/mF A01
CSCL 09B G3/62
NEE-32821
uncla:
22016
By
Kang G. Shin
Department of Electrical Engineering and Computer Science
The University of Michigan
Ann Arbor, Michigan 48109
(313) 733-0391
Prepared for:
NASA Langley Research Center
Hampton, Virginia 23885
Attention: R. W. Butler
o^
August 1985
'J''I
C+J'
August lb, 2985	 Kang G. Shin: Statue Report
1. INTRODUCTION
In this status report, we summarixe the research accomplishments, under the
auspices of NASA grants NAG-1-298, NAG-1 . 492, and NGT 23-005-801, in the area of
real-time computing during the period of September 1984 - August 1985. Since real-time
ccmputing systems usually require both high performance and high reliability, our
research effort has been focused on the design and analysis of computers that are ,t
and reliable.
2. SUMMARY OF ACCOMPLISIDAENTS
Due to the long-term nature of our projects, we have taken a step-by-step approach
to the design and analysis of real-time computing systems, beginning from the definition
Of new performance measures and analytical modeling/simulation to experimental valida-
tions. Validations have been pursued mainly via various experiments at the NASA AIR-
LAB. Two of the three such Airlab experiments that we have conducted during this
period are outlined below.
Application of our performance measures to the design and evaluation of real-time,
fault-tolerant computers has yielded several very interesting results as shown below and
elsew here.
2.1. Experiments on FTMP at the NASA Airlab
Our fundamental approach consists of both analytical modeling and experimenta,
validation. Although the analytical part has been far more advanced than the experi-
meutal counterpart (for both our case and others'), we feel that experimental validation
is difficult but cjecnfial.
f Ifffffn
1
August lb, 1986	 Kang G. Shin: Status Report
To this end, three experiments on FTMP were begun, and one of them has been
completed, and the remaining two are now in progress. These are (i) measurement of
fault latency in FTMP [131,[1111, (ii) validation and analysis of FTMP synchronization
protocols 1111,116), and (iii) investigation of error propagation in FTMP. (i) h as been
completed and (ii) and ( iii) are now in progress. Experiments (ii) and (iii) are described
below. ( See 1131, 1181 for a detailed description of (i).)
2.1.1. Evaluation of FTMP S)	 onisation
We have developed an analytic model that represents the state of a processing ele-
ment in a multiprocessor with time-shared buses. This model consists of several states
representing idle, processing, failed, communicating, and waiting states of a single pro-
censor or a single trizd of processors. Especially, the model emphasizes (i) validation of
processor synchronization protocols and (ii) the impacts of bur contention and failure
handling on computer performance, i.e. dynam;c failure probability, and mean and vari-
anc ! costs.
t
The transition rates between the model states will describe the effects of workload,
bus contention, architecture, and reliability on a multiprocessor system. To measure
these transition rates, we have adapted our model to the srchiteeture of FTMP and have
already conducted some experiments on FTMP at the NA SOA AIRLAB. During our
recent tips to the AIRLAB, we have been able to make direct measurements on the
FTMP to aid in arriving at experimental values for the transition rates. Measurements
are made through software functions and using the logic analyzer in the Lab. The meas-
urements obtained so far are preliminary. To obtain more concrete measurements,
experiments at Airlab are planned to continue in the coming months and years.
2	 I11
^ 1
}
rAugust lb, 1985	 KaLg G. Shins Status Report
With the results of FTMP experiments, we will have a better understanding of the
accuracy of the model and how it may be solved analytically. The validated model will
then be used as a dependable but economical tool for evaluation and design of real-time
avionics computers.
2.1.2. Measurement of Error Propal,ation in FTMP
The most important parameter in our error propagation model (discussed in our
last year's proposal) is the error propagation time for each un:t. Wher, a fault occurs in
the system, it will induce an error first in the faulty unit, and this error may propagate
and cause additional errors in other units prior to its detection. (This is analogous to
"snowballing effects".) In such a case, the error is said to propagate from a faulty point
It a detection, point. If the system is divided into many units, each of which may have
inputs from several other units .-nd outputs to several other units, then the error propa-
gation time is defined for each input/output pair of a unit as the time for an error to
propagate from an input port to an output port. The purpose of our current experi-
ments at the NASA AIRLAB is to directly measure error propagation times on FTMI
The processor region of an LRU in FTMP can be divided into the following units:
(1) a CAPS-6 bit-sliced microprocessor, (2) a cache memory including 8K PROM and 8K
RAM, (3) a system bus coupler, (4) a group of control and communication registers, and
(5) an interval timer and an address mapper. Basically, all units communicate via the
transfer bus, but only the CAPS-6 and the system bus coupler can initiate a bus
transfer. Since the error detection is built upon the system bus, any error in the proces-
sor region must propagate through the system bus coupler before the system detects
that error. Thus, we have to first measure the error propagation time for the system
bus coupler with inputs from the CAPS-6 and outputs to the system bus interface unit.
1111111
I=
i=
3 ti
1
A
L
^Z.
August lb, 1986	 Kang G. Shin: Status Report
The method we use to detect input and output errors is to compare the input and out-
put line values of the system bus coupler between LRU3 (upon which faults are injected)
and LRUO (which should be in the sam e. triad as LRU3 when faults are injected). By
inspecting the circuit diagram, we have defined 26 input lines and 4 output Imes for our
experiments. We are now building t'—, e comparison circuit for error detection, and
tecause the clocks in both LRUs are not exactly synchronized, we have encountered
some difficulties in designing the circuit. However, we believe that by carefully sampling
and buffering the input and output lines of the comparison circuit, it will be possible to
detect errors correctly.
These experiments are expected to be continued for the rest of this year and also
for the ccming years. As described above, we w.-11 begin experiments at low level
modules in FTMP and then expand them to higher level subsystems in FTMP as well as
other systems that will be available at the AIRLAB and elsewhere.
2.2. Application of Po-rformance Measures
We have applied our performance measures to various problems associated with the
design and analysis of fault-tolerant, real-time computing systems. As can be seen below
and from oar publication list, there are numerous important applications, indicating the
power and usefulness of our performance measures. NVe feel that these measures can be
applied to almost all, if not all, problems associated with design, validation, and analysis
of real-time computing systems. Consequently, the application, that we have dtait with
thus far are only a beginning. Nlore applications will be dis;ussed in the forthcoming
renewal proposal.
4
1
r
'I
vi
i^
August lb, 11185
	
Kang G. Shin: Status Report
2.2.1. Optimal Reconfiguration of a MvIti-Module System
We have developed a new quantitative approach to the problem of reconfiguring a
degradable multi-module system. The approach is concerned with both assigning acme
modules for computation and arranging others for reliability.
Conventionally, a fault-tolerant system performs reconfiguration only upon a sub-
system failure. Since there exists an inherent tradeoff between the computation capacity
and fault-tolerance of a multi-module computing system, the conventional approach is a
passive action and Goes not yield a configuration which provides an optimal compromise
for the tradeoff. Using the expected total reward as the optimal criterion, we have
shown the need and existence of an active reconfigttration strategy under which the Sys-
tem reconfigures itself on the basis of not only the occurrence of a failure but also the
progreaeion of the mi8sion.
Following the problem fcrmulation, we have investigated some important properties
of an optimal reconfiguration strategy which specify (1) the times at which the system
should undergo reconfiguration, and (ii) the configurations that the system should
change to. Then, the optimal reconfiguration problem is converted to integer nonlinear
knapsack and fractional programming problems. V1't have developed various algorithms
for solving these problem and worked out a uemonstrative example. Fee 171 for a
detailed description.
2.2.2. Scheduling with a Quick Recovery from Failure
Multiprocessors used in life-critical real-time *-stems must recover quickl y from
failure. Part of this recovery consists of switching to a new task schedule that ensures
that hard deadlines for critical tasks continue to be met. We have developed a dynamic
programming algorithm that ensures that backup, or contingency, schedules can be
5
'i T!I
	 3*t ,
.0
August 16, 1966
	
Kang G. Shin: Status Report
efficiently embedded within the original, "primary" schedule to ensure that bard dead-
lines continue to be met in the face of tip to a given maximum number of processor
failures. Several illustrative examples have also been worked out. See [4) for more
details.
2.2 1. Modeling of Real-Time Multiprenessors with Time- Shared Buses
In this project, the workload effects on computer performance are addressed for a
highly reliable unibus multiprocessor used in real -time control ( 11IJ16]. Because of the
strict performance criteria required by a system of this type, it would be desirable to be
able to determine the significant effects of workload distribution and scheduling on per-
formance.
As an approach to studying these effects, a modified stochastic Petri net (SPN) is
used to describe the synchronous operation of this system. From this model the vital
components affecting performance can be determiu-d. However, because of the complex-
ity in solving the modified SPN, a simpler model is constructed that presents the same
critical aspects. This model is a closed queueing network. It consists of multiserver 	 I
nodes and a non - preemptive priority queue. The use of this model for a specific applica-
tion requires the partitioning of the workload into job classes. The steady state solution
of the queue ;,ig model directly produces useful results, such as idle processing time, sys-
tem bus contention, and bus queueing times, that are necessary in any performance
evaluation. This model has been used in evaluating FTMP.
2.2.4. Optimal Checkpointing in Real -Time Systems
l a sing the basic concept of cbeckpointing in database systems, we have developed
analyt cal models for the design and evaluation of checkpointing in real-time systems,
d	 IA 
1
t^
R .i
	 77	 . . _..-	 \ r
August lb, 1985	 Kang G. Shin: Status Report
which usually require high periormbnce and reliability (12).
First, we have modeled the behavior of a real-time task under the common assump-
tion of perfect coverage of on-line detection mechanisms (which is termed a basic model).
Then, we have generalized the model (to an extended model) to include more realistic
cases, i.e., imperfect coverage* of on-line detection mechanisms and acceptance tests.
Finally, we have determined an optimal placement of checkpoints to minimize the Dean
task execution time while the probability of unreliable results (or lack of confidence) is
kept below the specified level. Consideration of both the case of imperfect coverages and
the probability of lack of ronfidence is a significazit departure from previous approache, ­
by others.
For the basic model we have Shown that zc. equal distant iaten-al is optimal,
whereat for the extended L-iodel tL:s is not necessa--ily true. For the latter we have
derived a numerical algorithm which produces a better solution than the common usual
solution, i.e., equi-distance (inter-c heck poi at) intervals.
2.2.5. Synchronisation of a Large Cock Network
Clock Synchronization is one of the main problems associated with the design of a
multiprocessor system, especially when there exist malicious faults. Although over the
past few years many different algor4hms have been propo4ed for overcoming this prob-
lem, they are not suitable for a large real-time multiprocessor system due to their exces-
sive time overhead and/or large number of interconnections.
To remedy this problem we have developed a new method that (1) requires little
time overhead by using the phase- locked clock synchronization, and (ii) uses only 20-
W0 of the total number of interconnections required by the other methods for almost
no loss in the synchronizing capabilities (10). This drastic reduction in the total number
IN
7 A
It
PMIII,
August 16, 1086	 Kang G. Shin: Status Report
	 i n
of interconnections is made possible by grouping the various clocks in the system into
mAny different clusters aed then treating the clusters themselves as single clock units as
far as the network is concerned. The method is significant in that regardless of their
sizr multiprocessor systems can be built at an inexpensive cost without sacrificing both
the synchronization and fault tolerance capabilities.
To show the feasibility of our meth-d, we have siso developed an example
hardware implementation.
11
2.2.0. Processor Tradeoff in Distributed ilea{-Time Systems
Optimizing the design of real-time distributed systems is important since the sys-
tems are frequently critical to life. This optimization is a difficult problem, and heuris-
tics and designer judgement are called for in the process. The chief cause of the difficulty
is the large number of parameters under t ae designer's control which impact perfor-
mance and life-cycle cost. We have studied the interplay between the more important
parameters in this project using two objective measures, i.e., the mean cost and the pro-
bability of dynamic failure that we have previously developed. Among these parameters
are the processor burn-in time and processor rep:acement policy. A central feature of
this work is a look at how the application requirements affect the optimality of the dis-
tributed systems: indeed, the application requirements are an integral part of the
analysis. See (9] for more details.	 1
2.2.7. Implications of the Interactive Convergence Algorithm
Fault-tolerant synchronization is crucial to the proper functioning of controller
computers. Ir this project, we have assessed the Interactive Convergence Algorithm of
Lamport and Melliar-Smith, and studied the interaction of tightness, fault tolerance, and
8 1
A`
August lb, 1086	 Kan& G. Shin: Status Report
overhead in the context of the SIFT aircraft control system. See 1141 for a detailed
description.
3. PARTICIPATING PERSONNEL
•
	
	
Yann-Ilang Lee: Ph. D. completed in December 1984. Currently working a.-4 a
researcher at IBM Thomas J. Watson Research Center.
•
	
	
Michael Woodbury: M. S. completed in December 1984. Currently working towards
the Ph. D. degree.
•	 Michael Lin: Second-year Ph. U. student.
•	 R. Parameswaran: M. S. expected in April 1988.
•	 James Dolter: M. S. expected in December 1985.
•	 Y. Muthuswamy: New M. S. student beginning Fali 1985.
4. TECHNICAL PAPERS AND REPORTS
The following technical papers/reports present in some detail the work accom-
plished thus far.
(11
	
	
C. M. Krishna, K. G. Shin, and Y. - H. Lee, "On Optimal Criteria for Check-
poiuts Placement," Communications of ACM, Vol. 27, No. 10, pp. 1008-1012,
October 1984.
[21	 K. G. Shin and Y. - H. Lee, "Analysis of Backward Error Recovery for Con-
current Processes with Recovery Blocks", IEEF, Trani. on Software Engineering,
Vol. SE-10, No. 6, pp. 992-700, November 1984.
13] K. G Shin, C. M. Krishna, and Y. - H. Lee, "A Unified Method for Evaluating
Real-Time Computer Controllers and Its Application," IEEE Trans. on
Automatic Control, Vol. AC-30, No. 4, pp. 357-389, April 1985.
141
	
	
K. G. Shin and C. M. Krishna, "The Processor Number- Power
 Tradeoff in a
Class of Multiprocessors," Proe. of the 5-th Int'l C'onf. on Diet.• ibuted Computing
Systems, pp. 321-328, May 1985.
.^
A
9
+^ 1
_ ^E'
August 16, lo g s	 Kang G. Shins Statu s Report
15J C. M. Krishna and K. G. Shin, "On Scheduling Tanks with a Quick Recovery
from Failure", Digest of FTCS-15, pp. 234-239, June 1985. Also to appear in
IEEE Tr ans. on Computers.
[6) C. M. Krishna, K. G. Shin, and R. W. Butler, "Ensuring Fault- Toleranc , of
Phase-Lock-d Clocks", IEEE Trans. obi Computers, Vol. C-34, No. 8, pp. 752-
756, August 1985.
171 Y. - H. Lee and K. G. Shin, "Optimal Reconfiguration Strategies for a Degrad-
able Multi-Module Computing System," accepted to J. of the ACM subject to a
minor revision.
181
	
	 1'. - H. Lee and K. G. Shin, "Optimal Design and Use of Retry in Fault Tolerant
Renl-Time Computer Systems", submitted to J. of ACM. (First revision Febru-
ary 1986.)
(9J
	
	
C. M. Krishna, K. G. Shan, and I. S. Bbandari, "Processor Tradeoff in Distri-
buted Real-Time Systems," to appear in IEEE Trani. on Computers.
[10) K. G. Shin and R. Parameswaran, "Svochronization of a Large Clock Network
in the Presence" of Malicious Failures," Proe. 1985 Real- T l*m.c Systems Symp. (to
appear). Also submitted to IEEE Trans. on Computer;.
1111
	
	
M. H. Woodbury and K. G. Shin, "Performance Modeling of Real-Time Mul-
tiprocessors with Time-Sharrd Bus ►^,, '­ submitted to IEEE Trans. on Computers.
I'
(12)
	
	
K. G. Shir, T. - H. Lin, and Y. - H. Lee, "Optimal Cherkpointing in Real-Time
Systems," submits;-A to IEEE Trans. on Computers.
[131
	
	
K. G. SLIn and Y. - H. Lee, "Measurement of Application of Fault Latency,"
submitted to IEEE Trans. on Computers.
[14]	 C. M. Krishna and K. G. Shin, "Operational Implications of the Interactive Con-
vergence" Algorithm," submitted to IEEE Trans. on Softtrare Enginerring.
1151
	
	 C. M. Krishna and K. G. Shin, "Performance Measures for Control Computers,"
submitted to IEEE Trans. on Computers.
[h]	 K. G. Shin, M. H. Woodbury, and Y. - H. Lee, "Modeling and Measurement of
Fault-Tolerant Multiprocessors," NASA Contractor's Report, May 1985.
1171	 Y. - H. Lee ; "Cbaracterization of Failure Handling in Fault- Tob-rant Multipro-
cesaor Systems," Ph.D. Thesis, The University of Michigan, November 1984.
10


Performance and evaluation of real-time multicomputer control systems

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server