An important issue in the effective use of parallel processing is the estimation of the speed-up one may expect as a function of the number of processors used. Amdahl's Law has traditionally provided a guideline to this issue, although it appears excessively pessimistic in the light of recent experimental results. In this note, Amdahl's Law is amended by giving a greater importance to the capacity of a program to make effective use of parallel processing, but also recognizing the fact that imbalance of the workload of each processor is bound to occur. An activity set model of parallel program behavior is then introduced along with the corresponding parallelism index of a program, leading to upper and lower bounds to the speed-up

Gelenbe, Erol

English

NASA Technical Reports Server

. 
Multiprocessor Speed-Up, Amdahl’s Law, and the 
Activity Set Model of Parallel Program Behavior 
Erol Gelvnbe 
December 1988 
Research Institute for Advanced Computer Science 
NASA Ames Research Center 
RIACS Technicai Report 88.37 
NASA Cooperative Agreement Number NCC 2-387 
(NASA-CR-185422) MULTfPiiOCESSOB SPEEC-UP, 
AMDAHL’S LAW, A N D  THE ACTIVlTY SET MODEL OF 
PARALLEL PROGBBM B E H A V I O R  LResearch I n s t .  
for  Advanced Computer S c i e n c e )  16 pCSCL V 9 B  
G 3 / b 2  
I 
N 89-26452 
U n c l a s  
0 2 1 7 9 2 5  
Research Institute for Advanced Computer Science 
\ 
https://ntrs.nasa.gov/search.jsp?R=19890017081 2020-03-20T01:27:02+00:00Z
Multiprocessor Speed-Up, Amdahl’s Law, and the 
Activity Set Model of Parallel Program Behavior 
Erol Gelenbe 
December 1988 
Research Institute for Advanced Computer Science 
NASA Ames Research Center 
RIACS Technical Report 88.37 
NASA Cooperative Agreement Number NCC 3-387 
It4 u It i p rocesso r S peed - Up , A m d a h 1 ’ s La w , a n d the 
Activity Set Model of ParalIel Program Behavior 
Erol Gelenbe* 
Research Institute for Advanced Computer Science 
NASA Ames Research Center 
RIACS Technical Report 88.37 
December 1988 
An important issue in the effective use of panllel processing is the estimation of the speed-up one 
may expect as a function of the number of processors used. A n d h i ’ s  Law has tradtionally 
provided a guideline to this issue, although it appexs excessively pessimistic in the light of recent 
experimantal results. In this note we first amend Amdahl’s Law by gwing a greater importance to 
the capacity of a program to make effective use of p d e l  processing, but also recognizing the 
fact that imbalance of the workload of each processor is bound to occur. We then introduce the 
Activity Set Model of parallel program behavior and the corresponding parallelism index of a 
program, leadmg to upper and lower bounds to the speed-up. 
* The author’s address is Universite Rene Descartes, Ecole des Hautes Etudes en Inforrnatique, 
45 rue des Saints Peres, 75006 Paris, France.,This research was supported in part by Cooperative 
Agreement NCC 2-387 between the National Aeronautics and Space Administration (NASA) and 
the Universities Space Research Association (USRA) whle the author was visiting the Center for 
Advanced Architectures at RIACS. 
1 .  Introduction 
Consider a program that is to be executed on a multiprocessor 
containing N identical processors. 
be equipped with one or more array processors attached for its private 
use. 
Each of these processors may itself 
Amdahl's law [1,2] estimates an upper bound of N/log N for the 2 
actual speed-up, or ratio of elapsed time with a single processor to the 
elapsed execution time with N processors. Recent experiments [ 3 ]  have 
shown that this may be an unnecessarily pessimistic estimate of speed-up 
and that values close to N may be obtained for specific applications. 
We have recently derived another approach to computing the estimat- 
ed maximum speed-up [4], assuming an unlimited number of processors, 
based on a model which considers the density p of precedence relations 
between tasks in programs. We obtain [4] an approximate formula 
(1 + p)/2p for the maximum speed-up, on the average, for such a family 
of programs assuming that the number of processors available is unlimit- 
ed. When p is close to 1, nearly all tasks are interdependent and the 
programs will in fact execute sequentially; the speed-up factor itself 
will be close to just 1 .  On the other hand, when p is very close to 0 
we are dealing with programs composed of many quasi-independent tasks 
and the maximum speed-up is very large. An "infinite" speed-up merely 
means that the average execution time for the program family remains 
nearly constant as the number of individual tasks in the program becomes 
very large. 
In this paper we shall begin by considering Amdahl's law and suggest 
some modifications or amendments that can serve to explain speed-up 
factors which are nearly linear in the number of processors. We shall 
2 
then return to the intrinsic processor independent behavior of parallel 
programs and introduce a new model of parallel program behavior, the 
Activity Set Model,' which may be used to describe the behavior of 
parallel programs and to derive bounds to the speed-up which can be 
expected when such programs are execdted or multiprocessors. 
We first consider a simple amendment to Amdahl's law, and derive a 
speed-up fotmula with N processors which is of the form 
N 
[ ( l  - € ) ( 1  + 6) + E log2N] 
where E is the probability that the programs cannot effectively use N 
processors. 6 i s  a measure of the unbalance between the workloads of 
each processor when N processors are used. Indeed, 6 / N  is the amount of 
time in excess of the optimistic equal run-time 1 / N  which the most loaded 
processor will take to run the task that has been assigned to it. Thus, 
when the computation has been organized so that E is very small, the 
speed-up can be as high as N / ( 1  + 6). These results are detailed in 
Section 2. 
In Section 3 ,  we consider an evaluation of speed-up based on 
intrinsic program behavior which leads to general bounds. 
Throughout the discussion we consider a program, or a family of 
programs, whose average run-time on a single processor is equal to 1 .  
'The name of this model is, of course, inspired by Peter Denning's 
Working Set Model used in paging. 
3 
2 .  Amendment t o  Amdahl's Law 
Assume t h a t  t h e  program, o r  family of programs, considered makes 
f u l l  use of t h e  N processo r s  w i th  p r o b a b i l i t y  ( 1  - E ) .  I t  w i l l  use only 
i p rocesso r s  (1  5 i S N - 1 )  wi th  p r o b a b i l i t y  c i ,  where 
N- 1 
1 
E =  2 E i  
If it uses  N p rocesso r s ,  one of t h e s e  w i l l  have t h e  g r e a t e s t  work- 
load so t h a t  t h e  run-time o f  t h e  program on N processo r s  i s  given by 
t h e  t i m e  e l apsed  f o r  t h e  most heav i ly  loaded one, o r  
1 6  
N N  
- + -  
Obviously, i f  a l l  t h e  p rocesso r s '  workload were p e r f e c t l y  balanced,  
t h e  e l apsed  time would be simply 1 / N .  
S i m i l a r l y ,  when only i p rocesso r s  a r e  used t h e  elapsed t i m e  w i l l  be 
i 1 - + -  
i 1 
6 
These t i m e s  should be viewed a s  average va lues  when a family of 
programs i s  considered.  
Thus, t h e  average elapsed time when N processo r s  a r e  a v a i l a b l e  i s  
given by t h e  formula 
So t h e  speed-up T(l)/T(N) = 1/T(N) i s  simply given by 
4 
N 
6 S =  N-1 
1 
[ ( l  - ~ ) ( 1  + 6) f N 2 + 
The Most Favorable Case 
Naturally, the greatest speed-up will be obtained if we set 
= E ,  E 
We then have for large enough N: 
= 0 for 1 S i 5 N - 2.  N- 1 i E 
,N 
1 + 6(1  - E)  + bN-l S z :  
This formltla expiains the quasi-linear speed-ups that can be 
encountered for some specific applications. 
The Least Favorable Case: Amdahl's Law 
We consider that the least favorable case for a family of programs 
is obtained by setting 
= 1 - E 1/N, for  all 1 5  i 5 N - 1 'i 
which implies that we are equally likely to make use of any number of 
processors less than N; this is the assumption made in Amdahl's law. 
then have 
We 
6N = 6 N N 6i' S =  
Z . + Z -  N 1  
N l  Since 2 - 2 log2N, we have i 1 
(3)  
Amendment to Amdahl's Law 
Let us consider a family of programs that can fully use all N 
processors with probability 1 - E .  If a program in this family cannot 
make full use of them, then assume that it is equally likely to use 
N - 1, N - 2, ... or just one processor. We then have E = &/N-1, 
1 6 i 2 N - 1, so that we obtain 
i 
N 
N-1 6i s 5  EN 
N- 1 [(l - &)(1 + 6) + -(log2(N - 1) + 2 i)] 1 
For large N, this bound becomes 
N 
(1 - E ) ( 1  + 6) + E log2N s s  
which is a useful compromise between the unnecessarily pessimistic form 
of Amdahl's Law, and the overly optimistic linear bound. 
In ( 6 )  program characteristics appear via the parameters E and 6 ,  
where E is the probability that a program is unable to make use of a l l  N 
processors, while 6 measures the imbalance between the average execution 
time of  parallel tasks. In the sequel we shall consider a simple 
representation o f  a parallel program's execution in virtual time. 
3 .  The Activity Set Model of Parallel Program Behavior 
Consider a program that is being executed with an unlimited number 
of processors. 
representing the number of active processes o r  tasks at some instant t, 
lying between the instant t = 0 when the program execution is initiated 
Its behavior may be characterized by a variable n(t) 
6 
and the instant t = T when it ends. Such a behavior is shown in 
Figure 1. n(t) is the size of the Activity Set at time t. 
40 
20 
10 
n(t) 
Size of the Activity Set 
- N  
t 
2 
(Virtual Time) 
Figure 1 .  
of virtual time t, between t = 0 when the program execution begins and 
t = T when it ends. 
Size of the Activity Set of a parallel program as a function 
The total work 
W = JTn(t)dt 
0 
(7) 
represents the amount of computational effort that must be accomplished 
by any set of processors in order to execute the program. 
The Activity Set of the program at time t is the set of parallel 
processes or tasks that are running simultaneously at time t. 
7 
The parallelism index No: 
1 T  N = - J- n(t)dt = W/T 
O T o  
is the average number of active processes in the program, or the average 
number of processors that it can use simultaneously. 
Time t in the expressions above should be considered to be virtual 
time, or program execution time from which the time spent for all 
interruptions (paging, I/O, other programs' execution time, etc.) has 
been deleted. 
If the program is to be run on a single processor whose speed is c 
times faster than any one of the initial set of processors, the program 
will now run sequentially in time 
T(c,l) = W/c = NOT/c (9 1 
If, on the other hand, we are limited to running the program on N 
processors, each of which has the same power as one of the processors in 
the initial unlimited set, we can derive some bounds on the execution 
time. 
Consider the quantity 
C(N) = JT(N - n(t))+dt 
0 
+ + where (x) = x if x L 0 and (x) = 0 if x < 0 .  
C(N) denotes the amount of additional work that N processors could 
provide during the interval [ O , T ] ,  or their excess capacity, when the 
program is executed with an unlimited number of processors. It is shown 
as the shaded area of Figure 1. Similarly, 
8 
D(N)  = JT(n(t) - N)+dt 
0 
is the work accomplished by the other processors being used when the 
total number of processors available is unlimited. 
When the total number of' processors is liuiited to N, the total 
execution time T(N) cannot be smaller than 
T(N) 2 T + [D(N) - C(N)]+/N 
since in the best case the excess capacity C(N) will be used to 
accommodate as much excess work D(N) as the processors can take. 
Similarly, an upper bound t o  T(N) can be derived by considering 
that all of the work D(N) will be accomplished on the N processors after 
time T: 
T(N) 5 T + D(N)/N 
The speed-up obtained by using N processors instead of a single 
processor (of  speed c = 1) can now be estimated from these bounds. It is 
given by the formula S = T(l)/T(N) so  that 
NOT NOT ' ' T + b(N)/N (D(N)-C(NI )+ 
N T +  
or 
9 
From these inequalities we see quite clearly that S can never 
S exceed N which should be obvious, but also that - is bounded from 
above by the multiplicative factor 
NO 0' 
1 
and from below by the multiplicative factor 
3 . 1  A Statistical Interpretation 
Clearly, the precise behavior of a program as given by the function 
n(t) is in general very difficult to predict since it will obviously be 
data dependent. 
function so that No, D(N)/T, C(N)/T will now have convenient statistical 
interpretations in terms of the statistical average or expected value 
E[*] taken over the finite time interval [O,T]: 
Thus, it is quite natural to treat n(t) as a random 
T + E[(n(t) - N)']
We shall now write in particular 
S 1 - >  
10 
3 . 2  A Numerical Example 
Within the framework of the statistical interpretation of n(t), 
consider the case where it is time independent and its distribution 
function is geometric over the interval [ O , T ] :  
This example describes a situation in which the program is always more 
likely to have a smaller number of active parallel tasks. 
must be choser! so that E[n(t)] = No, hence No = 1/ (1  - q) or 
q = (No - l)/No. 
Clearly, q 
We then have 
a0 
i- 1 
E[(n(t) - N]'] = Z (i - N)q (1  - q) 
i=N 
'IN. No N No - 
NO 
= 4 /(I - q)  = (- 
Hence 
S 1 
No No-I 
1 + -(- jN 
No 
On Figure 2 we show the lower bound (13) to the speed-up S as a 
function of the number of processors N for various values of No (the 
parallelism index). 
11 
. 
20 
15 
10 
5 
Lower bound to S 
N = 20 0 
N = 10 0 
N = 5  0 
> N  
5 10 15 20 
Figure 2. The lower bound for the speed-up S as a function of N for 
various values of N provided by equation (13). 0 
4 .  Conclusions 
In this note we have considered some amendments to Amdahl's law 
which take into account the fact the programs may be able to make effec- 
tive use of all of the processors which are available to them, but which 
also recognize the fact that imbalance in the partition of the workload 
between processors reduces the speed-up one could expect. 
We have then suggested examining the speed-up issue in terms of a 
representation of the intrinsic behavior of parallel program execution 
which we call the Activity Set Model. This model describes the set of 
simultaneously active parallel processes as a function of program virtual 
time. We show how the size of the Activity Set can be used to derive 
12 
bounds on the  speed-up of the program a s  a function of the number of 
processors  which are a v a i l a b l e  to i t .  
13 
References 
Amdahl's law has been introduced in [ l ] .  A more recent discussion of the 
problem and some pertinent modifications can be found in [ 2 ] .  Other 
relevant references and extensions can be found in [5 ,6 ,7 ,8 ,9 ] .  
Amdahl, G. M. "Validity of the Single-Processor Approach to Achiev- 
ing Large-Scale Computing Capabilities," AFIPS Conference Proceed- 
ings 30, MIPS Press, 483-485 (1967).  
Gustafson, J. L. "Reevaluating Amdahl's Law," Communications of 
the ACM. 31, 5 ,  532-533 (1988). 
Cosnard, M., Robert, Y., Tourancheau, B. "Evaluating Speed-ups on 
Distributed Memory Architectures," to appear in Parallel Computing. 
Gelenbe, E., Nelson, R., Philips, T., Tantawi, A. "Asymptotic 
Processing Time of a Model of Parallel Computation," Proceedings 
National Computer Conference USA, November 1986. 
Gustafson, J. L. "The Scaled-Sized Model: A Revision of Amdahl's 
Law," ICs Supercomputing '88. L. P. Kartashev and S .  I. Kartashev 
eds., International Supercomputing Institute Inc., 11, 130-133 
(1988).  
Gustafson, J. L . ,  Hawkinson, S . ,  Scott, K. "The Architecture of a 
Homogeneous Vector Supercomputer," Proceedings of ICCP 86,  IEEE 
Computer Society Press, 649-652 (1986). 
Moler, C. "Matrix Computations on Distributed Memory Multi- 
Processors," Hypercube Multiprocessors 1986, M. T. Heath ed., SIAM 
Editions, 161-180 (1986).  
Robert, Y., Tourancheau, B. "LU and QR Factorization on the FPS T 
Series Hypercube," CONPAR 8 8 ,  Manchester, U.K., September 1988. 
Saad, Y. "Gaussian Elimination on Hypercubes," Parallel Algorithms 
and Architectures, M. Cosnard et al. eds., North-Holland, 5-17 
(1986). 


Multiprocessor speed-up, Amdahl's Law, and the Activity Set Model of parallel program behavior

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server