Distributed computer systems in daily use are becoming more and more heterogeneous. Currently, much of the design and analysis studies of such systems assume homogeneity. This assumption of homogeneity has been mainly driven by the resulting simplicity in modeling and analysis. A simulation study is presented which investigated the effects of heterogeneity on scheduling algorithms for hard real time distributed systems. In contrast to previous results which indicate that random scheduling may be as good as a more complex scheduler, this algorithm is shown to be consistently better than a random scheduler. This conclusion is more prevalent at high workloads as well as at high levels of heterogeneity

El-Toweissy, Mohamed

Mukkamala, Ravi

Zeineldine, Osman

English

NASA Technical Reports Server

N91-25953
MEASURING THE EFFECTS OF HETEROGENEITY ON
DISTRIBUTED SYSTEMS
Mohamed E1-Toweissy Osman ZeinEll)ine Ravi Mukkamala
Department of Computer +qcienc<'
Old I)()tllilti(_ll IlIliv<.rsity
Norfolk, Virginia 22152!1
ABSTRACT
Distributed computer systems in daily use are be-
coming more and more heterogeneous. Currently,
much of the design and analysis studies of such sys-
tems assume homogeneity. This assninption of ho-
mogeneity has been mainly driven by the restllting
simplicity in modeling and analysis. In this paper,
we present a simulation study to investigate the ef-
fects of heterogeneity on scheduling algorithms for
hard real-time distributed systems. In contrast to
pervious results which indicate that random schedul-
ing may be as good as a more complex scheduler, our
algorithm is shown to be consistently better tlian a
random scheduler. This conclusion is more prevalent
at high workloads as well as at high levels of hetero-
geneity.
INTRODUCTION
With the adwmcing communication technologies
and the need for integration of global systems, het-
erogeneity is becoming a reality in distributed com-
puter systems, tIowever, most existing performance
studies of such systems still assume homogeneity; be
it in hardware (e.g., node speed) or in software (e.g.,
scheduling algorithms). Generally, such homogeneity
assumptions are dictated by the resulting simplicity
in modeling and analysis.
Clearly, heterogeneous systems are less analyti-
cally tractable than their homogeneous counterparts.
Typically, heterogeneity will result in increased num-
ber of variables in the context of analytical tech-
niques such ,as mathematical programming, proba-
bilistic analysis, and queuing theory. 'Finis is one re+i-
son for assuming homogeneity while using analyti-
cal techniques. In the case of simulation techniques,
however, it is possible for a modeler to introduce any
level of heterogeneity into the system. The prohlem
now lies in the complexity of interpretation of the
restllts. If the simulator was writl.en with the basic
objective of te_ting a hypothesis or coml)ariug the
perfornmnce of a :-;et of algorithnls, introducing het-
erogen,'it.y ',','ill sul,slant.ially increase efforts to sep-
arate ils effects front those of the algorithnl. Thus,
it tile(Icier i.s Inert' ]ik,,ly i.o ;t.SNIIU|{' ;! hotliogt?tl(+Oll.S
system.
With this in mind, w<+ have been investigating into
t,h[" t'll'<'cts tff h('terogeneity +m the l)erformance of
dist.rilitlied sysleilis+ ()lit initiM efforts, reliOrlt.¢+(I
in (Zeinl';ll)ine et al. 1991) and this paper, focus
on scheduling in hard real-time systems. For this
purpose, we haw" designed a tlistril)uted scheduler
alined at handling various heterogelteities; in partic-
ular, heterogeneities in tiodes, node tratl]c and local
scheduling algorithnls.
In the rest of this paper we present the systenl model.
Next., we discuss sonic is.sues related to the effectiw,-
ness of our algorithin. Ma.ior results with their t'_'-
st>of'live com'lusio,ls arc then portrayed. I"inally, w_"
highlight some reconmlcmlations for l'ut.ure work.
THE PI/OPOSED MODEL
For the I)Urlioses of scheduling, the distributed sys-
tem is tnodeled as atl'<'," of nodes as is shown in Fig-
ure 1. 'l'ht_ nodes ;it the Iow<'st level (level 01 ar<'
th<' l_r+Jc<.+.sUi9 Ilotlt's while tile uoth>s at the hight,r
levels I'elii'i,.,.ienl server<,; (or gnardians). A process-
ing node is responsil)le for executing arriving jol+s
when they l/leel, SOlile specified criteria (e.g., dead-
line). The processing nodes are grouped into clns-
lets, and ,,ach chisl.('r is /tssiglt_(I a unique serv<,r.
\_]h,,'n il _,(.rvl,r rl.ll,ivl._ ;i j<,l,, il. l.i'i('s Io eit.ln.r rq'di-
rect that job to a processing node within its chrsLer
or to its guardian. It is to be noted that this hier-
archicM structnre could be logical (i.e., some of the
processing nodes may thelnselves assulne the role of
the servers).
https://ntrs.nasa.gov/search.jsp?R=19910016639 2020-03-19T16:51:48+00:00Z

The system model has four components: jobs, pro-
cessing nodes, servers, and the communication sub-
system. A job is characterized by its arrival time,
execution time, deadline, and priority (if any). Tile
specifications of a processing node include its speed
factor, scheduling policy, external arrival rate (of
jobs), and job mix (due to heterogeneity). A server is
modeled by its speed and its node assigmnent policy.
Finally, the communication subsystem is represented
by the speeds of transmission and distances between
different nodes (processing and servers) in tile sys-
tem.
O perat ion
The flow diagram of tile scheduling algoritJnn is
shown in Figure 2. When a job with deadline arrives
(either from an external user or from a server) at
a processing node, the local scheduling algorithnl at
tile node decides whether or not to execute this job
locally. This decision is based on pending jobs in
the local queue (which are already guaranteed to be
executed within their deadlines), the requirements of
the new job, and the scheduling policies (e.g., FCFS,
SJF, SDF, SSF etc. (Zhao et al. 1987)). In case the
local scheduler cannot execute the new job, it either
sends the job to its server (if there is a possil)ility of
completion), or discard the job (if I.here is no such
chance of completion).
The level-1 server maintains a copy of the latest
information provided by each of its child nodes in-
eluding the load at the node and its scheduling pol-
icy. Using this information, the server should be able
to decide which processing nodes are eligible for exe-
cuting a job and meet its deadline. When more than
one candidate node is available, a random selection
is carried out among these nodes. If a server can-
not find a candidate node for executing tl,e job, it.
forwards the job to the level-2 server.
The information at the level-2 server consists of an
abstraction of the information available at each of the
level-1 servers. This server redirects an arriving job
to one of the level-1 servers. The choice of candidate
servers is dependent on the ability of these servers to
redirect a job to one of the processing nodes in their
cluster to meet the deadline of the job. (For more
details on operation and information contents at each
level, the reader is advised to refer to (ZeinEII)ine et
al. 1991)).
EXPERIMENT
In order to utilize the proposed scheduler ,_s a w>
hicle for our research on measuring the elfects of het-
erogeneigy, first it has to be proven effective. Conse-
quently, we have conducted several parametric stud-
ies to determine the sensitivity of our algorithm to
various parameters: the cluster size, the frequency
of propagation of load statistics (between levels), the
processing node scheduling policy (FCFS, SJF etc),
the communication delay (between nodes), and lhe
effects of information structures. For lack of space,
we present a sample of the results pertaining to the
first three of these parameters. Accordingly, all the
results reported here assume:
• the total number of processing nodes iS 100;
• equal load at all nodes;
• c¢.ln_ulnicali_m _l,'l;o' hplv,'_'_'ll ally n_>dps is Ih_'
Sillll('.
The perform;nice of lhe scheduler is measm'ed in
terms of the percentage of johs discarded by the al-
gorilhm (at levels 0, i & 2). The rate of arrivals
of jobs aml lheh' processing reqnirements are com-
binedly represented through a load factor. This load
factor refers to the load ou the overall system. Our
load consists of jobs from three types of execution
time constraints (10, 50 & 100) with slack (25, 35 &
300) respectively.
Discussion
\Ve now discus,_ ,mr _d_servn/ions regarding the
char;tcleristics of tl.' ,lisLrilmled schrdulit,g algo-
rithm (DSA) in lerms of the three selected param-
eters. In order to isolate t.he effect of one factor fi'om
others, the choice of parameters is made .jmlicio,,sly.
For example, in studying the effects of cluster size
(Figure 3), the updation period is chosen to be a
medium value of 200 (star=200). For each paralnet-
ric study we have two sets of runs, they differ in the
local scheduling policy at the processing uodes; one
set uses I"('I"S whih. the the other uses S.IF.
Clustm' size (2h,st,.'r size imlicates the u,tml>e,
of processing nodes being assigned to a level-I server.
In our study, we haw co,sidered three cluster sizos:
100, 50, and 10. A clusler of 100 nodes indicates
a centralized server structure where all the process-
ing 1redes at',' under one l,.'vel-1 server. In tiffs case,
level-2 server is al_seut. Similarly, in Ihe case of clus-
ter of 50 nodes, there are two level-1 servers, and one
level-2server. For 10-nodecluster, we have 10level-1
serw'rs. In athlitiou, we consider a completely decen-
tralized caLse represented by I,he ra,dom policy. In
this case, each processing node acl.s as its own server
and rat,domly selects a d,,stination node to execut,,
a job which it cannot locally guarantee.
Tim resti/ls ;ire plotted in Figore 3. These resli]ts
show that our algorithm is robust to variations in

cluster size. In addition, its performance is signifi-
cantly superior to a random policy.
Frequency of updations "File currency of in-
formation at a node about the rest of the system
plays a major role in performance, tlence, if the
state of processing nodes varies rapidly, then tile fre-
quency of status information exchange between the
levels should also be high. In order to determine the
sensitivity of the proposed algorithm to the period of
updating statistics at tile servers, we experimented
with four time periods: 25, 100, 200 and 500 units.
Tile results are summarized in Figure 4. From these
results, the following observations are drawn:
• our algorithm is extremely sensitive to changes
in period of infornaation exchanges between
servers and processing nodes;
• even in the worst case of 500 units, the per-
formance of our algorithm is significantly better
than the random policy.
Local scheduling policy Our third parametric
study is concerned with the effect of the scheduling
policy at the processing nodes. In this paper, we
report the results of the runs conducted with all the
processing nodes having the same scheduling policy;
either I"CI"S or SJI". I,ater iu our work, we plan 1o
experiment with different mixes of local scheduling
policies. We would also give each node the freedom to
select its scheduling policy in order to determine the
impact of node autonomy on the overall performance
of the system. This issue is of crucial importance,
since there is no scheduling policy that best fits all
working environments.
Revisiting the results of Figures 3 and 4, we can
observe the following:
• both FCFS and SJF behave similarly at light to
moderate loads, while SJ F is consistently better
than FCFS at high loads;
• the percentage of discarded jobs sharply in-
creases with the increase in the load factor for
both the Random and FCFS policies, ltowever,
for SJF the rate of increase in the percentage of
discarded jobs dramatically drops at high loads.
The reason for the above result is that, at. light to
moderate loads there is no build t,p at the processing
node queues, consequently, the dominant factor is the
jobs being processed at the node processors. This
behavior is the same for all policies, l[owever, when
the queues start to build up, the respective queue
policy prevails. Hence, for the SJF, the short jobs
with their relatively small slack, will have a better
probability of heing executed.
EFFECTS OF HETEROGENEITY
So far, our concentration has been on gaining bet-
ter insight into the Imhavior of our algorithm in ol"-
der to assess its viability and suitability to he able
to conduct further research. Proven effective, we re-
turn back to the objective for which the alger,Ibm
has been developed. The main goal of the current
phase of our studies is measuring the effects of hcf
eroge_eity on scheduling in hard real-time distrihmed
systems. For this purpose, we are pursuing multiple
experiments to measure l he effects of node hetero-
gel,eily (simply repr,,s, qd,ed by node speed), hetero-
gem'ity in scheduling algorilhllls, helm'ogencily in
loads as well as other syslenl heterogeneities. In this
paln.r , we present the etfi.cts of node helerogeneily.
(]{esulls oil olluq' lylws will hc r(.ported irl a s_._lU(:l
of papers).
We consider four different node speed distributions
(hetl, he,2, he,3, and hem). The ho,nogeneous case
(denoted by hem) represents a system with 100 nodes
having the same unit speeds. The three he.teroge-
neous case are represented by hetl, he,2, and he,3.
Each of these see are described by a set of <# of
nodes, speed factors> pairs. The average speed fac-
tor for all dis|ribution is 1.0. so the average systent
sp,.,'d is th,. s;IIIII'. 'l'hc t l_rcc h,,tol',)gCllCOllS c;l:.,(, dil'-
fer in their Sl>,!,rd [actor V;tli;tllC,., tilllS varying I,Imd,>
gree of heterogeneity. While bet3 represents a severe
case of heterogeneity, hell is more biased towards
homogeneity.
The results are illchlded in Figure 5. From lhese
results, we observe Ihal :-
• with our algorithm, even though the increase in
degree of heterogeneity resulted in an increase of
discarded jobs, the increase is not do signiticant.
Ih'nce, our algoi'ithtlt ;tpln,arn to I:.' robust t_>
re>de bet ,_rogem.il.ics.
• the performance of the random policy i.s ex-
tremely sensit.iwe Io the node heterogeneity. As
the hete,'ogeneity is increased, the. number of dis-
carded jobs is also significantly increased.
With the increase in node heterogeneity, the nun,her
of nodes with slow speed also increase. Thus, using
a rando,n policy, if a slow speed node is se.lected ran-
domly, t.l,,'n the' .i',h i,_ ,,,,,,',' lik,'ly tlf) I-' discar,h',l. II,
our alger,film , sire'," tl,,., server is aware of the hel.-
erogeneities, il. can suitably avoid a low speed node
when uecessary, l"ven in this case, there, is a te,ld,'ncy
for high-slwed nodes Io be overloaded and low speed
nodes to be under loaded, llence, the difference in
performance.

CONCLUSION
In this paper, we have presented a distributed
scheduling algorithm that can tolerate different types
of system heterogeneity. Following, we have con-
ducted several parametric studies with the objective
of evaluating the effectiveness of our algorithm. Ren-
dering its effectiveness, we have started pursuing our
studies toward our goal of determining the impact
of heterogeneity on the overall system performance.
Our initial step has been reported here, and it con-
centrates on the effect of node heterogeneity. Some
interesting results have been obtained. From these
results, we reach the following conclusions.
• Concerning the algorithm behavior: the algo-
rithm is robust to variations in tile cluster size;
besides, it elliciently utilizes tile availal)h: state
information; moreover, it is sensitive to the local
scheduling policy at the processing nodes.
• Concerning the effect of heterogeneity: the per-
formance of the algorithm tends to be invariant
with respect to node heterogeneity; in addition,
the algorithm has a large improvement over the
random selection in terms of the percentage of
discarded jobs.
Currently, we are studying the effects of hetero-
geneity in local scheduling algorithms and hetero-
geneities in loads on the performance of tile over-
all system. With heterogeneities in scheduling poli-
cies, each node may autonomously decide its own
scheduling policy (FCFS, SJF, etc.). Similarly, by
load heterogeneities we let the external load at a
node be independent of the other nodes. Similarly,
each node may autonomously decide its resources and
their speeds. We propose to measure the effects of
such heterogeneities in terms of the response time
and throughput. We conjecture that the performance
of random policies will continue to deteriorate under
these heterogeneities as compared to even simple re-
source allocation or execution policies.
ACKNOWLEDGEMENT
This research was sponsored in part by the NASA
Langley Research Center under contracts NAG-I-
ll14 and NAG-l-1154.
References
Biyabani, S.R.; J.A. Stankovic; and K. Ra-
mamritham. 1988. "The integration of deadline
and criticalness in hard real-time scheduling."
Proc. Real-time Systems Symposium, (DEC.),
152-160.
Chuang, J.Y. and J.W.S. Liu. 1988. "Algorithms
for scheduling periodic jobs to minimize aver-
age error." Proc. Real-time Systems Sympos_vm,
(DEC..), 142-151.
Craig, D.W. and C.M. Woodside. 1990. "'Th(: re-
jection rate for tasks with random arrivals, dead-
lines, and preemptive scheduling." IEEE TraT_s.
Software Engtneerillg SE-16, no. 10(OCT.),
1198-1208.
[7 Eager D.L.; E.D. Lazowska; and J. Zahorjan.
1986. "Adaptive load sharing iu homogeneous
distributed systems." IEEE Trans. Software En-
gtneertng, SE-12(May), no. 5, 662-675.
Rajkumar R.; L. Sha; and ,].P. l,ehoczky. 1'988.
"'l{c;d-l, iuw syuchrtmizati(m i)rotocols for tlml-
til)roccssors," Proc. Real-Irate Systems Sympo-
stum. (1)1':('.), 25!)-2(]9.
Shin, K.G.; C.M. Krishna; and Y.II. Lee. "Ol)-
titnal resource control in periodic real-time en-
vironments." Proc. Real-time Systems Sympo-
sium. (DEC), 33--II.
[] Stankovic 3. and h:. l_amamritham. L986.
"Evaluatiotl of a bidding algorithm for hard
real-time distributed sysletlls." IEEE Tran._.
Compute r._C-34, rio. i 2( Dee. ), 1130-1143.
[] ZeiuEIl)ille, O.; M. EI-Toweissy; amt R. Mukka-
mala. 1991. "' A Distributed Scheduling Algo-
rithm for Ileterogeueous Real-time Systems." To
apl)ear in Lecturer Notes 17_ ('ompnler Sc_euct,
Springer-Verlag.
[] Zhao, W.; K. Ranaanaritham; and J. Stankovic.
1987. "Scheduling tasks with resource require-
ments in hard-real time systems." IEEE Tra_s.
Software E_gil_eering SE-13, no. 5(May): 56-t-
577.

() Level 2 server
Level 1 server
Level 0 •
Proc. node
Figure 1 • System Model
from other level I
scr'vcrs
from other child
nodes
Jobs
from other servers
External jobs
k_
"--,r discard
•--- to a level I server
discard
to a processing
node in this
cluster
_d
!
Level 2 server
Level 1 server
I Processing
I node
parture
guaranteed job processor !
queue ___j !
I
il ___( _ granted
I
I gateway
I discard
Figure 2" Flow diagram of The Algorithm

3O
%
j 2s
o
b 2o
d 15
i
s 10
c
a 5
r
d
. cluster = 100 stat = 200
cluster = 50 corn = 5l t IO n de: horn ,,_
random J
0
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
load factor _ a
30
o i stat = 25 cluster = 100
25 stat = 100 com= 5
o stat = 200 node: horn /A
b 20 stat=500
random
S 10
C • " -
r
d
% 30
J 25
o
b 2o
d 15
i
s 10
c
a 5
r
d o
0 0.1 0.2 0.3 0.4 0.5 0,6 0.7 0.8 0.9
load factor _ a
,, hom
• hetl
* het2
•het3
% 3O
J
25
0
b
2O
d 15
i
s 10
c
a 5
r
d o
Figure 3:
30
25
20
Effect of cluster size Is
1o
5
/
• cluster = 100 star = 200 /
• cluster = 50 corn = 5 /
+ cluster = 10 node: horn/ //,
..=o. ///
0 0.1 0,2 0,3 0.4 0.5 0.6 0.70.S 0,9
load factor _ b
30
25
20
Figure 4:
Effect of updation period is
lO
3O
25
20
15
load factor
• horn
!hel Random(SJF)_,.,_
X
I0
Figure 5: oo
a Effect of Node Heterogeneity '
30
hetl • <50,1.0>, <25,1.5>,
2s
<25,0.5>
het2 • <50,0.5>, <50,1.5> 20
het3 • <20,0.25>, <20,0.5> 'is
<20,1.0>, <20,1.5>,
<20,1.75> lo
o o.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
load factor _ c
/
• stat=25 cluster=lO0 J fl
• stat= 100 corn=5 J J
÷ star= 200 node:hom)_ / ,7
,stat=500 ' / /
. a..om /
0 0.1 0.2 0.3 0.4 0.5 0.6 0,7 0.8 0.9
load factor _ b
• horn
• hetl
* het2
• het3
o.1 0.2 0.3 0.4 05 o s 0.7 o s
load factor _ b
0 :
.or. ///
heA Random (FCFS_.,,_r2 JjJ_
o o.1 0.2 0.3 0.4 o.s o.s o.7 o.s 0.9
load factor _ d



Measuring the effects of heterogeneity on distributed systems

Abstract

Similar works

Full text

Available Versions

NASA Technical Reports Server