A Distributed Approach for Fault Mitigation in Large Scale Distributed Systems

In a large scale real-time distributed system, a large number of components and the time criticality of tasks can contribute to complex situations. Providing predictable and reliable service is a paramount interest in such a system. For example, a single point failure in an electric grid system may lead to a widespread power outage like the Northeast Blackout of 2003. System design and implementation address fault avoidance and mitigation. However, not all faults and failures can be removed during these phases, and therefore run-time fault avoidance and mitigation are needed during the operation. Timing constraints and predictability of the system behavior are important concerns in a large scale system as well. This dissertation proposes several distributed fault tolerance mechanisms using multi-agent technologies to predict and mitigate faults with various frequencies and severities. Some faults are frequently observed over time and some are not. In general, frequent fault types often cause relatively less severe consequences. Rare faults, however, are extremely difficult to predict, yet the consequences can be catastrophic. A rare fault -- often indicated by repeated doses of common faults -- causes severe harm. In our preliminary study, we design distributed rational agents using a probabilistic prediction mechanism to discover faults in the CMS experiments at CERN. All fault-mitigating activities of the agents and application tasks are guaranteed by the urgency-based priority scheduling policy with multiple steps of feasibility tests. The experiment shows that the distributed approach provides 15% more system availability than centralized approaches. This dissertation also explores the problem of predicting rare events. Many adaptive fault tolerant mechanisms attempt to predict faults through learning from data. However, in order to train the system, we need a significant amount of training data, which is not easily available for rare fault events. We use the PNNL (Pacific Northwest National Laboratory) system failure data collected from about 1,000 nodes over 4 years. We find that the severity of observed fault events is power-law distributed and there are certain associations among these events. Based on the power-law observation, we generate training data for the machine learning algorithm developed in this dissertation. The algorithm incorporates the power-law distribution principle, Bayesian inference, and logistic regression to predict rare events as well as common ones. The logistic regression is used to predict the probability of each type of events and the Bayesian inference is used for finding associations among events. A new learning algorithm is deployed with fully distributed agents using a rational decision model. The simulation study based on the PNNL data shows that the new prediction algorithm provides 15$\%$ better system availability than the prediction using the simple update method that was used in our preliminary study; and it achieves more than 10 times less system loss caused by rare faults. Finally, we developed a comprehensive simulation library, named SWARM-eTOSSIM for cyber-physical systems research. The library provides a framework suitable for simulating power-aware real-time distributed networked systems with powerful simulation controls and graphical interface. We downsized the new fault-mitigation mechanism so that it can be ported to devices with limited resources, such as sensor network elements

Prototype of Fault Adaptive Embedded Software for Large-Scale Real-Time Systems

This paper describes a comprehensive prototype of large-scale fault adaptive embedded software developed for the proposed Fermilab BTeV high energy physics experiment. Lightweight self-optimizing agents embedded within Level 1 of the prototype are responsible for proactive and reactive monitoring and mitigation based on specified layers of competence. The agents are self-protecting, detecting cascading failures using a distributed approach. Adaptive, reconfigurable, and mobile objects for reliablility are designed to be self-configuring to adapt automatically to dynamically changing environments. These objects provide a self-healing layer with the ability to discover, diagnose, and react to discontinuities in real-time processing. A generic modeling environment was developed to facilitate design and implementation of hardware resource specifications, application data flow, and failure mitigation strategies. Level 1 of the planned BTeV trigger system alone will consist of 2500 DSPs, so the number of components and intractable fault scenarios involved make it impossible to design an `expert system' that applies traditional centralized mitigative strategies based on rules capturing every possible system state. Instead, a distributed reactive approach is implemented using the tools and methodologies developed by the Real-Time Embedded Systems group.Comment: 2nd Workshop on Engineering of Autonomic Systems (EASe), in the 12th Annual IEEE International Conference and Workshop on the Engineering of Computer Based Systems (ECBS), Washington, DC, April, 200

Design Methodology of Tightly Regulated Dual-Output LLC Resonant Converter Using PFM-APWM Hybrid Control Method

A dual-output LLC resonant converter using pulse frequency modulation (PFM) and asymmetrical pulse width modulation (APWM) can achieve tight output voltage regulation, high power density, and high cost-effectiveness. However, an improper resonant tank design cannot achieve tight cross regulation of the dual-output channels at the worst-case load conditions. In addition, proper magnetizing inductance is required to achieve zero voltage switching (ZVS) of the power MOSFETs in the LLC resonant converter. In this paper, voltage gain of modulation methods and steady state operations are analyzed to implement the hybrid control method. In addition, the operation of the hybrid control algorithm is analyzed to achieve tight cross regulation performance. From this analysis, the design methodology of the resonant tank and the magnetizing inductance are proposed to compensate the output error of both outputs and to achieve ZVS over the entire load range. The cross regulation performance is verified with simulation and experimental results using a 190 W prototype converter

Prototype of Fault Adaptive Embedded Software for Large-Scale Real-Time Systems

This paper describes a comprehensive prototype of large-scale fault adaptive embedded software developed for the proposed Fermilab BTeV high energy physics experiment. Lightweight self-optimizing agents embedded within Level 1 of the prototype are responsible for proactive and reactive monitoring and mitigation based on specified layers of competence. The agents are self-protecting, detecting cascading failures using a distributed approach. Adaptive, reconfigurable, and mobile objects for reliability are designed to be self-configuring to adapt automatically to dynamically changing environments. These objects provide a self-healing layer with the ability to discover, diagnose, and react to discontinuities in real-time processing. A generic modeling environment was developed to facilitate design and implementation of hardware resource specifications, application data flow, and failure mitigation strategies. Level 1 of the planned BTeV trigger system alone will consist of 2500 DSPs, so the number of components and intractable fault scenarios involved make it impossible to design an “expert system” that applies traditional centralized mitigative strategies based on rules capturing every possible system state. Instead, a distributed reactive approach is implemented using the tools and methodologies developed by the RealTime Embedded Systems group

Coordination-Driven Monolayer-to-Bilayer Transition in 2D Metal-Organic Networks

We report on monolayer-to-bilayer transitions in 2D metal−organic networks (MONs) from amphiphiles supported at the water−air interface. Functionalized calix[4]arenes are assembled through the coordination of selected transition metal ions to yield monomolecular 2D crystalline layers. In the presence of Ni(II) ions, interfacial self-assembly and coordination yields stable monolayers. Cu(II) promotes 2D coordination of a monolayer which is then diffusively reorganizing, nucleates, and grows a progressive amount of second layer islands. Atomic force microscopic data of these layers after transfer onto solid substrates reveal crystalline packing geometries with submolecular resolution as they are varying in function of the building blocks and the kinetics of the assembly. We assign this monolayer-to- bilayer transition to a diffusive reorganization of the initial monolayers owing to chemical vacancies of the predominant coordination motif formed by Cu2+ ions. Our results introduce a new dimension into the controlled monolayer-to-multilayer architecturing of 2D metal− organic networks

A More Appropriate Cardiac Troponin T Level That Can Predict Outcomes in End-Stage Renal Disease Patients with Acute Coronary Syndrome

Purpose: Cardiac troponin T (cTnT), a useful marker for diagnosing acute myocardial infarction (AMI) in the general population, is significantly higher than the usual cut-off value in many end-stage renal disease (ESRD) patients without clinically apparent evidence of AMI. The aim of this study was to evaluate the clinical usefulness of cTnT in ESRD patients with acute coronary syndrome (ACS). Materials and methods: Two hundred eighty-four ESRD patients with ACS were enrolled between March 2002 and February 2008. These patients were followed until death or June 2009. Medical records were reviewed retrospectively. The cut-off value of cTnT for AMI was evaluated using a receiver operating characteristic (ROC) curve. We calculated Kaplan-Meier survival curves, and potential outcome predictors were determined by Cox proportional hazard analysis. Results: AMIs were diagnosed in 40 patients (14.1%). The area under the curve was 0.98 in the ROC curve (p<0.001; 95% CI, 0.95-1.00). The summation of sensitivity and specificity was highest at the initial cTnT value of 0.35 ng/mL (sensitivity, 0.95; specificity, 0.97). Survival analysis showed a statistically significant difference in all-cause and cardiovascular mortalities for the group with an initial cTnT ≥0.35 ng/mL compared to the other groups. Initial serum cTnT concentration was an independent predictor for mortality. Conclusion: Because ESRD patients with an initial cTnT concentration ≥0.35 ng/mL have a poor prognosis, it is suggested that urgent diagnosis and treatment be indicated in dialysis patients with ACS when the initial cTnT levels are ≥0.35 ng/mL.ope

Mammary fibroadenoma in a lamb

A fibroadenoma was diagnosed in the left udder of a 3-month-old female Chios lamb. No recurrence was observed after surgery. Grossly, the tumor had a whitish-gray lobular appearance, and the lobules were interlaced with thin septa. Microscopically, the tumor was composed of proliferating fibroepithelial tissue, including differentiated ducts lined by whorls and interlacing bundles of abundant loose fibrovascular stroma. Immunohistochemistry revealed the ductal epithelium to be positive for pancytokeratin (AE1/AE3) and loose fibrovascular stroma was positive for vimentin and basal cells covering the ductal epithelium of alpha-smooth-muscle actin. Immunostaining for the estrogen and progesterone receptors was negative. A diagnosis of mammary fibroadenoma was made based on the histological and immunohistochemical findings

Generation and analysis of large-scale expressed sequence tags (ESTs) from a full-length enriched cDNA library of porcine backfat tissue

BACKGROUND: Genome research in farm animals will expand our basic knowledge of the genetic control of complex traits, and the results will be applied in the livestock industry to improve meat quality and productivity, as well as to reduce the incidence of disease. A combination of quantitative trait locus mapping and microarray analysis is a useful approach to reduce the overall effort needed to identify genes associated with quantitative traits of interest. RESULTS: We constructed a full-length enriched cDNA library from porcine backfat tissue. The estimated average size of the cDNA inserts was 1.7 kb, and the cDNA fullness ratio was 70%. In total, we deposited 16,110 high-quality sequences in the dbEST division of GenBank (accession numbers: DT319652-DT335761). For all the expressed sequence tags (ESTs), approximately 10.9 Mb of porcine sequence were generated with an average length of 674 bp per EST (range: 200–952 bp). Clustering and assembly of these ESTs resulted in a total of 5,008 unique sequences with 1,776 contigs (35.46%) and 3,232 singleton (65.54%) ESTs. From a total of 5,008 unique sequences, 3,154 (62.98%) were similar to other sequences, and 1,854 (37.02%) were identified as having no hit or low identity (<95%) and 60% coverage in The Institute for Genomic Research (TIGR) gene index of Sus scrofa. Gene ontology (GO) annotation of unique sequences showed that approximately 31.7, 32.3, and 30.8% were assigned molecular function, biological process, and cellular component GO terms, respectively. A total of 1,854 putative novel transcripts resulted after comparison and filtering with the TIGR SsGI; these included a large percentage of singletons (80.64%) and a small proportion of contigs (13.36%). CONCLUSION: The sequence data generated in this study will provide valuable information for studying expression profiles using EST-based microarrays and assist in the condensation of current pig TCs into clusters representing longer stretches of cDNA sequences. The isolation of genes expressed in backfat tissue is the first step toward a better understanding of backfat tissue on a genomic basis