Delay Distribution Based Remote Data Fetch Scheme for Hadoop Clusters in Public Cloud

Apache Hadoop and its ecosystem have become the de facto platform for processing large-scale data, or Big Data, because it hides the complexity of distributed computing, scheduling, and communication while providing fault-tolerance. Cloud-based environments are becoming a popular platform for hosting Hadoop clusters due to their low initial cost and limitless capacity. However, cloud-based Hadoop clusters bring their own challenges due to contradictory design principles. Hadoop is designed on the shared-nothing principle while cloud is based on the concepts of consolidation and resource sharing. Most of Hadoop\u27s features are designed for on-premises data centers where the cluster topology is known. Hadoop depends on the rack assignment of servers (configured by the cluster administrator) to calculate the distance between servers. Hadoop calculates the distance between servers to find the best remote server from which to fetch data from when fetching non-local data. However, public cloud environment providers do not share rack information of virtual servers with their tenants. Lack of rack information of servers may allow Hadoop to fetch data from a remote server that is on the other side of the data center. To overcome this problem, we propose a delay distribution based scheme to find the closest server to fetch non-local data for public cloud-based Hadoop clusters. The proposed scheme bases server selection on the delay distributions between server pairs. Delay distribution is calculated measuring the round-trip time between servers periodically. Our experiments observe that the proposed scheme outperforms conventional Hadoop nearly by 12% in terms of non-local data fetch time. This reduction in data fetch time will lead to a reduction in job run time, especially in real-world multi-user clusters where non-local data fetching can happen frequently

Imbalance state resolving considering flow types

This paper proposes a scheme to resolve the imbalance state in a network in which flow types, mice and elephant flows, are considered. A combination of link utilized rate and transmission delay of each link are considered as a link cost. In the proposed scheme, the load imbalance state is resolved by dividing the elephant flow into several subflows and injecting each subflow into multiple paths. The maximum utilization rate of the proposed scheme decreases 38.9%, compared to a conventional scheme

Self-adjustable domain adaptation in personalized ECG monitoring integrated with IR-UWB radar

To enhance electrocardiogram (ECG) monitoring systems in personalized detections, deep neural networks (DNNs) are applied to overcome individual differences by periodical retraining. As introduced previously [4], DNNs relieve individual differences by fusing ECG with impulse radio ultra-wide band (IR-UWB) radar. However, such DNN-based ECG monitoring system tends to overfit into personal small datasets and is difficult to generalize to newly collected unlabeled data. This paper proposes a self-adjustable domain adaptation (SADA) strategy to prevent from overfitting and exploit unlabeled data. Firstly, this paper enlarges the database of ECG and radar data with actual records acquired from 28 testers and expanded by the data augmentation. Secondly, to utilize unlabeled data, SADA combines self organizing maps with the transfer learning in predicting labels. Thirdly, SADA integrates the one-class classification with domain adaptation algorithms to reduce overfitting. Based on our enlarged database and standard databases, a large dataset of 73200 records and a small one of 1849 records are built up to verify our proposal. Results show SADA\u27s effectiveness in predicting labels and increments in the sensitivity of DNNs by 14.4% compared with existing domain adaptation algorithms

Optimization Approach to Minimize Backup Capacity Considering Routing in Primary and Backup Networks for Random Multiple Link Failures

This paper proposes an optimization approach that designs the backup network with the minimum total capacity to protect the primary network from random multiple link failures with link failure probability. In the conventional approach, the routing in the primary network is not considered as a factor in minimizing the total capacity of the backup network. Considering primary routing as a variable when deciding the backup network can reduce the total capacity in the backup network compared to the conventional approach. The optimization problem examined here employs robust optimization to provide probabilistic survivability guarantees for different link capacities in the primary network. The proposed approach formulates the optimization problem as a mixed integer linear programming (MILP) problem with robust optimization. A heuristic implementation is introduced for the proposed approach as the MILP problem cannot be solved in practical time when the network size increases. Numerical results show that the proposed approach can achieve lower total capacity in the backup network than the conventional approach

Dynamic load balancing with learning model for Sudoku solving system

This paper proposes a dynamic load balancing with learning model for a Sudoku problem solving system that has multiple workers and multiple solvers. The objective is to minimise the total processing time of problem solving. Our load balancing with learning model distributes each Sudoku problem to an appropriate pair of worker and solver when it is received by the system. The information of the estimated solution time for a specific number of given input values, the estimated finishing time of each worker, and the idle status of each worker are used to determine the worker-solver pairs. In addition, the proposed system can estimate the waiting period for each problem. Test results show that the system has shorter processing time than conventional alternatives

Flows Reduction Scheme Using Two MPLS Tags in Software-Defined Network

This paper proposes a scheme to reduce the number of flow entries permanently stored in an OpenFlow switch and the number of configuration messages from a controller in a software-defined network (SDN). In an SDN, a flow table in an OpenFlow switch is used to instruct packets. The flow table consists of flow entries decided by the controller. A flow request is sent from the OpenFlow switch to the controller if the incoming packet does not match any flow entry in the flow table. The controller\u27s central processing unit may be overloaded to handle user requests, since the user requests for different data types have been rapidly increasing. As a result, flow configuration in switches is delayed. Moreover, the control plane may be flooded by configuration messages of those requests. A scheme to permanently keep the flow entries in the switch can reduce the number of requests. However, a large number of permanent flow entries is required. Other switch features may be degraded, since there is not enough memory in the flow table to implement those features. In the proposed scheme, switches in the network are divided into multiple regions. In order to guide packets from sources to destinations, the flow table incorporating the concept of two multiprotocol label switching tags is re-designed. One tag directs a packet from a source switch to an edge switch in the destination region. The other tag directs the packet from that edge switch to another switch in the same region. A mathematical model for the proposed scheme is formulated as an integer linear programming to determine a set of switches in each region so that the total number of permanent flow entries in the network can be minimized. The performance of the proposed scheme is analyzed. Moreover, the proposed scheme is implemented and demonstrated via Japanese Science Information Network 5

Modulation-Adaptive Link-Disjoint Path Selection Model for 1 + 1 Protected Elastic Optical Networks

In elastic optical networks (EONs), an appropriate modulation technique is adapted according to the distance of an optical path. A robust modulation technique with a large number of spectrum slots is considered for longer distance optical paths, and a less robust modulation technique with a small number of spectrum slots is used for shorter distance optical paths. When an optical path is configured, the number of required spectrum slots is determined based on the nonlinear relationship between the optical path length and the number of utilized spectrum slots. Minimizing the total path lengths does not always minimize the total number of required spectrum slots for configuring an optical path, which decreases the spectrum utilization. This paper introduces a modulation-adaptive link-disjoint path selection model by considering a step function based on realistic modulation formats in order to minimize the total number of utilized spectrum slots in 1 + 1 protected EONs. We formulate the modulation-adaptive link-disjoint path selection problem as an integer linear programming (ILP). We prove that the modulation-adaptive link-disjoint path selection problem is NP-complete. By using an optimization solver, we solve the ILP problem for different backbone networks, namely, Japan Photonic Network (JPN48), German 17 Network, and COST 239 Network, within a practical time. Numerical results obtained from performance evaluation indicate that the introduced model reduces the number of utilized spectrum slots compared to the conventional schemes

グラフ圧縮による媒介中心性の計算手法

本論文はグラフの各点の媒介中心性を求める計算手法を提案する．それは次数が1である点をグラフから除き，圧縮されたグラフで計算する．提案手法が，従来の手法の次数が1である点が存在するグラフで生じる冗長な計算を回避し，計算量を削減することを示す．This paper proposes a computation method to find a betweenness centrality of each vertex of a graph. The method compresses the original graph by removing vertices whose degree is one from the graph. The betweenness centrality is then calculated from the compressed graph. This avoids avoid blackundancy of the computation in the conventional method without the graph compression. As a result, the calculation time is blackuced

Single tag scheme for segment routing in software-defined network

This paper proposes a scheme to reduce a size of a packet header for a segment routing (SR) scheme in a software-defined network (SDN). The SR scheme inserts a segment identification (SID) list into the packet header to indicate a path for the source–destination pair of the packet. The path can be split into different segments to suit the service requirement and the segments are carried by the SID-list whose length increases with the number of segments. This also increases the packet overhead, and an additional packet is needed if the packet length exceeds the maximum transmission unit (MTU). Moreover, it may not be possible to implement SR in SDN due to the limited number of stacked labels provided by the switch vendor. In the proposed scheme, the SID-list is replaced by a single tag to indicate a node edge, called a swapping node. The tag is replaced by a new tag at the swapping node. With this scheme, the size of SID-list is fixed and does not vary with the number of segments, and no additional packets are required. A mathematic model to balance the number of flow entries in each swapping node is introduced by minimizing the maximum number of flow entries in each swapping node over the network. We implement the proposed scheme on the transmission-Japan science information network (SINET5) and demonstrate confirms its functionality

HEAR: Approach for Heartbeat Monitoring with Body Movement Compensation by IR-UWB Radar

Further applications of impulse radio ultra-wideband radar in mobile health are hindered by the difficulty in extracting such vital signals as heartbeats from moving targets. Although the empirical mode decomposition based method is applied in recovering waveforms of heartbeats and estimating heart rates, the instantaneous heart rate is not achievable. This paper proposes a Heartbeat Estimation And Recovery (HEAR) approach to expand the application to mobile scenarios and extract instantaneous heartbeats. Firstly, the HEAR approach acquires vital signals by mapping maximum echo amplitudes to the fast time delay and compensating large body movements. Secondly, HEAR adopts the variational nonlinear chirp mode decomposition in extracting instantaneous frequencies of heartbeats. Thirdly, HEAR extends the clutter removal method based on the wavelet decomposition with a two-parameter exponential threshold. Compared to heart rates simultaneously collected by electrocardiograms (ECG), HEAR achieves a minimum error rate 4.6% in moving state and 2.25% in resting state. The Bland–Altman analysis verifies the consistency of beat-to-beat intervals in ECG and extracted heartbeat signals with the mean deviation smaller than 0.1 s. It indicates that HEAR is practical in offering clinical diagnoses such as the heart rate variability analysis in mobile monitoring