IPA: Inference Pipeline Adaptation to Achieve High Accuracy and Cost-Efficiency

Efficiently optimizing multi-model inference pipelines for fast, accurate, and cost-effective inference is a crucial challenge in ML production systems, given their tight end-to-end latency requirements. To simplify the exploration of the vast and intricate trade-off space of accuracy and cost in inference pipelines, providers frequently opt to consider one of them. However, the challenge lies in reconciling accuracy and cost trade-offs. To address this challenge and propose a solution to efficiently manage model variants in inference pipelines, we present IPA, an online deep-learning Inference Pipeline Adaptation system that efficiently leverages model variants for each deep learning task. Model variants are different versions of pre-trained models for the same deep learning task with variations in resource requirements, latency, and accuracy. IPA dynamically configures batch size, replication, and model variants to optimize accuracy, minimize costs, and meet user-defined latency SLAs using Integer Programming. It supports multi-objective settings for achieving different trade-offs between accuracy and cost objectives while remaining adaptable to varying workloads and dynamic traffic patterns. Extensive experiments on a Kubernetes implementation with five real-world inference pipelines demonstrate that IPA improves normalized accuracy by up to 35% with a minimal cost increase of less than 5%

Flange Wrinkling in Flexible Roll Forming Process

AbstractFlexible roll forming is an advanced sheet metal forming process for producing variable cross section profiles. Flange wrinkling at the transition zone where the cross section changes is a major defect in the flexible roll forming process. In this paper, the flange wrinkling at the transition zone is studied using finite element analysis. The results showed that the strip deformation at the transition zone can be considered as a combination of two strip deformations observed in the conventional roll forming process and the flanging process. According to finite element analysis results, when the flange wrinkling occurs, compressive longitudinal strain is smaller than the necessary compressive longitudinal strain calculated by mathematical modeling to obtain the intended profile geometry in the compression zone. Therefore, comparison of compressive longitudinal strain obtained from the finite element analysis and the necessary compressive longitudinal strain is a good criterion to predict the flange wrinkling occurrence. A flexible roll forming setup was developed. Longitudinal strain history is obtained from the finite element simulation and is compared with the experimental data from the flexible roll forming setup. Results show a good agreement and confirm the finite element analysis

The Effect of Post-discharge Telephone Training and Follow-up on Self-care Behaviors of Myocardial Infarction Patients

Background and purpose: Patients with myocardial infarction need to receive care and self-care ability. The aim of this study was to determine the effect of post-discharge education and follow-up on self-care behaviors of patients with myocardial infarction. Materials and Methods: In this quasi-experimental study, 116 patients with myocardial infarction were selected by convenience sampling method and randomly assigned to the two groups of control (n = 58) and intervention (n = 58). In the intervention group, a face-to-face training session was held first. Then telephone follow-up with training immediately after discharge, twice a week in the first month and once a week in the second month with the intervention group. The standard questionnaire of self-care behaviors in patients with myocardial infarction was used to collect information. SPSS software version 23 and descriptive statistics and Fisherchr('39')s independent and accurate t-tests were used to analyze the data. Results: The mean score of self-care in the intervention group in one month and two months after training compared to discharge time was 35.29 and 44.10 units, respectively. The results of LSD post hoc test showed that these differences were statistically significant (p <0.001). Using chi-square test, a statistically significant difference was observed between the frequency distribution of patients in the control and intervention groups in one month and two months after the intervention in terms of self-care. Conclusion: Due to its availability, the use of telephone follow-up can be used as a method to follow the training provided to patients

A Tale of Two Scales: Reconciling Horizontal and Vertical Scaling for Inference Serving Systems

Inference serving is of great importance in deploying machine learning models in real-world applications, ensuring efficient processing and quick responses to inference requests. However, managing resources in these systems poses significant challenges, particularly in maintaining performance under varying and unpredictable workloads. Two primary scaling strategies, horizontal and vertical scaling, offer different advantages and limitations. Horizontal scaling adds more instances to handle increased loads but can suffer from cold start issues and increased management complexity. Vertical scaling boosts the capacity of existing instances, allowing for quicker responses but is limited by hardware and model parallelization capabilities. This paper introduces Themis, a system designed to leverage the benefits of both horizontal and vertical scaling in inference serving systems. Themis employs a two-stage autoscaling strategy: initially using in-place vertical scaling to handle workload surges and then switching to horizontal scaling to optimize resource efficiency once the workload stabilizes. The system profiles the processing latency of deep learning models, calculates queuing delays, and employs different dynamic programming algorithms to solve the joint horizontal and vertical scaling problem optimally based on the workload situation. Extensive evaluations with real-world workload traces demonstrate over 10× SLO violation reduction compared to the state-of-the-art horizontal or vertical autoscaling approaches while maintaining resource efficiency when the workload is stable

Reconciling High Accuracy, Cost-Efficiency, and Low Latency of Inference Serving Systems

The use of machine learning (ML) inference for various applications is growing drastically. ML inference services engage with users directly, requiring fast and accurate responses. Moreover, these services face dynamic workloads of requests, imposing changes in their computing resources. Failing to right-size computing resources results in either latency service level objectives (SLOs) violations or wasted computing resources. Adapting to dynamic workloads considering all the pillars of accuracy, latency, and resource cost is challenging. In response to these challenges, we propose InfAdapter, which proactively selects a set of ML model variants with their resource allocations to meet latency SLO while maximizing an objective function composed of accuracy and cost. InfAdapter decreases SLO violation and costs up to 65 and 33, respectively, compared to a popular industry autoscaler (Kubernetes Vertical Pod Autoscaler)

IPA: Inference Pipeline Adaptation to Achieve High Accuracy and Cost-Efficiency

Efficiently optimizing multi-model inference pipelines for fast, accurate, and cost-effective inference is a crucial challenge in ML production systems, given their tight end-to-end latency requirements. To simplify the exploration of the vast and intricate trade-off space of accuracy and cost in inference pipelines, providers frequently opt to consider one of them. However, the challenge lies in reconciling accuracy and cost trade-offs. To address this challenge and propose a solution to efficiently manage model variants in inference pipelines, we present IPA, an online deep-learning Inference Pipeline Adaptation system that efficiently leverages model variants for each deep learning task. Model variants are different versions of pre-trained models for the same deep learning task with variations in resource requirements, latency, and accuracy. IPA dynamically configures batch size, replication, and model variants to optimize accuracy, minimize costs, and meet user-defined latency SLAs using Integer Programming. It supports multi-objective settings for achieving different trade-offs between accuracy and cost objectives while remaining adaptable to varying workloads and dynamic traffic patterns. Extensive experiments on a Kubernetes implementation with five real-world inference pipelines demonstrate that IPA improves normalized accuracy by up to 35% with a minimal cost increase of less than 5%

[Solution] IPA: Inference Pipeline Adaptation to achieve high accuracy and cost-efficiency

Efficiently optimizing multi-model inference pipelines for fast, accurate, and cost-effective inference is a crucial challenge in machine learning production systems, given their tight end-to-end latency requirements. To simplify the exploration of the vast and intricate trade-off space of latency, accuracy, and cost in inference pipelines, providers frequently opt to consider one of them. However, the challenge lies in reconciling latency, accuracy, and cost trade-offs. To address this challenge and propose a solution to efficiently manage model variants in inference pipelines, we present IPA, an online deep learning Inference Pipeline Adaptation system that efficiently leverages model variants for each deep learning task. Model variants are different versions of pre-trained models for the same deep learning task with variations in resource requirements, latency, and accuracy. IPA dynamically configures batch size, replication, and model variants to optimize accuracy, minimize costs, and meet user-defined latency Service Level Agreements (SLAs) using Integer Programming. It supports multi-objective settings for achieving different trade-offs between accuracy and cost objectives while remaining adaptable to varying workloads and dynamic traffic patterns. Navigating a wider variety of configurations allows IPA to achieve better trade-offs between cost and accuracy objectives compared to existing methods. Extensive experiments in a Kubernetes implementation with five real-world inference pipelines demonstrate that IPA improves end-to-end accuracy by up to 21% with a minimal cost increase. The code and data for replications are available at https: //github.com/reconfigurable-ml-pipeline/ipa

How hypoxia regulate exosomes in ischemic diseases and cancer microenvironment?

Exosomes, as natural occurring vesicles, play highly important roles in the behavior and fate of ischemic diseases and different tumors. Secretion, composition, and function of exosomes are remarkably influenced by hypoxia in ischemic diseases and tumor microenvironment. Exosomes secreted from hypoxic cells affect development, growth, angiogenesis, and progression in ischemic diseases and tumors through a variety of signaling pathways. In this review article, we discuss how hypoxia affects the quantity and quality of exosomes, and review the mechanisms by which hypoxic cell-derived exosomes regulate ischemic cell behaviors in both cancerous and noncancerous cells

Effects of the COVID-19 pandemic on lifestyle among Iranian population: A multicenter cross-sectional study

Background: Quarantine, an unpleasant experience, was implemented in many countries to limit the spread of Coronavirus disease 2019 (COVID-19), which it could associated whit lifestyle changes. The present study aimed to determine the changes in Iranian's lifestyle during COVID-19 pandemic. Materials and Methods: In the present cross-sectional study, 2710 Iranian people completed an online researcher-made questionnaire asking lifestyle regarding COVID-19, which includes five sections about physical activity, stress and anxiety, nutrition habit, sleep disorders, and interpersonal relationship in addition to demographic data from January to February 2021, using the multistage cluster sampling method. Results: The participants' mean age was 33.78 +/- 11.50 years and 68.3% of them were female. Traveling, sightseeing, and family visits have been eliminated from 91%, 83.5%, and 77.5% of participants' lives, respectively. There were increase in stress level (P < 0.001), weight of the participants (P < 0.001), sleep problems (P < 0.001), and healthier foods (P < 0.001) but decrease in interpersonal communication (P < 0.001) and the amount of physical activity (P < 0.001). Conclusion: In summary, this study indicates some changes in lifestyle of Iranian people, including changes in some eating practices, physical activity, social communication, and sleeping habits during the pandemic. However, as the COVID-19 pandemic is ongoing, a comprehensive understanding of these behaviors and habits can help develop interventions to mitigate the negative lifestyle behaviors during COVID-19 pandemic

How hypoxia regulate exosomes in ischemic diseases and cancer microenvironment?

Exosomes, as natural occurring vesicles, play highly important roles in the behavior and fate of ischemic diseases and different tumors. Secretion, composition, and function of exosomes are remarkably influenced by hypoxia in ischemic diseases and tumor microenvironment. Exosomes secreted from hypoxic cells affect development, growth, angiogenesis, and progression in ischemic diseases and tumors through a variety of signaling pathways. In this review article, we discuss how hypoxia affects the quantity and quality of exosomes, and review the mechanisms by which hypoxic cell-derived exosomes regulate ischemic cell behaviors in both cancerous and noncancerous cells