FaaSwap: SLO-Aware, GPU-Efficient Serverless Inference via Model Swapping

The dynamic request patterns of machine learning (ML) inference workloads have driven an increasing trend towards exploiting serverless computing for scalable ML model serving. However, today's serverless platforms lack efficient support for GPUs -- provisioning functions on GPUs incurs extremely high overhead, forcing them to keep long-running even when idling for reduced cold starts. This leads to significant resource waste to perform ML inference and hinders the pay-per-use billing for GPUs. In this paper, we present FaaSwap, a serverless platform enabling fine-grained, request-level GPU sharing for resource-efficient ML inference. FaaSwap leverages model swapping to support fast inference execution at low resource cost. It keeps models in a host which has a large amount of cheap memory and quickly swaps models to GPUs when requested, reducing per-function keep-alive cost and enabling efficient GPU sharing across much more functions. FaaSwap also supports swapping models between GPUs for load balancing and improved inference performance. In FaaSwap, we design sophisticated request scheduling and memory management algorithms that efficiently exploit model swapping to reduce GPU cost and meet latency service-level objectives (SLOs) for all inference functions. We have implemented and integrated FaaSwap into Alibaba Cloud Function Compute (FC), one of the world's largest commercial serverless platform. Evaluation results show that FaaSwap can achieve low-latency model swapping, efficiently share a GPU across hundreds of functions, and satisfy per-function latency SLOs at scale

PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization

Instruction tuning large language models (LLMs) remains a challenging task, owing to the complexity of hyperparameter selection and the difficulty involved in evaluating the tuned models. To determine the optimal hyperparameters, an automatic, robust, and reliable evaluation benchmark is essential. However, establishing such a benchmark is not a trivial task due to the challenges associated with evaluation accuracy and privacy protection. In response to these challenges, we introduce a judge large language model, named PandaLM, which is trained to distinguish the superior model given several LLMs. PandaLM's focus extends beyond just the objective correctness of responses, which is the main focus of traditional evaluation datasets. It addresses vital subjective factors such as relative conciseness, clarity, adherence to instructions, comprehensiveness, and formality. To ensure the reliability of PandaLM, we collect a diverse human-annotated test dataset, where all contexts are generated by humans and labels are aligned with human preferences. Our results indicate that PandaLM-7B achieves 93.75% of GPT-3.5's evaluation ability and 88.28% of GPT-4's in terms of F1-score on our test dataset. PandaLM enables the evaluation of LLM to be fairer but with less cost, evidenced by significant improvements achieved by models tuned through PandaLM compared to their counterparts trained with default Alpaca's hyperparameters. In addition, PandaLM does not depend on API-based evaluations, thus avoiding potential data leakage. All resources of PandaLM are released at https://github.com/WeOpenML/PandaLM

Nascent SecM Chain Outside the Ribosome Reinforces Translation Arrest

<div><p>SecM, a bacterial secretion monitor protein, contains a specific amino acid sequence at its C-terminus, called arrest sequence, which interacts with the ribosomal tunnel and arrests its own translation. The arrest sequence is sufficient and necessary for stable translation arrest. However, some previous studies have suggested that the nascent chain outside the ribosome affects the stability of translation arrest. To clarify this issue, we performed <i>in vitro</i> translation assays with HaloTag proteins fused to the C-terminal fragment of <i>E</i>. <i>coli</i> SecM containing the arrest sequence or the full-length SecM. We showed that the translation of HaloTag proteins, which are fused to the fragment, is not effectively arrested, whereas the translation of HaloTag protein fused to full-length SecM is arrested efficiently. In addition, we observed that the nascent SecM chain outside the ribosome markedly stabilizes the translation arrest. These results indicate that changes in the nascent polypeptide chain outside the ribosome can affect the stability of translation arrest; the nascent SecM chain outside the ribosome stabilizes the translation arrest.</p></div

In vitro translation of HaloTag proteins harbouring the arrest sequence.

<p>(<b>A</b>) Halo-L8-SecM_133–170 (lane 1), Halo-L17-SecM_133–170 (lane 2), Halo-L26-SecM_133–170 (lane 3), Halo-pD-L8-SecM_133–170 (lane 4) and Halo-SecM_1–170 (lane 5) were translated in the presence of HaloTag TMR Ligand using the PURExpress ΔRibosome Kit at 37°C for 20 min. Puromycin (1 mg/mL) was added to the reaction mixture at 0 min, and the reaction mixture was incubated at 37°C for 3 min. Aliquots were withdrawn before the addition of puromycin and after 3-min incubation and subjected to NuPAGE. Polypeptides labelled with HaloTag TMR Ligand were detected using Molecular Imager FX. Black and white arrowheads indicate the translation arrest products (polypeptidyl-tRNA) and released products, respectively. The results shown are representative of three independent experiments with similar results. (<b>B</b>) Myc-Halo-L8-SecM_133–170 (lane 1), myc-Halo-L17-SecM_133–170 (lane 2), myc-Halo-L26-SecM_133–170 (lane 3), myc-Halo-pD-L8-SecM_133–170 (lane 4) and myc-Halo-SecM_1–170 (lane 5) were translated in the absence of HaloTag TMR Ligand using the PURExpress ΔRibosome Kit at 37°C for 20 min. Puromycin (1 mg/mL) was added at 0 min, and the reaction mixture was incubated at 37°C for 3 min. Aliquots were withdrawn before the addition of puromycin and after a 3-min incubation and subjected to NuPAGE. Myc-tagged polypeptides were detected by western blotting with anti-c-myc-tag. Black and white arrowheads indicate the translation arrest products (polypeptidyl-tRNA) and released products, respectively. The results shown are representative of three independent experiments with similar results. (<b>C</b>) Fractions of translation arrest products in the absence (left) and the presence of puromycin (right). Filled bars, fluorescence detection using HaloTag TMR Ligand; open bars, detection by western blotting. Error bars represent the standard deviation (SD) of three independent experiments. The asterisk indicates statistical significance as determined by the Student's <i>t</i>-test (<i>p</i> < 0.05).</p

Lifetimes of the translation arrest of HaloTag proteins harbouring the arrest sequence.

<p>(<b>A</b>-<b>D</b>) Time-course analyses of polypeptidyl-tRNA remaining after the addition of puromycin. Halo-L17-SecM_133–170 (<b>A</b>), Halo-L26-SecM_133–170 (<b>B</b>), Halo-pD-L8-SecM_133–170 (<b>C</b>) and Halo-SecM_1–170 (<b>D</b>) were translated in the presence of HaloTag TMR Ligand using the PURExpress ΔRibosome Kit at 37°C for 20 min. Puromycin (1 mg/mL) was added to the reaction mixture at 0 min, and the mixture was incubated at 37°C. Aliquots removed at the indicated time points were subjected to NuPAGE. Polypeptides labelled with HaloTag TMR Ligand were detected using Molecular Imager FX. Black and white arrowheads indicate the translation arrest products (polypeptidyl-tRNA) and released products, respectively. (<b>E</b>) Plots of the fraction of polypeptidyl-tRNA remaining in the presence of puromycin as a function of time. Squares, Halo-L17-SecM_133–170; diamonds, Halo-L26-SecM_133–170; triangles, Halo-pD-L8-SecM_133–170; circles, Halo-SecM_1–170. Data points represent means ± SD of three independent experiments. The solid and dotted lines show the fit to the data obtained using a single exponential function. The lifetimes of the translation arrest of Halo-L17-SecM_133–170, Halo-L26-SecM_133–170, Halo-pD-L8-SecM_133–170 and Halo-SecM_1–170 were 5.6 ± 0.066, 11 ± 0.22, 9.4 ± 0.63 and 51 ± 1.6 min, respectively (the errors represent fitting errors). (<b>F</b>) Time-course analysis of myc-SecM_1–170 polypeptidyl-tRNA remaining after the addition of puromycin. Myc-SecM_1–170 was translated using the PURExpress ΔRibosome Kit at 37°C for 40 min. Puromycin (1 mg/mL) was added at 0 min, and the mixture was incubated at 37°C. Aliquots were withdrawn at indicated time points and subjected to NuPAGE. Myc-SecM_1–170 was detected by western blotting with anti-c-myc-tag. Black and white arrowheads indicate the translation arrest products (polypeptidyl-tRNA) and released products, respectively. (<b>G</b>) The fraction of myc-SecM_1–170 polypeptidyl-tRNA remaining in the presence of puromycin as a function of time. Data points with error bars represent means ± SD for three independent experiments. The solid line shows the fit to the data obtained using a single exponential function. The lifetime of the translation arrest of myc-SecM_1–170 was 48 min ± 4.3 min (the error corresponds to fitting error).</p

In vitro translation of HaloTag proteins with mutated arrest sequence.

<p>Each protein construct, with or without a mutation (R163A or P166A) in the arrest sequence, was translated in the presence of HaloTag TMR Ligand using the PURExpress ΔRibosome Kit at 37°C for 20 min. Puromycin (1 mg/mL) was added at 0 min, and the reaction mixture incubated at 37°C for 3 min. Aliquots were withdrawn before and 3 min after the addition of puromycin and subjected to NuPAGE. Polypeptides labelled with HaloTag TMR Ligand were detected using Molecular Imager FX. <b>A</b>, Halo-L8-SecM_133–170; <b>B</b>, Halo-L17-SecM_133–170; <b>C</b>, Halo-L26-SecM_133–170; <b>D</b>, Halo-pD-L8-SecM_133–170; <b>E</b>, Halo-SecM_1–170. Black and white arrowheads indicate the translation arrest products (polypeptidyl-tRNA) and released products, respectively. The results shown are representative of three independent experiments with similar results.</p

Optimization of Ni0.95−xZnxCo0.05Fe1.90Mn0.02O4 ceramics with promising magneto-dielectric properties for VHF antenna miniaturization

Magnetic, dielectric and DC conductive properties of Ni0.95−xZnxCo0.05Fe1.90Mn0.02O4 (with x=0-0.20 at an interval of 0.05) ferrite ceramics were studied, in order to develop magneto-dielectric materials with almost equal values of relative permeability and permittivity, for the miniaturization of HF (3–30MHz) and VHF (30–90MHz and 100–300MHz) antennas. The ferrite ceramics were prepared by using the conventional two-step sintering process. The real part of relative permeability is increased almost linearly with increasing concentration of Zn, while that of relative permittivity keeps nearly unchanged. It is found that promising magneto-dielectric materials, with close values of real permeability and permittivity over 30–90 MHz (VHF), can be obtained for the samples at Zn concentrations between x=0.05 and x=0.10

Rapid processing of ferrite ceramics with promising magneto-dielectric characteristics

Ferrite ceramics, Ni0.88Zn0.07Co0.05Fe1.98O4, with the addition of 4wt.% Bi2O3 as sintering aid, were fabricated by using a simple one-step processing without involving the step of calcination. X-ray diffraction (XRD) results indicated that single phase ferrite ceramics can be achieved after sintering at 1000∘C for 2h. The samples demonstrated relative densities in the range of 97–99%. Desired magneto-dielectric properties have been approached by adjusting the sintering temperature and sintering time duration. This technique is believed to be applicable to other ceramic materials.Published versio

Microfluidic preparation of optical sensors for biomedical applications

Abstract Optical biosensors are platforms that translate biological information into detectable optical signals, and have extensive applications in various fields due to their characteristics of high sensitivity, high specificity, dynamic sensing, etc. The development of optical sensing materials is an important part of optical sensors. In this review, we emphasize the role of microfluidic technology in the preparation of optical sensing materials and the application of the derived optical sensors in the biomedical field. We first present some common optical sensing mechanisms and the functional responsive materials involved. Then, we describe the preparation of these sensing materials by microfluidics. Afterward, we enumerate the biomedical applications of these optical materials as biosensors in disease diagnosis, drug evaluation, and organ‐on‐a‐chip. Finally, we discuss the challenges and prospects in this field