Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (MoE) Inference

Mixture-of-Experts (MoE) models have gained popularity in achieving state-of-the-art performance in a wide range of tasks in computer vision and natural language processing. They effectively expand the model capacity while incurring a minimal increase in computation cost during training. However, deploying such models for inference is difficult due to their large size and complex communication pattern. In this work, we provide a characterization of two MoE workloads, namely Language Modeling (LM) and Machine Translation (MT) and identify their sources of inefficiencies at deployment. We propose three optimization techniques to mitigate sources of inefficiencies, namely (1) Dynamic gating, (2) Expert Buffering, and (3) Expert load balancing. We show that dynamic gating improves maximum throughput by 6.21-11.23$\times$ for LM, 5.75-10.98$\times$ for MT Encoder and 2.58-5.71$\times$ for MT Decoder. It also reduces memory usage by up to 1.36$\times$ for LM and up to 1.1$\times$ for MT. We further propose Expert Buffering, a new caching mechanism that only keeps hot, active experts in GPU memory while buffering the rest in CPU memory. This reduces static memory allocation by up to 1.47$\times$. We finally propose a load balancing methodology that provides additional scalability to the workload

Dimethylarginine Dimethylaminohydrolase II Overexpression Attenuates LPS-Mediated Lung Leak in Acute Lung Injury

Acute lung injury (ALI) is a severe hypoxemic respiratory insufficiency associated with lung leak, diffuse alveolar damage, inflammation, and loss of lung function. Decreased dimethylaminohydrolase (DDAH) activity and increases in asymmetric dimethylarginine (ADMA), together with exaggerated oxidative/nitrative stress, contributes to the development of ALI in mice exposed to LPS. Whether restoring DDAH function and suppressing ADMA levels can effectively ameliorate vascular hyperpermeability and lung injury in ALI is unknown, and was the focus of this study. In human lung microvascular endothelial cells, DDAH II overexpression prevented the LPS-dependent increase in ADMA, superoxide, peroxynitrite, and protein nitration. DDAH II also attenuated the endothelial barrier disruption associated with LPS exposure. Similarly, in vivo, we demonstrated that the targeted overexpression of DDAH II in the pulmonary vasculature significantly inhibited the accumulation of ADMA and the subsequent increase in oxidative/nitrative stress in the lungs of mice exposed to LPS. In addition, augmenting pulmonary DDAH II activity before LPS exposure reduced lung vascular leak and lung injury and restored lung function when DDAH activity was increased after injury. Together, these data suggest that enhancing DDAH II activity may prove a useful adjuvant therapy to treat patients with ALI

High resolution mapping of QTLs for fruit color and firmness in Amrapali/Sensation mango hybrids

IntroductionMango (Mangifera indica L.), acclaimed as the ‘king of fruits’ in the tropical world, has historical, religious, and economic values. It is grown commercially in more than 100 countries, and fresh mango world trade accounts for ~3,200 million US dollars for the year 2020. Mango is widely cultivated in sub-tropical and tropical regions of the world, with India, China, and Thailand being the top three producers. Mango fruit is adored for its taste, color, flavor, and aroma. Fruit color and firmness are important fruit quality traits for consumer acceptance, but their genetics is poorly understood.MethodsFor mapping of fruit color and firmness, mango varieties Amrapali and Sensation, having contrasting fruit quality traits, were crossed for the development of a mapping population. Ninety-two bi-parental progenies obtained from this cross were used for the construction of a high-density linkage map and identification of QTLs. Genotyping was carried out using an 80K SNP chip array.Results and discussionInitially, we constructed two high-density linkage maps based on the segregation of female and male parents. A female map with 3,213 SNPs and male map with 1,781 SNPs were distributed on 20 linkages groups covering map lengths of 2,844.39 and 2,684.22cM, respectively. Finally, the integrated map was constructed comprised of 4,361 SNP markers distributed on 20 linkage groups, which consisted of the chromosome haploid number in Mangifera indica (n =20). The integrated genetic map covered the entire genome of Mangifera indica cv. Dashehari, with a total genetic distance of 2,982.75 cM and an average distance between markers of 0.68 cM. The length of LGs varied from 85.78 to 218.28 cM, with a mean size of 149.14 cM. Phenotyping for fruit color and firmness traits was done for two consecutive seasons. We identified important consistent QTLs for 12 out of 20 traits, with integrated genetic linkages having significant LOD scores in at least one season. Important consistent QTLs for fruit peel color are located at Chr 3 and 18, and firmness on Chr 11 and 20. The QTLs mapped in this study would be useful in the marker-assisted breeding of mango for improved efficiency

That was close! A counterfactual simulation model of causal judgments about decisions

How do people make causal judgments about other's decisions? Prior work has argued that judging causation requires going beyond what actually happened and simulating what would have happened in a relevant counterfactual situation. Here, we extend the counterfactual simulation model of causal judgments for physical events, to explain judgments about other agents' decisions. In our experiments, an agent chooses what path to take to reach a goal. In Experiment 1, participants either made hypothetical judgments about whether the agent would succeed were it to take a certain path, or counterfactual judgments about whether the agent would have succeeded had it taken a different path. In Experiment 2, participants made causal judgments about whether the agent succeeded or failed because of the path that it took. Our computational model accurately captured participants' judgments in both experiments and we find that causal judgments are better explained by counterfactuals rather than hypotheticals

A computational model of responsibility judgments from counterfactual simulations and intention inferences

How responsible someone is for an outcome depends on what causal role their actions played, and what those actions reveal about their mental states, such as their intentions. In this paper, we develop a computational account of responsibility attribution that integrates these two cognitive processes: causal attribution and mental state inference. Our model makes use of a shared generative planning algorithm assumed to approximate people's intuitive theory of mind about others' behavior. We test our model on a variety of animated social scenarios in two experiments. Experiment 1 features simple cases of helping and hindering. Experiment 2 features more complex interactions that require recursive reasoning, including cases where one agent affects another by merely signaling their intentions without physically acting on the world. Across both experiments, our model accurately captures participants' counterfactual simulations and intention inferences, and establishes that these two factors together explain responsibility judgments

A computational model of responsibility judgments from counterfactual simulations and intention inferences

How responsible someone is for an outcome depends on what causal role their actions played, and what those actions reveal about their mental states, such as their intentions. In this paper, we develop a computational account of responsibility attribution that integrates these two cognitive processes: causal attribution and mental state inference. Our model makes use of a shared generative planning algorithm assumed to approximate people's intuitive theory of mind about others' behavior. We test our model on a variety of animated social scenarios in two experiments. Experiment 1 features simple cases of helping and hindering. Experiment 2 features more complex interactions that require recursive reasoning, including cases where one agent affects another by merely signaling their intentions without physically acting on the world. Across both experiments, our model accurately captures participants' counterfactual simulations and intention inferences, and establishes that these two factors together explain responsibility judgments