3 research outputs found
Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Models
Deep learning recommendation models (DLRMs) are used across many
business-critical services at Facebook and are the single largest AI
application in terms of infrastructure demand in its data-centers. In this
paper we discuss the SW/HW co-designed solution for high-performance
distributed training of large-scale DLRMs. We introduce a high-performance
scalable software stack based on PyTorch and pair it with the new evolution of
Zion platform, namely ZionEX. We demonstrate the capability to train very large
DLRMs with up to 12 Trillion parameters and show that we can attain 40X speedup
in terms of time to solution over previous systems. We achieve this by (i)
designing the ZionEX platform with dedicated scale-out network, provisioned
with high bandwidth, optimal topology and efficient transport (ii) implementing
an optimized PyTorch-based training stack supporting both model and data
parallelism (iii) developing sharding algorithms capable of hierarchical
partitioning of the embedding tables along row, column dimensions and load
balancing them across multiple workers; (iv) adding high-performance core
operators while retaining flexibility to support optimizers with fully
deterministic updates (v) leveraging reduced precision communications,
multi-level memory hierarchy (HBM+DDR+SSD) and pipelining. Furthermore, we
develop and briefly comment on distributed data ingestion and other supporting
services that are required for the robust and efficient end-to-end training in
production environments
Recommended from our members
Blood lead concentration and its associated factors in preschool children in eastern Iran: a cross-sectional study
Background Lead is a toxic metal that affects almost every organ in the body. Children are more susceptible to lead toxicity because they ingest non-food items (pica), have oral exploratory habits, absorb more substantial amounts of ingested lead compared to adults, and have a developing central nervous system. This study describes venous blood lead concentrations (BLC) in young children living in Birjand, Iran. Methods A cross-sectional study was performed in 2016 on children 1-7 years of age who were referred to healthcare centers in Birjand City. Demographic information was obtained, and their BLC was tested using atomic absorption spectrometry (AAS). Results Four hundred children were tested. Their mean age was 52.37 +/- 23.77 months; their mean BLC was 2.49 +/- 2.64 mu g/dL (median 1.85 mu g/dL). Thirty-two (8%) children had a BLC > 5 mu g/dL. A logistic regression model revealed that per one unit of increase in age, the chance of an elevated BLC decreased by 3% (OR (95%CI): 0.97 (0.96-0.99),p < 0.01). The risks of an elevated BLC was 61% lower in girls compared to boys (OR (95%CI): 0.39 (0.17-0.92),p = 0.03). Further, per one rate of increase in the BMI, the chance of an elevated BLC was higher (OR (95%CI): 1.13 (1.02-1.24),p = 0.01). Children whose fathers were laborers had higher BLC than those with employee fathers (p = 0.01). Conclusion Of 400 children aged 1-7 years old living in Birjand, Iran, 8% had elevated BLC. BLC correlated with the child 's age, gender, body mass index, and father's occupation.Open access journalThis item from the UA Faculty Publications collection is made available by the University of Arizona with support from the University of Arizona Libraries. If you have questions, please contact us at [email protected]