ROAM: memory-efficient large DNN training via optimized operator ordering and memory layout

As deep learning models continue to increase in size, the memory requirements for training have surged. While high-level techniques like offloading, recomputation, and compression can alleviate memory pressure, they also introduce overheads. However, a memory-efficient execution plan that includes a reasonable operator execution order and tensor memory layout can significantly increase the models' memory efficiency and reduce overheads from high-level techniques. In this paper, we propose ROAM which operates on computation graph level to derive memory-efficient execution plan with optimized operator order and tensor memory layout for models. We first propose sophisticated theories that carefully consider model structure and training memory load to support optimization for large complex graphs that have not been well supported in the past. An efficient tree-based algorithm is further proposed to search task divisions automatically, along with delivering high performance and effectiveness to solve the problem. Experiments show that ROAM achieves a substantial memory reduction of 35.7%, 13.3%, and 27.2% compared to Pytorch and two state-of-the-art methods and offers a remarkable 53.7x speedup. The evaluation conducted on the expansive GPT2-XL further validates ROAM's scalability

Kenyon Alumni Magazine - Spring 2023

https://digital.kenyon.edu/kcab/1304/thumbnail.jp

Review of the Esduardo Mariscal Dance Theater production of The Secret Waltz of

Review of the Esduardo Mariscal Dance Theater production of The Secret Waltz of the Flies, being performed at Oak Street Theatre in Portland

Casco Bay Weekly : 31 October 1996

https://digitalcommons.portlandlibrary.com/cbw_1996/1044/thumbnail.jp

Casco Bay Weekly : 20 June 2002

https://digitalcommons.portlandlibrary.com/cbw_2002/1025/thumbnail.jp

The Space Between: Performance, the Body and Scholarship

The thesis is concerned with the interrelationships between the body, making sense of experience through performance, and the conceptual and scholarly understanding that people construct around experience. The lens through which these interrelationships are explored is phenomenology, both in terms of phenomenological theory per se and, more specifically, with theories related to performance and pedagogical process. The research question explores the relationship between the body, space/place and digital media through four cycles of participatory action research in which practice and theory are interrelated. The experience of (the body) in space and place is captured and re-created with digital media in the live performance space drawing attention to spatial and temporal anomalies that both de-stabilise and re-affirm what is it to be ‘now’ and ‘here.’ Ideas shift from the determined to the disintegrated, and the body moves between a critical engagement with experience and a pre-reflective and heightened consciousness of ‘being’ in performance – as maker, performer and viewer, and as learner, teacher and researcher. Answers to questions are replaced by gaps and spaces between – in which the known, the not known, and the imagined unfold and become exposed. Experiments shift from the body immersed in and subsumed by technology to the body, live (not mediatised) in performance, and again to the live as mediatised, exposing the phenomena that we encounter. Performance emerges as the body touched, sensed and multi-faceted in an in-between space of inter-relationships, inter-subjectivities and inter-medialities. The body is both fullness and void, coexistent and isolated – in suspense as it hovers and ‘is’ of all worlds. Investigations are devised and delivered, with students as co-researchers, through a teaching and learning model that guides and exposes, disrupts and transforms – creating a pedagogy of instability and discovery in order to reveal new and innovative performance

Casco Bay Weekly : 5 December 1996

https://digitalcommons.portlandlibrary.com/cbw_1996/1049/thumbnail.jp