Bellosa, Frank

Kupferschmied, Philipp

Stoess, Jan

KITopen

NUMA-Aware User-Level Memory Management forMicrokernel-Based Operating SystemsPhilipp Kupferschmied, Jan Stoess, Frank BellosaSystem Architecture GroupUniversity of KarlsruheIn the recent past there has been a growinginterest from research and practitioners in oper-ating system (OS) design with a minimal kernelbase. While minimal (or micro) kernel based sys-tems are a long-standing idea from a standpointof OS research, such systems are now success-fully employed in real-world scenarios, typically ei-ther as true microkernel systems or as hypervisors[5, 7]. At the same time, tremendous advancesin multi-processing and multi-core technology havelead to computer platforms where the typical mem-ory demands are so high that memory bandwidthis becoming a crucial bottleneck. The answer tothat problem is a non-uniform memory architecture(NUMA), where processors or subsets of processorsare connected to their own portion of main memoryvia a dedicated bus, thus alleviating overall loadon each of the buses. NUMA architectures typi-cally imply that the local portion of memory canbe accessed fast, whereas remote accesses are sig-nificantly slower (as an example, an AMD OpteronNUMA platform in our lab shows a remote-to-localread memory access ratio of about 1.4). From asoftware standpoint, data locality and placementare thus of paramout importance in such systems.Ideally, most of a processor’s working set shouldremain in its local memory. Common techniquesto work toward that goal are data replication andmigration.In this work, we explore the requirements fora microkernel-based, NUMA-aware operating sys-tem. On the one hand, designing a system withregard to NUMA architectures requires to improvethe locality of all data items where possible. Onthe other, the goal of maximum flexibility for a mi-crokernel requires to keep the changes to the micro-kernel itself to the minimum. We thus strive for anOS architecture which allows for flexible memorymanagement at user-level, but also improves thelocality of kernel data structures without requiringpolicies within the kernel.Our approach stays in contrast to virtual ma-chine systems such as Disco [2], since it enables ex-plicit application-controlled NUMA memory man-agement rather than to encapsulate applications invirtual machines and to hide memory-managementissues from them. It is also different to the Tornado[4] and K42 [1] approaches in that it restricts thekernel to providing simple and low-level primitivessuch as per-node address spaces, rather than usinga sophisticated clustered object system, which al-lows multiple-component objects to appear like asingle object.Our core kernel design principle is that the ker-nel should provide only low-level and localizedmemory-management abstractions, leaving most ofthe NUMA address space construction to user-level.Particularly, the kernel offers support for local ad-dress space construction only: An address spacealways belongs to the NUMA node on which it iscreated, and the kernel allocates all address spacemanagement information on that node. This leadsto more efficiency and better predictability of thebase address space construction primitives. How-ever, an address space can contain memory map-pings both to local and to remote memory. Ap-plications can thus construct cross-node NUMAaddress spaces by hand, that is, by creating logi-cally coherent per-node address spaces with iden-tical virtual-to-physical mappings, either to node-local or to remote memory. Synchronization of ad-dress spaces is left up to the applications; the kernel1First published in: Poster/WiP session of the EurosSys'09, Nuremberg, Germany, April 2009EVA-STAR (Elektronisches Volltextarchiv – Scientific Articles Repository) http://digbib.ubka.uni-karlsruhe.de/volltexte/1000026891 offers a cross-node communication primitive to al-low signaling and synchronization on address spaceupdates. Node-local regions (virtual addresses thatare mapped to different physical addresses on eachnode, a useful property e.g. for transparent codereplication) can easily be constructed by not syn-chronizing all mappings between related addressspaces. When a thread migrates from one nodeto another, it is also migrated into another addressspace. The bookkeeping of which address spacesbelong to the same application is performed at user-level.For evaluation, we have built a NUMA-aware OSprototype based on the L4 microkernel [6]. L4 of-fers address spaces as a first class abstraction. Theyare populated by user-level pagers, which can mappages of memory into an address space when apagefault occurs, and, later on, unmap them asyn-chronously if needed. The kernel represents ad-dress spaces by means of page tables; the map-ping dependencies are represented by means of amapping database (MDB) that keeps track of howphysical frames are mapped to address spaces. Weenhanced L4’s in-kernel representation of addressspaces by assigning a home node to each addressspace, set to the NUMA node, on which the ad-dress space is created. The kernel allocates mem-ory for page tables and corresponding MDB entrieson that home node. When a user-level applica-tion is parallely active on multiple nodes, we useone L4 address space on each node instead of usinga single, node-spanning address space. By settingthe home node appropriately, this solution ensuresthat the corresponding kernel data is allocated onthe NUMA node, on which the address space isactive. We propose to use a per-node pagefaulthandler, each of which is responsible for handlingpagefaults that occur on its node. For all map-pings that shall be global, that is, equally visible inall per-node address spaces, synchronization is per-formed between these pagers, and entirely at user-level: From the viewpoint of the kernel, the per-node address spaces are completely unrelated. Thiseliminates the need for an additional synchroniza-tion primitive within the kernel, the choice of whichwould depend on hardware characteristics such asthe remote-to-local latency ratio [3]. The user-level pagers can implement arbitrary synchroniza-tion mechanisms, based on shared memory, IPC,or combinations of both, and tailored towards thehardware and application characteristics.Synchronization between per-node addressspaces is performed lazily, by means of pagefaults.Consequently, n pagefaults will occur until a globalmapping is established on n nodes. However, ifparallel threads work mainly on local data, theactual number of pagefaults is comparable to usinga single address space only.A pager can only unmap mappings it has estab-lished. If a pager on one node wants to globallyunmap a page, it must notify all other pagers todo so as well, making the unmap-operation moreexpensive. Further investigations must show if thisoverhead is acceptable in realistic scenarios, and ifthere are more efficient solutions.References[1] J. Appavoo et al. Enabling scalable performancefor general purpose workloads on shared mem-ory multiprocessors. Technical report, IBM Re-search, 2003.[2] E. Bugnion et al. Disco: Running commod-ity operating systems on scalable multiproces-sors. ACM Transactions on Computer Systems,15(4), 1997.[3] E. M. Chaves et al. Kernel-kernel communica-tion in a shared-memory multiprocessor. Con-currency: Practice and Experience, 5(3), 1993.[4] B. Gamsa et al. Tornado: Maximizing localityand concurrency in a shared memory multipro-cessor operating system. In Proceedings of thethird Symposium on Operating Systems Designand Implementation. 1999.[5] G. Heiser. Hypervisors for consumer electron-ics. In Proceedings of the 6th IEEE ConsumerCommunications and Networking Conference.2009.[6] J. Liedtke. On microkernel construction. InProceedings of the 15th ACM Symposium onOperating System Principles. 1995.[7] VMWare. Virtualization overview. http://www.vmware.com/pdf/virtualization.pdf.2

NUMA-aware User-Level Memory Management for Microkernel-Based Operating Systems

https://publikationen.bibliothek.kit.edu/1000026891/2042432

NUMA-aware User-Level Memory Management for Microkernel-Based Operating Systems

Abstract

Similar works

Full text

Available Versions

KITopen