|Publication number||US7529906 B2|
|Application number||US 10/643,588|
|Publication date||May 5, 2009|
|Filing date||Aug 18, 2003|
|Priority date||Aug 18, 2003|
|Also published as||EP1665054A2, EP2284711A1, US20050044339, WO2005020088A2, WO2005020088A3|
|Publication number||10643588, 643588, US 7529906 B2, US 7529906B2, US-B2-7529906, US7529906 B2, US7529906B2|
|Original Assignee||Cray Incorporated|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (16), Non-Patent Citations (7), Referenced by (7), Classifications (7), Legal Events (4)|
|External Links: USPTO, USPTO Assignment, Espacenet|
This application is related to U.S. patent application Ser. No. 10/643,758, entitled “REMOTE TRANSLATION MECHANISM FOR A MULTINODE SYSTEM”, filed on even date herewith; to U.S. patent application Ser. No. 10/235,898 (now U.S. Pat. No. 6,922,766), entitled “REMOTE TRANSLATION MECHANISM FOR A MULTINODE SYSTEM”, filed Sep. 4, 2002; to U.S. patent application Ser. No. 10/643,744, entitled “Multistream Processing System and Method”, filed on even date herewith; to U.S. patent application Ser. No. 10/643,577, entitled “System and Method for Synchronizing Memory Transfers”, filed on even date herewith; to U.S. patent application Ser. No. 10/643,742, entitled “Decoupled Store Address and Data in a Multiprocessor System”, filed on even date herewith; to U.S. patent application Ser. No. 10/643,586 (now U.S. Pat. No. 7,334,110), entitled “Decoupled Scalar/Vector Computer Architecture”, filed on even date herewith; to U.S. patent application Ser. No. 10/643,585, entitled “Latency Tolerant Distributed Shared Memory Multiprocessor Computer”, filed on even date herewith; to U.S. patent application Ser. No. 10/643,754, entitled “Relaxed Memory Consistency Model”, filed on even date herewith; to U.S. patent application Ser. No. 10/643,758, entitled “Remote Translation Mechanism for a Multinode System”, filed on even date herewith; and to U.S. patent application Ser. No. 10/643,741 (now U.S. Pat. No. 7,437,521), entitled “Method and Apparatus for Local Synchronizations in a Vector Processor System”, filed on even date herewith, each of which is incorporated herein by reference.
The present invention relates memory management in computer systems, and more particularly to systems and methods for sharing memory using scalable hardware resources.
A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent document or the patent disclosure as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever. The following notice applies to the software and data as described below and in the drawings hereto: Copyright © 2003, Cray, Inc. All Rights Reserved.
Multiprocessor computer systems include a number of processing nodes connected together by an interconnection network. Typically, each processing node includes one or more processors, a local memory, and an interface circuit connecting the node to the interconnection network. The interconnection network is used for transmitting packets of information between processing nodes.
Distributed, shared-memory multiprocessor systems include a number of processing nodes that share a distributed memory element. By increasing the number of processing nodes, or the number of processors within each node, such systems can often be scaled to handle increased demand. In such a system, each processor is able to access local memory, or memory of other (remote) processing nodes. Typically, a virtual address is used for all memory accesses within a distributed, shared-memory multiprocessor system, and is translated into a physical address in the requesting node's translation look-aside buffer (TLB). Thus, the requesting node's TLB will need to contain address translation information for all the memory that the node is able to access (local or remote). This amount of address translation information can be substantial, and can result in much duplication of translation information throughout the multiprocessor system (e.g., if the same page of memory is accessed by 64 different nodes, the TLB used by each node will need to contain an entry for that page). This type of system does not scale efficiently to very large memories.
Therefore, there is a need for an address translation mechanism in a multi-processor system that addresses these and other shortcomings. As a result, there is a need in the art for the present invention.
The above-mentioned shortcomings, disadvantages and problems are addressed by the present invention, which will be understood by reading and studying the following specification.
One aspect of the systems and methods includes translating a virtual memory address into a physical memory address in a multi-node system that is initiated by providing the virtual memory address at a source node. A determination is made that a translation for the virtual memory address does not exist. A physical node to query is determined based on the virtual memory address. An emulated remote translation table (ERTT) segment is queried on the determined physical node to see if the ERTT segment may provide a translation. If the translation is received then the translation may be loaded into a TLB on the source node. Otherwise a memory reference error may be generated for the entity or application referencing the invalid virtual memory address.
A further aspect is that the ERTT segment may be located in generally accessible node memory, e.g. memory that is available for general purpose use by applications and the kernel.
The present invention describes systems, clients, servers, methods, and computer-readable media of varying scope. In addition to the aspects and advantages of the present invention described in this summary, further aspects and advantages of the invention will become apparent by reference to the drawings and by reading the detailed description that follows.
In the following detailed description of exemplary embodiments of the invention, reference is made to the accompanying drawings which form a part hereof, and in which is shown by way of illustration specific exemplary embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, and it is to be understood that other embodiments may be utilized and that logical, mechanical, electrical and other changes may be made without departing from the scope of the present invention.
Some portions of the detailed descriptions which follow are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the ways used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussions, terms such as “processing” or “computing” or “calculating” or “determining” or “displaying” or the like, refer to the action and processes of a computer system, or similar computing device, that manipulates and transforms data represented as physical (e.g., electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.
In the Figures, the same reference number is used throughout to refer to an identical component which appears in multiple Figures. Signals and connections may be referred to by the same reference number or label, and the actual meaning will be clear from its use in the context of the description.
The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is defined only by the appended claims.
In some embodiments of the invention, memory controller 110 includes a remote translation table (RTT) 112. The RTT 112 contains translation information for a virtual memory address space associated with one or more remote nodes. Further details on the operation of RTT 112 are found in U.S. patent application Ser. No. 10/235,898, entitled “REMOTE TRANSLATION MECHANISM FOR A MULTINODE SYSTEM”, which has been previously incorporated by reference.
Additionally, some embodiments of the invention include an emulated RTT (ERTT) 114. In
In some embodiments, a multiple processor unit 102 includes four processors 104.1-104.4 and four cache memory controllers 106. Although each multiple processor unit is shown in
In one embodiment, the hardware environment is included within the Cray X1 computer system, which represents the convergence of the Cray T3E and the traditional Cray parallel vector processors. The X1 is a highly scalable, cache coherent, shared-memory multiprocessor that uses powerful vector processors as its building blocks, and implements a modernized vector instruction set. In these embodiments, multiple processor unit 102 is a Multi-streaming processor (MSP). It is to be noted that
Physical page 208 comprises a page number of the physical page for an address. In some embodiments, page boundaries may range from 64 KB to 4 GB.
Node number 210 comprises a node number identifying the physical node where the page resides.
Reference count 212 represents a count of the number of processes currently mapping the page. As an example consider two processes A and B. A references pages P1 and P2, while B references pages P1 and P3. The reference count for the ERTT segment entry for page P1 will be 2 because both A and B reference page P1. The reference count for the ERTT segment entries for pages P2 and P3 will be 1 because only one of A or B references the page. Now assume that process B terminates. The reference count for page P3 goes to zero, the reference count for page P1 is set to 1 to reflect the fact that process B no longer references those pages. In some embodiments, when a reference count for an ERTT segment entry goes to zero, the entry is removed and made available for later reuse. In addition, in some embodiments a corresponding entry in the RTT is also cleared and made available for reuse.
Flags 214 comprise a set of flags for the entry. In some embodiments, the flags may include a lock flag indicating that the entry should be locked to prevent another process from altering the entry thereby providing for serial access to the ERTT segment entry. A valid flag indicates that the that the translation in the ERTT segment entry is a valid entry. A write flag indicates whether the page is writable or not.
In some embodiments of the invention, an ERTT segment may contain 16K entries, however no embodiment of the invention is limited to any particular number of entries.
In some embodiments of the invention, an ERTT header 302 is maintained by one of the nodes of the application. In some embodiments, the ERTT header 302 is maintained at a “well known” location in the group of nodes assigned to an application. However, in alternative embodiments, ERTT header 302 may be replicated in all nodes assigned to an application. Further, in some embodiments, ERTT 114 includes an ERTT header 302 that is maintained by virtual node 0 assigned to the application. ERTT header 302 maps virtual nodes to physical nodes. The index into ERTT header is a virtual node number, and the value at that location is the physical node number corresponding to the virtual node number. The ERTT segment 202 may then be located using the physical node number. Thus in the example shown, ERTT header 302 establishes a mapping between virtual nodes 0, 1 and 2 and physical nodes 1, 3 and 6 respectively.
The method begins when an ERTT is initialized (block 402). In some embodiments, an ERTT is initialized in node memory 108 when an application is assigned to run on one or more nodes. In some embodiments, an ERTT is maintained through the use of memory mapping functions such as “madvise”, “mmap”, “shmget” and the like. Further details on maintaining an RTT and systems and methods that are also applicable to maintaining an ERTT may be found in U.S. patent application Ser. No. 10/643,758, entitled “REMOTE TRANSLATION MECHANISM FOR A MULTINODE SYSTEM” which has been previously incorporated by reference. Additionally, the ERTT may be initialized through the normal page fault mechanism of the kernel. As page faults occur, the kernel reads pages into memory and loads the appropriate ERTT segment entry with a translation. In some embodiments, memory must be referenced locally to cause the ERTT to be loaded before a remote node attempts to access local memory.
At some point after the ERTT has been initialized, processes running on one of the nodes assigned to an application may attempt to access a virtual address that is not mapped by the TLB (Translation Lookaside Buffer) for the node. This cause a TLB miss which in turn causes memory management routines in the kernel to execute (block 404). For convenience, one or more kernel memory management routines executed upon a TLB miss will be referred to collectively as the TLBmiss routine.
The TLBmiss routine will then attempt to map the referenced page. In some embodiments, the TLBmiss routine uses predetermined bits of the virtual address as a virtual node number (block 406). The TLBmiss routine may the consult the ERTT header 302 in order to determine the physical node having the physical address corresponding to the virtual address (block 408). In some embodiments, if the virtual node maps to the same node that the application or kernel is running on, the standard operating system memory management routines may be used to make the translation to a physical page on the local node.
However, if the reference is to a remote node, the TLBmiss routine then uses the offset portion of the virtual address as an index to query the determined physical node's ERTT segment 202 to determine if the address is valid on the physical node (block 410). If the physical node's ERTT segment 202 has a translation for the virtual address, then the translation is placed in the referring (i.e. the local or source) node's TLB (block 414) In some embodiments, the ERTT segment entry may be placed directly into the TLB because the ERTT entry format matches the TLB format with respect to all significant bites.
Otherwise, an invalid reference has occurred and the kernel indicates a memory reference error to the application (block 416). This may cause the application to terminate.
Systems and methods for managing virtual address translation using scalable resources for an emulated remote translation table have been disclosed. The systems and methods described provide advantages over previous systems. One advantage is that because kernel memory management routines process the translation immediately upon a TLB miss, the application instruction causing the miss may be precisely determined and appropriate error processing may take place. This is unlike the RTT, where numerous instructions may take place between the time translation is requested and the time that an invalid reference is determined, thereby rendering the kernel unable to determine which instruction caused the invalid reference.
In addition, in some embodiments, the kernel always uses the ERTT for memory references. In some embodiments, entry into kernel mode turns off use of the RTT and turns on use of the ERTT for virtual to physical address translation. As a result, the kernel is typically protected from invalid references that would otherwise cause an undesirable kernel crash (also referred to as a kernel panic).
Furthermore, in some embodiments, applications may choose between using the ERTT or the RTT. If an application chooses to use the ERTT, then the system may assign the application to run on non-contiguous physical nodes that appear to the application to be contiguous through the use of virtual nodes, thereby increasing the flexibility and potentially improving the resource utilization of the nodes on the system. In addition, the use of virtual nodes does not require any changes or recompiles of the underlying application software in order to take advantage of the increased flexibility provided by the ERTT.
Although specific embodiments have been illustrated and described herein, it will be appreciated by those of ordinary skill in the art that any arrangement which is calculated to achieve the same purpose may be substituted for the specific embodiments shown. This application is intended to cover any adaptations or variations of the present invention.
The terminology used in this application is meant to include all of these environments. It is to be understood that the above description is intended to be illustrative, and not restrictive. Many other embodiments will be apparent to those of skill in the art upon reviewing the above description. Therefore, it is manifestly intended that this invention be limited only by the following claims and equivalents thereof.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US5430850||Jul 22, 1991||Jul 4, 1995||Massachusetts Institute Of Technology||Data processing system with synchronization coprocessor for multiple threads|
|US5446915||May 25, 1993||Aug 29, 1995||Intel Corporation||Parallel processing system virtual connection method and apparatus with protection and flow control|
|US5560029||Jul 21, 1992||Sep 24, 1996||Massachusetts Institute Of Technology||Data processing system with synchronization coprocessor for multiple threads|
|US5649141||Jun 30, 1995||Jul 15, 1997||Nec Corporation||Multiprocessor system for locally managing address translation table|
|US5860146||Jun 25, 1996||Jan 12, 1999||Sun Microsystems, Inc.||Auxiliary translation lookaside buffer for assisting in accessing data in remote address spaces|
|US5897664||Jul 1, 1996||Apr 27, 1999||Sun Microsystems, Inc.||Multiprocessor system having mapping table in each node to map global physical addresses to local physical addresses of page copies|
|US6003123||Feb 10, 1998||Dec 14, 1999||Massachusetts Institute Of Technology||Memory system with global address translation|
|US6101590 *||Oct 10, 1995||Aug 8, 2000||Micro Unity Systems Engineering, Inc.||Virtual memory system with local and global virtual address translation|
|US6105113 *||Aug 21, 1997||Aug 15, 2000||Silicon Graphics, Inc.||System and method for maintaining translation look-aside buffer (TLB) consistency|
|US6490671 *||May 28, 1999||Dec 3, 2002||Oracle Corporation||System for efficiently maintaining translation lockaside buffer consistency in a multi-threaded, multi-processor virtual memory system|
|US6684305 *||Apr 24, 2001||Jan 27, 2004||Advanced Micro Devices, Inc.||Multiprocessor system implementing virtual memory using a shared memory, and a page replacement method for maintaining paged memory coherence|
|US6922766||Sep 4, 2002||Jul 26, 2005||Cray Inc.||Remote translation mechanism for a multi-node system|
|US6925547 *||Dec 14, 2001||Aug 2, 2005||Silicon Graphics, Inc.||Remote address translation in a multiprocessor system|
|US20020169938||Dec 14, 2001||Nov 14, 2002||Scott Steven L.||Remote address translation in a multiprocessor system|
|US20020172199||Dec 14, 2001||Nov 21, 2002||Scott Steven L.||Node translation and protection in a clustered multiprocessor system|
|US20040044872 *||Sep 4, 2002||Mar 4, 2004||Cray Inc.||Remote translation mechanism for a multi-node system|
|1||"U.S. Appl. No. 10/235,898 Notice of Allowance mailed Mar. 15, 2005", 4 pgs.|
|2||"U.S. Appl. No. 10/235,898, Non Final Office Action mailed Jul. 7, 2004", 10 pgs.|
|3||"U.S. Appl. No. 10/235,898, Response filed Jan. 6, 2005 to Non Final Office Action mailed Jul. 7, 2004", 16 pgs.|
|4||Chen, Y. , et al., "UTLB: A Mechanism for Address Translation on Network Interfaces", Proceedings of the Eighth International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS), (1998),193-204.|
|5||Kontothanassis, L. , et al., "VM-based shared memory on low-latency, remote-memory-access networks", Proceedings of the ACM ISCA '97, (1997),157-169.|
|6||Scott, S. , "Synchronization and Communication in the T3E Multiprocessor", ASPLOS, vol. III, (1996),pp. 26-36.|
|7||Wood, D. A., et al., "An In-Cache Address Translation Mechanism", Proceedings of the 13th Annual International Symposium on Computer Architecture, (1986),358-365.|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US8095771||Apr 7, 2008||Jan 10, 2012||Microsoft Corporation||Method and system for caching address translations from multiple address spaces in virtual machines|
|US8909946||May 18, 2006||Dec 9, 2014||Microsoft Corporation||Efficient power management of a system with virtual machines|
|US9218047||Dec 8, 2014||Dec 22, 2015||Microsoft Technology Licensing, Llc||Efficient power management of a system with virtual machines|
|US9489035||Dec 15, 2015||Nov 8, 2016||Microsoft Technology Licensing, Llc||Efficient power management of a system with virtual machines|
|US20070112999 *||May 18, 2006||May 17, 2007||Microsoft Corporation||Efficient power management of a system with virtual machines|
|US20080215848 *||Apr 7, 2008||Sep 4, 2008||John Te-Jui Sheu||Method and System For Caching Address Translations From Multiple Address Spaces In Virtual Machines|
|US20160034392 *||Mar 28, 2013||Feb 4, 2016||Hewlett-Packard Development Company, L.P.||Shared memory system|
|U.S. Classification||711/206, 711/147|
|International Classification||G06F12/08, G06F12/00|
|Cooperative Classification||G06F9/3836, G06F9/3834, G06F9/384|
|May 27, 2004||AS||Assignment|
Owner name: CRAY INC., WASHINGTON
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:SHEETS, KITRICK;REEL/FRAME:014673/0127
Effective date: 20040510
|Jan 12, 2010||CC||Certificate of correction|
|Aug 21, 2012||FPAY||Fee payment|
Year of fee payment: 4
|Oct 11, 2016||FPAY||Fee payment|
Year of fee payment: 8