|Publication number||US7500049 B2|
|Application number||US 11/263,628|
|Publication date||Mar 3, 2009|
|Filing date||Oct 31, 2005|
|Priority date||Oct 31, 2005|
|Also published as||CN101371232A, CN101371232B, DE112006002582T5, US20070101076, WO2007053668A2, WO2007053668A3|
|Publication number||11263628, 263628, US 7500049 B2, US 7500049B2, US-B2-7500049, US7500049 B2, US7500049B2|
|Inventors||Martin Dixon, Michael Cornaby, Michael Fetterman, Per Hammarlund|
|Original Assignee||Intel Corporation|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (13), Non-Patent Citations (5), Referenced by (8), Classifications (10), Legal Events (3)|
|External Links: USPTO, USPTO Assignment, Espacenet|
Embodiments of the present invention relate to data processing in a processor-based system, and more particularly to executing processor operations transparently to an operating system (OS).
Systems are typically formed of hardware and software components. Typical hardware includes a processor and related circuitry, including chipsets, memories, input/output (I/O) devices and the like. Software components typically include an OS and basic input/output system (BIOS) programs, low-level drivers, and higher-level applications such as user-level applications for performing desired tasks, such as word processing, database processing, scientific computing and the like.
Generally, the OS is the primary scheduler of activities on the system and is fully aware of various processes being executed on the processor. As a result, when additional features or extensions are added to hardware such as the processor, OS support is required in the way of drivers or other software so that the OS can monitor execution of the enhanced hardware. When additional processor features or extensions are to be visible to each process being executed on the processor, the OS may choose to virtualize the feature or extension, such that each process perceives that it has its own private access or copy of the feature or extension.
Upon initiation of a process, the OS provides a process control block (PCB), which is a structure to represent the process in a privileged level of memory (i.e., inaccessible to user-level applications). The PCB may include various information regarding the executing process, such as identification information, state information, register values, memory information, and other such information. Providing such information and maintaining coherency between this information in the processor and the process control block maintained by the OS is a cumbersome and performance sensitive activity.
When certain instructions are added to an instruction set architecture (ISA) additional, extended state may be available in a processor. Further when enhancements to hardware (e.g., processor hardware such as registers and the like) are incorporated, OS support is needed. This support may be in the form of drivers for current OS, or new OS service packs, and future OS versions may include additional code to support the enhancements. Also, additional storage space in a PCB or other OS data structure is needed whenever a new feature is added to a processor.
These extensions can also cause a performance impact on various activities, such as a context switch between two processes. If the additional state due to these extensions is unlikely to be used frequently by most processes, the OS may implement so-called lazy save and restore mechanisms which can be used to postpone and sometimes eliminate the context switches of the additional state due to these extensions, thus saving time. However, in a multiprocessor system, such as a symmetric multiprocessor (SMP) system, these mechanisms are more cumbersome, and typically an OS will instead perform a full state save upon a context switch, which can be a relatively expensive process. Such saves both implicate the OS and are inefficient.
Accordingly, a need exists for an improved manner of implementing enhancements to hardware such as processor extensions.
In various embodiments, application memory (i.e., user-level accessible memory) may be used as storage for some of the per-process or per-thread state information of a processor, rather than requiring the operating system (OS) to provide storage in its PCB or other OS structures. The state information may correspond to architectural state information such as control and configuration registers, physical registers, other control structures, and the like. In some implementations, at least some of these registers may be vector-sized registers substantially wider than scalar registers of the processor. In this manner, an OS may be unaware of additional state added to the processor. Furthermore, the additional state can be used by separate applications without any interaction or coordination between the applications. This in turn enables the use of new processor extensions such as new modes of operation, instruction extensions and the like, on existing operating systems.
Using embodiments of the present invention, processor state may be saved and restored without OS involvement. The state may include N bytes of state, where N can be determined by a given processor feature to be implemented. Furthermore, the mechanisms described herein may be operating system agnostic and transparent, and may perform state save and restore operations with higher performance than if the OS performed full saves and restores on each context switch. To that end, the mechanisms may have minimal overhead to set up and use, and may be used by multiple processes without need for the OS to context switch this managed state. Accordingly, new registers and features may be implemented in a processor without additional storage requirements in an OS process control block. In other words, processor extensions, which may include additional hardware, state, and/or instructions (collectively herein “extensions”) to perform new features or functions, may be supported. Furthermore, such support may be provided without OS support. That is, such extensions can be included in a processor and proper execution can occur transparently to the OS. Additionally, processor state need not be saved immediately upon the transition of control to another application. Accordingly, the performance overhead of maintaining state can be minimized, and only context switches which are absolutely necessary will occur. If the nature of the extension is such that it is reasonably unlikely that multiple processes will be simultaneously using the extension, then context switches for this new state are generally avoided. This is especially appropriate, if much extended state is not used, or not used frequently, by most applications.
In this manner, extended (e.g., processor) resources may be virtualized on a per-thread or per-process basis, requiring minimal support from user code, while being entirely OS transparent. In this manner, multiple threads and/or processes may use the extended resources efficiently, with minimal overhead on context changes.
Generally, registers which are available to processes within an OS are explicitly saved and restored by the OS during a context switch. Typically, the OS knows where the backing storage for a process's registers is located, but the hardware does not. Here, however, the processor knows the location of the backing store, and in some embodiments, the OS may not. Instead of explicit saves and restores during a context switch, the hardware maintains one or more bits of state which indicate where the currently authoritative copy of the new extended state resides, either in the hardware's registers, or in the backing store. In some embodiments, only one state bit is used to represent all of the extended state, which is saved and restored as needed by the hardware. In other embodiments, the new extended state may be broken up into smaller pieces, each of which has such a state bit. The hardware can dynamically detect the need to either save or restore a given piece of extended state, i.e., based on the state corresponding to one of these state bits, and transparently perform the save or restore as necessary, and update the state bit accordingly.
When the backing store is the authoritative copy, and the user attempts to access some of the extended state, the hardware transparently restores the state into the processor's registers before continuing. This action might provoke a page fault, as the backing store is not necessarily present. Thus, all new instructions that access such extended state are specified to allow for such page faults.
When the user attempts to modify some of the extended state, the hardware first verifies that the backing store is currently present, writable, and marked as dirty. If it is not, then the appropriate page fault is signaled. Thus, all new instructions that can modify such extended state are specified to allow for such page faults. When such verification occurs, the processor may cache a positive result until certain events occur, thus eliminating the need for frequent verifications. As a side effect of such verification, the processor discovers a physical translation of the virtual address the user specified for the backing store; this physical translation may be retained by the hardware, and used as described below.
In some embodiments, when the user attempts to modify some of the extended state, the processor asserts ownership of the corresponding cache lines in the backing store. The method of doing so varies, depending on the nature of the memory system involved.
In some embodiments, one or more dirty bits are associated with pieces of the extended state. Such dirty bits are cleared when values are restored from the backing store, and when values are saved back to the backing store. Such dirty bits are set whenever instructions are executed which change the value of the corresponding extended register(s).
Whenever the authoritative copy of extended state resides in the processor's register(s), and it is dirty, there is always a valid physical translation of the current backing store stored in the processor. The processor may respond to memory transactions targeted at these addresses. The processor may respond to a memory read request with the values stored in the registers. Thus, the registers are essentially acting as a specialized cache, caching only the memory that is being used as a backing store. Therefore, for each cache line that is included in the backing store, the processor maintains the per cache line state required of whatever memory system exists in the system.
The location of the backing store may potentially be changed by several events. In some embodiments, the user may explicitly change the location. In some embodiments, the OS may be aware of the backing store pointer, and may change it as part of a context switch. As the backing store pointer can be a virtual address, every change of address space may potentially change the physical translation of the backing store. Immediately before any of these events, the authoritative copy of the extended state may exist either in backing store, or in the processor's registers. If the authoritative copy is in the backing store, then no further action is required when the pointer is changed. If the authoritative copy is in the processor's registers, and it is not marked as dirty, then again, no further action is required. If the authoritative copy is in the processor's registers, and it is marked as dirty, some embodiments may immediately save the dirty state out to the backing store. Other embodiments may provide for a means to postpone this state save, and potentially eliminate it. As one example, an implementation might choose to allow the virtual address pointing to the backing store to be updated, but indicate that the physical translation is no longer in synchronization with this new pointer. As long as no attempts are made to access the new extended state, the old physical translation and the extended registers themselves continue to act as a “normal” cache for the cache lines they map, responding to any snoop traffic, and the like. These registers or cache lines can be invalidated by normal cache traffic if another agent asserts ownership. When some subsequent attempt is made to access the extended state, the current backing store pointer is translated (as indicated above), and then compared with the old physical translation. If it is the same, and the extended state cache tags are still valid, then no state save or restore is required; they are completely eliminated, and the bits are set to indicate that the authoritative copy is located in the registers. However, if the translations do not match, the “cache lines” which are still valid are flushed to memory before the new translation is used to fetch the corresponding register values from the new backing store.
As an example implementation, register state information used by a processor may have a backing store located at a user-specified location in user-level paged memory. During process initialization, a backing store area may be requested, e.g., from the OS as an allocated block in memory. While this area is requested from the OS, the OS is unaware of the use of the area as a backing store. The user specifies the address of this storage by setting a register to point to it. In some embodiments, this register is saved and restored by the OS as part of a normal context switch; other embodiments exist which remove this requirement.
Thus, there are two locations at which copies of the extended state can exist: in the extended registers themselves, and (assuming the user has provided a pointer) in the backing store. Whenever the pointer to the backing store is set, the authoritative copy of the state is understood to be in the backing store. Any attempt to use or access the new extended state will cause it to be transparently loaded into the register.
Before each use of the extended state, the processor checks that the pages of memory are present and writeable and marked as dirty, as seen by the OS. If user memory is accessed via a paging system, the processor remembers the translation of the user pages required to hold the new state. This can be viewed as a special purpose translation lookaside buffer (TLB). Once this check is done, and until the processor would normally be required to drop or flush such a TLB entry, the processor can allow unconstrained access to the new state and new features. In some embodiments, the processor may allow its new/extended state to be modified without regard for maintaining consistency with this backing store. By thus relaxing coherency requirements, certain benefits may be realized in a given implementation. In some embodiments, the processor may keep both the pointer to the backing store (a linear or virtual address) and the translated physical address in a structure similar to a TLB. On a current protection level (CPL) change however, the processor may respond to snoops on this area with appropriate responses depending upon the status of the state information (e.g., dirty or clean). On a CPL change in an Intel Architecture (IA-32) environment, the OS may invalidate write permissions to a given page (i.e., storing a backing store) and thus on the resumption from a more privileged state to a less privileged state (e.g., from the OS to the application), the processor once again checks that the pages of memory are present and writeable.
Referring now to
While not shown in
Referring back to
Still referring to
However when a privilege level changes, e.g., when control passes from an application program back to the OS, consistency is maintained between the processor state information in the processor and the backing store (block 60). That is when modifications are made to the processor state information in a different privilege level, the dirty data may be written back to memory, and more specifically to the backing store, to maintain consistency. In different embodiments, various manners of maintaining consistency may be implemented.
When a privilege level changes from a higher privilege level to a lower privilege level, e.g., when control passes from the OS back to the application, the presence of the backing store in memory must be verified. That is, the memory page or pages on which the application has its backing store must be present in memory and writable by the application. Furthermore, in certain implementations it may be necessary to test that the backing store is setup and initialized prior to the usage of any extensions or instructions that utilize the new state.
In various implementations, when a process is initiated that is to use certain processor extensions or instructions, and more particularly where such extensions or instructions are not supported by a given OS, a user-level backing store may be set up and used to implement such extensions or instructions without involving the OS. Referring now to
Still referring to
Next, the processor may read the state present in the backing store and load it into the processor (block 135). Accordingly, desired state information is now loaded in the processor and further execution of the application or process is possible. Thus execution of the process may continue. During execution, it may be determined whether there is a change in a privilege level, e.g., a current protection level (CPL) (diamond 140). If not, any changes to processor state information occurring as a result of execution of the current process or any other process that is of the same privilege level are not written back to memory. In other words, no consistency is maintained between the processor state information in the processor and the backing store (block 145).
If instead at diamond 140 it is determined that a privilege level has changed, control may pass to diamond 150. There, during execution of the new process of a different privilege level, the processor may determine whether it receives a snoop request for one or more processor state locations (diamond 150). If not, continued execution of the new process occurs in a loop with diamond 150. However, if a snoop request is received for a location of the processor state information, next it may be determined whether the hit is for a dirty location (diamond 155). If the snooped location is not dirty (i.e., clean), meaning that the state of the register and the state of the memory are known to be coherent, a hit response may be sent (block 160), and further execution of the new process may continue via a loop back to diamond 150.
If instead at diamond 155 the snoop request hits a dirty location, meaning that the state of the register and the state of the memory are not known to be coherent, control passes to block 170. There, a snoop response is sent with the dirty data (block 170). In some implementations, the dirty location also may be written back to memory, and more specifically to the backing store. In some embodiments, data may be supplied to the backing store (i.e., memory) via either microcode or a hardware mechanism. In such manner, on a context switch consistency between the processor state information and the backing store may be maintained. Accordingly, the processor state information storage in the processor acts as a cache memory. To that end, in various implementations a processor may include an additional snoop filter (if a snoop filter is already present) to handle incoming snoop requests for these processor state locations. Furthermore, the processor state locations may be extended with one or more indicators to indicate the presence of valid and/or dirty data, although the scope of the present invention is not so limited.
Using embodiments of the present invention, backing store data may be swapped out to a lower level of a memory hierarchy (e.g., a disk) on a task switch from a process using the backing store to a different process scheduled by an OS. Referring now to
Still referring to
Based on the snoop results, it may be determined whether a hit occurs to a location in the processor (diamond 230). For example, a specialized snoop filter within the processor may be used to perform snoops on these state storage locations in the processor. As described above, such locations may be extended with one or more indicators to indicate their status (i.e., valid, dirty, or the like). If there is no hit at diamond 230, a miss occurs and control may pass back to the DMA agent, as will be discussed further below. Next, if a hit has occurred it may be determined whether the hit corresponds to a dirty location (diamond 235). If the location corresponding to the hit is not dirty, a hit response is sent, and control may pass back to the DMA agent. If instead at diamond 235 it is determined that a hit occurs to a dirty location, the first processor may move the dirty data from state storage in the processor (e.g., in a register file) to the backing store in memory (block 240).
Furthermore, in some implementations the processor may send one or more responses to the DMA agent (e.g., hit modified (HITM) responses) that identify the modified location memory, as well as provide the dirty data to the DMA agent. In either event, when the dirty data is available to the DMA agent (either directly or via the backing store), the DMA agent may complete writing the paged memory to disk (block 270). Block 270 is also performed when the backing store locations miss the snoop filter (or are not dirty), receiving control from diamonds 230 and 235 discussed above. While described with this particular implementation in the embodiment of
In some implementations, a process executing on a first processor may be migrated to a second processor, for example, as directed by an OS. If the process implicates extended processor resources and use of a backing store, the backing store may be restored to the second processor for proper operation on the second processor. Referring now to
Thus at block 320, the OS schedules a process B that does not use the extended registers. Furthermore, the OS schedules migration of process A to a second processor. Accordingly, as shown in
Then during execution of process A on the second processor, when extended registers are used, one or more RFO's may be issued by the second processor (block 365). That is, during execution of process B, register state information may be fetched in on demand access to the registers via a cache coherency mechanism, for example. Accordingly, the second processor may send one or more requests for ownership (RFO) to the first processor. Accordingly, the first processor may snoop the processor state locations within the processor during the course of executing its other (i.e, new process) (block 325).
Based on the snoop results, it may be determined whether a hit occurs to a location in the processor (diamond 330). If not, a miss may be sent to the second processor, and control in the second processor passes to block 370, discussed below. If instead a hit occurs at diamond 330, next it may be determined whether the hit corresponds to a dirty location in the processor (diamond 335). For example, as described above a specialized snoop filter within the processor may be used to snoop the processor state storage locations. If the location is clean, an appropriate response is sent to the second processor, and control in the second processor passes to block 370. If instead at diamond 335 it is determined that a hit occurs to a dirty location, the first processor may move the dirty data from state storage (e.g., a register file) in the first processor to the backing store in memory (block 345). Further, the first processor may send one or more responses to the second processor (e.g., hit modified (HITM) responses) that identify the modified storage location, and in some implementations may also provide the dirty data to the requesting agent (i.e., the second processor).
After handling the snoop responses, the first processor may continue execution of process B (block 350). Furthermore, the second processor may continue execution of process A (block 370). Accordingly, process A thus completes on the second processor (block 375).
In some implementations, a process may be executed using multiple threads or sub-processes. To enable use of extended register state information in such multiple threads or sub-processes, one or more additional storage spaces in application memory may be provided. That is, a sub-process of a currently running process may request a separate backing store from the OS to enable replication of the backing store information, thus allowing both processes to continue unimpeded. In such manner, both threads may have a private set of register state to avoid a fault condition.
Embodiments may be implemented in many different processor architectures. Referring now to
As shown in
Still referring to
As further shown in
When all needed data for a μop is present in register file 430, the μop may be executed via one of execution units 440. In various implementations different execution units may be present. For example, integer, floating point, address generation, single instruction multiple data (SIMD), and store data (STD) units may be present, although the scope of the present invention is not so limited. After execution, result data may be provided back to register file 430 for storage until the instruction retires. Then, the result data may be written back to a desired location (e.g., of a memory hierarchy).
While shown with this implementation in the embodiment of
As further shown in
In such manner, control registers 450 may provide architectural state information that is used to handle processor extensions without the need for OS support. One representative register 455 is shown in
If a hit occurs within directory 465, snoop filter 460 may snoop the physical location in register file 430 or control registers 450 to determine a state of the location. If the location is dirty, snoop filter 460 may so indicate to the requesting agent and further provide the dirty data to both the requesting agent and to the backing store in memory. In such manner, snoop filter 460 may maintain coherency between the contents of these locations and a requesting agent, such as a DMA agent or another processor of a multi-processor system. While shown with this particular implementation in the embodiment of
Referring now to
As shown in
Note that in various embodiments, different granularities of the various state bits may be provided. For example as shown in
Still referring to
While shown with this particular implementation in the embodiment of
Accordingly, in various embodiments space may be allocated for instruction set additions and extensions to a processor architecture without a need for OS support. Embodiments may be implemented in various processor architectures, including for example, chip multiprocessors (CMPs), small core arrays, other multicore processors, coprocessors or other such systems.
Thus embodiments may be implemented in many different system types. Referring now to
First processor 570 and second processor 580 may be coupled to a chipset 590 via P-P interfaces 552 and 554, respectively. As shown in
In turn, chipset 590 may be coupled to a first bus 516 via an interface 596. In one embodiment, first bus 516 may be a Peripheral Component Interconnect (PCI) bus, as defined by the PCI Local Bus Specification, Production Version, Revision 2.1, dated June 1995 or a bus such as a PCI Express bus or another third generation I/O interconnect bus, although the scope of the present invention is not so limited.
As shown in
Embodiments may be implemented in code and may be stored on a storage medium having stored thereon instructions which can be used to program a system to perform the instructions. The storage medium may include, but is not limited to, any type of disk including floppy disks, optical disks, compact disk read-only memories (CD-ROMs), compact disk rewritables (CD-RWs), and magneto-optical disks, semiconductor devices such as read-only memories (ROMs), random access memories (RAMs) such as dynamic random access memories (DRAMs), static random access memories (SRAMs), erasable programmable read-only memories (EPROMs), flash memories, electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, or any other type of media suitable for storing electronic instructions.
While the present invention has been described with respect to a limited number of embodiments, those skilled in the art will appreciate numerous modifications and variations therefrom. It is intended that the appended claims cover all such modifications and variations as fall within the true spirit and scope of this present invention.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US5428779 *||Nov 9, 1992||Jun 27, 1995||Seiko Epson Corporation||System and method for supporting context switching within a multiprocessor system having functional blocks that generate state programs with coded register load instructions|
|US5768557||Jun 13, 1996||Jun 16, 1998||Intel Corporation||Low cost writethrough cache coherency apparatus and method for computer systems without a cache suppporting bus|
|US6075938||Jun 10, 1998||Jun 13, 2000||The Board Of Trustees Of The Leland Stanford Junior University||Virtual machine monitors for scalable multiprocessors|
|US6145049 *||Dec 29, 1997||Nov 7, 2000||Stmicroelectronics, Inc.||Method and apparatus for providing fast switching between floating point and multimedia instructions using any combination of a first register file set and a second register file set|
|US6434677||Jun 1, 1999||Aug 13, 2002||Intel Corporation||Method and apparatus for altering data length to zero to maintain cache coherency|
|US6434683 *||Nov 7, 2000||Aug 13, 2002||Storage Technology Corporation||Method and system for transferring delta difference data to a storage device|
|US6456891 *||Oct 27, 1999||Sep 24, 2002||Advanced Micro Devices, Inc.||System and method for transparent handling of extended register states|
|US6487630||Feb 26, 1999||Nov 26, 2002||Intel Corporation||Processor with register stack engine that dynamically spills/fills physical registers to backing store|
|US6629157||Jan 4, 2000||Sep 30, 2003||National Semiconductor Corporation||System and method for virtualizing the configuration space of PCI devices in a processing system|
|US6651163 *||Mar 8, 2000||Nov 18, 2003||Advanced Micro Devices, Inc.||Exception handling with reduced overhead in a multithreaded multiprocessing system|
|US6671762 *||Dec 29, 1997||Dec 30, 2003||Stmicroelectronics, Inc.||System and method of saving and restoring registers in a data processing system|
|US20020056024 *||Feb 26, 1999||May 9, 2002||Tuan H. Bui||Processor with register stack engine that dynamically spills/fills physical registers to backing store|
|US20050120160||Oct 25, 2004||Jun 2, 2005||Jerry Plouffe||System and method for managing virtual servers|
|1||"Itanium Processor Family Performance Advantages: Register Stack Architecture" by Scott Townsend http://www.intel.com/cd/ids/developer/asmo-na/eng/20314.htm?prn=Y Date Unknown.|
|2||"Performance Advantages of the Register Stack in Intel Itanium Processors" Rakvic, et al. http://systems.cs.colorado.edu/EPIC2/papers/s2-1-rakvic.pdf 2nd Workshop on Explicitly Parallel Instruction Computing Architecture and Compilers, Monday, Nov. 18, 2002. Istanbul, Turkey.|
|3||Boggs, D.; Baktha, A.; Hawkins, J; Marr, D.T.; Miller, J.A.; Roussel, P.; Singhal, R.; Toll, B; Venkatraman, K.S. "The Microarchitecture of the Intel(R) Pentium(R) 4 Processor on 90nm Technology." Intel Technology Journal. http://developer.intel.com/technology/itj/2004/volume08issue01/ (Feb. 2004).|
|4||GPTO official communication dated Oct. 13, 2008 in related foreign patent application, with English language translation, pp. 1-16.|
|5||PCT/US2006/042619 International Search Report with Written Opinion of the International Searching Authority Mailed Apr. 4, 2008.|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US7899904 *||Apr 30, 2008||Mar 1, 2011||Lsi Corporation||Hardware processing of regular expressions|
|US7996663||Dec 27, 2007||Aug 9, 2011||Intel Corporation||Saving and restoring architectural state for processor cores|
|US8086801 *||Apr 8, 2009||Dec 27, 2011||International Business Machines Corporation||Loading data to vector renamed register from across multiple cache lines|
|US8234407 *||Jun 30, 2009||Jul 31, 2012||Oracle America, Inc.||Network use of virtual addresses without pinning or registration|
|US20080270342 *||Apr 30, 2008||Oct 30, 2008||Tarari, Inc.||Hardware processing of regular expressions|
|US20090172369 *||Dec 27, 2007||Jul 2, 2009||Stillwell Jr Paul M||Saving and restoring architectural state for processor cores|
|US20100262781 *||Apr 8, 2009||Oct 14, 2010||International Business Machines Corporation||Loading Data to Vector Renamed Register From Across Multiple Cache Lines|
|US20100332789 *||Jun 30, 2009||Dec 30, 2010||Sugumar Rabin A||Network Use of Virtual Addresses Without Pinning or Registration|
|U.S. Classification||711/101, 712/225, 711/122|
|Cooperative Classification||G06F11/2033, G06F11/1438, G06F9/463|
|European Classification||G06F9/46G4, G06F11/14A8L, G06F11/20P2S|
|Jan 17, 2006||AS||Assignment|
Owner name: INTEL CORPORATION, CALIFORNIA
Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:DIXON, MARTIN;CORNABY, MICHAEL;FETTERMAN, MICHAEL;AND OTHERS;REEL/FRAME:017464/0417;SIGNING DATES FROM 20051025 TO 20051117
|Sep 3, 2012||FPAY||Fee payment|
Year of fee payment: 4
|Aug 18, 2016||FPAY||Fee payment|
Year of fee payment: 8