US 7406048 B2
Methods, configured storage media, and systems are provided for communications using two or more disparate networks in parallel to provide load balancing across network connections, greater reliability, and/or increased security. A controller provides access to two or more disparate networks in parallel, through direct or indirect network interfaces. When one attached network fails, the failure is sensed by the controller and traffic is routed through one or more other disparate networks. When all attached disparate networks are operating, one controller preferably balances the load between them.
1. A controller which controls access to multiple independent disparate networks in a parallel network configuration, the disparate networks comprising at least one private network and at least one network based on the Internet, the controller comprising:
a site interface connecting the controller to a site;
at least two network interfaces which send packets toward the disparate networks; and
a packet path selector which selects between network interfaces, using at least two known location address ranges which are respectively associated with disparate networks, according to at least: a destination of the packet, an optional presence of alternate paths to that destination, and at least one specified criterion for selecting between alternate paths when such alternate paths are present;
wherein the controller receives a packet through the site interface and sends the packet through the network interface that was selected by the packet path selector.
2. The controller of
3. The controller of
4. The controller of
5. The controller of
6. The controller of
7. A method for combining connections for access to disparate parallel networks, the method comprising the steps of:
receiving at a controller a packet which has a first site IP address as source address and a second site IP address as destination address;
selecting, within the controller on a per-packet basis, between a path through an Internet-based network and a path through a private network that is not Internet-based; and
forwarding the packet along the selected path toward the second site.
8. The method of
9. The method of
10. The method of
11. The method of
12. The method of
13. A method for controlling access to multiple independent disparate networks in a parallel network configuration, the disparate networks comprising at least one private network and at least one network based on the Internet, the method comprising the steps of:
receiving a packet through a site interface that connects a controller to a site;
selecting between at least two network interfaces of the controller which use at least two known location address ranges which are respectively associated with disparate networks, according to at least: a destination of the packet, an optional presence of alternate paths to that destination, and at least one specified criterion for selecting between alternate paths when such alternate paths are present; and
sending the packet through the selected network interface.
14. The method of
15. The method of
16. The method of
17. The method of
18. The method of
19. A controller for combining connections for access to disparate parallel networks, the controller comprising:
a site interface configured for receiving a packet which has a first site IP address as source address and a second site IP address as destination address; and
a packet path selector which selects, within the controller on a per-packet basis, between a path through an Internet-based network and a path through a private network that is not Internet-based;
wherein the controller receives a packet through the site interface and sends the packet through the network interface that was selected by the packet path selector.
20. The controller of
21. The controller of
22. The controller of
23. The controller of
24. The controller of
This application is a continuation of forthcoming U.S. Pat. No. 6,775,235 (application Ser. No. 10/361,837), which claims priority to commonly owned U.S. provisional patent application Ser. No. 60/355,509 filed Feb. 8,2002, which provisional application is also incorporated herein by reference. U.S. Pat. No. 6,775,235 is a continuation-in-part of U.S. patent application Ser. No. 10/034,197 filed Dec. 28, 2001, which claims priority to U.S. provisional patent application Ser. No. 60/259,269 filed Dec. 29, 2000, each of which applications is also incorporated herein by reference.
The present invention relates to computer network data transmission, and more particularly relates to tools and techniques for communications using disparate parallel networks, such as a virtual private network (“VPN”) or the Internet in parallel with a point-to-point, leased line, or frame relay network, in order to help provide benefits such as load balancing across network connections, greater reliability, and increased security.
Organizations have used frame relay networks and point-to-point leased line networks for interconnecting geographically dispersed offices or locations. These networks have been implemented in the past and are currently in use for interoffice communication, data exchange and file sharing. Such networks have advantages, some of which are noted below. But these networks also tend to be expensive, and there are relatively few options for reliability and redundancy. As networked data communication becomes critical to the day-to-day operation and functioning of an organization, the need for lower cost alternatives for redundant back-up for wide area networks becomes important.
Frame relay networking technology offers relatively high throughput and reliability. Data is sent in variable length frames, which are a type of packet. Each frame has an address that the frame relay network uses to determine the frame's destination. The frames travel to their destination through a series of switches in the frame relay network, which is sometimes called a network “cloud”; frame relay is an example of packet-switched networking technology. The transmission lines in the frame relay cloud must be essentially error-free for frame relay to perform well, although error handling by other mechanisms at the data source and destination can compensate to some extent for lower line reliability. Frame relay and/or point-to-point network services are provided or have been provided by various carriers, such as AT&T, Qwest, XO, and MCI WorldCom.
Frame relay networks are an example of a network that is “disparate” from the Internet and from Internet-based virtual private networks for purposes of the present invention. Another example of such a “disparate” network is a point-to-point network, such as a T1 or T3 connection. Although the underlying technologies differ somewhat, for purposes of the present invention frame relay networks and point-to-point networks are generally equivalent in important ways, such as the conventional reliance on manual switchovers when traffic must be redirected after a connection fails, and their implementation distinct from the Internet. A frame relay permanent virtual circuit is a virtual point-to-point connection. Frame relays are used as examples throughout this document, but the teachings will also be understood in the context of point-to-point networks.
A frame relay or point-to-point network may become suddenly unavailable for use. For instance, both MCI WorldCom and AT&T users have lost access to their respective frame relay networks during major outages. During each outage, the entire network failed. Loss of a particular line or node in a network is relatively easy to work around. But loss of an entire network creates much larger problems.
Tools and techniques to permit continued data transmission after loss of an entire frame relay network that would normally carry data are discussed in U.S. patent application Ser. No. 10/034,197 filed Dec. 28, 2001 and incorporated herein. The '197 application focuses on architectures involving two or more “private” networks in parallel, whereas the present application focuses on architectures involving disparate networks in parallel, such as a proprietary frame relay network and the Internet. Note that the term “private network” is used herein in a manner consistent with its use in the '197 application (which comprises frame relay and point-to-point networks), except that a “virtual private network” as discussed herein is not a “private network”. Virtual private networks are Internet-based, and hence disparate from private networks, i.e., from frame relay and point-to-point networks. To reduce the risk of confusion that might arise from misunderstanding “private network” to comprise “virtual private network” herein, virtual private networks will be henceforth referred to as VPNs. Other differences and similarities between the present application and the '197 application will also be apparent to those of skill in the art on reading the two applications.
Various architectures involving multiple networks are known in the art. For instance,
A configuration like that shown in
Alternatively, one of the frame relay networks may be a backup which is used only when the other frame relay network becomes unavailable. In that case, it may take even skilled network administrators several hours to perform the steps needed to switch the traffic away from the failed network and onto the backup network, unless the invention of the '197 application is used. In general, the necessary Private Virtual Circuits (PVCs) must be established, routers at each site 102 must be reconfigured to use the correct serial links and PVCs, and LANs at each site 102 must be reconfigured to point at the correct router as the default gateway.
Although two private networks are shown in
Internet-based communication solutions such as VPNs and Secure Sockets Layer (SSL) offer alternatives to frame relay 106 and point-to-point leased line networks such as those using an ISDN link 204. These Internet-based solutions are advantageous in the flexibility and choice they offer in cost, in service providers, and in vendors. Accordingly, some organizations have a frame relay 106 or leased line connection (a.k.a. point-to-point) for intranet communication and also have a connection for accessing the Internet 500, using an architecture such as that shown in
But better tools and techniques are needed for use in architectures such as that shown in
Organizations are still looking for better ways to use Internet-based redundant connections to backup the primary frame relay networks. Also, organizations wanting to change from frame relay and point-to-point solutions to Internet-based solutions have not had the option of transitioning in a staged manner. They have had to decide instead between the two solutions, and deploy the solution in their entire network communications system in one step. This is a barrier for deployment of Internet-based solutions 500/502, since an existing working network would be replaced by a yet-untested new network. Also, for organizations with several geographically distributed locations a single step conversion is very complex. Some organizations may want a redundant Internet-based backup between a few locations while maintaining the frame relay network for the entire organization.
It would be an advancement in the art to provide new tools and techniques for configuring disparate networks (e.g., frame relay/point-to-point WANs and Internet-based VPNs) in parallel, to obtain benefits such as greater reliability, improved security, and/or load-balancing. Such improvements are disclosed and claimed herein.
The present invention provides tools and techniques for directing packets over multiple parallel disparate networks, based on addresses and other criteria. This helps organizations make better use of frame relay networks and/or point-to-point (e.g., T1, T3, fiber, OCx, Gigabit, wireless, or satellite based) network connections in parallel with VPNs and/or other Internet-based networks. For instance, some embodiments of the invention allow frame relay and VPN wide area networks to co-exist for redundancy as well as for transitioning from frame relay/point-to-point solutions to Internet-based solutions in a staged manner. Some embodiments operate in configurations which communicate data packets over two or more disparate WAN connections, with the data traffic being dynamically load-balanced across the connections, while some embodiments treat one of the WANs as a backup for use mainly in case the primary connection through the other WAN fails.
Other features and advantages of the invention will become more fully apparent through the following description.
To illustrate the manner in which the advantages and features of the invention are obtained, a more particular description of the invention will be given with reference to the attached drawings. These drawings only illustrate selected aspects of the invention and its context. In the drawings:
The present invention relates to methods, systems, and configured storage media for connecting sites over multiple independent parallel disparate networks, such as frame relay networks and/or point-to-point network connections, on the one hand, and VPNs or other Internet-based network connections, on the other hand. “Multiple” networks means two or more such networks. “Independent” means routing information need not be shared between the networks. “Parallel” does not rule out all use of NNIs and serial networks, but it does require that at least two of the networks in the configuration be in parallel at the location where the invention distributes traffic, so that alternate data paths through different networks are present. “Frame relay networks” or “private networks” does not rule out the use of an ISDN link or other backup for a particular frame relay or point-to-point private network, but it does require the presence of multiple such networks;
With continued attention to the illustrative network topology for one embodiment of the invention shown in
Therefore, locations A, B, and C can communicate with each other over the frame relay network 106, and locations A and B (but not C) can also communicate with each other over the VPN connection 604. Communication between locations A and C, and communication between locations B and C, can take place over the frame relay network 106 only. Communication between locations A and B can take place over frame relay network 106. It can also take place over one of the lines 1-and-3 pair, or the lines 2-and-3 pair, but not both at the same time. Traffic can also travel over lines 2 and 4, but without a VPN tunnel. When the source and destination IP address pairs are the same between locations A and B but different types of networks connect those locations, as in
The invention may thus be configured to allow the organization to achieve the following goals, in the context of
To better understand the invention, consider the operation of controller device 602 at location A. The controller 602 examines the IP data traffic meant to go through it and makes determinations and takes steps such as those discussed below.
If the traffic is destined for the Internet 500, send the traffic over the Internet using lines 1 and/or 2. Load balancing decisions that guide the controller 602 in distributing packets between the lines can be based on criteria such as the load of a given network, router, or connection relative to other networks, routers, or connections, to be performed dynamically in response to actual traffic. Load-balancing may be done through a round-robin algorithm which places the next TCP or UDP session on the next available line, or it may involve more complex algorithms that attempt to measure and track the throughput, latency, and/or other performance characteristics of a given link or path element. Load-balancing is preferably done on a per-packet basis for site-to-site data traffic over the Internet or frame relay net, or done on a TCP or UDP session basis for Internet traffic, as opposed to prior approaches that use a per-department and/or per-router basis for dividing traffic. Load-balancing algorithms in general are well understood, although their application in the context of the present invention is believed to be new.
If the traffic is destined for location B, then there are at least three paths from the current location (A) to location B: frame relay line 5, VPN line 1, or Internet line 2. In some embodiments, the invention determines whether the three connections are in load-balance mode or on-failure backup mode or a combination thereof. For a load-balance mode, the controller 602 chooses the communication line based on load-balancing criteria. For backup mode, it chooses the communication line that is either the preferred line or (if the preferred line is down) the currently functional (backup) line.
By contrast with the preceding, if the traffic is destined for location C, then the controller 602 at site A sends the traffic on the frame relay line, line 5.
Now let us look at the operation of the controller device 602 at location B. The device examines the IP data traffic sent to it and makes determinations like the following:
To operate as discussed herein, the invention uses information about the IP address ranges in the locations reside as input data. For instance, a packet destined for the Internet 500 is one whose destination address is not in any of the address ranges of the known locations (e.g., locations A, B, and C in the example of
Address ranges can be specified and tested by the controller 602 using subnet masks. The subnet masks may be of different lengths (contain a different number of one bits) in different embodiments and/or in different address ranges in a given embodiment. For instance, class B and class C addresses may both be used in some embodiments.
As another example, consider the illustrative network topology shown in
Suppose that the address ranges used by the routers in the
Without the invention, a topology like
Traditionally, such necessary match-ups of packets with routers were done by inflexible approaches such as sending all traffic from a given department, building, or local area network to a specified router. Manual and/or tedious reconfiguration was needed to change the destination address used in packets from a given source LAN such as one at site A, so this approach allowed load-balancing only on a very broad granularity, and did not load-balance dynamically in response to actual traffic. In particular, difficult reconfiguration of network parameters was needed to redirect packets to another router when the specified router went down.
By placing inventive modules 602 between locations and their routers as illustrated in
Unlike the configuration shown in
Another difference between the inventive approach and prior approaches is the narrow focus of some prior art on reliability. The present document takes a broader view, which considers load balancing and security as well as reliability. Configurations like those shown in
In some embodiments, a network at a location T is connected to a controller 602 for a location R but is not necessarily connected to the controller 602 at another location S. In such cases, a packet from location T addressed to location S can be sent over the network to the controller at location S, which can then redirect the packet to location T by sending it over one or more parallel disparate networks. That is, controllers 602 are preferably, but not necessarily, provided at every location that can send packets over the parallel independent networks of the system.
In some embodiments, the controller 602 at the receiving end of the network connection between two sites A and B has the ability to re-sequence the packets. This means that if the lines are of dissimilar speeds or if out-of-sequence transmission is required by security criteria, the system can send packets out of order and re-sequence them at the other end. Packets may be sent out of sequence to enhance security, to facilitate load-balancing, or both. The TCP/IP packet format includes space for a sequence number, which can be used to determine proper packet sequence at the receiving end (the embodiments are dual-ended, with a controller 602 at the sending end and another controller 602 at the receiving end). The sequence number (and possibly more of the packet as well) can be encrypted at the sending end and then decrypted at the receiving end, for enhanced security. Alternately, an unused field in the TCP/IP header can hold alternate sequence numbers to define the proper packet sequence.
In the operation of some embodiments, the controller 602 on each location is provided with a configuration file or other data structure containing a list of all the LAN IP addresses of the controllers 602 at the locations, and their subnet masks. Each controller 602 keeps track of available and active connections to the remote sites 102. If any of the routes are unavailable, the controller 602 preferably detects and identifies them. When a controller 602 receives IP traffic to any of the distant networks, the data is sent on the active connection to that destination. If all connections are active and available, the data load is preferably balanced across all the routers. If any of the connections are unavailable, or any of the routers are down, the traffic is not forwarded to that router; when the routes become available again, the load balancing across all active routes preferably resumes.
In some embodiments, load balancing is not the only factor considered (or is not a factor considered) when the controller 602 determines which router should receive a given packet. Security may be enhanced by sending packets of a given message over two or more disparate networks. Even if a packet sniffer or other eavesdropping tool is used to illicitly obtain data packets from a given network, the eavesdropper will thus obtain at most an incomplete copy of the message because the rest of the message traveled over a different network. Security can be further enhanced by sending packets out of sequence, particularly if the sequence numbers are encrypted.
The controller 602 also includes a packet path selector 704, which may be implemented in custom hardware, or implemented as software configuring semi-custom or general-purpose hardware. The path selector 704 determines which path to send a given packet on. In the configuration of
The controller 602 also includes two or more disparate network interfaces 706, namely, there is at least one interface 706 per network to which the controller 602 controls access. Each interface 706 can be implemented as a direct interface 706 or as an indirect interface 706; a given embodiment may comprise only direct interfaces 706, may comprise only indirect interfaces 706, or may comprise at least one of each type of interface.
An indirect interface 706 may be implemented, for instance, as a direct frame relay connection over land line or wireless or network interfaces to which the frame relay routers can connect, or as a point-to-point interface to a dedicated T1, T3, or wireless connection. One suitable implementation includes multiple standard Ethernet cards, in the controller 602 and in the router, which connect to each other. An external frame relay User-Network Interface (UNI) resides in a router 105 of a network 106; a similar Ethernet card resides in the Internet router 104. Each such Ethernet card will then have a specific IP address assigned to it. The controller can also have a single Ethernet card with multiple IP addresses associated with different routers and LANs. An indirect interface 706 may connect to the network over fiber optic, T1, wireless, or other links.
A direct interface 706 comprises a standard connection to the Internet 500, while another direct interface 706 comprises a standard connection to a VPN. One direct interface 706 effectively makes part of the controller 602 into a UNI by including in the interface 706 the same kind of special purpose hardware and software that is found on the frame relay network side (as opposed to the UNI side) of a frame relay network router. Such a direct frame relay network interface 706 is tailored to the specific timing and other requirements of the frame relay network to which the direct interface 706 connects. For instance, one direct interface 706 may be tailored to a Qwest frame relay network 106, while another direct interface 706 in the same controller 602 is tailored to a UUNet network 106. Another direct interface 706 comprises standard VPN components.
An indirect interface 706 relies on special purpose hardware and connectivity/driver software in a router or other device, to which the indirect interface 706 of the controller 602 connects. By contrast, a direct interface 706 includes such special purpose hardware and connectivity/driver software inside the controller 602 itself. In either case, the controller provides packet switching capabilities for at least redundancy without manual switchover, and preferably for dynamic load-balancing between lines as well.
An understanding of methods of the invention will follow from understanding the invention's devices, and vice versa. For instance, from
One then connects the site interface 702 to a site 102 to receive packets from a computer (possibly via a LAN) at the site 102. Likewise, one connects a first network interface 706 to a first router for routing packets to a first network, and a second network interface 706 to a second router for routing packets to a second network, with the networks being disparate to each other. A third, fourth, etc. network may be similarly connected to the controller 602 in some embodiments and/or situations.
The connected disparate networks are parallel to one another (not serial, although additional networks not directly connected to the controller 602 may be serially connected to the parallel disparate networks). The connected disparate networks are independent of one another, in that no routing information need be shared between them, to make them parallel (NNIs can still be used to connect networks in serial to form a larger independent and parallel network). A mistake in the routing information for one network will thus not affect the other network.
After the connections are made (which may be done in a different order than recited here), one sends 802 a packet to the site interface 702. The controller 602 then sends the packet through the one (or more—copies can be sent through multiple networks) network interface 706 that was selected by the packet path selector 704. The packet path selector 704 can maintain a table of active sessions, and use that table to select a path for packets in a given session. The packet path selector 704 does not need a session table to select paths for site-to-site traffic, because the controller 602 on the other site knows where to forward the site-to-site-packets.
During an address range information obtaining step 900, address ranges for known locations are obtained. Address ranges may be specified as partial addresses, e.g., partial IP addresses in which some but not all of the address is specified. Thus, “198.x.x.x” indicates an IP address in which the first field is 198 and the other three address fields are not pinned down, corresponding to the range of addresses from 22.214.171.124 to 126.96.36.199. Each address range has an associated network; a network may have more than one associated contiguous range of addresses which collectively constitute the address range for that network. The locations reachable through the network have addresses in the address range associate with the network. Since part of the address specifies the network, a location reachable through two networks has two addresses, which differ in their network-identifying bits but are typically the same in their other bits. Address ranges may be obtained 900 by reading a configuration file, querying routers, receiving input from a network administrator, and/or other data gathering means.
During a topology information obtaining step 902, topology information for the system of parallel disparate networks is obtained. The topology information specifies which one or more networks can be used (if functioning) to reach which known locations. With regard to
If necessary, a connection forming step is performed, e.g., to obtain a virtual circuit between two sites 102. The controller 602 then checks the status of each connection and updates the information for available communication paths.
The controller 602 at each location will go through the address range information obtaining step, topology information obtaining step and connection forming step. More generally, the steps illustrated and discussed in this document may be performed in various orders, including concurrently, except in those cases in which the results of one step are required as input to another step. Likewise, steps may be omitted unless required by the claims, regardless of whether they are expressly described as optional in this Detailed Description. Steps may also be repeated, or combined, or named differently.
During a packet receiving step 904, the controller 602 at a given source location receives a packet to be sent from that location to the destination site 102. In some cases, multiple packets may be received in a burst. The packet comes into the controller 602 through the site interface 702.
During a determining step 906, the controller 602 (or some other device used in implementing the method) looks at the packet destination address to determine whether the destination address lies within a known address range. That is, the destination address is compared to the known location address ranges that were obtained during step 900, in order to see whether the destination location is a known location. Only packets destined for known locations are potentially rerouted by the invention to balance loads, improve security, and/or improve reliability. Packets destined for unknown locations are simply sent to the network indicated in their respective destination addresses, which is the Internet 500 in the examples given herein but could also be some other “catch-all” network. Although they are not rerouted, such packets may nonetheless be counted as part of the load balancing calculation.
During a path selecting step 908, the path selector 704 selects the path over which the packet will be sent; selection is made between at least two paths, each of which goes over a different network 106 than the other. The disparate networks are independent parallel networks. This path selecting step 908 may be performed once per packet, or a given selection may pertain to multiple packets. In some embodiments, selecting a network will also select a path, as in the system shown in
As indicated, the path selection may use 910 load balancing as a criterion for selecting a path, use 912 network status (up/down) and other connectivity criteria (e.g., router status, connectivity status) as a criterion for selecting a path, and/or use 914 division of packets between disparate networks for enhanced security as a criterion for selecting a path. These steps may be implemented in a manner consistent with the description above of the path selector 704 given in the discussion of
The description of systems and methods likewise extend to corresponding computer-readable media (e.g., RAM, ROM, other memory chips, disks, tape, Iomega ZIP or other removable media, and the like) which are configured by virtue of containing software to perform an inventive method, or software (including any data structure which is uniquely suited to facilitate performance of an inventive method. Articles of manufacture within the scope of the present invention thus include a computer-readable storage medium in combination with the specific physical configuration of a substrate of the computer-readable storage medium, when that substrate configuration represents data and/or instructions which cause one or more computers to operate in a specific and predefined manner as described and claimed herein.
No change to packet source IP address or destination IP address need by done by the controller in a topology like that shown in
However, packet addresses are modified during operation of a configuration like that shown in
During an address modifying step 916, the packet destination address is modified as needed to make it lie within an address range (obtained during step 900) which is associated with the selected path to the selected network (selected during step 908). For instance, if a packet is received 904 with a destination address corresponding to travel through the Internet but the path selection 908 selects a path for the packet through a frame relay network 106 to the same destination, then the packet's destination IP address is modified 916 by replacing the IP address with the IP address of the appropriate interface of the controller at Site B. Also the packet's source IP address is replaced with the IP address of the appropriate interface of the source controller. This modifying step may be viewed as optional, in the sense that it need not be performed in every embodiment. But it is required in the sense that a system embodiment of the invention which is claimed with a limitation directed to destination address modification must be at least capable of performing the modifying step, and a method embodiment which is claimed with a limitation directed to the modifying step must perform the modifying step on at least one packet.
With regard to both
The controller 602 at the destination site goes through the steps described above in reverse order as needed. The controller 602 receives the packet from the source location through one of the network interfaces. Packet resequencing may be needed in either the
The present invention provides methods and devices for placing frame relay and other private networks in parallel with VPNs and other Internet-based networks, thereby providing redundancy without requiring manual switchover in the event of a network failure. Load-balancing between lines and/or between networks may also be performed. For instance, the invention can be used to provide reliable, efficient, and secure point-to-point connections for private networks 106 in parallel with a VPN and an SSL Internet connection. Some prior art approaches require network reconfiguration each time a frame relay circuit fails, and some have complex router configurations to handle load balancing and network failures. This requires substantial effort by individual network customers to maintain connectivity, and they will often receive little or no help from the frame relay carriers, or not receive prompt service from a VPN provider. Instead, well-trained staff are needed at each location, as are expensive routers. By contrast, these requirements are not imposed by the present invention.
As used herein, terms such as “a” and “the” and item designations such as “connection” or “network” are generally inclusive of one or more of the indicated item. In particular, in the claims a reference to an item normally means at least one such item is required.
The invention may be embodied in other specific forms without departing from its essential characteristics. The described embodiments are to be considered in all respects only as illustrative and not restrictive. Headings are for convenience only. The claims form part of the specification. The scope of the invention is indicated by the appended claims rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be embraced within their scope.