US 20060028999 A1
The present invention is a computer system and a method for gathering, processing and analysis of network information resulting in presentation and visualization of packet networks in the form of individual virtual flows, sometimes also called connections or sessions, containing their statistical characteristics in a time-sampled dynamics. The system, deployed as a separate device or co-hosted with other network devices, collects and processes information from all valid packets in network, classifies and maps gathered statistics to the statistics of relevant virtual flows. The statistical information is further processed by the system to provide near-real presentation, as well as stored in a searchable database for future analyses. The invention to be used by network engineers and administrators as a tool for a near real-time control of network traffic, as an analytical tool for solving network bottlenecks, network performance optimization and troubleshooting analyses, cutting costs by optimizing network layout, appropriate organization of traffic and intelligent configuration of QoS, routers and other network devices.
1. A computer system for gathering, processing and analysis of network information resulting in presentation and visualization of packet networks in a time-dependent dynamics, comprising:
at least one network interface unit, containing NIC, which collects all valid data-link network packets (or parts thereof required for gathering the statistics) and, optionally, retrieves virtual flow statistical and identity information from the packets or parts thereof;
at least one information processing unit, which retrieves, (if not done by the network interface units), the virtual flow statistical and identity information from the packets/parts thereof, maps and processes the information each time-sampling interval into any configurable combination of statistics counters chosen from virtual flow, OSI layer-2 and layer-3 address, network devices, OSI levels 3, 4, 5 and 6 protocol, OSI level-7 application and aggregate-virtual-flow based counters;
at least one data presentation and visualization unit to convert the said statistics into appropriate data and graphical formats useful for a customer, and to provide GUI for a near-real time presentation as well as for results of historical searches, alerts and analytical processing;
at least one data storage unit which records each time-sampling interval the chosen configurable combination (in the information processing unit) of statistics counters into searchable files or databases, and enables network troubleshooting, optimization analyses and detection of anomalies.
2. The computer system as defined in
3. The computer system as defined in
4. The computer system as defined in
5. The computer system as defined in
6. The computer system as defined in
7. The computer system as defined in
8. The computer system as defined in
9. The computer system as defined in
10. The computer system as defined in
11. The computer system as defined in
12. A use of the computer system defined in
13. A use of the computer system defined in
14. The computer system as defined in
15. A method for visualization of a plurality of communication networks, comprising:
gathering the virtual flow statistical and identity information from all datalink packets in the network or relevant parts of these packets;
mapping and processing said information each time-sampling interval into any configurable combination of statistics counters chosen from virtual flow, OSI layer-2 and layer-3 address, network devices, OSI levels 3, 4, 5 and 6 protocol, OSI level-7 application and aggregate-virtual-flow based counters;
near-real time presentation of said statistics in its time sampled dynamics in a data and graphical formats useful for a customer;
recording each time-sampling interval the configured (on the stage of information mapping and processing) combination of statistics counters into searchable files or databases, and proceeding with network troubleshooting, optimization analyses and detection of anomalies;
filtering only relevant information at each of the above-mentioned stages;
temporary storage of the necessary parts of datalink packets and their further analysis, e.g. for suspected traffic;
detailed processing of the collected historical statistics in order to reveal anomalies and to dispatch appropriate triggers;
screening the gathered packets or their parts and/or virtual flow statistics to discover signatures of viruses, worms, intrusion attempts or DOS/DDOS attacks and to trigger notifications and/or dispatch blocking the virtual flows with malicious traffic;
detailed processing of the collected historical statistics to reveal network bottlenecks and/or network poor performance and to trigger relevant recommendations for network engineers and administrators;
16. The method as defined in
17. The method as defined in
18. The method as defined in
The present invention relates generally to computers and packet networks and in particular to network monitoring, gathering of statistical information and using it for network troubleshooting and improvement of networks performance and traffic optimization.
TCP/IP networks operate with OSI-4 connection-oriented transport protocol TCP/IP and connectionless protocol UDP/IP. The packets running in networks can be logically assembled to so-called streams, also known as sessions or flows, hereafter virtual flows (VFs). Several VFs related to the same application task can be logically combined into virtual super-flow (VSF), e.g. FTP protocol control and data VFs compose an FTP VSF. There is an exact mapping between a VF and a layer-4 connection-oriented protocol session, e.g. TCP-session. The VF is also applicable to sessionless protocols, for example UDP, whereas VF is characterized by a set of parameters, such as source and destination IP-addresses, source and destination ports and IP-protocol (hereafter this set of parameters is called VF-identity parameters, or VFID). For layer-4 session-keeping protocols, e.g. TCP/IP, the virtual flow is started with the first control packet of a session (SYN) and is completed either by a last one (ACK after FIN or RST), or by a sufficiently long configurable timeout. In the case of layer-4 connectionless protocols, e.g. UDP/IP, the virtual flow is started with the first packet having a unique VFID and is completed by a sufficiently long configurable timeout.
Network administrators and engineers have a rather limited set of tools to visualize and control their networks. Their main tools are sniffer/data analyzer type products, which are capable of capturing and presenting packets running in a network, like network protocol analyzer Ethereal (www.ethereal.com), complex network analyzer Sniffer from Sniffer Technologies (www.sniffer.com), Sniffer Portable from Network Associates (www.networkassociates.com) or LanPro network analyzer from Radcom (www.radcom.com). Most sniffing type products can combine collected packets into application-related flows. VF/VSF level capabilities of sniffer/data analyzers are mostly used for protocol decoding and application level statistics of some VFs calculated off-line. Although being very useful tools, the devices are inferior in their capability to present near real-time flow related parameters (e.g. throughput, number of packets per second) for all virtual flows running in the network. Some information about the network may be learned from QoS boxes (e.g. manufactured by Packeteer, Allot, etc.) or routers with QoS capabilities (Cisco), deployed as the gateway devices to the outside Internet and providing a lot of useful information about the traffic passed through them, whereas all other LAN flows remain completely “invisible”. The effectiveness of QoS box deployment may be improved and sometimes even becomes unnecessary, if flow visualization of networks, including historical data, could be available for detailed analyses of network events.
Systems, devices and methods, disclosed in U.S. Pat. Nos. 6,108,782, 6,453,345, 6,459,682, 6,615,262, 6,661,778, EP 1341345, U.S. patent application 2001/0021176, 2002/0032717, 2003/0055950, and WO 01/71545, 02/21802, WO 02/33892 failed to provide detailed data for each individual virtual flow, especially retransmission data, RTT, server response delay, reasons for VFs completions (e.g whether server or client is timed out, server-side or client side initiated disconnect, etc.), changes in throughput and other flow-statistics counters within a flow lifetime and other important for network engineers information. Computer system and method disclosed in U.S. Pat. No. 6,453,345 is based on a permanent storage of packets running in networks to provide current and historical aspects of network statistics, which requires sophisticated storage devices. All mentioned prior art has failed to provide inexpensive and, therefore, affordable solution for most companies for configurable presentation of the whole network picture in a near real-time and does not teach how to obtain detailed information necessary for networks troubleshooting and optimization, detection of anomalies and a time-sampled historical searchable view on the total network as well as on each individual VF, VSF, AGVF or any other logical flow.
Network administrators and engineers lack instrumentation to “watch” what is currently running in their networks to perform in-depth analyses of the traffic, networks performance optimization and troubleshooting, to reveal network anomalies and to obtain historical information about the traffic, e.g. in the last hour, night, or a time period between certain dates, or at the date and time of an important sometimes disastrous event in the network.
It is the object of the present invention to provide a method and computer system able to supply a network administrator or engineer with near real-time information/statistics as well as with historical data relating to all virtual flows running in the network and also derived information regarding various logical flows in the network.
An aspect of the present invention, is a computer system, deployed as a passive network device, which monitors LAN/WAN traffic without being physically on packet routes, collects and processes valid packets from the network, retrieves statistical information from the packets, assembles and maps the information to a VF-statistics, stores said information in a searchable database and outputs VF-statistics and the derived OSI layer-2 and layer-3 addresses, network-devices, OSI levels 3, 4, 5, 6 protocols, OSI level-7 applications and aggregate-virtual-flow based statistics to a near-real time GUI presentation.
Yet another aspect of the invention is deployment of the computer system physically on the packet routes (active deployment), enabling it not only to collect statistical information, store it to a database and analyze the traffic, but also to apply results of the analyses actively by performing traffic modifications, e.g. by dropping a worm related VFs to prevent the worm spreading.
Another aspect of the present invention is a co-hosting the invented system on the same computer and the same NIC (and normal functioning) with other network tools such as sniffers, firewalls, QoS and IDS systems. It is worth to mention that the invention enables passive deployment of the invented system with the above-mentioned network tools without limitations, whereas the active deployment of the system encompassing active network tools like firewalls, QoS and IDS may cause limitations or require coordination of performance activities between the invented system and the tools.
Another aspect of the present invention relates to further processing VF-based information into the application, network protocols and host related information, by making application/protocols classification of all VFs in the network, whereas the destination/source address of each host (IP-address in ip-networks) is an integral part of VFID. Keeping all VF data, including VFID and statistics counters, in a searchable database enables an easy access to any application, network protocol or host based statistics. According to this aspect of the invention a topology of the networks, from which the system collects statistics, may be reconstructed using IP-addresses of all hosts, stored per each VF in the database, and either netmask inputs from network administrators, or netmask discovery techniques. A network topology map resulting from the reconstruction is a useful and convenient GUI, which in combination with the capability of the invented system to depict on the map in near real-time statistics regarding applications, protocols, throughputs, retransmissions, RTT (Round-Trip Time), numbers of connections and packets, other parameters with relation to network elements and their interconnections, creates real visualization of network dynamics. The invented system provides a network administrator or an engineer with the means necessary for real control of network, enables bottleneck analyses and troubleshooting, re-planning and network layout optimization.
It is yet another aspect of the present invention providing an analytical agent, which is capable of revealing network bottlenecks and/or network poor performance and of triggering relevant recommendations for network optimization. Statistical information regarding all VFs running in the network is collected for each time sampling period, which is normally configurable from seconds to tens of seconds. Data for each VF, which represents a collection of statistics for at least one time sampling period, is kept by the system long enough enabling historical searches. Thus, an administrator may easily obtain time-dependent throughput data for a very important long running VF including times when there was insufficient bandwidth. It is easy to figure out the sources and reasons of extra retransmissions, to locate the most bandwidth-consuming hosts and applications at peak hours and to gain deep understanding of the nature of the load on a web-server at different hours, etc.
A one more aspect of the present invention relates to processing of VF-based information to the aggregate-virtual-flows (AGVFs) information by combining VFs with a certain common parameter (e.g. by combining VFs with a source or destination IP being related to a certain subnet), thereby providing a subnet-level visualization of the traffic and network events. It may be extremely useful for network personnel to keep track of a AGVF, combining VFs by a certain common type of service or functionality. For example, it may be useful in networks served by several Internet providers to monitor the SLA per each provider by arranging AGVF per provider. Another possible application of the aspect of the invention is monitoring traffic from a company central office to its affiliated premises by configuring an AGVF for each remote office.
Yet another aspect of the invention is an availability control of network elements and network services. Absence of VFs, originating from a certain network element (NE) and/or broken VFs full of retransmissions towards the NE, trigger configurable NE availability alerts. It may be easily configured to monitor availability of a certain type of applications/services, running on a NE or on a group of NE to trigger alerts when the applications/services are malfunctioning.
Another aspect of the present invention relates to a time-sampled storage of statistical information regarding each individual VF in a searchable database. Once in a configurable amount of time VF-based and derived (OSI layer-2 and layer-3 addresses, network-devices, OSI levels 3, 4, 5, 6 protocols, OSI level-7 applications and aggregate-virtual-flow based) statistical information is summarized and stored in a database, so that for all sessions with a lifetime more than a sampling time, a historical view on each statistics counter may be retrieved to provide graphs and tables of parameters (e.g throughput, retransmissions, RTT, etc). Such historical view can, for example, reveal throughput starvation for an important VF at certain hours to be remedied by re-scheduling of the less important traffic from the peek hours or changing QoS-related policies in a router/QoS-box or by any other means. Various configurable searches in the database may provide a crucial information for network engineers and administrators by highlighting applications and hosts with most bandwidth consumption at peek-hours, network elements with a maximum connections to/from them, reasons for web-server connection requests not being served at certain hours, retransmissions peeks originating from a group of servers at certain hours, etc.
Another aspect of the present invention relates to network security. This is possible to accomplish because all VF-related information is stored in a database or in recoverable to database file storage formats and may be examined. Unusual patterns of behavior, like huge amount of VFs from Internet to a certain computer, normally serving only LAN-residents, or lots of opened connections from a certain machine, will set of the system's alerts and actions configured by administrator.
A one more aspect of the present invention relates to improving network security. Keeping a full VFs history backlog enables to reveal fingerprints (VFs) of an intrusion to a computer in the network, which occurred at a known time in the past. Spreading a worm in the network generates an anomalous flow with a great number of VFs from a worm-sourcing computer to all other NEs. Worm spreading pattern may be alerted, helping to prevent it and/or reveal computer from which the worm spreads. Patterns of DOS/DDOS attacks may be easily highlighted causing an alert for action to be undertaken.
Another aspect of the present invention is a use of the available statistical information for billing purposes, thereby enabling different and more flexible billing methods than the ones cited in prior arts, allowing charging of customers based on the amount of data cleared from retransmission or, interalia, taking some other statistical VF parameters into consideration.
Yet another aspect of the invention is a use of the collected statistical data to monitor QoS conditions in a network, including monitoring SLA (service level agreement) with providers.
In some embodiments the NUI 11 deploys an Intermediate Driver 23 to be inserted between NIC Driver 22 and TCP/IP stack. The Intermediate Driver 23 provides TCP/IP-like interface towards NIC Driver 22 and NIC-driver-like interface towards NIU Driver 25 and/or Drivers of Other Network Tools 26 such as sniffers, firewalls, QoS and IDS systems. The Intermediate Driver 23 intercepts packets on the path from NIC Driver 22 to TCP/IP stack and acts to ensure delivery of a copy of each packet to the NUI Driver 25 as well as to the Drivers of Other Network Tools 26. Intermediate Driver 23 enables co-hosting on the same NIC and independent proper functioning of the invented system and the other network tools. In some other embodiments the NUI Driver 25 itself accomplishes the functions of the Intermediate Driver.
In some embodiments the packets collected by NIUs 11 are passed through a configurable Filter 24 with rules enabling further treatment of only relevant packets to/from certain IP addresses, networks, ports or selected by any other configurable parameters. The Filter 24 is configured and activated, when it is required to limit the amount of incoming packets and statistics information, e.g. to decrease load on the system by collecting, processing, presenting and storing only the information of interest, thereby filtering an irrelevant traffic.
In other embodiments, when the invented system is deployed in a passive mode, all packets (or only filtered ones) are processed in the NIU Driver 25 used by the system to retrieve relevant statistics, which is passed to the IPU 12. In some other embodiments, whenever the system is deployed as active or passive, packets are passed to IPU 12 without filtering.
The IPU 12, showed in detail at
In general, the VF-context enables to calculate for each sub-flow the following statistics counters for each time sampling period as well as VF life-time averages: a number of packets passed, packets throughput in second, packets size, a distribution of packet sizes, packets latency and the latency jitter, bytes passed, bytes throughput, average timeout between packets and counters for packets bursting, etc. VF context for TCP/IP traffic additionally enables calculation of retransmitted packets, retransmitted packets throughput in second, retransmitted bytes, retransmitted throughput, effective throughput (throughput cleaned from retransmissions), RTT and RTT jitter. VF context for TCP/IP performs permanent overview of TCP-session in both directions (for each sub-flow), including milliseconds accurate timing for each packet, inspection and analyses of TCP-header packet sequence number and acknowledgment number to follow retransmission and in some cases reasons for retransmissions and to be used for RTT estimations. The retransmission, RTT and TCP header flag bits (RST, SYN, FIN, ACK) information are used to figure out reasons for VFs completions, such as server or client side timeout, server-side or client side initiated disconnect, etc.
If the statistics is collected on the level of AGVFs, each AGVF on configuration arranges an AGVF-context to keep the counters. The first packet for each VF and the first packet from each side of a for bi-directional flows is classified to figure out whether the traffic matches rules configured for any AGVF, and when it does, all packets assigned to the sub-flow will be used to update statistics counters for an appropriate AGVF.
When the configured statistics is collected on the level of applications, a VF is classified by transferring packets to an application classifier. If the VF is recognized to belong to an application of interest, the VF statistics is used to update the counters in the application statistics context. Some of the application-specific parameters may be kept in the VF context to enable a further VSF reconstruction and an advanced analyses of application traffic.
Collection of statistics based on IP addresses is accomplished by arranging a data structure further named a map of IP-contexts, which contains a context per active IP-address in the network with two sub-contexts for inbound and outbound traffic, respectively. Statistics of an IP-context is updated using VFs sources or destined to the IP-address. When the last VF with a certain IP-address is removed from the system, so does the IP-context after its statistics were collected.
On each configurable time-sampling timeout, which is from seconds to tens of seconds, all statistics from all VF-contexts, AGVF-contexts, IP-contexts and application-contexts kept in Maps 33 is summarized, calculated, collected and passed to the DPVU 3 and the DSU 4 units.
The DPVU 3 is shown in details at
The DSU 4, detailed at
The invention may be used by network engineers and administrators as a tool for a near real-time control of network traffic, as an analytical tool for solving network bottlenecks, network performance optimization and troubleshooting analyses, cutting costs by optimizing network layout, appropriate organization of traffic and intelligent configuration of QoS, routers and other network devices.