NetFlow vs. sFlow vs. IPFIX vs. NetStream. Network Traffic Analysis and Network Traffic Monitoring.

Network monitoring is a systematic effort to monitor parameters of a computer network in order to detect issues that degrade network performance. Network Traffic Analysis is used to deduce information from patterns collected during network monitoring.

Network Traffic Analysis

Packet analysis and flow analysis are two technologies that we can choose from when we perform traffic analysis on the network.

Packets Analysis versus Flow Analysis

Packet Analysis uses packet capturing technologies such as SPAN, RSPAN, ERSPAN to get raw copies of traffic. Packet analysis is suitable for the in depth analysis of a specific conversation as the full packet header along with the payload is collected. In contrast to packet analysis, flow analysis is about collecting the metadata from network traffic used for statistical purpose (e.g. top talkers, traffic by protocols, bandwidth usage etc.).

What is a Flow?

A flow is a sequence of packets sharing the same properties that are sent between a sending and a receiving hosts. For instance, when we watch live streaming video, packets sent from server to PC create a flow as they are part of the same conversation.

What is NetFlow?

NetFlow is a Cisco proprietary network protocol used for flow analysis. NetFlow collects and aggregates information about network traffic flowing through a device with an enabled NetFlow feature. The output of NetFlow are flow records that are sent to a centralized place in a network (flow collector) as NetFlow messages.

Why do we need NetFlow?

Flow statistics collected by the NetFlow protocol are typically used for:

• bandwidth monitoring
• network threat detection (DoS attacks) and forensic analysis
• accounting or billing based on usage
• investigation of network problems that cause congestion and slowness of applications.

What are the NetFlow Infrastructure Elements?

• NetFlow exporters
• NetFlow collector
• Analysis application

How are the Flows Created?

NetFlow enabled devices (NetFlow exporters) create NetFlow records aggregating packets into flows based on the criteria below:

• IP Source Address
• IP Destination Address
• Source Layer 4 port
• Destination Layer 4 port
• Class of Service
• IP Protocol
• Source Interface

 

Each packet that is going to be forwarded is examined for the above parameters. The first unique packet creates a flow as an entry in the NetFlow cache (flow record). The packet is then forwarded out of the router. The other packets matching the same parameters are aggregated to this flow and the bytes counter for the flow increases. If any of the parameters is not matched, a new flow is created in the cache.

NetFlow Record is Created

There are hundreds of thousands flows recorded in the NetFlow cache. Obviously, flows do not live in cache forever, instead they are exported from the cache to a flow collector on a regular basis. A flow is exported when it is inactive for a certain time e. g. no new packets are received for the flow. By default, the inactive flow timer is set to 15 seconds. The flow is also exported when it is long lived (active) and lasts longer than the active timer.  By default, the active timer is set to 30 minutes. For instance, a large file download that lasts longer than 30 minutes may be broken into multiple flows. It is a role of the flow collector to combine these flows showing the total download.

NetFlow Collector

Multiple NetFlow exporters send records periodically to one or more NetFlow collectors using the User Datagram Protocol (UDP) or Stream Control Transmission Protocol (SCTP) when the reliable transport is required. The role of a collector is to gather, record, combine or aggregate the exported flows to produce the reports used for traffic and security analysis.

The Analysis Applications

Such applications analyze the received flow data for the purpose of an intrusion detection or traffic profiling. They are also responsible for the presentation of data and the creation of reports.

NetFlow variations by other Vendors

Vendors other than Cisco have their own versions of Netflow, such as jFlow (Juniper), rFow (Ericcson), sFlow (HP) etc.

Sampled Flow aka sFlow

Although sFlow stands for Sampled Flow, actual packets are being sampled here instead of flows. Sampling involves either copying headers of packets, or extracting features from packet. The goal of packet sampling and filtering is to forward only certain packets. Based on a defined sampling rate, an average of 1 out of n packets is randomly sampled. sFlow agent running within a router or a switch, packages interface counters and flow samples into sFlow datagrams. The datagrams are continuously sent using UDP to the sFlow Collector where they are further analyzed.

NetFlow versus sFlow

SFlow has an ability to monitor L2-L7 headers, the ability to monitor L2 headers (MAC, VLAN ID) has been added to NetFlow v9.

Packet sampling is hardware based and is performed by switching ASICs, achieving wire speed performance. It makes sFlow a scalable technology which is able to monitor the links with the speed of up to 10 GBps.

sFlow datagrams are continuously sent across the network in real-time, while the export of NetFlow records depends on active/inactive timers. It may take up to 30 minutes to export flow when NetFlow is used. Obviously, sFlow is better in traffic visibility than NetFlow. It makes sFlow good at massive DoS attacks detection, as the sampled network patterns are sent on the fly to the sFlow collector.

Nevertheless, measurements provided by sFlow are only an approximation of the real traffic because sampled packets do not reflect all network traffic. As a result, sFlow lacks accuracy provided by NetFlow as it cannot track every network communication. The accuracy, however is highly required in digital forensics so sFlow cannot fully qualify for forensic investigation.

IPFIX

Internet Protocol Flow Information Export (IPFIX) is a standard for exporting the information about network flows from devices. It is derived from Cisco’s proprietary NetFlow v9. A metering process generates flow records collecting data packets at an Observation Point, filters them and aggregates information about these packets. Flow records are sent by the Exporting process running on exporter as IPFIX messages encapsulated by layer 4 protocols (SCTP, UDP or TCP)  to a collector. The messages are pushed to the collector without any interaction by the collector.

IPFIX can be used to export any traffic information from L2-L7 to flow collector. It is a flexible protocol that supports variable length fields. It allows to collect information such as  http url or host (e.g. facebook.com) as well as the user-defined data types. For instance, syslog or SNMP data or even room temperature values can be continuously exported to the collector inside the IPFIX messages.

NetStream

NetStream is a NetFlow equivalent brought by Huawei. NetStream infrastructure consists of:
– NetStream data exporter (NDE)
– NetStream collector (NSC)
– NetStream data analyzer (NDA)

Note:  NSC and NDA are typically integrated into one server.

The NDE samples packets in order to reduce the impact on device performance. For instance, when NDE is set to packet-based random sampling, the NDE randomly samples a packet from a specified number of packets transmitted. If the number of packets is set to 100, the NDE randomly samples a packet from every 100 packets.

Note:  IPv4 NetStream defines a flow based on the seven criteria just like NetFlow, so packets with the same 7-tuple information are marked as one flow.

Flow records are stored in NetStream cache on NDE. When a NetStream flow is aged out, NDE exports flow statistics from cache to NSC using NetStream packets. Similar to NetFlow, NetStream flows are aged out based on active and inactive timers. When RST or FIN flag are received, a particular flow is immediately aged out from cache as well. Also, when bytes aging mode is enabled, NDE ages out the flow once upper bytes limit is reached.

NDE periodically exports flow statistics to NSC. NSC collects and parses packets from multiple NDEs and stores them to the database.

NDA is a traffic analysis tool. It extracts statistics from NSC, processes statistics, and generates a report. The report can be used for traffic accounting, network planning, and attack monitoring.

---

Conclusion:

It is important to mention that all of these technologies (NetFlow, SFlow etc.) have their strengths and weaknesses in terms of scalability, performance, accuracy and protocol coverage in the estimation of network traffic parameters. But what is equally or more important for network professionals is the quality, versatility, power and the ease of use of the traffic analysis applications that actually analyze the collected flows / packets info and present engineers with readable data and reports.

Read this article in French – NetFlow versus sFlow versus IPFIX versus NetStream. L’analyse et la surveillance du trafic réseau.