NetFlow data and its fields vs packet capture

When it comes to keeping tabs on networks, we’ve got two heavy hitters: NetFlow and packet capture. They are both crucial for understanding what’s going on in network traffic, but they do it in their own unique ways, especially when we dig into data fields.

In this article, we are diving into the differences between NetFlow versions 5 and 9 and packet capture, shining a spotlight on the intricate details of NetFlow’s flow records. We will discuss what makes NetFlow stand out compared to the reliable workhorse, packet capture, and giving network pros a solid overview of their strengths and quirks.

1. Packet Capture

Packet capture, also known as network sniffing or packet analysis, involves capturing and storing network packets for subsequent analysis. Packet capture provides a more granular level of detail compared to NetFlow. It captures the entire packet payload, including headers, payloads, and protocols. As a result, packet capture allows for in-depth analysis of network communications, including application layer data and packet-level anomalies, identifying security threats.

Packet capture files store detailed information about each captured packet. Figure 1 shows a captured IPv4 packet, which represents an HTTP response sent from a web server with IP 188.184.100.182 to a client with IP 10.10.10.2. This packet contains Layer 2, 3, 4 and 7 protocol headers, along with a payload highlighted in the orange box. After the packet is processed by the client, the payload, an HTML page, is displayed in the client’s web browser.

Unlike NetFlow, which creates flows based on specific fields from headers, packet capture stores all the header attributes populated and used by each layer protocol (e.g., data, network, transport, or application) to move a packet toward its destination.

Packet capture has its own limitations, primarily in terms of resource consumption and scalability. Due to the need to capture and store every packet, it leads to higher storage requirements and increased network overhead. This can be a significant challenge in large-scale networks with high traffic volumes, as it requires substantial storage capacity and processing power.

2. NetFlow

NetFlow provides insights into network traffic patterns, allowing administrators to identify top talkers, monitor bandwidth utilization, and detect anomalies. NetFlow offers several advantages over packet capture, primarily in terms of scalability and resource utilization. Unlike packet capture, which stores every packet, NetFlow aggregates flow data, significantly reducing the amount of data that needs to be stored and processed. This makes NetFlow more scalable, especially in large network environments with high traffic volumes.

While NetFlow provides valuable information, it does have certain limitations. One drawback is its lack of packet-level detail. NetFlow data does not capture the entire packet payload, making it challenging to perform deep packet inspection or analyze application-specific information. This limitation restricts the ability to troubleshoot complex network issues or investigate application-layer anomalies.

2.1 How NetFlow Creates Flows

Flow refers to a sequence of packets with shared characteristics, identified by common attributes such as source and destination IP addresses, ports, protocol, and more. To understand the data fields in NetFlow, it’s essential to delve into the process of creating flows within the router’s flow cache.

Upon receiving a packet on a network interface, a new flow record is generated in the router’s cache. The packet counter for the flow is set to 1, and the byte count is increased by the length of the packet.

In the case of a second packet arriving with all seven fields matching the entry in the cache, the packet counter for the existing flow record is incremented to 2, and the byte count is increased by the length of the second packet. The router is designed to remove the flow from its cache and export it to the collector when no further packets with the same fields are received during the configured time interval.

However, if a packet is received on the interface and not all seven attributes match an existing flow record in the cache, a new flow is then created in the cache to accommodate this distinct packet. This process ensures that the router effectively manages and tracks network flows based on specific attributes.

2.2 NetFlow version 5 Data Fields

NetFlow version 5 monitors network traffic by tracking seven essential fields (key fields) that define a flow:

IP source address
IP destination address
Source TCP or UDP port
Destination TCP or UDP port
Layer 3 protocol type
Type of Service
Router interface

NetFlow v5 employs a set of fixed seven attributes of IP information to identify a flow. In addition to these key attributes, it also collects and exports non-key fields.

Non-key fields, in contrast, encompass additional attributes that do not contribute to defining flows but are still captured for subsequent analysis. It’s important to note that a change in the value of a non-key field does not result in the creation of a new flow. Instead, the values in non-key fields are appended to flows, providing supplementary information about the traffic within those flows, beyond the essential flow-defining (key) attributes.

Examples of non-key fields include the number of bytes and packets. The following is a comprehensive list of key and non-key fields in NetFlow v5:

Source and Destination IP address
IP address of a next-hop router
SNMP index of input and output interface
Number of packets in the flow
Number of Layer 3 bytes in the packets of the flow
System Uptime at the start, and the time the last packet of the flow was received
TCP/UDP source and destination port number
Cumulative OR of TCP flags
IP protocol type (for example, TCP = 6; UDP = 17)
IP type of service (ToS)
Autonomous system number of the source and destination, either origin or peer
Source and destination address prefix mask bits

2.3 NetFlow version 9 Data Fields

NetFlow Version 9 adopted a template-based approach, moving away from the fixed set of fields in version 5. This change allows greater flexibility in capturing different types of data by defining custom key and non-key fields.

NetFlow v9 or Flexible NetFlow enables users to customize key fields for flow identification. To achieve this, the “match” command within the flow record configuration is used. In our example below, a flow is defined by a set of six criteria: IPv4 protocol, source and destination IP addresses, source and destination transport ports, and an application name. These attributes, along with non-key fields, are exported in flow records to a flow collector.

flow record RECORD-1
   match ipv4 protocol
   match ipv4 source address
    match ipv4 destination address
    match transport source-port
    match transport destination-port
    match application name
    collect counter packets long
    collect interface input
    collect interface output
    collect counter bytes
    collect flow direction

Non-key fields are specified using the “collect” command within the flow record configuration, and they do not define flows themselves. Instead, they dictate what supplementary information about each flow should be collected, contributing to a comprehensive understanding of network traffic and its characteristics. In our example, a flow is enriched by the following non-key fields:

collect counter packets: Collects and counts the number of packets in the flow.
collect counter bytes: Collects and counts the total number of bytes in the flow.
collect interface input: Collects the input interface index through which the flow entered the device.
collect interface output: Collects the output interface index through which the flow exited the device.
collect flow direction: Collects information about the flow direction, indicating whether it is inbound or outbound.
collect application name: Collects the name of the application associated with the flow.

Here is a list of key and non-key fields for NetFlow v9 [1]:

NetFlow v9 is designed with a flexible structure. To facilitate mutual comprehension between the collector and exporter, NetFlow Version 9 consistently transmits template data. This template precisely delineates the fields that will be incorporated in forthcoming data flow sets. Figure 2 illustrates Template ID 258, showcasing the six key fields and five non-key fields configured for the flow record RECORD-1.

Figure 3 illustrates an IPv4 packet containing NetFlow v9 data transmitted from the exporter to the flow collector. The fields within this packet are defined by FlowSet 1 template 258, as depicted in Figure 2. Within the flow record, there are three distinct flows. Flows 2 and 3 correspond to HTTP traffic originating from client 10.10.10.2 and directed towards the web server at 188.184.100.182. These two flows involve a total of 5 packets sent to the server, totaling 343 bytes.

Notably, unlike the packet capture displayed in Figure 1, this representation only includes specific data fields extracted from headers and outlined in the template, without displaying payload data.

Conclusion

NetFlow data and packet capture serve distinct purposes in network monitoring. Packet capture, with detailed payload analysis, is effective for deep inspection but faces challenges in scalability and resource usage. 

NetFlow, particularly version 9, offers efficiency and scalability by aggregating flow data based on customizable templates. Although it lacks packet-level details, NetFlow excels in monitoring traffic patterns and detecting anomalies. The periodic transmission of template data ensures effective communication between collectors and exporters. In summary, packet capture provides detailed insights, while NetFlow version 9 offers scalability for comprehensive network monitoring.

While NetFlow provides valuable data on network traffic, it alone may not be sufficient for proper network traffic monitoring. A comprehensive NetFlow analysis tool is essential to translate this data into actionable insights. In this context, Noction Flow Analyzer is a practical tool that complements NetFlow’s capabilities and can be of great use to network professionals. NFA efficiently processes large volumes of flow, SNMP and BGP data, offers a myriad of filtering, grouping and display options, making it suitable for companies looking for a reliable and cost-effective network traffic monitoring solution.