Carrier Grade NAT (CGNAT)
- Scott Leishman
- Gavin Shearer (Deactivated)
Configuration
Simple Example
Required Configuration
Address-group to match subscriber (source) addresses
set resources group address-group AG_MATCH1
set resources group address-group AG_MATCH1 address 10.10.1.0/28
set resources group address-group AG_MATCH1 address-range 10.10.1.20 to 10.10.1.30
set resources group address-group AG_MATCH1 address 10.10.1.10 |
NAT Pool to specify Public Addresses and how they are Allocated
set service nat pool NAT_POOL1 entry RANGE1 ip-address range start 10.10.3.1
set service nat pool NAT_POOL1 entry RANGE1 ip-address range end 10.10.3.63
set service nat pool NAT_POOL1 entry RANGE2 ip-address prefix 10.10.3.64/28
set service nat pool NAT_POOL1 type CGNAT
set service nat pool NAT_POOL1 address-allocation round-robin
set service nat pool NAT_POOL1 address-pooling paired
set service nat pool NAT_POOL1 port dynamic-block-allocation block-size 128
set service nat pool NAT_POOL1 port dynamic-block-allocation max-blocks-per-subscriber 8
set service nat pool NAT_POOL1 port allocation sequential
set service nat pool NAT_POOL1 port range start 1024
set service nat pool NAT_POOL1 port range end 65535 |
CGNAT Policy pulls together the 'match' address-group and the NAT pool
set service nat cgnat policy POLICY1 match source address-group AG_MATCH1
set service nat cgnat policy POLICY1 priority 10
set service nat cgnat policy POLICY1 translation pool NAT_POOL1 |
Assign the CGNAT Policy to the interface which routes to the external network
set service nat cgnat interface dp0p1s2 policy POLICY1 |
Optional Configuration
Session Timeouts
set service nat cgnat session-timeout tcp partially-open 240
set service nat cgnat session-timeout udp partially-open 240
set service nat cgnat session-timeout other partially-open 20
set service nat cgnat session-timeout udp established 1800
set service nat cgnat session-timeout tcp established 1800
set service nat cgnat session-timeout other established 30 |
Per-port established-state timeouts are available per port
set service nat cgnat session-timeout tcp port 53 established 10 |
Logging
set service nat cgnat log event subscriber
set service nat cgnat log event session
set service nat cgnat log event resource-constraint
set service nat cgnat log event port-block-allocation
set service nat pool NAT_POOL1 select event port-block-allocation
set service nat cgnat policy POLICY1 select event session all-subscribers
set service nat cgnat policy POLICY1 select event session creation
set service nat cgnat policy POLICY1 select event session deletion
set service nat cgnat policy POLICY1 select event subscriber |
Note that enabling session logging will switch CGNAT to record destination address and port in sub-sessions (2-tuple sessions in tables within 3-tuple sessions). This will affect performance.
set resources group address-group <name>
CGNAT uses an address-group to match inside subscriber source addresses.
Parameter | Format | Description | Details |
|---|---|---|---|
address | <h:h:h:h:h:h:h:h/x> | IPv6 subnet to match | not applicable for CGNAT |
<h:h:h:h:h:h:h:h> | IPv6 address to match | not applicable for CGNAT | |
<x.x.x.x/x> | IP subnet to match | The first and last addresses in the prefix will be matched. For example, 10.10.1.0/28 will match address range 10.10.1.0 to 10.10.1.15. | |
<x.x.x.x> | IP address to match | ||
address-range | <h:h:h:h:h:h:h:h> to <h:h:h:h:h:h:h:h> | IPv6 address | not applicable for CGNAT |
<x.x.x.x> to <x.x.x.x> | IPv4 address | ||
description | <text> | Address-group description |
set service nat pool <name>
A NAT pool is used to specify the public addresses that CGNAT will use to translate inside source addresses to.
Command | Format | Details |
|---|---|---|
entry <range-name> ip-address range start | <x.x.x.x> | IPv4 Address |
entry <range-name> ip-address range end | <x.x.x.x> | IPv4 Address |
entry <range-name> ip-address prefix | <x.x.x.x/x> | IPv4 address and mask. First and last addresses are used provided they are not '.0' or '.255'. For example, 10.10.3.0/28 would give range 10.10.3.1-10.10.3.15. |
entry <range-name> ip-address subnet | <x.x.x.x/x> | IPv4 subnet and mask. First and last addresses are never used. For example, 10.10.3.0/28 would give range 10.10.3.1-10.10.3.14. |
type | CGNAT | Only option. Required for future compatibility. |
address-allocation | round-robin | Addresses are allocated from the NAT pool in a round-robin fashion. Only option. Required for future compatibility. |
address-pooling | paired | Internal IP addresses are paired with external IP addresses. An internal IP address will never use more than one external IP address at any given time. Only option. Required for future compatibility. |
port dynamic-block-allocation block-size | <64..4096> | The number of ports in a block (multiple of 64 only). |
port dynamic-block-allocation max-blocks-per-subscriber | <1..32> | Maximum number of port-blocks per subscriber |
port allocation | random | Allocate ports randomly from within the current 'active' port-block. |
sequential | Allocate ports sequentially from within the current 'active' port-block. | |
port range start | <0..65535> | Start of port rang. It is recommended that this is set no lower than 1024. |
range end | <0..65535> | End of port range. |
blacklist address-group | <addr-grp-name> | The address-group identifying the pool addresses being blacklisted. Adding external IP addresses or prefixes to a blacklist address group will cause sessions creating using that external address to be cleared, and for that address to no longer be used to create new sessions. It is useful on a live system where the administrator wants to prevent an address in the NAT pool form being used for CGNAT mappings. This has an advantage over reconfiguring the NAT pool range to exclude that address in that reconfiguring the NAT pool range will cause all sessions using that NAT pool to be torn down. |
set service nat cgnat policy <name>
A CGNAT policy ties together an address-group for matching inside source addresses with a NAT pool for specifying the set of external public addresses to use. A CGNAT policy is the thing that is attached to an interface,
Command | Format | Description |
|---|---|---|
match source address-group | <addr-grp-name> | Address group name |
translation pool | <nat-pool-name> | NAT pool name |
priority | <1..9999> | The priority of CGNAT policy. Multiple CGNAT policies on an interface are evaluated in order of priority, with the lowest numerical priority first. |
set service cgnat session-timeout
Established-state timeout values may be configured per port (a destination port in outbound traffic).
other | partially-open | <10..240> | Non-TCP/UDP 'partially-open' timeout in seconds |
established | <30..1800> | Non-TCP/UDP 'established' timeout in seconds | |
udp | partially-open | <10..240> | UDP 'partially-open' timeout in seconds |
established | <30..1800> | UDP 'established' timeout in seconds | |
tcp | partially-open | <10..240> | TCP 'partially-open' timeout in seconds |
partially-closed | <10..240> | TCP 'partially-closed' timeout in seconds | |
established | <30..14400> | TCP 'established' timeout in seconds |
Other configuration commands
set service cgnat ...
cpu-affinity event session | <0..65535> | Number of the core to process session events on |
disable-hairpinning | Disable hairpinning in CGNAT device | |
export event ... | Used to control logging | |
max-dest-per-session | <1..64> | Max number of destination records that can be associated with a session |
max-sessions | <1..33554432> | System-wide maximum number of sessions |
select warning event resource-constraint ... | ||
snat-alg-bypass | Enable SNAT ALG traffic to bypass CGNAT |
Operational Commands
show cgnat summary
The error and summary statistics will count every packet that matches either of the following two criteria:
outbound packet whose source address matches a CGNAT policy, or
inbound packet who's destination address is a CGNAT public address
CGNAT Summary
Sessions:
Active sessions 2
Sessions created ~2
Sessions destroyed ~0
Active sub-sessions 4
Sub-sessions created ~4
Sub-sessions destroyed ~0
Maximum table size 33554432
Table full No
Public address mapping table:
Used 2
Subscriber address table:
Used 2
Max 65536
Out:
Translated packets ~12
bytes ~336
Did not match CGNAT policy 0
Untranslatable packets 0
Hairpinned packets 0
In:
Translated packets ~0
bytes ~0
Unknown source addr or port ~0
Did not match CGNAT session 0
Did not match CGNAT pool 0
Session hash tables:
Created 1
Destroyed 0
PCP sessions created 0
PCP errors 0
Memory allocation failures 0
Resource limitation failures 0
Thread contention errors 0
Packet buffer errors 0
|
Counts marked with a tilde (~) are approximate. These are collated from each CGNAT policy every time the "show cgnat summary" command is run. The policy counts are only updated approximately every 20 secs from similar counts in the subscriber data structures.
Sessions | |
|---|---|
Active sessions | Active sessions. An original global atomic counter. This is not simply 'sessions created less sessions destroyed'. |
Sessions created | Sum of 'sessions created' counts on all CGNAT policies. |
Sessions destroyed | Sum of 'sessions destroyed' counts on all CGNAT policies. |
Active sub-sessions | Count of current sub-sessions. An original global atomic counter. This is not simply 'sub-sessions created less sub-sessions destroyed'. |
Sub-sessions created | Sum of 'sub-sessions created' counts on all CGNAT policies. |
Sub-sessions destroyed | Sum of 'sub-sessions destroyed' counts on all CGNAT policies. |
Maximum table size | Configurable maximum session table size. (This value applies to sessions, not sessions plus sub-sessions) |
Table full | Set when the 'active sessions' count exceeds 'Maximum table size'. |
Public address mapping table | |
Used | Active public addresses in use by CGNAT. An original global atomic counter. There is no maximum value for this table. Its size is constrained by the number of addresses in the NAT pool used by CGNAT. |
Subscriber address table | |
Used | Active CGNAT subscribers. An original global atomic counter. |
Max | Maximum CGNAT subscribers allowed in the subscriber address table. Configurable. Default 64k. |
Out | |
Translated packets | Count of outbound packets translated. Sum of 'Out packets' counts on all CGNAT policies. |
Translated bytes | Total byte count of the outbound translated packets. (IP header and payload). Sum of 'Out bytes' counts on all CGNAT policies. |
Did not match CGNAT policy | Subscriber address did not match any CGNAT policy on the outbound interface. |
Untranslatable packets | Untranslatable packets. Sum of error counts 'Untranslatable IP protocol' and 'Untranslatable ICMP message' |
Hairpinned packets | Count of hairpinned packets. |
In | |
Translated packets | Count of inbound packets translated. Sum of 'In packets' counts on all CGNAT policies. |
Translated bytes | Total byte count of the inbound translated packets. (IP header and payload). Sum of 'In bytes' counts on all CGNAT policies. |
Unknown source addr or port | Sum of 'In, unknown source' counts on all CGNAT policies. Counts inbound packets whose dest address and port have matched a session, but whose source address and port have failed to match a sub-session. Only applies when sub-sessions are being recorded for this session. |
Did not match CGNAT session | Destination address, port and protocol did not match a CGNAT session and the destination address does not match a CGNAT public address. |
Did not match CGNAT pool | Destination address does not match a CGNAT public address. |
Session Hash Tables | |
Created | A hash table may be created for each main session in order to store sub-sessions if:
Creation and maintenance of these tables is an expensive operation. If many hash tables are being created and destroyed then performance may degrade somewhat. These counters are designed to provide some indication that this may be occurring. Typically multiple sub-sessions per main session will occur with peer-to-peer gaming. |
Destroyed | |
PCP sessions created | Sessions created by the Port Control Protocol |
PCP errors | Failures when trying to create a PCP session. |
Memory allocation failures | Sum of all memory allocation failures. Use "show cgnat error" to see details. |
Resource limitation failures | Sum of all resource limitation failures. Use "show cgnat error" to see details. |
Thread contention errors | Sum of all thread contention errors. Use "show cgnat error" to see details. |
Packet buffer errors | Sum of all packet buffer errors. Use "show cgnat error" to see details. |
show cgnat error
$ show cgnat error
CGNAT Global Counts In Out
Unable to translate packet:
Subscriber address did not match a CGNAT policy 0 0
Packet did not match a CGNAT session 0 0
Destination address did not match CGNAT pool 0 0
CGNAT bypassed by SNAT-ALG packets 0 0
Untranslatable IP protocol 0 0
Untranslatable ICMP message 0 0
Resource limitations:
Subscriber port-block limit 0 0
No free port-blocks on selected public address 0 0
No free public addresses in NAT pool 0 0
Subscriber table full 0 0
Session table full 0 0
Dest session table full 0 0
Memory allocation failures:
Failed to allocate session 0 0
Failed to allocate destination session 0 0
Failed to allocate port block 0 0
Failed to allocate public address 0 0
Failed to allocate subscriber address 0 0
Thread contention errors:
Lost race to insert session into table 0 0
Lost race to insert destination session into table 0 0
Subscriber address destroyed while waiting for lock 0 0
Packet buffer errors:
IP header not available in message buffer 0 0
L4 header not available in message buffer 0 0
Prepare message buffer for header change failed 0 0
PCP errors: .
PCP invalid or missing argument 0 0
PCP public address and port not available 0 0
Other:
ICMP Echo Request for CGNAT public address 0 0
Unknown 0 0 |
N/A = Means that count is not applicable in this direction
OK = Means that count is applicable in this direction
Unable to translate packet: | In | Out | Comment |
|---|---|---|---|
Subscriber address did not match a CGNAT policy | N/A | OK | Subscriber address did not match a CGNAT policy |
Packet did not match a CGNAT session | OK | N/A | Inbound pkts that do not match a CGNAT session and whose dest addr does match a CGNAT policy. These are dropped. |
Destination address did not match CGNAT pool | OK | N/A | Inbound pkts that do not match a CGNAT session and whose dest addr does not match a CGNAT policy. These are forwarded untranslated. |
CGNAT bypassed by SNAT-ALG packets | N/A | OK | Pkt do not match a CGNAT session and does not match a CGNAT policy and SNAT is cfgd and pkt matches an ALG session or tuple. Pkt is forwarded unchanged. |
Untranslatable IP protocol | OK | OK | Protocol is not one of the following: TCP, UDP, UDP-lite, DCCP or ICMP |
Untranslatable ICMP message | OK | OK | ICMP type is not one of: echo-req, echo-reply, dest unreach, redirect, time-exceeded, or parameter-problem |
Resource limitations: | |||
Subscriber port-block limit | N/A | OK | CGNAT policy config contains the max number of port-blocks that anyone subscriber may use at any one time. Once this is reached, further mapping requests will fail, and this count will increment. |
No free port-blocks on the selected public address | N/A | OK | Counts mapping failures that occur when a subscriber is within his port-block limit but there are no free port-blocks on the paired public address. This can occur when 1. the max port-blocks per subscriber and port-block size equate to more than 64512 ports, or 2. More than one subscriber is using the same public address, which will occur if there are totally unused public addresses. |
No free public addresses in NAT pool | N/A | OK | All port-blocks on all public addresses are in use. This only affects new subscribers, i.e. subscribers that do not already have a paired public address. |
Subscriber table full | N/A | OK | Affects new subscribers. The size of the subscriber hash table is configurable. Default is 64k. |
Session table full | N/A | OK | Max number of main sessions (tuple is: source/subscriber addr, source port, and protocol aka 3-tuple session). The size of the session table is configurable. Default is 32m. |
Dest session table full | N/A | OK | Max number of destination records, or sub-sessions, per main session. Configurable. Default is 64. The tuple is a destination address and port. |
Memory allocation failures: | Note that there can be a delay between a chunk of memory no longer being visible in a show output and the memory for that chunk actually being freed. Background garbage collectors and RCU callbacks mean this delay can be in the order of tens of seconds. | ||
Failed to allocate session | N/A | OK | |
Failed to allocate destination session | N/A | OK | |
Failed to allocate port block | N/A | OK | |
Failed to allocate public address | N/A | OK | |
Failed to allocate subscriber address | N/A | OK | |
Thread contention errors: | |||
Lost race to insert session into table | N/A | OK | This occurs if two forwarding threads both try and create an identical session (src addr, src port, and protocol) at the same time. The first to be added to the session table 'wins' the race. The losing thread will drop the packet and release the mapping it had previously obtained. |
Lost race to insert destination session into table | N/A | OK | This occurs if two forwarding threads both try and create an identical sub-session (dest addr, dest port) on the same main session at the same time. The first to be added to the sub-session table 'wins' the race. The losing thread will drop the packet. |
Public address destroyed while waiting for lock | N/A | N/A | No longer applicable. |
Subscriber address destroyed while waiting for lock | N/A | OK | This may occur if two forwarding threads try and create new mappings for the same subscriber while at the same time the subscriber table is being cleared. This is unlikely ever to occur. |
Packet buffer errors: | |||
IP header not available in message buffer | OK | OK | Pkt does not contain all of the IP header. (Note that the system reassembles IP fragments before CGNAT sees the pkt). |
L4 header not available in message buffer | OK | OK | Pkt does not contain all of the layer 4 header (e.g. TCP, UDP etc.). This can also apply to pkts embedded within ICMP error messages. i.e. we will attempt to translate any such embedded pkts if there is enough of the pkt to allow us to do so. |
Prepare message buffer for header change failed | OK | OK | It is possible for the system to receive a pkt into multiple pkt buffers. CGNAT will detect this and attempt to coalesce these such that the l3 and l4 headers are in the same pkt buffer. If this fails then the pkt is dropped. This typically happens when there are no pkt buffers in the buffer pool. |
Cannot advance beyond end of message buffer | N/A | N/A | No longer applicable. |
PCP errors: | In/Out direction is meaningless for PCP. We store any errors in the 'Out' counter for convenience. | ||
PCP invalid or missing argument | N/A | OK | |
PCP public address and port not available | N/A | OK | |
Other: | |||
ICMP Echo Request for CGNAT public address | OK | N/A | An ICMP echo request has been sent to a CGNAT public address, and no CGNAT session exists for that flow. Reply with an ICMP echo-reply and drop the request pkt. This is not actually an error. No translation takes place. |
Unknown | Should never occur. |
show nat pool
v@vm-rtr2-1:~$ show nat pool
NAT Pool POOL1
Active Yes
Type cgnat
User count 1
User addresses 29
Addresses:
Address pooling paired
Address allocation round-robin
Address count 79
Contention ratio 1:2.72
Address Ranges:
RANGE2 (prefix)
Range 10.10.3.64 - 10.10.3.79
Address count 16
Prefix: 10.10.3.64/28
RANGE1 (range)
Range 10.10.3.1 - 10.10.3.63
Address count 63
Last Allocated Address:
TCP none
UDP 10.10.3.65
Other none
Ports:
Port allocation sequential
Port range 1024-65535
Port count 64512
Port-block size 128
Max port-blocks per user 8
Logging:
Log port-block allocation Yes
Log all No
Translation Mappings:
Active 2
Total requests 2
Ok 2
Failed 0
Port Block Allocation:
Active 2
Total 2
Total Freed 0
Total Failures 0
Failures exceeding max 0 |
Active | Yes/No. A NAT pool may become inactive momentarily as it is reconfigured or deleted |
Type | Always 'cgnat' |
User count |