A network connection failing in AWS does not always mean the server is down. An EC2 instance can be healthy, the application can be running, and the route to the destination can exist, yet a connection can still be rejected somewhere along the network path.
Two AWS features that frequently become part of this investigation are Security Groups and Network Access Control Lists (NACLs). Although both are used to control network traffic, they have different scopes, rule behaviour, and troubleshooting requirements.
Knowing where each control operates can significantly reduce the time required to diagnose connectivity problems.
Security Groups and NACLs Are Not the Same Firewall
The easiest way to understand the difference is to look at what each one protects.
A Security Group is associated with a network interface and controls traffic for the resource using that interface.
A Network ACL is associated with a subnet and controls traffic entering or leaving that subnet.
A simplified architecture looks like this:
Client
|
v
Route
|
v
Subnet
|
+—- Network ACL
|
v
EC2 Network Interface
|
+—- Security Group
|
v
Operating System
|
v
Application
Because these controls operate at different levels, a connection can pass one and still be stopped by the other.
How Security Groups Handle Traffic
Security Groups are commonly used to define which resources can communicate with an EC2 instance.
For example, suppose a web server listens on TCP port 443. Its security group could allow HTTPS traffic from a load balancer or another trusted network.
A rule might look like:
Protocol: TCP
Port: 443
Source: 10.10.1.0/24
Action: Allow
Security Groups are stateful. When an allowed connection is established, the return traffic does not require a separate rule simply to permit the response.
Security Groups use allow rules. There is no explicit deny rule that can be added to a security group.
This makes them particularly useful for controlling access to individual workloads.
How Network ACLs Handle Traffic
Network ACLs work at the subnet boundary rather than directly on an EC2 instance.
A NACL can contain both allow and deny rules. Each rule has a number, and AWS evaluates matching rules in ascending numerical order.
For example:
Rule 100 → Allow TCP 443 from 10.10.0.0/16
Rule 200 → Deny TCP 443 from 0.0.0.0/0
Traffic matching rule 100 is allowed before AWS reaches rule 200.
NACLs are stateless. Allowing traffic in one direction does not automatically permit the response in the opposite direction.
This distinction becomes important when troubleshooting TCP connections.
Scenario 1: The Security Group Is Missing the Application Port
Imagine an EC2 instance hosting an internal API on port 8080.
The server is running, but another EC2 instance receives:
Connection timed out
The first checks show that both servers are in the same VPC and that the route table contains the expected local route.
The destination security group contains rules for SSH and HTTPS but nothing for TCP 8080.
In this situation, the security group is the access-control layer preventing the connection.
The solution is to permit TCP 8080 from the appropriate source rather than opening the service to every IP address.
Scenario 2: The Security Group Allows Traffic, but the NACL Rejects It
Now consider the opposite situation.
The destination security group allows the required port, but the subnet’s NACL contains a deny rule affecting the client network.
The security group cannot override the NACL.
The traffic must satisfy both controls.
This produces an important troubleshooting rule:
An allow rule at one layer does not cancel a deny at another layer.
When a security group appears correct, move one level outward and inspect the NACL associated with the subnet.
Scenario 3: The Request Leaves, but the Response Does Not Return
Stateless NACL behaviour becomes particularly visible with TCP.
Suppose an EC2 instance initiates a connection to a service. The outbound request is permitted, but the response is blocked.
Because the NACL does not maintain connection state, the return traffic must satisfy the appropriate NACL rule in the opposite direction.
This is why NACL troubleshooting should never focus only on the initial request.
Check:
Source → Destination
Destination → Source
Both directions matter.
Security Groups simplify this situation because their stateful behavior allows return traffic for an established permitted connection.
Scenario 4: The Correct Security Group Is Being Checked on the Wrong Interface
Another common mistake is investigating a security group that is not actually controlling the traffic in question.
An EC2 instance can have more than one network interface, and security group associations are applied to network interfaces.
When troubleshooting, verify the actual interface involved:
EC2 Instance
|
+– ENI
|
+– Security Group A
+– Security Group B
Check the interface’s private IP address and attached security groups before modifying any rules.
This prevents unnecessary changes to unrelated security groups.
Scenario 5: NACL Rule Ordering Creates an Unexpected Denial
Consider this configuration:
Rule 90 → Allow TCP 443 from 10.0.0.0/16
Rule 100 → Deny TCP 443 from 10.0.0.0/16
The first matching rule determines the result.
The later deny rule does not override the earlier allow.
When investigating a NACL, therefore, examine the complete rule set rather than looking for a single rule that appears to allow the connection.
A Practical Investigation Process
When an EC2 connection fails, avoid immediately changing firewall rules. Work through the network path in sequence.
1. Identify Both Endpoints
Record:
- Source IP
- Destination IP
- Source subnet
- Destination subnet
- VPC IDs
- Destination port
- Protocol
This establishes exactly what traffic you are investigating.
2. Validate the Route
Check the route table associated with the source subnet.
Confirm that the destination address has a valid route and that the selected target is correct.
If the route is missing, neither a Security Group nor a NACL change will solve the problem.
3. Inspect the Destination NACL
Determine which NACL is attached to the destination subnet.
Review inbound rules and confirm that the connection is permitted.
Then check the source subnet’s NACL for the return path.
4. Inspect Security Groups
Check the security groups attached to the relevant network interface.
For an inbound connection, verify that the destination allows the source and port.
For outbound traffic, verify the source-side rules as well.
5. Check the Host
AWS networking may be working while the operating system blocks the connection.
On Linux, useful checks include:
ss -lntp
to determine whether the service is listening, and:
nft list rule-set
or:
iptables -L -n -v
to inspect host-level filtering.
6. Test the Actual Service
Testing the application port is more useful than relying exclusively on ICMP.
For example:
nc -vz 10.0.2.25 8080
For an HTTP service:
curl -I http://10.0.2.25:8080
These tests help distinguish a network filtering problem from an application problem.
Using VPC Flow Logs
When the configuration looks correct but traffic still fails, VPC Flow Logs can provide additional evidence.
Flow logs can show information such as source and destination addresses, ports, protocols, and whether observed traffic was accepted or rejected.
They are especially useful when investigating intermittent connectivity or determining whether traffic is reaching the expected network interface.
However, Flow Logs should be considered together with route tables, NACLs, Security Groups, host firewalls, and application logs.
A Simple Way to Remember the Difference
Think of the two controls this way:
Security Group
↓
“Can this resource communicate?”
Network ACL
↓
“Can this subnet accept or send this traffic?”
Security Groups provide stateful, resource-level filtering.
NACLs provide stateless, subnet-level filtering with ordered allow and deny rules.
Neither replaces the other. Both can participate in determining whether a connection succeeds.
Conclusion
Troubleshooting blocked traffic in AWS requires understanding how Security Groups and Network ACLs work together. Although both control network access, their different scopes and rule behaviour mean that a connection can be permitted at one layer and blocked at another.
Start by verifying the route between the source and destination, then inspect the NACLs associated with both subnets and the Security Groups attached to the relevant network interfaces. Pay particular attention to NACL rule ordering, stateless return traffic, and the specific ports and IP addresses involved. If AWS network controls appear correct, continue investigating host-level firewalls, listening services, and application logs.
Tools such as VPC Flow Logs, ss, nc, and curl can help narrow down where the connection is failing. Rather than making broad firewall changes, apply targeted rule adjustments and test connectivity after each change. A structured troubleshooting approach helps identify the actual source of blocked traffic, minimize unnecessary security changes, and maintain reliable communication across AWS workloads.
Facing issues?
Our technical support
engineers can solve it.
