By default resources, you launch on the cloud (EC2, RDS, and others) cannot communicate with your local networks like home or office. To allow this you can create a Site-to-Site VPN. This VPN connection will be established between your router and AWS VPC.

Creating VPN between networks is well documented. However, you can have issues configuring your home router. At home, I have a Unifi Dream Machine router, which is designed for small networks, but has features that match advanced routers for offices. One of them is a Site-to-Site VPN using the IPSec protocol.

Setup VPN on AWS ☁️

The first step is to create a VPN connection on AWS. For this blog I will use… the default VPC 🙂. To configure it AWS requires to define 3 components: Customer Gateway, Virtual Private Gateway and VPN connection.

The Customer Gateway is basically just an entity that holds the information about your home router - public IP. The Virtual Private Gateway is the virtual entity on the VPC side, that allows configuring routing to that gateway. That VPG is attached to the VPN. So the last entity is the VPN connection which brings it all together and establishes the VPN tunnels between your home or office and VPC.

Create base components 🏗️

To start your VPN connection start by defining Customer Gateway. Go to the VPC tab, find the Customer Gateway panel and click the button to Create. The required field is IP address. Write here your public IP address where your router is running. Also, it is good to name this gateway. I named mine home.

To quickly check you public IP you can open https://ifconfig.co

The next step is to create a Virtual Private Gateway. Go to the Virtual Private Gateway panel and click Create. It just asks for a name. Let’s name it also home. The VPG state will be detached. I will come back to it later.

Create a VPN connection 🔒

This is the time to start defining VPN. Open the Site-to-Site VPN connection panel and click Create VPN Connection. The form will have 3 panels: details and tunnel options.

Details start from defining the gateway on the VPC side. Choose Virtual private gateway and in the form select your VPG.

Next select Customer gateway. Here you define with which router the VPN will be established.

There is the last configuration to set: routing. It is a section where you can define which local networks the VPN will be used for. On the screenshot, I marked this as a point 1.

AWS allows for two options: dynamic routing based on BGP, and statically defined. In my case, I am using static. In the prefixes, I am putting my local network prefixes(like 192.168.1.0/24). You are allowed to put multiple networks here.

Configure Tunnel Options ⚙️

Tunnel options allow defining the IPSec parameters. AWS creates 2 tunnels, to which you can connect and each of them you can configure differently. Or the same 🙂.

What I am always choosing is:

• Encryption algorithms: AES-256 (for both phases)
• Integrity algorithms: SHA2-256, SHA2-384, SHA2-512
• DH groups: all above 14
• IKE Version: ikev2

Those parameters will be crucial for setting up our Unifi router.

When you finish these changes you can create the VPN and wait a few seconds. The VPN connection should be ready.

Configure VPC Routing 🛣️

Previously I mentioned that the Virtual Private Gateway state is not attached to any VPC. We can reassign the VPN connection between VPCs by changing attachments.

To attach go back to Virtual Private Gateway and select your VPG and in Actions, button find Attach to VPC. Select your VPC and your VPG should be available to configure routing on VPC.

On VPC the last thing to configure is routes. As we have assigned VPG, and networks from our home are known (configured on VPN with static routes), we can configure automated propagation of those rules to VPC route tables. To configure this perform:

1. Go to Route tables
2. Select routes assigned to VPC, or those which should have access to your home.
3. Select Actions button, and choose Edit route propagation,
4. Select Virtual private gateway

After a few seconds, the new route should be added As you can see, the last route is “Propagated”, and its target is my virtual private gateway.

Setup VPN on Unifi 🏠

Having configured AWS VPC, the last part is to configure our router. In my home, I have Unifi Dream Machine, with the latest software (Network 7.1).

To create a VPN connection:

• Go to Settings > Teleport & VPN,
• Scroll down to Site-to-Site VPN and click Create,

Start filling out the form. The Pre-Shared Key you could configure in Tunnel Options. If you have skipped this, go to the AWS VPN tab, and click Download Configuration. In this file you will find the PSK to fill in point 1 and the Remote IP Address (point 4). In the Remote Gateway/Subnets(point 3) put AWS VPC network addressing. In my case, it was 172.31.0.0/16.

To align encryption options with Tunnel Options on AWS, select Manual in Advanced configuration and customize. Configure your parameters according to how you have configured them on AWS. I recommend using AES-256 and higher DH Groups and use the above image as an example.

Last step - testing 🪛

To check if you have a working VPN connection create an EC2 instance on this VPC.

I have created an instance with IP 172.31.34.95. This instance has a Security Group rule allowing for All Traffic from my home network. When the VPN works and instance is up, a simple ping can prove that everything is configured:

$ ping 172.31.34.95 -c 5
PING 172.31.34.95 (172.31.34.95) 56(84) bytes of data.
64 bytes from 172.31.34.95: icmp_seq=1 ttl=63 time=38.1 ms
64 bytes from 172.31.34.95: icmp_seq=2 ttl=63 time=38.2 ms
64 bytes from 172.31.34.95: icmp_seq=3 ttl=63 time=36.4 ms
64 bytes from 172.31.34.95: icmp_seq=4 ttl=63 time=38.3 ms
64 bytes from 172.31.34.95: icmp_seq=5 ttl=63 time=38.1 ms

--- 172.31.34.95 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4005ms
rtt min/avg/max/mdev = 36.380/37.810/38.285/0.718 ms

If you are not sure that you are pinging EC2 take a look on ping response times. On local networks they are much lower.

Summary ☑️

Setting up VPN allows making your infrastructure secure. You don’t have to expose ports on the public internet to have access to cloud machines.

FAQ ❓

I have two network VLANs at home, should I configure some firewall rules if I don’t want to allow access from one of them?

Let’s assume you have two networks: home and guest. If the guest network should not have access to resources to VPN, on AWS VPN, in Static IP Prefixes configuration you have to set only home network subnet.

Another thing are Security Groups, where we define allowed networks. If you will not set too wide network range, then it will also block access.

Istio is a complex system. For the applications, the main component is the sidecar container Istio-Proxy, which proxies all traffic from all containers in Pod. And this can lead to some issues.

This post describes one of the most complicated problems I have encountered in my career.

The problem - Connection Reset 🐛

During Istio rollout on a huge system, with more than 40 different microservices, on a single endpoint, QA engineers found a bug. It was a POST endpoint, which was returning chunked data.

Istio was returning error 502, in logs an additional flag was visible: upstream_reset_before_response_started. The application logs confirmed that the result was correct.

In legacy Istio versions of the presented problem Istio were returning 503 error with UC flag.

Analyzing issue ⛏️

Let’s see the curl response and look at Istio-proxy logs:

kubectl exec -it curl-0 -- curl http://http-chunked:8080/wrong -v
< HTTP/1.1 502 Bad Gateway
< content-length: 87
< content-type: text/plain
< date: Sun, 24 Apr 2022 12:28:28 GMT
< server: istio-envoy
< x-envoy-decorator-operation: http-chunked.default.svc.cluster.local:8080/*
upstream connect error or disconnect/reset before headers. reset reason: protocol error

$ kubectl logs http-chunked-0 -c istio-proxy
[2022-04-24T12:23:37.047Z] "GET /wrong HTTP/1.1" 502 UPE upstream_reset_before_response_started{protocol_error} - "-" 0 87 1001 - "-" "curl/7.80.0" "3987a4cb-2e0e-4de6-af66-7e3447600c73" "http-chunked:8080" "10.244.0.17:8080" inbound|8080|| 127.0.0.6:39063 10.244.0.17:8080 10.244.0.14:35500 - default

Time for spying 🕵🏻‍♂️

To analyze the traffic we can use tcpdump and Wireshark. Istio-proxy runs as a sidecar, which routes whole incoming and outgoing traffic to pod through own proxy.

To sniff traffic there are 3 ways:

1. Running tcpdump in istio-proxy container,
2. Using kubectl plugin ksniff - a plugin to kubectl to dump packets from pod, github repo,
3. Adding additional container to pod, with root permission and tcpdump installed,

The first option will not work by default, because istio-proxy runs without root permission. The third is the backup if 1 and 2 would not work. Let’s try ksniff.

What is ksniff 🛠️

ksniff in three words is a plugin that:

• figures out what node is running pod with an app,
• deploys an own pod with an affinity to that node, bound to the host network,
• opens Wireshark on your laptop with a packet stream from the application.

Let’s execute it to sniff our application:

kubectl sniff http-chunked-0 -c istio-proxy -p -f '-i lo' -n default

Important parameters

• -p is a parameter to support sniffing even if the pod is non-privileged. See docs,
• -f '-i lo' passes filter to tcpdump, we want to sniff localhost interface inside the Pod.

If there is no issue, our system has Wireshark in PATH, ksniff should open a new window

Finding the root cause 🔎

Wireshark will continuously follow with new packet records. It makes it hard to figure out our particular call. We can use filters to help with searching. Knowing the request path, method, response code - we can use it to find our packet using filter:

http.request.uri == "/wrong"

It shows only a single packet, our request. Wireshark allows to show the whole TCP conversation:

• click right click on the packet,
• go to Conversation Filter,
• select TCP.

Wireshark will write a filter to show the whole communication between istio-proxy container and the application container!

Let’s see the above image. The first 3 records are the three-way handshake packets. Later is our GET request. The most interesting happens in the last two packets. Application container returns response HTTP 200 OK. istio-proxy then closes the connection with RST packet.

This is what we saw in the logs. The flag was upstream_reset_before_response_started{protocol_error}. But why? This still does not explain.

Swiss knife by Wireshark 🪛

It is hard to read the HTTP protocol from multiple packet bodies. But Wireshark also has a solution for that. We can see data from L7, the application one. In our case, it is the HTTP protocol.

Click with the right mouse on a single packet, go to the Follow tab, and select TCP Stream:

Now we can check what the request from istio-proxy looked like, and what was the response from the app. Do you have an idea from the above picture?

Look closer at the response, there is a double Transfer-Encoding header. One starts from uppercase, the second one does not.

Double transfer-encoding header - what does it mean❔

Searching over Istio issues I found this answer. The most important are the first 2 points:

1. two transfer-encoding: chunked is equivalent to transfer-encoding: chunked, chunked as per RFC,
2. transfer-encoding: chunked, chunked doesn’t have the same semantic as transfer-encoding: chunked

Why the response was taken as double-chunked? According to Transfer Codings in Section 4, transfer-coding names are case-insensitive.

Summary 📓

As you see, Istio stands as a guard 👮‍♂️ of the HTTP protocol. If the app is returning a double-chunked response, then Istio requires it, otherwise, it rejects processing the request. curl ignores this inconsistency.

This issue was one of the most difficult tasks, which I ever had :-)

Infrastructure to reproduce and example app 🏭

In Github repository I created example infrastructure to reproduce the problem.

Bootstrap of the infrastructure installs ArgoCD, Istio and the App. The sample app exposes two endpoints:

• /correct - endpoint, which creates a streamed response,
• /wrong - is doing same as above, but additionally it set value of the Transfer-Encoding header to Chunked(uppercase).

I would like to thank Przemysław for his help and for showing me how to use Wireshark efficiently during this issue.🤝🏻