Network monitoring and analysis in OCI using VTAP and OpenSearch

Published in

Oracle Developers

11 min readJun 30, 2022

In this article, we’ll learn a bit about two new features/service in OCI: Virtual Test Access Point (VTAP) and OpenSearch. To learn more about VTAP, read Misha Kasvin’s blog post or Ben Woltz’s example. But if you’re in a hurry, I’ll summarize it here for you: VTAP allows you to copy (or mirror) network traffic at a specific point in your network, filter the traffic you are interested in and send it to a network analytics tool for further analysis.

A VTAP consists of 3 components:

A source
One or more capture filters which allows you to focus on the desired type of network traffic you want to mirror
A target

To illustrate this with an example, we’ll capture an incoming traffic using VTAP and plug the captured network data into OCI’s OpenSearch service. Conceptually, this is what we’ll try to do:

Our workload is a micro-service that runs on OKE and connects to a DB Service for persistence.
It is publicly accessible via an HTTP Load Balancer which is instantiated by an Ingress Controller.
We will set up a VTAP on the HTTP Load Balancer to mirror the network traffic arriving on the HTTP Load Balancer to a VM running Wireshark. In a future article, we’ll also add VTAPs on the worker nodes and DB System but let’s not get ahead of ourselves.
On the Wireshark VM, we’ll capture the mirrored traffic, write it to a file and then ship it to an OpenSearch
An admin user will access the OpenSearch Dashboard to analyse the traffic

Setting up the infrastructure

In order to achieve the above, we need to set up the following infrastructure:

A private OKE cluster with an ingress controller deployed
Private worker nodes running a simple microservice. For this article, running the ACME website will be sufficient.
A private network load balancer that will receive the mirrored traffic from the VTAP and ship it to a host running Wireshark
A private OpenSearch service
A bastion host or you can also use the OCI Bastion Service to access the Wireshark VM and OpenSearch

I’m using terraform-oci-oke to create my cluster. In terms of additional security rules, we also need to add the following additional rules in NSGs except for OpenSearch for which we’ll use a security list:

You can use Terraform to add them as extensions to additional NSGs or security lists.

Deploy the Ingress Controller

We’ll use ingress-nginx as the Ingress Controller:

helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginxhelm show values ingress-nginx/ingress-nginx > nginx.yaml

Edit the nginx.yaml and add the following service annotations:

service.beta.kubernetes.io/oci-load-balancer-security-list-management-mode: "None"
oci.oraclecloud.com/oci-network-security-groups: "<public-lb-nsg-id>"

Dry run the installation to verify you’ve set your service annotations correctly:

helm install --dry-run nginx ingress-nginx/ingress-nginx -f nginx.yaml

If you’ve done this well, you can proceed with the installation:

helm install nginx ingress-nginx/ingress-nginx -n ingress-nginx --create-namespace -f nginx.yaml

This will create a public HTTP Load Balancer.

Deploy the ACME website service (optional)

We’ll now deploy ACME website service. It’s a simple website that runs a single page and is ideal for testing deployment.

kubectl apply -f https://raw.githubusercontent.com/hyder/okesamples/master/ingresscontrollers/acme/acme-website.yaml

Create an ingress, making sure you set a correct value for the host parameter below:

kubectl apply -f - <<EOF
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: acme-website
  annotations:
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
  - host: f.q.d.n
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: acme-website
            port:
              number: 80
EOF

Then follow the steps in this previous article to add a DNS record. Test your service in a browser/curl:

Create OpenSearch Cluster

1 thing to note is that OpenSearch does not yet use NSGs so you’ll have to either create a security list and associate it with the opensearch subnet.

Once the cluster is provisioned, you can retrieve the endpoint on the details page:

Note there are 2 API endpoints:

The OpenSearch API endpoint
The OpenSearch Dashboard API endpoint

Similarly, there are 2 private IP addresses:

The Private IP which is actually the API endpoint IP address
The OpenSearch Dashboard private IP

We’ll be using both, so remember where to obtain them.

Deploy a Wireshark VM

Create a compute instance using the Ubuntu Platform image, deploy it to the private operator subnet and also ensure you’ve selected the operator NSG so you can ssh to it through the bastion host.

Once provisioned, ssh to the Wireshark VM and install tshark, a terminal-based Wireshark:

ssh -i /path/private/key -J opc@<bastion-public-ip> ubuntu@<wireshark-private-ip>sudo apt update && sudo apt install -y tshark

When prompted to allow non-superusers to capture packets, select “Yes”.

Give the ubuntu user permission to use Wireshark:

sudo usermod -aG wireshark $USER

Logout of your terminal session and log back in and check if you can run tshark without any problems:

ubuntu@tap-wireshark:~$ tshark -i ens3
Capturing on 'ens3'
 ** (tshark:3266) 04:46:41.473914 [Main MESSAGE] -- Capture started.
 ** (tshark:3266) 04:46:41.473964 [Main MESSAGE] -- File: "/tmp/wireshark_ens3FKTJO1.pcapng"
    1 0.000000000    10.0.0.11 → 10.0.0.4     SSH 286 Server: Encrypted packet (len=220)
    2 0.000167397     10.0.0.4 → 10.0.0.11    TCP 66 42930 → 22 [ACK] Seq=1 Ack=221 Win=463 Len=0 TSval=1035716911 TSecr=861840753

In the OCI Console, navigate to the Wireshark instance page and click on “Attached VNICs”. We’ll create a secondary VNIC in the opensearch subnet to send our traffic data from the network load balancer exclusively to this. Doing so helps us isolate the traffic data from others e.g. ssh to the instance itself. When creating the VNIC, also select the wireshark NSG and ensure you leave the “Skip source/destination check” unchecked.

At this point, the VNIC is created but not configured. Let’s download and run the script to configure the second VNIC:

curl https://docs.oracle.com/en-us/iaas/Content/Resources/Assets/secondary_vnic_all_configure.sh -Ochmod +x secondary_vnic_all_configure.sh sudo ./secondary_vnic_all_configure.sh -c

You should see something like the following:

Info: adding IP config for VNIC MAC 02:00:17:00:c9:c6 with id ocid1.vnic......
Info: added IP address 10.0.2.135 on interface ens5 with MTU 9000
Info: added rule for routing from 10.0.2.135 lookup ort1 with default via 10.0.2.129

Now check that the new network interface has an IP Address:

ubuntu@tap-wireshark:~$ ip address show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 02:00:17:00:e1:d6 brd ff:ff:ff:ff:ff:ff
    altname enp0s3
    inet 10.0.0.11/29 metric 100 brd 10.0.0.15 scope global ens3
       valid_lft forever preferred_lft forever
    inet6 fe80::17ff:fe00:e1d6/64 scope link
       valid_lft forever preferred_lft forever
3: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc fq_codel state UP group default qlen 1000
    link/ether 02:00:17:00:c9:c6 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    inet 10.0.2.135/27 scope global ens5
       valid_lft forever preferred_lft forever
    inet6 fe80::17ff:fe00:c9c6/64 scope link
       valid_lft forever preferred_lft forever

Excellent!

Our OpenSearch cluster should be ready so we can check if we can connect to the OpenSearch cluster from the Wireshark VM. Copy the OpenSearch API Endpoint from the cluster detail page, replace the bold below with it and run a curl:

curl -k https://<OpenSearch API Endpoint>:9200{
  "name" : "opensearch-master-2",
  "cluster_name" : "opensearch",
  "cluster_uuid" : "EB998EMAQSuE37d71g8VcA",
  "version" : {
    "distribution" : "opensearch",
    "number" : "1.2.4",
    "build_type" : "tar",
    "build_hash" : "f0ac9204ab08d677ebcd4dfad4b66ff159991f46",
    "build_date" : "2022-03-14T22:05:06.345077Z",
    "build_snapshot" : false,
    "lucene_version" : "8.10.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "The OpenSearch Project: https://opensearch.org/"
}

Finally, we must open the UDP port on 4789 in ufw to allow the network data from the Network Load Balancer:

sudo ufw allow from 10.0.2.0/27 to any port 22 proto tcp
sudo ufw allow from 10.0.2.0/27 to any port 4789 proto udpsudo ufw enable

List the open ports to make sure:

ubuntu@wireshark:~$ sudo ufw status
Status: activeTo                         Action      From
--                         ------      ----
22/tcp                     ALLOW       10.0.2.0/27
4789/udp                   ALLOW       10.0.2.0/27

Create the Network Load Balancer for VTAP

Let’s now create a Network Load Balancer before we create the vtap. Navigate to Networking > Load Balancers and create a network Load Balancer:

On the listener, select UDP:

On the backend set page, ensure that both the load balancer and the backends option for preserving source/destination Header are unchecked:

You can then add the backends. On the backend page, select the Wireshark VM and ensure you select the IP address of the 2nd VNIC:

Selecting the correct VNIC for the wireshark backend

In the health check policy section, select ‘TCP’ on port 22:

We are now ready to create the VTAP.

Create the VTAP

Navigate to the VCN page and create a VTAP:

Also create a capture filter for http and https while you’re at it:

By default, the VTAP will be stopped so you need to start it. Now we need to test if we are able receive the mirrored traffic. On the Wireshark VM, let’s use tcpdump for a quick and dirty test:

$ sudo tcpdump -nn -i ens5 udp
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens5, link-type EN10MB (Ethernet), snapshot length 262144 bytes

Hit the http load balancer IP address with curl or with your browser and you should get a 404 on the response if you haven’t deployed the acme website. That’s ok. The ingress controller is just responding with a 404. However, on the tcpdump output, every time you hit the load balancer in your browser, you should see an output:

12:15:25.132960 IP 10.0.2.18.18066 > 10.0.2.15.4789: VXLAN, flags [I] (0x08), vni 129057
IP 123.123.123.123.58177 > 10.0.2.56.80: Flags [S], seq 79568112, win 64240, options [mss 1440,nop,wscale 8,nop,nop,sackOK], length 0
12:15:25.153063 IP 10.0.2.18.18066 > 10.0.2.15.4789: VXLAN, flags [I] (0x08), vni 129057
IP 123.123.123.123.58177 > 10.0.2.56.80: Flags [.], ack 1967181215, win 517, length 0
12:15:25.157493 IP 10.0.2.18.18066 > 10.0.2.15.4789: VXLAN, flags [I] (0x08), vni 129057
IP 123.123.123.123.58177 > 10.0.2.56.80: Flags [P.], seq 0:348, ack 1, win 517, length 348: HTTP: GET / HTTP/1.1
12:15:25.198446 IP 10.0.2.18.18066 > 10.0.2.15.4789: VXLAN, flags [I] (0x08), vni 129057
IP 123.123.123.123.58177 > 10.0.2.56.80: Flags [P.], seq 348:650, ack 280, win 516, length 302: HTTP: GET /favicon.ico HTTP/1.1
12:15:25.198446 IP 10.0.2.18.18066 > 10.0.2.15.4789: VXLAN, flags [I] (0x08), vni 129057
IP 123.123.123.123.58177 > 10.0.2.56.80: Flags [F.], seq 650, ack 280, win 516, length 0
12:15:25.219993 IP 10.0.2.18.18066 > 10.0.2.15.4789: VXLAN, flags [I] (0x08), vni 129057
IP 123.123.123.123.58177 > 10.0.2.56.80: Flags [.], ack 281, win 516, length 0

There are a few IP addresses here. Let’s unpack them:

10.0.2.28: private IP address of the NLB
10.0.2.15: private IP address of the secondary VNIC of the Wireshark VM
10.0.2.56: private IP address of the HTTP Load Balancer
123.123.123.123: my (fake) public IP address when accessing the load balancer.

Let’s capture the traffic with Wireshark for a bit of offline analysis:

ubuntu@tap-wireshark:~$ tshark -i ens5 -w http.pcap udp

scp the file to your laptop and let’s do a bit of analysis with Wireshark. Click on Statistics > Endpoints> Map > Open in browser. My colleague Rakesh Singh very helpfully used a private VPN to generate traffic from all over the world and this is the captured traffic visualized in Wireshark:

Access the OpenSearch Dashboards

Now that we know our network traffic is coming in nicely, we want to do some analysis using OpenSearch Dashboards.

First, we need to extract the data from pcap format so we can load it into OpenSearch. Wireshark has a few output options such as ek, json, jsonraw. Let’s convert to JSON so we can then upload it to OpenSearch:

tshark -T ek -j "http tcp ip" -P -V -r http.pcap > http.json

This is basically reading from the http.pcap file we saved earlier, filtering on the HTTP, IP and TCP protocols and saving the resulting JSON into the http.json file.

Once we’ve converted this into JSON, we can upload the data to OpenSearch:

curl -k -H "Content-Type: application/x-ndjson" -XPOST https://<opensearch_api_endpoint>:9200/_bulk --data-binary "@http.json"

Generate a mapping file to be loaded as a template for packets-* index in order to convert Wireshark types into elasticsearch types:

tshark -G elastic-mapping --elastic-mapping-filter ip > template.json

As before, upload the template mapping to OpenSearch:

curl -k -H "Content-Type: application/x-ndjson" -XPOST https://<opensearch_api_endpoint>:9200/_template/packets?include_type_name=true --data-binary "@template.json"

Let’s now access the OpenSearch dashboard via the bastion:

ssh -C -t -L 127.0.0.1:5601:10.0.2.156:5601 -L 127.0.0.1:9200:10.0.2.158:9200 opc@<bastion_ip>

The first IP address (10.0.2.156) above is the OpenSearch Dashboard private IP address whereas the 2nd IP address (10.0.2.158) is the OpenSearch private IP. You need to create a tunnel to the right target with the right port. Once created, you can access the OpenSearch Dashboard locally: https://localhost:5601

Click on the left menu > Stack Management:

Then, click on Index Patterns > Create index pattern. In index pattern name, type packets-* and click on ‘Next step’:

Select ‘timestamp’ as the time field and create the index pattern.

Now click on the menu again, then click on Discover. You should be able to see your captured traffic in OpenSearch Dashboard:

From here, you can create your own nice visualizations and dashboards for further analysis.

Summary

This post introduces you to:

a new feature of VCN — Virtual Test Access Points aka VTAP
how to configure the NSGs and security rules for using VTAP
how to capture the mirrored traffic using Wireshark
converting the captured Wireshark trace to a JSON format
pushing the converted trace to an OCI OpenSearch Dashboard

In this exercise, we are capturing the traffic from 1 source only (the HTTP Load Balancer). We are also doing a few steps manually. In a future article, we’ll try to capture multiple traces simultaneously and automate the upload of the trace.

I hope you find this post useful. I would like to thank my colleagues Shaun Levey, Avi Miller, Andrea Marchesini and Rakesh Singh for their timely contributions and most helpful suggestions to this post.

Come on by and discuss in our public Slack!