Blog

Kubernetes and NSX-T Predefined Rules

Kubernetes and NSX-T Predefined Rules

Micro-segmentation is a crucial part of any Kubernetes production environment. With Kubernetes and NSX-T, micro-segmentation is realized as NSX-T Distributed Firewall Rules. There are two approaches to create the rules. One, and by far the most popular, is to create rules as Kubernetes NetworkPolicies. This approach has the creation and deletion of the rules driven by Kubernetes. This approach certainly has its benefits and as I stated is by far the most popular. However, there is a second approach that might fit a few use-cases and will be covered here. This second approach, and not as well documented, is to create predefined rules in NSX-T that have a dynamic membership criteria. These approaches could be used together or separately as a security policy solution. Lets walk through an example of the second approach and see if it might fit a use-case for your organization.

Assumptions

1) NSX-T 2.4.x or 2.5.x installed and operational as the SDN for Kubernetes
2) A Kubernetes environment such as Pivotal Container Service (PKS) 1.7 installed and operational
3) Deny all DFW rule in place

Micro-segmentation on Kubernetes and NSX-T Overview

A quick review of how Kubernetes and NSX-T work together. The integration between Kubernetes and NSX-T is performed through the NCP (NSX-T Container Plugin). Out of the box Kubernetes has no controller to handle the NetworkPolicies. The NCP listens on the Kubernetes cluster and makes REST calls back to NSX-T for various functions like creating Distributed Firewall Rules, or what will be demonstrated in this blog adding members to a predefined group. The dynamic membership between NSX-T Distributed Firewall Rules and Kubernetes is defined using Scope and Tag on the NSX-T side and Labels on the Kubernetes side. This approach could be used to define access to generic infrastructure services like DNS, NTP, etc.

NSX-T Predefined Rule

As described above, an NSX-T firewall rule can be configured that allows for dynamic membership from Kubernetes resources. The link between NSX-T and a Kubernetes resource is the NSX-T scope/tag construct and the Kubernetes label. Lets demonstrate this by creating a firewall rule that gives any member of a NSGroup access to a set of DNS servers that have the INFRA-SERVICES=TRUE key/value.

NSGroup

Here we will define a group to be used for the dynamic membership (ie..the source). The Kubernetes resources will be dynamically added to this group when the appropriate Kubernetes label is applied.
  1. Login to NSX-T Manager.
  2. Navigate to Advanced Networking and Security > Inventory > Groups
  3. Select ‘+Add’ group
  4. On the General tab enter the Name ‘INFRA-SERVICES’
  5. On the Membership Criteria tab
    a. Select ‘Logical Port’ for the criteria
    b. Select tag equals ‘TRUE’
    c. Select scope equals INFRA-SERVICES
  6. Select Add

IP Set

Here we will define an IP Set that will contain the destination DNS servers. For simplicity lets just use old faithful, 8.8.8.8 and 8.8.4.4.
  1. Login to NSX-T Manager.
  2. Navigate to Advanced Networking and Security > Inventory > Groups > IP Sets
  3. Select ‘+Add’ IP Set
  4. On the Genral tab enter the name ‘DNS-IPSET’
  5. On the Members tab add ‘8.8.8.8’,8.8.4.4′
  6. Select Add

Firewall Rule

Now lets define a firewall rule that allows the source group, INFRA-SERVICES, to have access to the destination IP-Set, DNS-IPSET.
  1. Login to NSX-T Manager.
  2. Navigate to Advanced Networking and Security > Security > Distributed Firewall
  3. Select ‘Default Layer3 Section’ and click Add Rule
  4. Select Name and enter ‘dns-rule’
  5. Select widget next to Source and then Edit Rule Source
  6. On the Container Object tab
    a. Select NS Group as Object Type
    b. Select ‘INFRA-SERVICES’ in the available list
    c. Move INFRA-SERVICES to the Selected Objects panel
    d. Select OK
  7. Select widget next to Destination and then Edit Rule Destination
  8. On the Container Object tab
    a. Select IP Set as Object Type
    b. Select ‘DNS-IPSET’ in the available list
    c. Move IDNS-IPSET to the Selected Objects panel
    d. Select OK
  9. Leave remaining fields with defaults and select Publish

Kubernetes Labels

Now that we have our firewall rule in place, lets see how to make use of it from Kubernetes. As mentioned earlier, NSX-T scope and tag equates to Kubernetes labels. In order to allow pods to have access to the DNS servers listed in the IP Set, it is simply a matter of adding the correct label. The steps below demonstrate adding the label and testing access to the DNS servers.

Test Pod

Here is a simple busybox test pod we can use to performa ping test. Note it currently does not have the label INFRA-SERVICES=TRUE. Without this label, assuming all other deny rules are in place, the first ping test will fail. 
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: test-box
  name: test-box
spec:
  containers:
  - image: busybox
    name: test-box
    command: ["sh", "-c", "sleep 1h"]
Deploy the pod 
$ kubectl apply -f test-box.yaml
Open a shell to the container and execute a ping test. 
$ kubectl exec -it test-box -- /bin/sh  
/ # ping 8.8.8.8  
PING 8.8.8.8 (8.8.8.8): 56 data bytes  
^C  
--- 8.8.8.8 ping statistics ---  
13 packets transmitted, 0 packets received, 100% packet loss  
/ # exit

Allow Permissions

Now in order to grant permissions to the test pod, add the label, ‘INFRA-SERVICES=TRUE’. By adding the label the NCP plugin will make a call to NSX-T and the pod will be dynamically added as a member to the INFRA-SERVICES group. 
$ kubectl label pod test-box INFRA-SERVICES=TRUE  
pod/test-box labeled  
$ kubectl get pod test-box --show-labels  
NAME       READY   STATUS    RESTARTS   AGE   LABELS  
test-box   1/1     Running   0          28m   INFRA-SERVICES=TRUE,run=test-box
Open a shell to the container and execute a ping test. 
$ kubectl exec -it test-box -- /bin/sh  
/ # ping 8.8.8.8  
PING 8.8.8.8 (8.8.8.8): 56 data bytes  
64 bytes from 8.8.8.8: seq=0 ttl=108 time=7.778 ms  
64 bytes from 8.8.8.8: seq=1 ttl=108 time=4.049 ms  
64 bytes from 8.8.8.8: seq=1 ttl=108 time=4.246 ms (DUP!)  
64 bytes from 8.8.8.8: seq=1 ttl=108 time=4.914 ms (DUP!)  
...  
^C  
--- 8.8.8.8 ping statistics ---  
12 packets transmitted, 12 packets received, 33 duplicates, 0% packet loss  
round-trip min/avg/max = 3.989/4.737/8.100 ms  
/ #

Conclusion

In summary, this post has demonstrated a second approach to defining secuirty policies between Kubernetes and NSX-T. The predefined rules allow a network administrator to create generic rules that could give access to generic services from an application running on Kubernetes.
Shout out to my colleague jwelsh who showed me this feature. Hope this has been interesting. Happy Kubing.
Tags: , , ,