Cross Cluster Communications Made Easy with the Netmaker Operator

Cross-Cluster Communications with Netmaker 

Establishing Kubernetes cross-cluster communications is challenging for many organizations. Netmaker makes it simple, without complex full-cluster implementations. Netmaker utilizes two custom objects, the Ingress Service Proxy and the Egress Service Proxy, to establish inbound and outbound connectivity between Kubernetes clusters. 

By deploying Netmaker’s an Ingress Service Proxy on Cluster A, and a Netmaker’s Egress Service Proxy on Cluster B, a service defined on Cluster A becomes consumable from Cluster B, a model can be applied to any number of services in either direction, making them accessible cross-cluster using standard Service DNS.

How It Works

By combining the use of Ingress and Egress Proxy Service Objects, you can expose any arbitrary number of services across any number of clusters securely over a private network.

The end result is a standard Service object on Cluster A which can access a standard Service object on Cluster B.

Ingress Proxy (Cluster A)

After creating a Service object on Cluster A with Netmaker’s Ingress annotations, the following occurs in the background::

1. The Netmaker Operator creates a netclient proxy sidecar that connects to the Netmaker network.
2. The Service Receives a private Netmaker IP address. 
3. The proxy listens on that IP for incoming connections.
4. The proxy forwards traffic to the specified Service.

Egress Proxy (Cluster B)

After creating a Service object on Cluster B with Netmaker’s Egress annotations, the following occurs in the background:

1. The Netmaker Operator creates a netclient sidecar that connects to the Netmaker network.
2. The proxy listens to incoming traffic on the created Service.
3. The proxy forwards traffic from Cluster B to the Ingress proxy in Cluster A.
4. The Ingress proxy on Cluster A forwards traffic to the target destination. 

Step-by-Step Setup with Example

In this example, we will deploy a Database in Cluster A, and consume that Database from an App Server running in Cluster B, using the Netmaker Operator, the Ingress Proxy Service, and the Egress Proxy Service.

This example assumes you have already set up Netmaker, and deployed the Netmaker Operator on two Kubernetes clusters. To access the example YAML files, visit https://github.com/gravitl/netmaker-k8s-ops 

Step 1: Deploy a Database in Cluster A

Deploy a database and expose it to the Netmaker network using an ingress proxy Service:

kubectl apply -f examples/cross-cluster-database-example.yaml --context=cluster-a

Key Configuration:

annotations:

  netmaker.io/ingress: "enabled"

  netmaker.io/ingress-dns-name: "postgres.cluster-a.netmaker.internal"

Step 2: Deploy Application Server in Cluster B

Deploy an application server and grant it to access the database via an egress proxy Service:

# Apply the server example to Cluster B

kubectl apply -f examples/cross-cluster-server-example.yaml --context=cluster-b

Key Configuration:

annotations:

  netmaker.io/egress: "enabled"

  netmaker.io/egress-target-dns: "postgres.cluster-a.netmaker.internal"

Step 3: Verify Application Can Connect

‍

Confirm egress proxy pod is deployed:

kubectl get pods -l app=netmaker-egress-proxy --context=cluster-b

‍

Test database connection from Cluster B:

kubectl exec -it db-test-client -- psql -h postgres-cluster-a.default.svc.cluster.local -U postgres -d mydb --context=cluster-b

‍

Test application server endpoints:

kubectl exec -it app-test-client -- curl http://app-server.default.svc.cluster.local/health --context=cluster-b

kubectl exec -it app-test-client -- curl http://app-server.default.svc.cluster.local/db-test --context=cluster-b

‍