Gloo Gateway mTLS mode

Secure the connection between the Gloo Gateway control plane and the gateway proxy with mutual TLS (mTLS).

About the control and data plane communication

Gloo Gateway and the Envoy gateway proxy communicate with each other by using the xDS protocol, which by default sends messages via plaintext. Because the Envoy configuration in the gateway proxy can contain sensitive data, you might want to encrypt the communication between the Gloo Gateway control plane and the gateway proxy, especially if these components run in separate clusters.

To secure the communication between the Gloo Gateway control and data plane, you can configure these components to use mutual TLS authentication (mTLS). This way, the control and data plane both must present a valid certificate to validate their identity before a connection can be established. After a connection is established, the xDS communication is automatically encrypted.

Cert generation job

Gloo Gateway comes with a built-in certGen job that, if enabled, automatically creates self-signed TLS certificates that the gloo and gateway proxy pods can use for their mTLS communication. The TLS certificates are stored in the gloo-mtls-certs Kubernetes secret. The gloo and gateway proxy pods are configured to mount this secret as a Kubernetes volume. You can enable the certGen job by using the following values in the Gloo Gateway Helm chart.

global:
  glooMtls:
    enabled: true
gateway:
  certGenJob:
    enabled: true

If you plan to bring your own TLS certificates to secure the communication, disable the certGen job and store your certificates in the gloo-mtls-certs Kubernetes secret in the gloo-system namespace. Leave the glooMtls setting enabled. Then, the gloo and gateway proxy automatically mount the gloo-mtls-certs Kubernetes secret that has your certificates.

Cert rotation cron job

You can rotate the TLS certificates by updating the gloo-mtls-certs secret. The Gloo Gateway SDS container is configured to monitor this secret and to automatically update the Envoy configuration if new certificates are detected.

If you use self-signed TLS certificates, you can automate the cert rotation process with a buit-in Gloo Gateway cron job. The cron job runs through several stages to generate and roll out the new TLS certificates. You have the option to instruct Gloo Gateway to wait between stages to ensure that your workloads have enough time to pick up certificate changes.

The job works as follows:

1. The cert rotation job creates new TLS credentials, including a Certificate Authority (CA) certificate that is used to sign the new server certificate and private key.
2. The new PEM-encoded CA certificate is added to the gloo-mtls-certs secret alongside the old CA certificate that is about to be rotated out, so that both CA certificates are accepted temporarily.
3. Gloo Gateway waits for the duration that is set in gateway.certGenJob.rotationDuration before continuing to the next step so that workloads in the cluster can pick up this change.
4. The old PEM-encoded server certificate and private key are replaced with the new server certificate and private key.
5. Gloo Gateway waits for the duration that is set in gateway.certGenJob.rotationDuration before continuing to the next step.
6. The old CA certificate is removed from the gloo-mtls-certs secret. All workloads now use the new TLS credentials.

To enable the cron job, set gateway.certGenJob.cron.enabled=true in your Helm values file and set up a schedule for your cron job. For example, you might use the following values to run the job every 5 minutes.

global:
  glooMtls:
    enabled: true
gateway:
  certGenJob:
    enabled: true
    runOnUpdate: true # Run the job during updates. 
    forceRotation: true # Create new certificates, even if the old ones are still valid. 
    rotationDuration: 65s # The time for Gloo Gateway to wait for all pods to pick up the certificate changes. 
    cron:
      enabled: true 
      schedule: "*/5 * * * *"

Before you begin

1. Follow the Get started guide to install Gloo Gateway, set up a gateway resource, and deploy the httpbin sample app.

2. Get the external address of the gateway and save it in an environment variable.

   export INGRESS_GW_ADDRESS=$(kubectl get svc -n gloo-system gloo-proxy-http -o jsonpath="{.status.loadBalancer.ingress[0]['hostname','ip']}")
   echo $INGRESS_GW_ADDRESS  

   kubectl port-forward deployment/gloo-proxy-http -n gloo-system 8080:8080

Enable mTLS

Use self-signed TLS certificates to secure the connection between the gloo and gateway proxy pods.

1. Get the Helm values for your current Gloo Gateway installation.

   helm get values gloo -n gloo-system -o yaml > gloo-gateway.yaml
   open gloo-gateway.yaml

2. Add the following values to the Helm value file to enable the Gloo Gateway mTLS mode with self-signed TLS certificates and automatic TLS certificate rotation.

   
   global: 
     glooMtls:
       enabled: true
   gateway:
     certGenJob:
       enabled: true
       runOnUpdate: true
       forceRotation: true
       cron:
         enabled: true
         schedule: "*/5 * * * *" 
       rotationDuration: 120s

   Value	Description
   global.glooMtls.enabled	Enable or disable the Gloo Gateway mTLS mode.
   certGenJob.enabled	Enable or disable the Gloo Gateway cert generation job that generates self-signed TLS certificates that the gloo and gateway proxy pods can use for their mTLS communication. The TLS certificates are stored in the gloo-mtls-certs Kubernetes secret. The gloo and gateway proxy pods are configured to mount this secret as a Kubernetes volume.
   certGenJob.runOnUpdate	Re-generate new certificates during a Gloo Gateway update.
   certGenJob.forceRotation	Rotate TLS certificates, even if the old ones are still valid.
   certGenJob.cron.enabled	Enable or disable automatic certificate rotation. If enabled, new self-signed certificates are generated and stored in the gloo-mtls-certs Kubernetes secret.
   certGenJob.cron.schedule	The schedule for your certificate rotation cron job. In this example, the cron job is configured to run every 5 minutes.
   certGenJob.rotationDuration	The certificate rotation cron job runs through several stages to generate and roll out the new TLS certificates. Use this field to configure a time that Gloo Gateway waits between these stages. The default wait time is 65 seconds.

   
   global: 
     glooMtls: 
       enabled: true
   gloo:
     gateway:
       certGenJob:
         enabled: true
         runOnUpdate: true
         forceRotation: true
         cron:
           enabled: true
           schedule: "*/5 * * * *" 
         rotationDuration: 120s

   Value	Description
   global.glooMtls.enabled	Enable or disable the Gloo Gateway mTLS mode.
   certGenJob.enabled	Enable or disable the Gloo Gateway cert generation job that generates self-signed TLS certificates that the gloo and gateway proxy pods can use for their mTLS communication. The TLS certificates are stored in the gloo-mtls-certs Kubernetes secret. The gloo and gateway proxy pods are configured to mount this secret as a Kubernetes volume.
   certGenJob.runOnUpdate	Re-generate new certificates during a Gloo Gateway update.
   certGenJob.forceRotation	Rotate TLS certificates, even if the old ones are still valid.
   certGenJob.cron.enabled	Enable or disable automatic certificate rotation. If enabled, new self-signed certificates are generated and stored in the gloo-mtls-certs Kubernetes secret.
   certGenJob.cron.schedule	The schedule for your certificate rotation cron job. In this example, the cron job is configured to run every 5 minutes.
   certGenJob.rotationDuration	The certificate rotation cron job runs through several stages to generate and roll out the new TLS certificates. Use this field to configure a time that Gloo Gateway waits between these stages. The default wait time is 65 seconds.

3. Upgrade your Gloo Gateway installation.

   helm upgrade -n gloo-system gloo gloo/gloo \
    -f gloo-gateway.yaml \
    --version=1.21.6

   helm upgrade -n gloo-system gloo glooe/gloo-ee \
   -f gloo-gateway.yaml \
   --version=1.21.6

4. Verify that your gloo pod is injected with an Envoy and SDS sidecar.

   • Envoy sidecar: The Envoy sidecar terminates TLS connections on the xdsBindAddr. When you enable Gloo Gateway mTLS mode, the xDS bind address is changed to 127.0.0.0:9977. This setup ensures that only the Envoy sidecar can connect to the gloo container. The Envoy container is configured to read the gloo-mtls-certs Kubernetes secret that was created by the certgen job.
   • SDS sidecar: The SDS sidecar allows Envoy to rotate certificates without restarting the pod. The container is configured to read the gloo-mtls-secret Kubernetes secret that was created by the certgen job.

   kubectl get deployment gloo -n gloo-system -o yaml

   Example output:

   ...
   - env:
      - name: ENVOY_SIDECAR
         value: "true"
        image: quay.io/solo-io/gloo-ee-envoy-wrapper:<version>@sha..
        imagePullPolicy: IfNotPresent
        name: envoy-sidecar
        ports:
        - containerPort: 9977
          name: grpc-xds
          protocol: TCP
        - containerPort: 9976
          name: rest-xds
          protocol: TCP
          ...
        volumeMounts:
        - mountPath: /etc/envoy
          name: envoy-sidecar-config
        - mountPath: /etc/envoy/ssl
          name: gloo-mtls-certs
          readOnly: true
   - env:
      - name: GLOO_MTLS_SDS_ENABLED
          value: "true"
        image: quay.io/solo-io/sds-ee:<version>@sha..
        imagePullPolicy: IfNotPresent
        name: sds
        ports:
        - containerPort: 9988
          name: validation
          protocol: TCP
        ...
        volumeMounts:
        - mountPath: /etc/envoy/ssl
          name: gloo-mtls-certs
          readOnly: true

5. Verify that your gateway proxy pod is injected with an SDS sidecar. The SDS sidecar is configured to read the gloo-mtls-secret Kubernetes secret that was created by the certgen job and to apply certificate changes to the pod.

   kubectl get deployment gloo-proxy-http -n gloo-system -o yaml

   Example output:

   - env:
      - name: POD_NAME
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.name
      - name: POD_NAMESPACE
        valueFrom:
          fieldRef:
            apiVersion: v1
            fieldPath: metadata.namespace
      - name: GLOO_MTLS_SDS_ENABLED
        value: "true"
      - name: LOG_LEVEL
        value: info
        - name: START_STATS_SERVER
          value: "true"
      ...
      volumeMounts:
      - mountPath: /etc/envoy/ssl
        name: gloo-mtls-certs
        readOnly: true

6. Review the configmap for your gateway proxy. Verify that you see an Envoy TLS configuration. This configuration instructs the gateway proxy to use TLS certificates the secure the xDS communication.

   kubectl get configmap gloo-proxy-http  -n gloo-system -ojson

   Example output:

   
    clusters:
      - name: gloo.gloo-system.svc.cluster.local:9977
        transport_socket:
          name: envoy.transport_sockets.tls
          typed_config:
            "@type": type.googleapis.com/envoy.api.v2.auth.UpstreamTlsContext
            common_tls_context:
              tls_certificate_sds_secret_configs:
                - name: server_cert
                  sds_config:
                    api_config_source:
                      api_type: GRPC
                      grpc_services:

7. Enterprise installation only: Review the extauth and rate limiting server configurations.

   1. View the extauth configuration. Verify that the extauth pod is configured with an additional Envoy and SDS container. The Envoy sidecar terminates TLS traffic and encrypts outgoing traffic. The SDS sidecar watches the gloo-mtls-certs Kubernetes secret and is used to handle the TLS certificate rotation.

      The extauth sidecars:

      • Listen on 127.0.0.1:9955 and route traffic to Gloo Gateway’s xDS port.
      • Listen on 0.0.0.0:8083 and route traffic to 127.0.0.1:8084, the extauth’s server port.

      kubectl get deployment extauth -n gloo-system -o yaml
      kubectl get configmap extauth-sidecar-config -n gloo-system -o yaml

   2. View the rate limit configuration. Verify that the rate limiter pod is configured with an additional Envoy and SDS container. The Envoy sidecar terminates TLS traffic and encrypts outgoing traffic. The SDS sidecar watches the gloo-mtls-certs Kubernetes secret and is used to handle the TLS certificate rotation.

      • The Envoy sidecar listens on 127.0.0.1:9955 and routes traffic to Gloo Gateway’s xDS port.

      kubectl get deployment rate-limit -n gloo-system -o yaml
      kubectl get configmap rate-limit-sidecar-config  -n gloo-system -o yaml

Review logs

You can review the Envoy and SDS sidecar container logs for each component by running the following commands:

Envoy:

kubectl logs -n gloo-system deploy/gloo envoy-sidecar
kubectl logs -n gloo-system deploy/extauth envoy-sidecar
kubectl logs -n gloo-system deploy/rate-limit envoy-sidecar

SDS:

kubectl logs -n gloo-system deploy/gloo sds
kubectl logs -n gloo-system deploy/gloo-proxy-http sds
kubectl logs -n gloo-system deploy/extauth sds
kubectl logs -n gloo-system deploy/rate-limit sds

Access the Envoy admin interface

Each Envoy sidecar has an administration interface that is available on port 8001. You can use this endpoint to verify that the TLS certificates are properly injected into the container.

1. Port-forward the gloo deployment on port 8001.

   kubectl port-forward -n gloo-system deploy/gloo 8001

2. Open the certs endpoint and review the certificates that the container is configured with.