Migrate from Edge to K8s Gateway proxies
Learn how to move workloads from Gloo Edge gateway proxies to Kubernetes Gateway API proxies.
Before you begin
Before you start migrating your workloads and trying out Gloo Gateway with the Kubernetes Gateway API, it is recommended to get familiar with the custom resources that the Kubernetes Gateway API uses to configure routing to an app. These Kubernetes resources include Gateway, HTTPRoute, and ReferenceGrant resources. When migrating to the Kubernetes Gateway API, you use these Kubernetes resources in addition to Gloo custom resources so that you can leverage the capabilities that you are familiar with from the Gloo Edge API.
To learn more about the Kubernetes Gateway API integration in Gloo Gateway, see the Gloo Gateway overview.
Step 1: Install Gloo Gateway
The Kubernetes Gateway API support was introduced in Gloo Gateway version 1.17.0. Make sure that you upgrade your Gloo Gateway setup to that version or later before you begin with the migration. You can follow the Get started guide for install instructions.
To migrate workloads between Kubernetes Gateway API and Gloo Edge gateway proxies, you can take two general approaches:
- Same cluster (recommended): Set up a gateway proxy that uses the Kubernetes Gateway API in the same cluster as your existing Gloo Edge gateway proxies. Then, update the routing resources to use the Kubernetes Gateway API gateway proxy. This approach is the least disruptive to your workload setup. However, the Gloo Edge and Kubernetes Gateway API gateway proxies share the same Helm values file and Settings resource. If you decide to run both proxy types in the same cluster, any existing settings in Gloo Edge, such as the Istio integration, automatically apply to the Kubernetes Gateway API gateway proxy. Similarly, any incorrect values or settings can impact or lead to failures in both gateway proxies.
- Separate clusters: Set up a gateway proxy that uses the Kubernetes Gateway API in a separate cluster. Then, update the routing resources or migrate your workloads to the new cluster. This approach temporarily takes up more resources and might involve redeploying your workloads to the new cluster. However, you might find this setup easier to keep separate and test before migrating.
Step 2: Translate Gloo Edge configuration
After you installed Gloo Gateway and enabled the Kubernetes Gateway API integration, it is now time to convert any existing Gloo Edge configuration into Kubernetes Gateway API-native configuration. In Gloo Edge, you typically use Upstreams, VirtualServices, and Gateways to configure routing for your apps. When using the Kubernetes Gateway API, you must change the configuration as discussed in the following sections for each routing resource.
Upstreams
If you currently use Upstreams to route traffic from your Gloo Edge proxy to a Kubernetes service, you have the option to continue using Upstreams or to remove these Upstreams when migrating to the Kubernetes Gateway API. The Upstream or Kubernetes service is referenced in the HTTPRoute resource directly.
For Upstreams that point to different destination types, such as Google Cloud Run or AWS EC2 instances, or Upstreams that configure additional settings, such as HTTP/2 or traffic shadowing, you must continue using Upstreams to represent these destinations.
Kubernetes service destinations
The following image shows an Upstream resource that represents a Kubernetes service (spec.kube). No additional Upstream settings are configured. As depicted in the image, the Upstream can be removed and the Kubernetes service is referenced in the backendRefs section of the HTTPRoute resource. Keep in mind that you also have the option to continue using Upstreams to represent Kubernetes services.
spec.kube) and sets additional values, such as useHttp2: true, you must continue using Upstreams.
Other destination types and additional Upstream configurations
The following image shows an Upstream that represents a different destination type, such as a Google Cloud Run instance (spec.gcp). Kubernetes Gateway API does not have native support for these destination types. Because of that, you must continue using Upstreams. Note that you must also continue using Upstreams if you set additional settings, such as useHttp2: true regardless of the destination type.
VirtualServices
VirtualServices are a central component in the Gloo Edge API. You use VirtualServices to configure gateway listeners, set up routing rules for your apps, configure policies for virtual hosts and routes, and define the backing destinations that fulfill the traffic.
When migrating to the Kubernetes Gateway API, you separate out specific VirtualService configuration into its own resource as depicted in the following image. For example, any route-level policies are moved to a RouteOption resource that is then referenced in the HTTPRoute resource that defines the matchers for your apps.
Review the following topics to learn more about how to translate configuration in the VirtualService:
Matchers and target destinations
To set up routing to your apps in Gloo Edge, you typically use VirtualServices to define the routes to your apps by using the spec.virtualHost.routes section. Routes include the virtual host domain (spec.virtualHost.domains) and the conditions you want to match incoming traffic on, such as a certain prefix path, header, query parameter, or HTTP method (spec.virtualHost.routes.matchers). To route traffic to your backing destination, you must define the target service in the spec.virtualHost.routes.routeAction section.
When migrating to the Kubernetes Gateway API, you create an HTTPRoute resource to set up routing for your apps. An HTTPRoute typically includes all the routes that make up your app. But you can also combine routing rules for multiple apps in the same HTTPRoute if you prefer less resources that you need to manage. You use the spec.hostnames section to define any hostnames for your apps, and the spec.rules.backendRefs section to specify the target destination you want to route requests to. You can choose between routing to a Kubernetes service directly or to an Upstream resource that represents your destination.
In the following image, you have a VirtualService that defines two routes. The first route matches on the /route-a prefix path and routes traffic to the petstore Upstream. The second route matches traffic when certain headers are present in the request. Matching traffic is forwarded to the httpbin Upstream.
During the migration to the Kubernetes Gateway API, you create an HTTPRoute resource to specify the routing rules. In this example, the routing rules for the petstore and httpbin apps are combined in one HTTPRoute. To learn more about how to configure request matching in the Kubernetes Gateway API, check out the Request matching guides. When defining the target destination, you can decide if you want to use Upstreams or route to the Kubernetes service directly. For more information about what option is right for you, see Upstreams. The following example assumes that you want to keep the Upstream for the petstore app. However, the Upstream for httpbin is removed and you route to the Kubernetes service directly.
RouteOptions
In Gloo Edge, you typically use VirtualServices to define the traffic management, resiliency, and security policies that you want to apply to one or multiple routes. These policies are specified in the spec.virtualHost.routes.options section of your VirtualService.
When migrating to the Kubernetes Gateway API, you move these policies to a separate RouteOption resource. The policy configuration does not change and can be migrated as-is. Then, you can use the targetRefs field in the RouteOption to attach the policy to one or more routes in an HTTPRoute, or you can also attach the RouteOption to a specific route by using the ExtensionRef filter in the HTTPRoute resource. If you plan to attach multiple different types of policies to a route and you also want to reuse the same policy across different routes, it is recommended to use the ExtensionRef filter to attach the policies. For more information about your policy attachment options, see Policy attachment.
In the following image, you have a VirtualService that defines two routes for the petstore app (route-a and route-b). route-a applies a fault injection and retry policy. route-b adds request and response headers. During the migration to the Kubernetes Gateway API, you can decide if you want to create a RouteOption resource for each policy, or to combine multiple policies in one RouteOption resource. The option that is right for you depends on the complexity of your setup and how you want to manage policies for your routes. In the following example, a RouteOption resource is created for each route.
To set up routing to the apps, you must create an HTTPRoute resource. The HTTPRoute defines the path to your apps and the Kubernetes service or Upstream you want to route requests to. To attach the policies to each route, you reference the corresponding RouteOption in the ExtensionRef filter in the HTTPRoute resource. Note that you have other options to attach policies to your route. For more information, see the RouteOption documentation.
VirtualHostOptions
In Gloo Edge, you can use a VirtualService to define traffic management, resiliency, and security policies for a specific virtual host. These policies are specified in the spec.virtualHost.options section of your VirtualService.
When migrating to the Kubernetes Gateway API, you move these policies to a separate VirtualHostOption resource. The policy configuration does not change and can be migrated as-is. Then, you use the targetRefs field in the VirtualHostOption to attach the policy to one or multiple hosts on a gateway proxy. In general, it is recommended to add all policies for a specific host to the same VirtualHostOption resource as it allows you to manage these policies more easily. However, if you decide to create multiple VirtualHostOption resources and attach them to the same host, these policies are merged in a specific order and sometimes ignored if policy conflicts exist. For more information, see Conflicting policies and merging.
In the following image, you have a VirtualService that defines a CORS and a transformation policy for all hosts. During the migration to the Kubernetes Gateway API, you create a VirtualHostOption resource and add the CORS and transformation policy to it. To attach the policies to the gateway proxy, you use the targetRefs section. You have the option to attach the policies to a particular host or all hosts that the gateway serves. For more information, see Policy attachment.
Gateways
Gloo Edge Gateways define the listeners and any listener-level policies that you want to apply to the gateway proxy. When migrating to the Kubernetes Gateway API, you separate out specific Gateway configuration into its own resource as depicted in the following image. For example, you move listener-level policies to a ListenerOption or HttpListenerOption resource. These resources are then applied to the Gateway as depicted in the following image.
Review the following topics to learn more about how to translate configuration in the Gateway:
Listeners
Gateways in Gloo Edge define the protocols and ports that the gateway proxy listens on for incoming traffic. These protocol-port combinations are also referred to as a gateway listener. Listeners can also be configured in VirtualServices. For example, you might added an sslConfig to your VirtualService to indicate the TLS certificate that is presented to the client when HTTPS traffic for a particular host reaches the gateway.
When migrating to the Kubernetes Gateway API, you typically define all of your listeners on the Gateway resource. You have the option to create a Gateway resource for each host or to use a wildcard for the hostname so that you can specify the host and routes on the HTTPRoute resource. In both cases you use the spec.parentRefs section in the HTTPRoute resource to indicate which Gateway serves the traffic for HTTPRoute routes.
In the following image, you have a Gloo Edge VirtualService that routes HTTPS traffic to the httpbin Upstream. During the migration, you create a Gateway resource to define the HTTPS listener on the gateway. Then, you create an HTTPRoute resource to configure routing to the httpbin app. In this example, the httpbin Upstream is removed. Instead, you route traffic to the httpbin service directly. To ensure that HTTPS traffic is forwarded to the httpbin app, you reference the Gateway resource in the spec.parentRefs section of your HTTPRoute.
ListenerOptions
To add extra configuration and policies to a specific listener in Gloo Edge, such as access logging or proxy protocol, you typically use a Gateway resource and specify this configuration in the spec.options section. When migrating to the Kubernetes Gateway API, you move the policies to a ListenerOption resource. The policy configuration does not change and can be migrated as-is. Then, you use the targetRefs section in the ListenerOption to attach the policy to one or all gateway listeners.
In the following image, you have a Gloo Edge Gateway resource that specifies an access log policy and enables proxy protocol for the HTTP listener. During the migration to the Kubernetes Gateway API, you create a ListenerOption resource and add the access log and proxy protocol to it. To attach the ListenerOption to one or all listeners on the gateway proxy, you reference the gateway in the spec.targetRefs section of the ListenerOption. The example assumes that you want to attach the ListenerOption resource to all listeners. But you can also specify a specific listener. For more information, see Policy attachment.
HttpListenerOptions
For policies that you want to apply to an HTTP or HTTPS listener only, such as local rate limiting or custom HTTPConnectionManager settings, you typically use the spec.httpGateway.options section in the Gloo Edge Gateway resource.
When migrating to the Kubernetes Gateway API, you move the policies to an HttpListenerOption resource. The policy configuration does not change and can be migrated as-is. Then, you use the targetRefs section in the HttpListenerOption to attach the policy to one or all HTTP and HTTPS listeners on the gateway.
In the following image, you have a Gloo Edge Gateway resource that specifies a local rate limiting policy. During the migration, you create an HttpListenerOption resource and move the policy to the spec.options section. To attach the HttpListenerOption to one or all HTTP and HTTPS listeners on the gateway proxy, you reference the gateway in the spec.targetRefs section of the HttpListenerOption. The example assumes that you want to attach the HttpListenerOption resource to all HTTP/HTTPS listeners. But you can also specify a specific listener. For more information, see Policy attachment.
Gateway customization
You might have customized your Gloo Edge gateway proxy, such as by adding annotations to allow integration with AWS Elastic LoadBalancers or changing the service type and ports in the gateway proxy. To configure these settings, you typically use the Gloo Edge Helm chart.
When migrating to the Kubernetes Gateway API, you add these custom settings to a GatewayParameters resource in the same namespace as your Gateway. Note that the configuration might be different from what you are used to in the Gloo Edge Helm chart. Check out the Gateway customization guides for guidance on how to customize the gateway. Another common reason for using a GatewayParameters resource is to customize the name of the Istio control plane (istiod) when enabling the Istio integration. For more information, see Istio integration.
In the following example, custom AWS annotations are added to gateway proxy in the Gloo Gateway Helm chart. During the migration, you create a GatewayParameters resource to add the AWS annotations. To spin up a gateway proxy that uses these annotations, you create a Gateway resource that references the GatewayParameters in the metadata.annotations section.
AuthConfig
To configure external auth for a certain route or all the routes that the gateway serves, you typically create an AuthConfig resource and reference that config in the spec.virtualHost.options or spec.virtualHost.routes.options sections of your VirtualService resource.
During the migration, the AuthConfig resource remains the same and does not need to change. However, to attach the AuthConfig to a route or gateway host, you must create a RouteOption or VirtualHostOption resource and reference the AuthConfig in the spec.options.extauth section.
In the following image, you have a basic auth policy that is configured by using an AuthConfig and Gloo Edge VirtualService. During the migration, the AuthConfig remains the same and can be used as-is. To attach the AuthConfig, you create a VirtualHostOption or RouteOption resource. When using a RouteOption resource, you must reference the RouteOption in the HTTPRoute that configures routing for your app. For options for how to attach the RouteOption resource to your routes, see Policy attachment.
RateLimitConfig
To configure global rate limiting, you typically create a RateLimitConfig resource and reference that config in the spec.virtualHost.option or spec.virtualHost.routes.options sections of your VirtualService resource.
During the migration, the RateLimitConfig resource remains the same and does not need to change. However, to attach the RateLimitConfig to a route or gateway host, you must create a RouteOption or VirtualHostOption resource and reference the RateLimitConfig in the spec.options.rateLimitConfigs section.
In the following image, you have a rate limit policy in a RateLimitConfig resource that limits access to 1 per minute. During the migration, the RateLimitConfig is not changed. To attach the RateLimitConfig to a host or route, you create a VirtualHostOption or RouteOption resource. Note that when using a RouteOption resource, you must reference the RouteOption in the HTTPRoute that configures routing for your app. For options for how to attach the RouteOption resource to your routes, see Policy attachment.
Step 3: Migrate your configuration
Now that you have all of your target configuration, you can apply them in a test environment. It is recommended to follow these general steps:
- Set up your gateways and listeners by using the Gateway and GatewayParameters resources.
- Create an HTTPRoute resource to set up routing to your apps. Note that you might need to create a ReferenceGrant if the HTTPRoute resource is in a different namespace than the target destination.
- Attach any gateway-level policies, such as CORS, CSRF, or Data loss prevention by using VirtualHostOption, ListenerOption, and HttpListenerOption resources.
- Attach any route-level policies, such as fault injection, retries, or header manipulation by using RouteOption resources.
- Test routing to your apps and verify that your policies are applied correctly.