From 3915a254ff0141c8f8afd8e0d4c038d5f13bfa26 Mon Sep 17 00:00:00 2001 From: salonichf5 <146118978+salonichf5@users.noreply.github.com> Date: Fri, 2 May 2025 12:04:51 -0600 Subject: [PATCH 1/9] docs: scaling control plane and data plane pods --- content/ngf/how-to/scaling.md | 69 +++++++++++++++++++ .../how-to/upgrade-apps-without-downtime.md | 2 +- 2 files changed, 70 insertions(+), 1 deletion(-) create mode 100644 content/ngf/how-to/scaling.md diff --git a/content/ngf/how-to/scaling.md b/content/ngf/how-to/scaling.md new file mode 100644 index 000000000..a9fe12294 --- /dev/null +++ b/content/ngf/how-to/scaling.md @@ -0,0 +1,69 @@ +--- +title: Scaling control plane and data plane +weight: 700 +toc: true +type: how-to +product: NGF +docs: DOCS-0000 +--- + +Scaling the control plane and data plane has its own set of trade-offs. This guide walks you through how to scale each component effectively and helps you decide when to scale the data plane versus creating a new gateway, based on your traffic patterns. + +--- + +### Scaling the data plane + +Data plane constitutes of a single container running both agent and nginx processes. Agent recieves configuration from control plane over a streaming RPC. +Every gateway created, provisions a new NGINX deployment with its own configuration. We have a couple of options on how to scale data plane deployments. You can do so either by increasing the number of replicas for data plane pod or creating a new gateway to provision a new data plane. + +#### When to create a new gateway vs Scale Data plane replicas + +When using NGINX Gateway Fabric, understanding when to scale the data plane vs when to create a new gateway is key to managing traffic effectively. + +Scaling data plane replicas is ideal when you need to handle more traffic without changing the configuration. For example, if you're routing traffic to `api.example.com` and notice an increase in load, you can scale the replicas from 1 to 5 to better distribute the traffic and reduce latency. All replicas will share the same configuration from the gateway used to set up the data plane, making configuration management easy. However, this approach can be limiting if you need to customize configurations for different use cases. Additionally, a fault in the configuration can affect all replicas, creating a potential single point of failure for that gateway. + +You can increase the number of replicas for an NGINX deployment by modifying the field `nginx.replicas` in the `values.yaml` or add the `--set nginx.replicas=` flag to the `helm install` command. Below is an example to do so: + + ```text + helm install ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric --create-namespace -n nginx-gateway --set nginx.replicas=5 + ``` + +Creating a new gateway is beneficial when you need distinct configurations, isolation, or separate policies. For example, if you're routing traffic to a new domain `admin.example.com` and require a different TLS certificate, stricter rate limits, or separate authentication policies, creating a new gateway is the right approach. It allows safe experimentation with isolated configurations and makes it easier to enforce security boundaries and apply specific routing rules. However, this comes with increased resource overhead since a new gateway creates a new deployment which can introduce complexity when managing multiple configurations if not well-organized. + +--- + +### Scaling the control plane + +Control plane is a kubernetes deployment in one container running the controller. It communicates with the agent (data plane) over gRPC to deliver configurations. With leader election enabled, the control plane can be scaled horizontally by running multiple replicas, although only the pod with leader lease can actively manage configuration status updates. + +Scaling the control plane can be beneficial in the following scenarios: + + 1. *Higher Availability* - When the control plane pod crashes, runs out of memory, or goes down during an upgrade, it can interrupt configuration delivery. By scaling to multiple replicas, leader election makes sure another pod can quickly step in and take over, keeping things running smoothly with minimal downtime. + 2. *Faster Configuration Distribution* - As the number of connected NGINX instances grows, a single control plane pod may become a bottleneck in handling connections or streaming configuration updates. Scaling the control plane improves concurrency and responsiveness when delivering configuration over gRPC. + 3. *Improved Resilience* - Running multiple control plane replicas provides fault tolerance. Even if the leader fails, another replica can quickly take over the leader lease, preventing disruptions in config management and status updates. + +To scale the control plane, use the `kubectl scale` command on the control plane deployment to increase or decrease the number of replicas. For example, the following command scales the control plane deployment to 3 replicas: + + ```text + kubectl scale deployment -n nginx-gateway ngf-nginx-gateway-fabric --replicas 3 + ``` + +#### Known risks around scaling control plane + +When scaling the control plane, it's important to understand how status updates are handled for data plane pods. +NGINX instances receive configurations from the control plane pods. However, only the leader control plane pod is allowed to write status updates to gateway resources. If an NGINX instance connects to a non-leader pod, the resource status may not be updated, which can result in configurations not being applied correctly. To prevent this, make sure leader election is stable, monitor for frequent leader changes, and avoid scaling the control plane pods excessively unless needed. + +To identify which control plane pod currently holds the leader election lease, retrieve the leases in the same namespace as the control plane pods. For example: + + ```text + kubectl get leases -n nginx-gateway + ``` + +The current leader lease is held by the pod `ngf-nginx-gateway-fabric-b45ffc8d6-d9z2g_2ef81ced-f19d-41a0-9fcd-a68d89380d10`: + + ```text + NAME HOLDER AGE + ngf-nginx-gateway-fabric-leader-election ngf-nginx-gateway-fabric-b45ffc8d6-d9z2g_2ef81ced-f19d-41a0-9fcd-a68d89380d10 16d + ``` + +--- \ No newline at end of file diff --git a/content/ngf/how-to/upgrade-apps-without-downtime.md b/content/ngf/how-to/upgrade-apps-without-downtime.md index e66edd9ba..71e570f0b 100644 --- a/content/ngf/how-to/upgrade-apps-without-downtime.md +++ b/content/ngf/how-to/upgrade-apps-without-downtime.md @@ -1,6 +1,6 @@ --- title: Upgrade applications without downtime -weight: 500 +weight: 600 toc: true type: how-to product: NGF From e61a1dc9f2b6accdcf297e6991ecbaca733af5ae Mon Sep 17 00:00:00 2001 From: salonichf5 <146118978+salonichf5@users.noreply.github.com> Date: Wed, 7 May 2025 00:31:45 +0530 Subject: [PATCH 2/9] Update content/ngf/how-to/scaling.md Co-authored-by: Saylor Berman --- content/ngf/how-to/scaling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/ngf/how-to/scaling.md b/content/ngf/how-to/scaling.md index a9fe12294..3e15c3cfa 100644 --- a/content/ngf/how-to/scaling.md +++ b/content/ngf/how-to/scaling.md @@ -7,7 +7,7 @@ product: NGF docs: DOCS-0000 --- -Scaling the control plane and data plane has its own set of trade-offs. This guide walks you through how to scale each component effectively and helps you decide when to scale the data plane versus creating a new gateway, based on your traffic patterns. +Users can scale both the NGINX Gateway Fabric control plane and data planes separately. This guide walks you through how to scale each component effectively and helps you decide when to scale the data plane versus creating a new gateway, based on your traffic patterns. --- From 3744e0a193daf5842f0a6c4a89e83db48fef1cb7 Mon Sep 17 00:00:00 2001 From: salonichf5 <146118978+salonichf5@users.noreply.github.com> Date: Wed, 7 May 2025 00:33:27 +0530 Subject: [PATCH 3/9] Apply suggestions from code review Co-authored-by: Saylor Berman --- content/ngf/how-to/scaling.md | 16 +++++++++------- 1 file changed, 9 insertions(+), 7 deletions(-) diff --git a/content/ngf/how-to/scaling.md b/content/ngf/how-to/scaling.md index 3e15c3cfa..a9adc9add 100644 --- a/content/ngf/how-to/scaling.md +++ b/content/ngf/how-to/scaling.md @@ -14,13 +14,13 @@ Users can scale both the NGINX Gateway Fabric control plane and data planes sepa ### Scaling the data plane Data plane constitutes of a single container running both agent and nginx processes. Agent recieves configuration from control plane over a streaming RPC. -Every gateway created, provisions a new NGINX deployment with its own configuration. We have a couple of options on how to scale data plane deployments. You can do so either by increasing the number of replicas for data plane pod or creating a new gateway to provision a new data plane. +Every Gateway object that gets created results in a new NGINX deployment being provisioned with its own configuration. There are a couple of options on how to scale data plane deployments. You can do so either by increasing the number of replicas for the data plane deployment or by creating a new Gateway to provision a new data plane. #### When to create a new gateway vs Scale Data plane replicas When using NGINX Gateway Fabric, understanding when to scale the data plane vs when to create a new gateway is key to managing traffic effectively. -Scaling data plane replicas is ideal when you need to handle more traffic without changing the configuration. For example, if you're routing traffic to `api.example.com` and notice an increase in load, you can scale the replicas from 1 to 5 to better distribute the traffic and reduce latency. All replicas will share the same configuration from the gateway used to set up the data plane, making configuration management easy. However, this approach can be limiting if you need to customize configurations for different use cases. Additionally, a fault in the configuration can affect all replicas, creating a potential single point of failure for that gateway. +Scaling data plane replicas is ideal when you need to handle more traffic without changing the configuration. For example, if you're routing traffic to `api.example.com` and notice an increase in load, you can scale the replicas from 1 to 5 to better distribute the traffic and reduce latency. All replicas will share the same configuration from the Gateway used to set up the data plane, making configuration management easy. You can increase the number of replicas for an NGINX deployment by modifying the field `nginx.replicas` in the `values.yaml` or add the `--set nginx.replicas=` flag to the `helm install` command. Below is an example to do so: @@ -28,17 +28,17 @@ You can increase the number of replicas for an NGINX deployment by modifying the helm install ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric --create-namespace -n nginx-gateway --set nginx.replicas=5 ``` -Creating a new gateway is beneficial when you need distinct configurations, isolation, or separate policies. For example, if you're routing traffic to a new domain `admin.example.com` and require a different TLS certificate, stricter rate limits, or separate authentication policies, creating a new gateway is the right approach. It allows safe experimentation with isolated configurations and makes it easier to enforce security boundaries and apply specific routing rules. However, this comes with increased resource overhead since a new gateway creates a new deployment which can introduce complexity when managing multiple configurations if not well-organized. +The other way to scale data planes is by creating a new Gateway. This is is beneficial when you need distinct configurations, isolation, or separate policies. For example, if you're routing traffic to a new domain `admin.example.com` and require a different TLS certificate, stricter rate limits, or separate authentication policies, creating a new Gateway could be a good approach. It allows safe experimentation with isolated configurations and makes it easier to enforce security boundaries and apply specific routing rules. --- ### Scaling the control plane -Control plane is a kubernetes deployment in one container running the controller. It communicates with the agent (data plane) over gRPC to deliver configurations. With leader election enabled, the control plane can be scaled horizontally by running multiple replicas, although only the pod with leader lease can actively manage configuration status updates. +The control plane builds configuration based on defined Gateway API resources and sends that configuration to the NGINX data planes. With leader election enabled by default, the control plane can be scaled horizontally by running multiple replicas, although only the pod with leader lease can actively manage configuration status updates. Scaling the control plane can be beneficial in the following scenarios: - 1. *Higher Availability* - When the control plane pod crashes, runs out of memory, or goes down during an upgrade, it can interrupt configuration delivery. By scaling to multiple replicas, leader election makes sure another pod can quickly step in and take over, keeping things running smoothly with minimal downtime. + 1. *Higher Availability* - When a control plane pod crashes, runs out of memory, or goes down during an upgrade, it can interrupt configuration delivery. By scaling to multiple replicas, another pod can quickly step in and take over, keeping things running smoothly with minimal downtime. 2. *Faster Configuration Distribution* - As the number of connected NGINX instances grows, a single control plane pod may become a bottleneck in handling connections or streaming configuration updates. Scaling the control plane improves concurrency and responsiveness when delivering configuration over gRPC. 3. *Improved Resilience* - Running multiple control plane replicas provides fault tolerance. Even if the leader fails, another replica can quickly take over the leader lease, preventing disruptions in config management and status updates. @@ -50,8 +50,10 @@ To scale the control plane, use the `kubectl scale` command on the control plane #### Known risks around scaling control plane -When scaling the control plane, it's important to understand how status updates are handled for data plane pods. -NGINX instances receive configurations from the control plane pods. However, only the leader control plane pod is allowed to write status updates to gateway resources. If an NGINX instance connects to a non-leader pod, the resource status may not be updated, which can result in configurations not being applied correctly. To prevent this, make sure leader election is stable, monitor for frequent leader changes, and avoid scaling the control plane pods excessively unless needed. +When scaling the control plane, it's important to understand how status updates are handled for Gateway API resources. +All control plane pods can send NGINX configuration to the data planes. However, only the leader control plane pod is allowed to write status updates to Gateway API resources. This means that if an NGINX instance connects to a non-leader pod, and an error occurs when applying the config, that error status will not be written to the Gateway object status. To help mitigate the potential for this issue, ensure that the number of NGINX data plane pods equals or exceeds the number of control plane pods. This increases the likelihood that at least one of the data planes is connected to the leader control plane pod. This way if an applied configuration has an error, the leader pod will be aware of it and status can still be written. + +There is still a chance (however unlikely) that one of the data planes connected to a non-leader has an issue applying its configuration, while the rest of the data planes are successful, which could lead to that error status not being written. To identify which control plane pod currently holds the leader election lease, retrieve the leases in the same namespace as the control plane pods. For example: From 4d1c8474faa1c2bd1875815c1b9bee5abf00bbcf Mon Sep 17 00:00:00 2001 From: salonichf5 <146118978+salonichf5@users.noreply.github.com> Date: Tue, 6 May 2025 13:59:44 -0600 Subject: [PATCH 4/9] update based on reviews --- content/ngf/how-to/scaling.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/content/ngf/how-to/scaling.md b/content/ngf/how-to/scaling.md index a9adc9add..9277d40da 100644 --- a/content/ngf/how-to/scaling.md +++ b/content/ngf/how-to/scaling.md @@ -13,7 +13,7 @@ Users can scale both the NGINX Gateway Fabric control plane and data planes sepa ### Scaling the data plane -Data plane constitutes of a single container running both agent and nginx processes. Agent recieves configuration from control plane over a streaming RPC. +The data plane is the NGINX deployment that handles user traffic to backend applications. Every Gateway object that gets created results in a new NGINX deployment being provisioned with its own configuration. There are a couple of options on how to scale data plane deployments. You can do so either by increasing the number of replicas for the data plane deployment or by creating a new Gateway to provision a new data plane. #### When to create a new gateway vs Scale Data plane replicas @@ -22,12 +22,20 @@ When using NGINX Gateway Fabric, understanding when to scale the data plane vs w Scaling data plane replicas is ideal when you need to handle more traffic without changing the configuration. For example, if you're routing traffic to `api.example.com` and notice an increase in load, you can scale the replicas from 1 to 5 to better distribute the traffic and reduce latency. All replicas will share the same configuration from the Gateway used to set up the data plane, making configuration management easy. -You can increase the number of replicas for an NGINX deployment by modifying the field `nginx.replicas` in the `values.yaml` or add the `--set nginx.replicas=` flag to the `helm install` command. Below is an example to do so: +There are two ways to modify the number of replicas for an NGINX deployment: + +- Modifying the field `nginx.replicas` in the `values.yaml` or add the `--set nginx.replicas=` flag to the `helm install` command. Below is an example to do so: ```text helm install ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric --create-namespace -n nginx-gateway --set nginx.replicas=5 ``` +- Editing the `NginxProxy` resource attached to the data plane deployment. You can specify the number of replicas in the field `kubernetes.deployment.replicas` of the nginxProxy resource: + + ```text + kubectl edit nginxproxies.gateway.nginx.org ngf-proxy-config -n nginx-gateway + ``` + The other way to scale data planes is by creating a new Gateway. This is is beneficial when you need distinct configurations, isolation, or separate policies. For example, if you're routing traffic to a new domain `admin.example.com` and require a different TLS certificate, stricter rate limits, or separate authentication policies, creating a new Gateway could be a good approach. It allows safe experimentation with isolated configurations and makes it easier to enforce security boundaries and apply specific routing rules. --- @@ -40,7 +48,7 @@ Scaling the control plane can be beneficial in the following scenarios: 1. *Higher Availability* - When a control plane pod crashes, runs out of memory, or goes down during an upgrade, it can interrupt configuration delivery. By scaling to multiple replicas, another pod can quickly step in and take over, keeping things running smoothly with minimal downtime. 2. *Faster Configuration Distribution* - As the number of connected NGINX instances grows, a single control plane pod may become a bottleneck in handling connections or streaming configuration updates. Scaling the control plane improves concurrency and responsiveness when delivering configuration over gRPC. - 3. *Improved Resilience* - Running multiple control plane replicas provides fault tolerance. Even if the leader fails, another replica can quickly take over the leader lease, preventing disruptions in config management and status updates. + 3. *Improved Resilience* - Running multiple control plane replicas provides fault tolerance. Even if the pod holding the leader lease fails, another pod can quickly step in and take over, preventing disruptions in status updates. To scale the control plane, use the `kubectl scale` command on the control plane deployment to increase or decrease the number of replicas. For example, the following command scales the control plane deployment to 3 replicas: From 540a7959869205ee221933c7ca6f3f402233e73c Mon Sep 17 00:00:00 2001 From: salonichf5 <146118978+salonichf5@users.noreply.github.com> Date: Wed, 7 May 2025 21:36:47 +0530 Subject: [PATCH 5/9] Apply suggestions from code review Co-authored-by: Alan Dooley --- content/ngf/how-to/scaling.md | 75 +++++++++++++++++++++-------------- 1 file changed, 45 insertions(+), 30 deletions(-) diff --git a/content/ngf/how-to/scaling.md b/content/ngf/how-to/scaling.md index 9277d40da..9433e26f7 100644 --- a/content/ngf/how-to/scaling.md +++ b/content/ngf/how-to/scaling.md @@ -1,5 +1,5 @@ --- -title: Scaling control plane and data plane +title: Scaling the control plane and data plane weight: 700 toc: true type: how-to @@ -7,38 +7,48 @@ product: NGF docs: DOCS-0000 --- -Users can scale both the NGINX Gateway Fabric control plane and data planes separately. This guide walks you through how to scale each component effectively and helps you decide when to scale the data plane versus creating a new gateway, based on your traffic patterns. +This document describes how you can separately scale the NGINX Gateway Fabric control plane and data plane. + +It provides guidance on how to scale each plane effectively, and when you should do so based on your traffic patterns. ---- ### Scaling the data plane -The data plane is the NGINX deployment that handles user traffic to backend applications. -Every Gateway object that gets created results in a new NGINX deployment being provisioned with its own configuration. There are a couple of options on how to scale data plane deployments. You can do so either by increasing the number of replicas for the data plane deployment or by creating a new Gateway to provision a new data plane. +The data plane is the NGINX deployment that handles user traffic to backend applications. Every Gateway object created provisions its own NGINX deployment and configuration. + +You have two options for scaling the data plane: + +- Increasing the number of replicas for an existing deployment +- Creating a new Gateway for a new data plane + +#### When to increase replicas or create a new Gateway + +Understanding when to increase replicas or create a new Gateway is key to managing traffic effectively. -#### When to create a new gateway vs Scale Data plane replicas +Increasing data plane replicas is ideal when you need to handle more traffic without changing the configuration. -When using NGINX Gateway Fabric, understanding when to scale the data plane vs when to create a new gateway is key to managing traffic effectively. +For example, if you're routing traffic to `api.example.com` and notice an increase in load, you can scale the replicas from 1 to 5 to better distribute the traffic and reduce latency. -Scaling data plane replicas is ideal when you need to handle more traffic without changing the configuration. For example, if you're routing traffic to `api.example.com` and notice an increase in load, you can scale the replicas from 1 to 5 to better distribute the traffic and reduce latency. All replicas will share the same configuration from the Gateway used to set up the data plane, making configuration management easy. +All replicas will share the same configuration from the Gateway used to set up the data plane, simplifying configuration management. There are two ways to modify the number of replicas for an NGINX deployment: -- Modifying the field `nginx.replicas` in the `values.yaml` or add the `--set nginx.replicas=` flag to the `helm install` command. Below is an example to do so: +First, by modifying the field `nginx.replicas` in the `values.yaml` or adding the `--set nginx.replicas=` flag to the `helm install` command. - ```text - helm install ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric --create-namespace -n nginx-gateway --set nginx.replicas=5 - ``` +```text +helm install ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric --create-namespace -n nginx-gateway --set nginx.replicas=5 +Secondly, by editing the `NginxProxy` resource attached to the data plane deployment. You can specify the number of replicas in the field `kubernetes.deployment.replicas` of the nginxProxy resource: -- Editing the `NginxProxy` resource attached to the data plane deployment. You can specify the number of replicas in the field `kubernetes.deployment.replicas` of the nginxProxy resource: +```text +kubectl edit nginxproxies.gateway.nginx.org ngf-proxy-config -n nginx-gateway - ```text - kubectl edit nginxproxies.gateway.nginx.org ngf-proxy-config -n nginx-gateway - ``` -The other way to scale data planes is by creating a new Gateway. This is is beneficial when you need distinct configurations, isolation, or separate policies. For example, if you're routing traffic to a new domain `admin.example.com` and require a different TLS certificate, stricter rate limits, or separate authentication policies, creating a new Gateway could be a good approach. It allows safe experimentation with isolated configurations and makes it easier to enforce security boundaries and apply specific routing rules. +The alternate way to scale the data plane is by creating a new Gateway. This is is beneficial when you need distinct configurations, isolation, or separate policies. + +For example, if you're routing traffic to a new domain `admin.example.com` and require a different TLS certificate, stricter rate limits, or separate authentication policies, creating a new Gateway could be a good approach. + +It allows for safe experimentation with isolated configurations and makes it easier to enforce security boundaries and specific routing rules. ---- ### Scaling the control plane @@ -46,9 +56,9 @@ The control plane builds configuration based on defined Gateway API resources an Scaling the control plane can be beneficial in the following scenarios: - 1. *Higher Availability* - When a control plane pod crashes, runs out of memory, or goes down during an upgrade, it can interrupt configuration delivery. By scaling to multiple replicas, another pod can quickly step in and take over, keeping things running smoothly with minimal downtime. - 2. *Faster Configuration Distribution* - As the number of connected NGINX instances grows, a single control plane pod may become a bottleneck in handling connections or streaming configuration updates. Scaling the control plane improves concurrency and responsiveness when delivering configuration over gRPC. - 3. *Improved Resilience* - Running multiple control plane replicas provides fault tolerance. Even if the pod holding the leader lease fails, another pod can quickly step in and take over, preventing disruptions in status updates. +1. _Higher availability_ - When a control plane pod crashes, runs out of memory, or goes down during an upgrade, it can interrupt configuration delivery. By scaling to multiple replicas, another pod can quickly step in and take over, keeping things running smoothly with minimal downtime. +1. _Faster configuration distribution_ - As the number of connected NGINX instances grows, a single control plane pod may become a bottleneck in handling connections or streaming configuration updates. Scaling the control plane improves concurrency and responsiveness when delivering configuration over gRPC. +1. _Improved resilience_ - Running multiple control plane replicas provides fault tolerance. Even if the pod holding the leader lease fails, another pod can quickly step in and take over, preventing disruptions in status updates. To scale the control plane, use the `kubectl scale` command on the control plane deployment to increase or decrease the number of replicas. For example, the following command scales the control plane deployment to 3 replicas: @@ -56,24 +66,29 @@ To scale the control plane, use the `kubectl scale` command on the control plane kubectl scale deployment -n nginx-gateway ngf-nginx-gateway-fabric --replicas 3 ``` -#### Known risks around scaling control plane +#### Known risks when scaling the control plane When scaling the control plane, it's important to understand how status updates are handled for Gateway API resources. -All control plane pods can send NGINX configuration to the data planes. However, only the leader control plane pod is allowed to write status updates to Gateway API resources. This means that if an NGINX instance connects to a non-leader pod, and an error occurs when applying the config, that error status will not be written to the Gateway object status. To help mitigate the potential for this issue, ensure that the number of NGINX data plane pods equals or exceeds the number of control plane pods. This increases the likelihood that at least one of the data planes is connected to the leader control plane pod. This way if an applied configuration has an error, the leader pod will be aware of it and status can still be written. + +All control plane pods can send NGINX configuration to the data planes. However, only the leader control plane pod is allowed to write status updates to Gateway API resources. + +If an NGINX instance connects to a non-leader pod, and an error occurs when applying the config, that error status will not be written to the Gateway object status. + +To mitigate the potential for this issue, ensure that the number of NGINX data plane pods equals or exceeds the number of control plane pods. + +This increases the likelihood that at least one of the data planes is connected to the leader control plane pod. If an applied configuration has an error, the leader pod will be aware of it and status can still be written. There is still a chance (however unlikely) that one of the data planes connected to a non-leader has an issue applying its configuration, while the rest of the data planes are successful, which could lead to that error status not being written. To identify which control plane pod currently holds the leader election lease, retrieve the leases in the same namespace as the control plane pods. For example: - ```text - kubectl get leases -n nginx-gateway - ``` +```text +kubectl get leases -n nginx-gateway The current leader lease is held by the pod `ngf-nginx-gateway-fabric-b45ffc8d6-d9z2g_2ef81ced-f19d-41a0-9fcd-a68d89380d10`: - ```text - NAME HOLDER AGE - ngf-nginx-gateway-fabric-leader-election ngf-nginx-gateway-fabric-b45ffc8d6-d9z2g_2ef81ced-f19d-41a0-9fcd-a68d89380d10 16d - ``` +```text +NAME HOLDER AGE +ngf-nginx-gateway-fabric-leader-election ngf-nginx-gateway-fabric-b45ffc8d6-d9z2g_2ef81ced-f19d-41a0-9fcd-a68d89380d10 16d --- \ No newline at end of file From 639e36c176262c39879e175758a46792ea1dd9af Mon Sep 17 00:00:00 2001 From: salonichf5 <146118978+salonichf5@users.noreply.github.com> Date: Wed, 7 May 2025 10:08:39 -0600 Subject: [PATCH 6/9] fix pod name --- content/ngf/how-to/scaling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/ngf/how-to/scaling.md b/content/ngf/how-to/scaling.md index 9433e26f7..f06eda886 100644 --- a/content/ngf/how-to/scaling.md +++ b/content/ngf/how-to/scaling.md @@ -85,7 +85,7 @@ To identify which control plane pod currently holds the leader election lease, r ```text kubectl get leases -n nginx-gateway -The current leader lease is held by the pod `ngf-nginx-gateway-fabric-b45ffc8d6-d9z2g_2ef81ced-f19d-41a0-9fcd-a68d89380d10`: +The current leader lease is held by the pod `ngf-nginx-gateway-fabric-b45ffc8d6-d9z2g`: ```text NAME HOLDER AGE From eb1ad77db94b64ccc0f293ee3e32321fa6a7a283 Mon Sep 17 00:00:00 2001 From: salonichf5 <146118978+salonichf5@users.noreply.github.com> Date: Wed, 7 May 2025 13:59:40 -0600 Subject: [PATCH 7/9] fix formatting issues --- content/ngf/how-to/scaling.md | 21 ++++++++++++--------- 1 file changed, 12 insertions(+), 9 deletions(-) diff --git a/content/ngf/how-to/scaling.md b/content/ngf/how-to/scaling.md index f06eda886..53b56a15f 100644 --- a/content/ngf/how-to/scaling.md +++ b/content/ngf/how-to/scaling.md @@ -33,15 +33,17 @@ All replicas will share the same configuration from the Gateway used to set up t There are two ways to modify the number of replicas for an NGINX deployment: -First, by modifying the field `nginx.replicas` in the `values.yaml` or adding the `--set nginx.replicas=` flag to the `helm install` command. +First, at the time of installation you can modify the field `nginx.replicas` in the `values.yaml` or add the `--set nginx.replicas=` flag to the `helm install` command -```text +```shell helm install ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric --create-namespace -n nginx-gateway --set nginx.replicas=5 -Secondly, by editing the `NginxProxy` resource attached to the data plane deployment. You can specify the number of replicas in the field `kubernetes.deployment.replicas` of the nginxProxy resource: +``` -```text -kubectl edit nginxproxies.gateway.nginx.org ngf-proxy-config -n nginx-gateway +Secondly, you can update the `NginxProxy` resource while NGINX is running to modify the `kubernetes.deployment.replicas` field and scale the data plane deployment dynamically: +```shell +kubectl edit nginxproxies.gateway.nginx.org ngf-proxy-config -n nginx-gateway +``` The alternate way to scale the data plane is by creating a new Gateway. This is is beneficial when you need distinct configurations, isolation, or separate policies. @@ -58,11 +60,10 @@ Scaling the control plane can be beneficial in the following scenarios: 1. _Higher availability_ - When a control plane pod crashes, runs out of memory, or goes down during an upgrade, it can interrupt configuration delivery. By scaling to multiple replicas, another pod can quickly step in and take over, keeping things running smoothly with minimal downtime. 1. _Faster configuration distribution_ - As the number of connected NGINX instances grows, a single control plane pod may become a bottleneck in handling connections or streaming configuration updates. Scaling the control plane improves concurrency and responsiveness when delivering configuration over gRPC. -1. _Improved resilience_ - Running multiple control plane replicas provides fault tolerance. Even if the pod holding the leader lease fails, another pod can quickly step in and take over, preventing disruptions in status updates. To scale the control plane, use the `kubectl scale` command on the control plane deployment to increase or decrease the number of replicas. For example, the following command scales the control plane deployment to 3 replicas: - ```text + ```shell kubectl scale deployment -n nginx-gateway ngf-nginx-gateway-fabric --replicas 3 ``` @@ -82,13 +83,15 @@ There is still a chance (however unlikely) that one of the data planes connected To identify which control plane pod currently holds the leader election lease, retrieve the leases in the same namespace as the control plane pods. For example: -```text +```shell kubectl get leases -n nginx-gateway +``` The current leader lease is held by the pod `ngf-nginx-gateway-fabric-b45ffc8d6-d9z2g`: -```text +```shell NAME HOLDER AGE ngf-nginx-gateway-fabric-leader-election ngf-nginx-gateway-fabric-b45ffc8d6-d9z2g_2ef81ced-f19d-41a0-9fcd-a68d89380d10 16d +``` --- \ No newline at end of file From 1499b66cd887d2d682bcac0878005f41f2911650 Mon Sep 17 00:00:00 2001 From: salonichf5 <146118978+salonichf5@users.noreply.github.com> Date: Thu, 8 May 2025 01:42:45 +0530 Subject: [PATCH 8/9] Update content/ngf/how-to/scaling.md Co-authored-by: Saylor Berman --- content/ngf/how-to/scaling.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/content/ngf/how-to/scaling.md b/content/ngf/how-to/scaling.md index 53b56a15f..25d3c5457 100644 --- a/content/ngf/how-to/scaling.md +++ b/content/ngf/how-to/scaling.md @@ -33,7 +33,7 @@ All replicas will share the same configuration from the Gateway used to set up t There are two ways to modify the number of replicas for an NGINX deployment: -First, at the time of installation you can modify the field `nginx.replicas` in the `values.yaml` or add the `--set nginx.replicas=` flag to the `helm install` command +First, at the time of installation you can modify the field `nginx.replicas` in the `values.yaml` or add the `--set nginx.replicas=` flag to the `helm install` command: ```shell helm install ngf oci://ghcr.io/nginx/charts/nginx-gateway-fabric --create-namespace -n nginx-gateway --set nginx.replicas=5 From 375b1a2ad1e5c3c09ee440f550ff75463048deda Mon Sep 17 00:00:00 2001 From: salonichf5 <146118978+salonichf5@users.noreply.github.com> Date: Wed, 7 May 2025 14:14:02 -0600 Subject: [PATCH 9/9] fix space --- content/ngf/how-to/scaling.md | 1 - 1 file changed, 1 deletion(-) diff --git a/content/ngf/how-to/scaling.md b/content/ngf/how-to/scaling.md index 25d3c5457..35cd132da 100644 --- a/content/ngf/how-to/scaling.md +++ b/content/ngf/how-to/scaling.md @@ -51,7 +51,6 @@ For example, if you're routing traffic to a new domain `admin.example.com` and r It allows for safe experimentation with isolated configurations and makes it easier to enforce security boundaries and specific routing rules. - ### Scaling the control plane The control plane builds configuration based on defined Gateway API resources and sends that configuration to the NGINX data planes. With leader election enabled by default, the control plane can be scaled horizontally by running multiple replicas, although only the pod with leader lease can actively manage configuration status updates.