Battle for Resources or the SSA Path to Kubernetes Diplomacy

Introduction In the world of Kubernetes, resource management is more than just creating, deleting, or updating objects. It's an intricate dance involving numerous tools, operators, and users. As our infrastructure grows, it becomes increasingly challenging to maintain control, necessitating the adoption of more advanced and sophisticated resource management and control systems and approaches. In this article, I will not delve into the basic methods of creating resources, as that is a rather trivial task. Instead, I would like to share my experience in optimizing resource update paths, which have proven to be immensely valuable in managing resources within large and complex Kubernetes clusters, as well as during the development of operators. Update "How do I update a resource?" This is often the first question that comes to mind for every developer, DevOps engineer, or anyone who has interacted with Kubernetes. And the first answer that comes to mind is UPDATE - or, in Kubernetes terminology, APPLY. UPDATE APPLY This approach is entirely correct. The kubectl apply command is an incredibly powerful and convenient tool: you simply modify the desired part of the resource manifest and apply it, and Kubernetes handles the rest. kubectl apply Let's apply the following deployment manifest as an example: > cat test-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: gpa-app spec: replicas: 1 selector: matchLabels: app: gpa-app template: metadata: labels: app: gpa-app spec: containers: - name: gpa-app image: nginx:1.21.0 ports: - containerPort: 80 > kubectl apply -f test-deployment.yaml > cat test-deployment.yaml apiVersion: apps/v1 kind: Deployment metadata: name: gpa-app spec: replicas: 1 selector: matchLabels: app: gpa-app template: metadata: labels: app: gpa-app spec: containers: - name: gpa-app image: nginx:1.21.0 ports: - containerPort: 80 > kubectl apply -f test-deployment.yaml However, the complexity comes when we want to automate the regular checking and updating of a resource, for example, by encapsulating the logic within a microservice. In such a scenario, you always need to keep the source manifest to introduce changes and apply them. Storing the manifest directly within the microservice isn't an ideal solution for several reasons: Any necessary manifest change, unrelated to the microservice's routine logic, would require code modification, building a new microservice image, and redeploying it. This introduces unnecessary downtime and operational inconvenience. If the manifest within the cluster is manually modified, these changes will be overwritten by the stale values from the microservice's internal manifest on its next application cycle. Any necessary manifest change, unrelated to the microservice's routine logic, would require code modification, building a new microservice image, and redeploying it. This introduces unnecessary downtime and operational inconvenience. If the manifest within the cluster is manually modified, these changes will be overwritten by the stale values from the microservice's internal manifest on its next application cycle. A more robust solution is to implement a logic that first retrieves (GET) the current state of the resource's manifest, updates the necessary fields, and then applies the changes back to the cluster: GET Executing the GET command to retrieve the current state Executing the GET command to retrieve the current state GET > kubectl get deployment gpa-app -o yaml apiVersion: apps/v1 kind: Deployment metadata: annotations: ... creationTimestamp: "2025-08-25T23:43:37Z" generation: 1 name: gpa-app namespace: default resourceVersion: "495672" uid: 2dec1474-988d-431b-9cf2-e8f41a624517 spec: replicas: 1 ... selector: matchLabels: app: gpa-app ... template: metadata: creationTimestamp: null labels: app: gpa-app spec: containers: - image: nginx:1.21.0 imagePullPolicy: IfNotPresent name: gpa-app ports: - containerPort: 80 protocol: TCP ... ... status: ... > kubectl get deployment gpa-app -o yaml apiVersion: apps/v1 kind: Deployment metadata: annotations: ... creationTimestamp: "2025-08-25T23:43:37Z" generation: 1 name: gpa-app namespace: default resourceVersion: "495672" uid: 2dec1474-988d-431b-9cf2-e8f41a624517 spec: replicas: 1 ... selector: matchLabels: app: gpa-app ... template: metadata: creationTimestamp: null labels: app: gpa-app spec: containers: - image: nginx:1.21.0 imagePullPolicy: IfNotPresent name: gpa-app ports: - containerPort: 80 protocol: TCP ... ... status: ... Cleaning the manifest and updating the replicas count Cleaning the manifest and updating the replicas count > cat test-deployment-upd.yaml apiVersion: apps/v1 kind: Deployment metadata: name: gpa-app spec: replicas: 2 # new replicas count selector: matchLabels: app: gpa-app template: metadata: labels: app: gpa-app spec: containers: - name: gpa-app image: nginx:1.21.0 ports: - containerPort: 80 > cat test-deployment-upd.yaml apiVersion: apps/v1 kind: Deployment metadata: name: gpa-app spec: replicas: 2 # new replicas count selector: matchLabels: app: gpa-app template: metadata: labels: app: gpa-app spec: containers: - name: gpa-app image: nginx:1.21.0 ports: - containerPort: 80 applying changes applying changes > kubectl apply -f test-deployment-upd.yaml > kubectl apply -f test-deployment-upd.yaml And this is a very popular solution that's sufficient for most use cases. However, we're discussing large, complex, and high-load systems where an extra request to the Kubernetes API can be an expensive operation. In the example above, we are making two separate requests every time: a GET followed by an APPLY. GET APPLY To delve even deeper into this situation, let's consider the presence of microservices or other systems that subscribe to and react to resource events. In this scenario, we would constantly be spamming the cluster with events, even when no actual changes are being made to the resource. This situation does not apply to the standard kubectl apply command because the utility itself performs a validation of the applied manifest by using information stored in the resource's annotations. Specifically, it uses the kubectl.kubernetes.io/last-applied-configuration annotation to intelligently determine what has changed and send only the necessary updates. This situation does not apply to the standard kubectl apply command because the utility itself performs a validation of the applied manifest by using information stored in the resource's annotations. Specifically, it uses the kubectl.kubernetes.io/last-applied-configuration annotation to intelligently determine what has changed and send only the necessary updates. kubectl.kubernetes.io/last-applied-configuration One of the most straightforward solutions is to first check for changes and then apply the new manifest only if the resource has genuinely been modified. To summarize everything discussed above, the logic for implementing a microservice or an operator for resource updates should be as follows: Get the current manifest of the resource from the cluster. Modify the retrieved resource. Compare the modified resource with its original state, and if it has changed, apply the new manifest. Get the current manifest of the resource from the cluster. Modify the retrieved resource. Compare the modified resource with its original state, and if it has changed, apply the new manifest. Let's name this approach GET-CHECK-APPLY GET-CHECK-APPLY Collaborative Resource Management Everything we've discussed so far represents a simple and elegant solution for a scenario where a microservice/user manages a resource. But what if multiple contributors are involved in modifying that same resource? This brings us to the main topic of this article: how to resolve the challenges of shared resource management politely and diplomatically. The first obvious step is to distribute attribute ownership among the contributors. For example, one service might be responsible for watching and updating the image, while another manages the number of replicas. “service-a” > cat test-deployment-service-a.yaml apiVersion: apps/v1 kind: Deployment metadata: name: gpa-app spec: ... template: ... spec: containers: - name: gpa-app image: nginx:1.21.0 # belongs to service-A ... > cat test-deployment-service-a.yaml apiVersion: apps/v1 kind: Deployment metadata: name: gpa-app spec: ... template: ... spec: containers: - name: gpa-app image: nginx:1.21.0 # belongs to service-A ... “service-b” > cat test-deployment-service-b.yaml apiVersion: apps/v1 kind: Deployment metadata: name: gpa-app spec: replicas: 3 # belongs to service-B ... > cat test-deployment-service-b.yaml apiVersion: apps/v1 kind: Deployment metadata: name: gpa-app spec: replicas: 3 # belongs to service-B ... Unfortunately, the GET-CHECK-APPLY approach will not be effective in this scenario. Since it operates on the entire resource manifest, a collision can occur when multiple services are working concurrently. Specifically, between the GET and APPLY steps of one service, another service might apply its own changes, which would then be overwritten by the first service's final APPLY. GET-CHECK-APPLY GET APPLY APPLY Patch as a solution for collaborative resource management The most straightforward and obvious path to solve the problem of collaborative resource management is to use PATCH. This approach works well for two main reasons: PATCH Field ownership is already distributed among the contributors. By using PATCH, each service can take responsibility for a specific set of fields, preventing conflicts. PATCH allows targeted update of only the required attributes. Instead of updating the entire manifest, you can send a partial update with just the fields you need to change. This is far more efficient and avoids overwriting modifications made by other services. Field ownership is already distributed among the contributors. By using PATCH, each service can take responsibility for a specific set of fields, preventing conflicts. PATCH PATCH allows targeted update of only the required attributes. Instead of updating the entire manifest, you can send a partial update with just the fields you need to change. This is far more efficient and avoids overwriting modifications made by other services. PATCH > cat test-deployment-service-a.yaml spec: replicas: 3 > kubectl patch deployment gpa-app --patch-file test-deployment-service-a.yaml > cat test-deployment-service-a.yaml spec: replicas: 3 > kubectl patch deployment gpa-app --patch-file test-deployment-service-a.yaml > cat test-deployment-service-b.yaml spec: template: spec: containers: - name: gpa-app image: nginx:1.22.0 > kubectl patch deployment gpa-app --patch-file test-deployment-service-b.yaml > cat test-deployment-service-b.yaml spec: template: spec: containers: - name: gpa-app image: nginx:1.22.0 > kubectl patch deployment gpa-app --patch-file test-deployment-service-b.yaml > cat kubectl get deployment gpa-app -o yaml apiVersion: apps/v1 kind: Deployment metadata: annotations: ... generation: 4 # one of the indicators that the resource has changed resourceVersion: "520392" name: gpa-app ... spec: replicas: 3 # changed replicas count by service-A ... template: ... spec: containers: - image: nginx:1.22.0 # changed image by service-B ... ... status: ... > cat kubectl get deployment gpa-app -o yaml apiVersion: apps/v1 kind: Deployment metadata: annotations: ... generation: 4 # one of the indicators that the resource has changed resourceVersion: "520392" name: gpa-app ... spec: replicas: 3 # changed replicas count by service-A ... template: ... spec: containers: - image: nginx:1.22.0 # changed image by service-B ... ... status: ... Unfortunately, we still can't abandon the GET-CHECK step. Like APPLY, PATCH also triggers a resource version change, which generates an event and creates noise, bothering our "neighbors" (other services and systems). GET-CHECK APPLY PATCH As a result, we've found that GET-CHECK-PATCH is more convenient than GET-CHECK-APPLY for collaborative work`. GET-CHECK-PATCH GET-CHECK-APPLY However, despite these improvements, this logic still feels quite cumbersome: To update a resource, we always make two separate API calls: GET and PATCH (or APPLY). We must implement complex logic to compare the initial state with the new state and decide whether to proceed. To update a resource, we always make two separate API calls: GET and PATCH (or APPLY). GET PATCH APPLY We must implement complex logic to compare the initial state with the new state and decide whether to proceed. In Kubernetes circles, this GET-CHECK-PATCH(APPLY) approach is known as Client-Side Apply (CSA), where all the logic for merging, conflict resolution, and validation is performed on the client side, and only the final result is applied. GET-CHECK-PATCH(APPLY) Client-Side Apply (CSA) While the client has significant control over the resource management process, many tools remain unavailable to it. For example, a client cannot prevent another from overwriting a set of fields that it owns. Kubernetes SSA Starting with Kubernetes v1.22, a highly effective and powerful declarative mechanism was introduced: Server-Side Apply (SSA). Server-Side Apply (SSA) SSA significantly simplifies collaborative resource management by moving the responsibility for updating, validating, and consolidating logic to the API server itself. The client only sends the desired state of the resource, and the Kubernetes API server handles all the complex logic under the hood. A key feature introduced with SSA is the mechanism of shared field management. The Kubernetes API server now knows which client is managing which field within a resource's specification. When a client sends a manifest using SSA, the API server checks if that client owns the fields it's trying to modify. If the field is unowned or already belongs to that client, the change is applied successfully. However, if another client owns the field, the API server will return an error, alerting you to a conflict or overwriting the owner based on your settings. SSA usage completely eliminates the need to use the GET-CHECK-PATCH(APPLY) approach. With SSA, you send the desired state, specify the client's name (field manager), and you receive the server's response. GET-CHECK-PATCH(APPLY) field manager It's important to note that using PATCH instead of APPLY of the entire manifest is still a best practice, as it allows your service to "claim" ownership of only the specific fields it manages. PATCH APPLY We can use the same patch files from the previous example, changing the replicas and image, and apply them using SSA. SSA > kubectl patch deployment gpa-app --patch-file test-deployment-service-a.yaml --field-manager=service-a > kubectl patch deployment gpa-app --patch-file test-deployment-service-b.yaml --field-manager=service-b > kubectl patch deployment gpa-app --patch-file test-deployment-service-a.yaml --field-manager=service-a > kubectl patch deployment gpa-app --patch-file test-deployment-service-b.yaml --field-manager=service-b > kubectl get deployment gpa-app -o yaml apiVersion: apps/v1 kind: Deployment metadata: annotations: ... generation: 6 # one of the indicators that the resource has changed resourceVersion: "534637" name: gpa-app ... spec: replicas: 1 # changed replicas count by service-A ... template: ... spec: containers: - image: nginx:1.21.0 # changed image by service-B ... ... status: ... > kubectl get deployment gpa-app -o yaml apiVersion: apps/v1 kind: Deployment metadata: annotations: ... generation: 6 # one of the indicators that the resource has changed resourceVersion: "534637" name: gpa-app ... spec: replicas: 1 # changed replicas count by service-A ... template: ... spec: containers: - image: nginx:1.21.0 # changed image by service-B ... ... status: ... To view a list of all managed fields, you need to expand the command kubectl get, by adding the --show-managed-fields flag kubectl get --show-managed-fields > kubectl get deployment gpa-app -o yaml --show-managed-fields apiVersion: apps/v1 kind: Deployment metadata: annotations: ... ... managedFields: - apiVersion: apps/v1 fieldsType: FieldsV1 fieldsV1: f:spec: f:replicas: {} # confirmation that the spec.replicas belong to service-a manager: service-a operation: Update time: "2025-08-26T00:23:50Z" - apiVersion: apps/v1 fieldsType: FieldsV1 fieldsV1: f:spec: f:template: f:spec: f:containers: k:{"name":"gpa-app"}: f:image: {} # confirmation that the spec.template.spec.containers.[0].image belong to service-b manager: service-b operation: Update time: "2025-08-26T00:24:05Z" ... name: gpa-app ... spec: ... > kubectl get deployment gpa-app -o yaml --show-managed-fields apiVersion: apps/v1 kind: Deployment metadata: annotations: ... ... managedFields: - apiVersion: apps/v1 fieldsType: FieldsV1 fieldsV1: f:spec: f:replicas: {} # confirmation that the spec.replicas belong to service-a manager: service-a operation: Update time: "2025-08-26T00:23:50Z" - apiVersion: apps/v1 fieldsType: FieldsV1 fieldsV1: f:spec: f:template: f:spec: f:containers: k:{"name":"gpa-app"}: f:image: {} # confirmation that the spec.template.spec.containers.[0].image belong to service-b manager: service-b operation: Update time: "2025-08-26T00:24:05Z" ... name: gpa-app ... spec: ... As you've seen, Kubernetes has "claimed" the replicas field for "service-a" and the image field for "service-b". replicas image This is the core of SSA's field management. If you now try to override the entire manifest again with SSA, Kubernetes will return an error because it detects a conflict. > kubectl apply -f test-deployment.yaml --field-manager=service-c --server-side error: Apply failed with 1 conflict: conflict with "service-a" using apps/v1: .spec.replicas Please review the fields above--they currently have other managers. Here are the ways you can resolve this warning: * If you intend to manage all of these fields, please re-run the apply command with the `--force-conflicts` flag. * If you do not intend to manage all of the fields, please edit your manifest to remove references to the fields that should keep their current managers. * You may co-own fields by updating your manifest to match the existing value; in this case, you'll become the manager if the other manager(s) stop managing the field (remove it from their configuration). See https://kubernetes.io/docs/reference/using-api/server-side-apply/#conflicts > kubectl apply -f test-deployment.yaml --field-manager=service-c --server-side error: Apply failed with 1 conflict: conflict with "service-a" using apps/v1: .spec.replicas Please review the fields above--they currently have other managers. Here are the ways you can resolve this warning: * If you intend to manage all of these fields, please re-run the apply command with the `--force-conflicts` flag. * If you do not intend to manage all of the fields, please edit your manifest to remove references to the fields that should keep their current managers. * You may co-own fields by updating your manifest to match the existing value; in this case, you'll become the manager if the other manager(s) stop managing the field (remove it from their configuration). See https://kubernetes.io/docs/reference/using-api/server-side-apply/#conflicts It correctly identifies that you, as the new applicant, do not own the fields that are already claimed by "service-a" and "service-b." This behavior is a key advantage of SSA, as it prevents unintentional overwrites and ensures that shared resources are updated collaboratively and safely. Conclusion By diving deep into Kubernetes resource management, it becomes clear that the evolution from Client-Side Apply to Server-Side Apply is not just a command-line change. The SSA is a fundamental shift in the philosophy of interacting with a cluster. While SSA may seem like a silver bullet, it does have its complexities, requiring a deeper understanding of Kubernetes architecture for a successful implementation. Client-Side Apply Server-Side Apply For a long time, CSA was our reliable companion. It gets the job done but comes with certain limitations. Its reliance on the kubectl.kubernetes.io/last-applied-configuration annotation makes it vulnerable to conflicts and errors, especially in complex, automated environments. In the hands of a single developer, CSA can be an effective tool for quick, simple operations. However, as soon as multiple systems or individuals try to manage the same resource simultaneously, its fragility becomes obvious. CSA can lead to unpredictable results, race conditions, and, as a consequence, cluster instability. kubectl.kubernetes.io/last-applied-configuration SSA solves these problems by moving complex logic to the API server. The field ownership management feature is a game-changer. The API server is no longer just executing commands; it becomes an intelligent arbiter that knows exactly who is responsible for which fields. This makes collaboration safe by preventing accidental overwrites and conflicts. For developers building operators and controllers, SSA is not just an option but a necessity. It unleashes the creation of robust and scalable systems that can coexist within the same cluster without interfering with each other. So, when should you use each approach? CSA can still be helpful in scenarios where you're manually managing resources and don't expect outside interference. It's light and straightforward for one-off operations. SSA is the new standard for all automated systems, operators, and teams working in high-load or shared environments. It is the path toward truly declarative, safe, and predictable state management for your cluster. CSA can still be helpful in scenarios where you're manually managing resources and don't expect outside interference. It's light and straightforward for one-off operations. CSA SSA is the new standard for all automated systems, operators, and teams working in high-load or shared environments. It is the path toward truly declarative, safe, and predictable state management for your cluster. SSA Ultimately, understanding these two approaches is key to working effectively and without errors in Kubernetes. By choosing Server-Side Apply, you're not just using a new command; you're adopting a modern, reliable, and more innovative way to manage your infrastructure. Thank you! Thank you!