# k8s-objectmatcher Administrator & Developer Guide This guide is the deep, source-based reference for `cisco-open/k8s-objectmatcher`. The library is small — one Go package (`patch`), four production source files, two test files, ~25 KB of source — but it sits in the critical path of Kubernetes controllers. A bug in the diff calculation can silently turn a healthy reconciler into a hot loop that thrashes the API server, or worse, into a quiet reconciler that fails to apply real changes. This document covers the architecture in detail, the full configuration surface, the patch-calculation algorithm step by step, the integration risk model, and the developer workflow for adding support for new resource kinds or custom cleaners. The audience is operator authors, controller-runtime users, library maintainers, and security reviewers tracing how patch payloads flow through the system. ## 1. Architecture Overview `k8s-objectmatcher` is a **pure-function diff engine** with two side effects on Kubernetes objects: stamping a last-applied annotation and reading it back. Everything else — the strategic-merge / JSON-merge dispatch, the per-kind cleaners, the null-stripping pass — operates on in-memory `[]byte` buffers without touching the cluster. ### 1.1 What the library does 1. **Marshals** both the current (live) object and the desired (modified) object into JSON. 2. **Applies** a configurable chain of `CalculateOption` filters that delete or normalise fields on both byte streams symmetrically. 3. **Strips nulls** and zero-valued `IntOrString` types via `DeleteNullInJson`. 4. **Reads** the last-applied snapshot from the live object's annotation (`banzaicloud.com/last-applied` by default), which the controller itself stamped on the previous reconcile. 5. **Computes** a three-way merge patch — strategic for typed objects, JSON-merge for `*unstructured.Unstructured`. 6. **Re-applies** the patch locally and re-diffs to suppress `$setElementOrder` noise on the strategic-merge path (and the JSON- merge path runs the same trick). 7. **Returns** a `*PatchResult` with the patch, the cleaned current bytes, the cleaned modified bytes, and the original snapshot. The controller compares `PatchResult.IsEmpty()` against `"{}"` and either skips the update (in-sync) or calls `client.Update` (real change). ### 1.2 ASCII architecture diagram ```text Controller Reconcile Loop currentObject (from API) modifiedObject (built locally) patch.PatchMaker 1. json.Marshal(currentObject) 2. json.Marshal(modifiedObject) 3. apply CalculateOption chain IgnoreStatusFields IgnoreField IgnoreVolumeClaimTemplateTypeMetaAndStatus (custom) 4. DeleteNullInJson on both 5. Annotator.GetOriginalConfiguration(currentObject) type switch on currentObject unstructured.Unstructured everything else BaseJSONMergePatcher K8sStrategicMergePatcher CreateThreeWay CreateThreeWayMergePatch JSONMergePatch then apply locally and then apply locally, CreateTwoWayMergePatch CreateMergePatch to collapse setElementOrder noise PatchResult Patch Current Modified Original IsEmpty() true false skip update client.Update SetLastAppliedAnnotation ``` ### 1.3 Component inventory | Component | Path | Role | |---|---|---| | `Annotator` | `patch/annotation.go` | Reads / writes the last-applied annotation. Handles zip+base64 with a legacy plain-value fallback. | | `PatchMaker` | `patch/patch.go` | Orchestrates the diff. The only public entry point most users touch. | | `Maker` (interface) | `patch/patch.go` | Single method `Calculate`. Decouples consumers from the concrete `*PatchMaker`. | | `PatchResult` | `patch/patch.go` | The return value of `Calculate`. Plain struct with four byte slices. | | `CalculateOption` | `patch/deletenull.go` | `func([]byte, []byte) ([]byte, []byte, error)`. The plug-in slot for filters. | | Per-kind cleaners | `patch/deletenull.go` | `IgnoreStatusFields`, `IgnoreField`, `IgnoreVolumeClaimTemplateTypeMetaAndStatus`, plus the unexported helpers `deleteStatusField`, `deleteDataField`, `deleteVolumeClaimTemplateFields`, `DeleteNullInJson`. | | `K8sStrategicMergePatcher` | `patch/patch_makers.go` | Default `StrategicMergePatcher`. Wraps `k8s.io/apimachinery/pkg/util/strategicpatch`. Has a `PreconditionFuncs` slice. | | `BaseJSONMergePatcher` | `patch/patch_makers.go` | Default `JSONMergePatcher`. Wraps `evanphx/json-patch` and `k8s.io/apimachinery/pkg/util/jsonmergepatch`. | | `init()` in deletenull.go | `patch/deletenull.go` | Registers a custom json-iterator encoder for `intstr.IntOrString` to write `nil` instead of `0` when the value is unset. | > [!NOTE] > The `init()` block in `patch/deletenull.go` overrides the global > json-iterator encoder for `intstr.IntOrString`. Importing the `patch` > subpackage anywhere in your binary changes the marshal behaviour of > json-iterator everywhere — this is intentional (so `Calculate` sees > stable bytes) but worth knowing if you also use json-iterator > elsewhere. ### 1.4 Technology stack | Layer | Component | Version | |---|---|---| | Language | Go | 1.22 | | JSON | `github.com/json-iterator/go` | v1.1.12 | | Strategic merge | `k8s.io/apimachinery/pkg/util/strategicpatch` | v0.19.16 | | JSON merge | `github.com/evanphx/json-patch` | v5.9.0+incompatible | | JSON-merge orchestration | `k8s.io/apimachinery/pkg/util/jsonmergepatch` | (vendored from apimachinery) | | Error wrapping | `emperror.dev/errors` | v0.8.1 | | Annotation access | `k8s.io/apimachinery/pkg/api/meta` (`MetadataAccessor`) | v0.19.16 | | Compression | `archive/zip` (stdlib) | n/a | | Encoding | `encoding/base64` (stdlib) | n/a | The integration test module (`tests/`) is a separate Go module with its own `go.mod` and pulls in `k8s.io/client-go` v0.21.3, `k8s.io/api` v0.21.3, and `k8s.io/apiextensions-apiserver` v0.21.3 — these are *not* runtime dependencies of `patch`. ### 1.5 Data flow for the main workflow ```text Controller Annotator PatchMaker Patcher impls Reconcile(req) Get(current) <-- API -- desired = build() SetLastApplied(desired) -> GetModified(desired,false) <- modifiedNoAnn -> SetOriginal(desired,modifiedNoAnn) Calculate(current, desired, opts...) json.Marshal(current) json.Marshal(desired) apply each opt DeleteNullInJson * 2 GetOriginal(current) <-- read annotation, base64+unzip --- if *unstructured.Unstructured CreateThreeWayJSONMergePatch -> jsonMergePatcher if patch != "{}": MergePatch -> jsonMergePatcher CreateMergePatch -> jsonMergePatcher else CreateThreeWayMergePatch -> strategicMergePatcher if patch != "{}": StrategicMergePatch -> strategicMergePatcher CreateTwoWayMergePatch -> strategicMergePatcher return PatchResult{Patch, Current, Modified, Original} if IsEmpty(): return Update(desired) -- API -> ``` ## 2. System Requirements Because the library is a Go module with no daemon and no CLI, the only runtime requirements are those of the controller binary that imports it. ### 2.1 Build requirements | Item | Version | Notes | |---|---|---| | Go toolchain | 1.22+ | Set in `go.mod` (`go 1.22`). | | Module proxy | Default `proxy.golang.org` | No private replacements needed. | | Network for `go get` | yes | One-time fetch; vendor afterwards if your build needs to be hermetic. | ### 2.2 Runtime requirements (from the calling controller) | Item | Requirement | Notes | |---|---|---| | Memory | None measurable beyond JSON marshal buffers | Each `Calculate` allocates 4× the object size temporarily (current, modified, patch, original). | | CPU | Negligible per call | Strategic merge is the heaviest step; even large StatefulSets finish in single-digit milliseconds. | | File system | None | Library is pure in-process. | | Kubernetes version | Any version where `apimachinery`'s strategic-merge logic still applies — effectively all supported versions (1.21+). | The library tracks `apimachinery` v0.19.16 in `go.mod`; later API server versions are forward-compatible because the patch metadata lookup is reflected from the Go struct, not from the API server. | | API permissions | Whatever the controller already has | The library makes no API calls of its own. | > [!NOTE] > The library's pinned `apimachinery v0.19.16` in `go.mod` is the > baseline for strategic-merge metadata. When a controller imports > `apimachinery v0.30+`, Go's MVS picks the newer version and that's > what's actually compiled. The pin only matters for builds that > downgrade — which is rare. ### 2.3 Object-size constraints The annotation system imposes Kubernetes' standard 256 KiB per-annotation cap. The library compresses with `archive/zip` before base64 encoding, so the effective ceiling is roughly 1–2 MiB of raw JSON (compression ratio depends on object content; ConfigMaps with text-heavy `data` compress well, dense binary fields don't). ## 3. Configuration Reference The library has **no config file** and **no environment variables**. Configuration happens at the call site — option arguments to `Calculate` and constructor arguments to `NewAnnotator` / `NewPatchMaker`. This section documents every knob. ### 3.1 `CalculateOption` reference | Option | Signature | Effect on byte stream | Implemented by | |---|---|---|---| | `IgnoreStatusFields()` | `() CalculateOption` | `delete(obj, "status")` on both sides | `deleteStatusField` | | `IgnoreField(field string)` | `(string) CalculateOption` | `delete(obj, field)` on both sides — top-level only | `deleteDataField` | | `IgnoreVolumeClaimTemplateTypeMetaAndStatus()` | `() CalculateOption` | Walks `spec.volumeClaimTemplates[*]` and **sets** `kind=""`, `apiVersion=""`, `status={"phase":"Pending"}` on each | `deleteVolumeClaimTemplateFields` | Application order matters because each option sees the byte stream mutated by the previous one. Order is the slice index in the `opts ...CalculateOption` variadic argument to `Calculate`. > [!IMPORTANT] > `IgnoreField` only affects **top-level** keys. To suppress a nested > field (e.g. `spec.replicas`, `metadata.labels[someKey]`), you must > implement a custom `CalculateOption`. This is a known gap — see > [#57](https://github.com/cisco-open/k8s-objectmatcher/issues/57). ### 3.2 `Annotator` configuration `NewAnnotator(key string)` is the only constructor. The single config input is the annotation key. Defaults: | Symbol | Value | Where set | |---|---|---| | `LastAppliedConfig` | `"banzaicloud.com/last-applied"` | `patch/annotation.go:32` | | `DefaultAnnotator` | `NewAnnotator(LastAppliedConfig)` | `patch/annotation.go:34` | > [!IMPORTANT] > Changing the annotation key in a running operator orphans the existing > snapshots on every live object. Plan a backfill: read every managed > object, call `SetLastAppliedAnnotation` with the new key, and > `Update`. Until the backfill completes, those objects' diffs degrade > to two-way merges and may produce large, server-defaulted patches. ### 3.3 `PatchMaker` configuration `NewPatchMaker(annotator *Annotator, smp StrategicMergePatcher, jmp JSONMergePatcher) Maker` takes three injected dependencies: | Argument | Default | Purpose | |---|---|---| | `annotator *Annotator` | `DefaultAnnotator` | Reads the last-applied snapshot during three-way merge. | | `smp StrategicMergePatcher` | `&K8sStrategicMergePatcher{}` | Three operations: `StrategicMergePatch`, `CreateTwoWayMergePatch`, `CreateThreeWayMergePatch`. The default delegates to `k8s.io/apimachinery/pkg/util/strategicpatch`. | | `jmp JSONMergePatcher` | `&BaseJSONMergePatcher{}` | Three operations: `MergePatch`, `CreateMergePatch`, `CreateThreeWayJSONMergePatch`. Default wraps `evanphx/json-patch` and `apimachinery/pkg/util/jsonmergepatch`. | The interfaces are deliberately small so you can substitute fakes in tests or wrap the defaults with metrics / tracing. ### 3.4 `K8sStrategicMergePatcher.PreconditionFuncs` The default strategic-merge implementation has a single configurable field: ```go type K8sStrategicMergePatcher struct { PreconditionFuncs []mergepatch.PreconditionFunc } ``` Precondition functions run on the patch and can short-circuit the merge with an error. The two stock functions in `k8s.io/apimachinery/pkg/util/mergepatch` are: - `RequireKeyUnchanged(key)` — error if the patch attempts to change the named key. - `RequireMetadataKeyUnchanged(key)` — same, but scoped to `metadata`. ```go maker := patch.NewPatchMaker( patch.DefaultAnnotator, &patch.K8sStrategicMergePatcher{ PreconditionFuncs: []mergepatch.PreconditionFunc{ mergepatch.RequireKeyUnchanged("apiVersion"), mergepatch.RequireMetadataKeyUnchanged("uid"), }, }, &patch.BaseJSONMergePatcher{}, ) ``` These run only on the `CreateThreeWayMergePatch` and `CreateTwoWayMergePatch` paths. Use them to harden against logic bugs that would otherwise produce a patch that mutates an immutable field. ### 3.5 The full configuration surface, in one table | Where | Symbol | Type | Purpose | |---|---|---|---| | Package | `LastAppliedConfig` | `const string` | Default annotation key | | Package | `DefaultAnnotator` | `*Annotator` | Pre-built annotator with the default key | | Package | `DefaultPatchMaker` | `Maker` | Pre-built `PatchMaker` with default deps | | Per-call | `opts ...CalculateOption` | variadic | Applied left-to-right inside `Calculate` | | Constructor | `NewAnnotator(key)` | constructor arg | Custom annotation key | | Constructor | `NewPatchMaker(annotator, smp, jmp)` | constructor args | Custom annotator and patcher impls | | Field | `K8sStrategicMergePatcher.PreconditionFuncs` | `[]mergepatch.PreconditionFunc` | Patch validation guards | There are **no** other knobs. No timeouts (the library doesn't make network calls), no logger injection (errors are returned, not logged), no rate limits. ## 4. Deployment & Scaling Deployment of `k8s-objectmatcher` is deployment of the controller binary that imports it. There is nothing to deploy separately. ### 4.1 Build deployment checklist - [x] `go get github.com/cisco-open/k8s-objectmatcher/patch` - [x] Pin the version in `go.mod` (don't track `latest` in production controllers) - [x] Run `go mod vendor` if your build is hermetic - [x] Cross-compile to your target arch (Linux/AMD64 and Linux/ARM64 both work without changes) - [x] Build the controller container image as usual - [x] Deploy via your normal operator-distribution channel (Helm, OLM, raw manifests) ### 4.2 Scaling considerations The library is **stateless and pure** — there is nothing to scale horizontally or vertically. Performance characteristics: | Metric | Cost | |---|---| | Memory per `Calculate` | ~4× the JSON size of the object (current, modified, patch, original transient buffers) | | CPU per `Calculate` | dominated by `strategicpatch.CreateThreeWayMergePatch` — ~milliseconds for typical objects | | GC pressure | high — every call allocates fresh maps for unmarshal + marshal cycles | | Lock contention | none — the library has no shared mutable state | For controllers reconciling thousands of objects per second, the GC allocations are usually the limiting factor. There is no built-in pooling of `[]byte` or `map[string]interface{}` buffers; consider profiling with `-memprofile` and a `sync.Pool` wrapper if you hit hot-path GC pressure. > [!TIP] > If you reconcile the same object frequently and the snapshot rarely > changes, cache the `PatchResult.Patch == "{}"` decision at the > controller layer keyed by `(uid, resourceVersion)`. The library > itself does no caching — by design, since determining whether > `resourceVersion` changes equate to semantic changes is the very > question the library exists to answer. ### 4.3 Versioning policy The repo follows Go module versioning. Major version bumps would require a path change to `/v2` per the Go module rules. As of this writing, the library lives at the v0/v1 module path (`github.com/cisco-open/k8s-objectmatcher`). ## 5. Operations Operating a controller that uses this library is operating any other controller — there's no dedicated runbook for the library itself. That said, three operational concerns deserve their own subsections. ### 5.1 Observability The library does **not** log. Every error is returned wrapped via `emperror.dev/errors` and propagated up to the caller. The controller is responsible for logging: ```go result, err := maker.Calculate(cur, des, patch.IgnoreStatusFields()) if err != nil { logger.Error(err, "patch calculation failed", "kind", cur.GetObjectKind().GroupVersionKind().Kind, "name", cur.GetName()) return err } if !result.IsEmpty() { logger.V(1).Info("applying patch", "patch", string(result.Patch), "size", len(result.Patch)) } ``` > [!NOTE] > Logging `result.Patch` at `V(1)` is invaluable for debugging — but > see Section 7.3 below for the security implication: patches can > contain secret material if your controller manages Secrets or > Secret-bearing objects. Log them at `V(2)` and through a > redacting log sink in production. ### 5.2 Annotation hygiene The last-applied annotation grows with the object. For a Deployment or StatefulSet, expect 2–10 KiB of base64-encoded zip per object. Per-namespace etcd consumption can become noticeable at thousands of objects, but the per-object annotation cap (256 KiB) is the more important limit. | Symptom | Likely cause | Mitigation | |---|---|---| | `annotation: too long` from API server | Object > ~1.5 MiB raw JSON | Use `IgnoreField` to drop a large `data` block before annotating, or split the object | | Annotation absent on live objects | First reconcile didn't call `SetLastAppliedAnnotation` | Backfill loop on startup | | Annotation never updates | Controller updates via `Patch` instead of `Update`, and the patch doesn't include the annotation | Switch to `Update`, or add the annotation to the `Patch` payload | ### 5.3 Upgrading the library in a running operator Drop-in safe for the supported version line. The annotation format has been **stable since the zip+base64 migration**, and the read path keeps a fallback for plain JSON values, so upgrading from any older Banzai Cloud or `cisco-open` build works without flag days. The strategic-merge / JSON-merge dispatch table is also stable. The public surface — `PatchMaker`, `Annotator`, `CalculateOption`, `PatchResult` — is treated as part of the contract. > [!TIP] > When upgrading, run a `go test ./patch/...` against a unit-test > harness with representative objects from your production cluster. > The library has 100% public-API coverage in its own tests; layered > tests in your controller catch any subtle behaviour changes that > matter to *your* CRDs. ## 6. Performance The hot path is `Calculate`. Cost analysis: ### 6.1 Per-call cost breakdown | Step | Cost | Notes | |---|---|---| | `json.Marshal(currentObject)` | O(N) where N is JSON size | json-iterator is ~2× faster than stdlib | | `json.Marshal(modifiedObject)` | O(N) | same | | Each `CalculateOption` | O(N) — Unmarshal + delete + Marshal | The shipped options round-trip the entire object; for large objects this is the heaviest step | | `DeleteNullInJson` × 2 | O(N) — recursive walk | Visits every key once | | `GetOriginalConfiguration` | O(annot_size) — base64 + unzip | ~half the size of original JSON | | Strategic merge `CreateThreeWayMergePatch` | O(N) plus reflection over the Go type | Dominant for typed objects | | Strategic merge re-apply + two-way | O(N) again | Doubles the strategic-merge cost; required to suppress `$setElementOrder` noise | | JSON merge `CreateThreeWayJSONMergePatch` | O(N) | Cheaper than strategic — no reflection | A typical Deployment runs `Calculate` in 1–3 ms on modern hardware. StatefulSets with many `volumeClaimTemplates` are slower because `IgnoreVolumeClaimTemplateTypeMetaAndStatus` walks the templates list in addition to the rest of the spec. ### 6.2 Allocation profile The dominant allocations are: 1. The `map[string]interface{}` returned by `json.Unmarshal` inside each option and inside `DeleteNullInJson`. 2. The `[]byte` slices returned by every `Marshal`. A single `Calculate` against a Deployment allocates ~30–50 KiB of heap. The GC handles this fine in normal operation; controllers running at >1000 reconciles/sec/CPU may want to look at hot-path profiling. > [!NOTE] > The library does not pool buffers internally. If you need pooling, > wrap the `Maker` interface in your own implementation that pools > the `[]byte` outputs of `Calculate` for downstream logging. ### 6.3 Benchmarking There are no shipped benchmarks. To measure your own: ```go func BenchmarkCalculate(b *testing.B) { cur := buildDeployment("cur") des := buildDeployment("des") maker := patch.DefaultPatchMaker b.ResetTimer() for i := 0; i < b.N; i++ { _, _ = maker.Calculate(cur, des, patch.IgnoreStatusFields()) } } ``` Run with `go test -bench=. -benchmem ./yourpkg/...`. ## 7. Security `k8s-objectmatcher` has **no network surface** — it never opens a socket, never makes an HTTP call, never reads a file. The security posture is therefore framed in terms of **integration risk** (what happens when the calling controller misuses the library) and **data exposure** (what's in the patch payloads that leave the library's address space). ### 7.1 Integration risk: the buggy-patcher problem A bug in patch calculation propagates to a destructive reconcile. Two failure modes are worth calling out: | Failure mode | Mechanism | Blast radius | |---|---|---| | **Spurious non-empty patch** | A `CalculateOption` mutates one side but not the other; the diff sees the asymmetry as a real change | Hot reconcile loop, API-server pressure, audit-log noise | | **Spurious empty patch** | An option deletes a field on both sides that *should* have triggered an update | Real changes go undetected; the cluster silently drifts from desired state | The first failure is loud and self-correcting (you'll notice in metrics and back off). The second is silent and dangerous. Custom `CalculateOption`s must be **symmetric** — mutate `current` and `modified` the same way — and unit-tested with deliberately mutated inputs. > [!CAUTION] > A custom `CalculateOption` that deletes a field only on the > `modified` side will silently mask real updates. Always assert in > tests that the option produces *identical* mutations on both byte > slices for inputs that are equal. ### 7.2 Operator security posture The library inherits the operator's RBAC and the operator's threat model. Specifically: - The operator typically holds verbs `get`, `list`, `watch`, `create`, `update`, `patch`, `delete` on the resources it manages. - A bug in the library (or in the controller's use of it) cannot exceed those verbs — `Calculate` produces bytes, but only the controller's `client.Update` call commits them. - The annotation it stamps is a benign metadata field. It does not carry capability or trust beyond what's already in the object. > [!WARNING] > Do **not** treat the last-applied annotation as a source of truth > for security decisions (e.g. "this object was previously approved > because it has our annotation"). Annotations can be written by any > principal with `update` on the object — including humans with > kubectl access. The annotation is a controller-internal cache, not > an authentication signal. ### 7.3 Patch payloads in audit logs and observability pipelines The patch bytes that flow through `Calculate` are **the diff of two Kubernetes objects**, which means they can carry sensitive material for any object type that holds secrets: | Object kind | Sensitive content in the patch | |---|---| | `Secret` | The full `data` map (base64-encoded plaintext) when changed | | `ConfigMap` | Embedded credentials, connection strings | | Pod template specs | `env` values, mounted secret references, SA token path | | CRDs storing credentials inline | Whatever the CRD schema permits | If your controller logs `result.Patch` (a useful debugging practice) and your log pipeline forwards everything to a centralized store (Loki, Splunk, Datadog), you have just exfiltrated secret material to that pipeline. Same for the audit log if you `kubectl describe` the patch payload through some intermediary. **Recommendations:** 1. Log `result.Patch` only at high verbosity (e.g. `V(2)` in logr/klog). 2. Configure a redacting log sink that strips `Secret.data`, `ConfigMap.data` for known-sensitive keys, and `env` values matching `*_PASSWORD`, `*_TOKEN`, `*_KEY` patterns. 3. For controllers that manage `Secret` objects, consider replacing the patch payload with a hash for log purposes: ```go import "crypto/sha256" if !result.IsEmpty() { hash := sha256.Sum256(result.Patch) logger.V(1).Info("patch needed", "patch_sha256", fmt.Sprintf("%x", hash[:8])) } ``` > [!CAUTION] > Audit logs and tracing spans are also exposure surfaces. > `result.String()` formats the entire `Patch`, `Current`, `Modified`, > and `Original` — that's *four* copies of the (possibly secret- > bearing) object, one of them base64-decoded from the annotation. > Never put `result.String()` in a span attribute. ### 7.4 Supply chain The library has a small, well-known dependency graph: | Dep | Trust | |---|---| | `k8s.io/apimachinery` | Kubernetes core, signed releases | | `github.com/json-iterator/go` | Mature, widely used | | `github.com/evanphx/json-patch` | Reference JSON-patch implementation, used by Kubernetes itself | | `emperror.dev/errors` | Small wrapper, low risk | The license inventory is enforced by `.licensei.toml` (Apache-2.0, MIT, BSD, MPL-2.0). License audits are routine — see the file for the exact ignore list. ## 8. Developer Internals This section documents the internals so contributors can extend the library without breaking the contract. ### 8.1 Repository structure | Path | Role | |---|---| | `patch/annotation.go` | `Annotator` type, `NewAnnotator`, the four annotation methods, the zip+base64 codec | | `patch/annotation_test.go` | Unit test for the empty-annotation edge case | | `patch/deletenull.go` | `CalculateOption` type, the three shipped options, `DeleteNullInJson`, the unexported `delete*Field` helpers, and the json-iterator `intstr.IntOrString` encoder override | | `patch/patch.go` | `Maker` interface, `PatchMaker` struct, `Calculate`, `PatchResult`, `unstructuredJsonMergePatch` (private) | | `patch/patch_makers.go` | `StrategicMergePatcher` interface, `JSONMergePatcher` interface, the two default impls | | `patch/patch_test.go` | Unit tests for the unstructured JSON-merge code path | | `tests/` | Integration tests against a real or simulated API server (separate Go module) | | `docs/legacy.md` | Retained historical documentation | | `Makefile` | `go test`, lint, license-check targets | | `.golangci.yml` | linter config (gocyclo min-complexity 15, misspell, gofmt, goimports enabled) | | `.licensei.toml` | License-audit allow/ignore lists | ### 8.2 The strategic-merge flow, step by step The flow inside `(*PatchMaker).Calculate` for a typed object (the "everything else" branch of the type switch): 1. **Marshal current and modified** with json-iterator (which uses the custom `IntOrString` encoder registered in `init()` — so `IntOrString{IntVal: 0, StrVal: ""}` becomes `null`, not `0`). 2. **Apply each `CalculateOption`** in order. Each option round- trips through Unmarshal + delete + Marshal, which means the per-option cost is O(N) regardless of how small the deletion is. 3. **`DeleteNullInJson`** walks the entire JSON tree and removes any key whose value is `nil` or zero-valued (per `isZero` — which treats float, int64, bool as **never** zero so they aren't accidentally elided when the user intentionally set 0/false). 4. **`Annotator.GetOriginalConfiguration(currentObject)`** reads the annotation. The implementation: - Reads `annots[a.key]`. - Tries base64 decode. If the result has content type `application/zip`, unzips and returns the inner bytes. - If base64 decode fails, falls back to returning the raw annotation bytes (legacy plain-JSON path). 5. **Type switch on `currentObject`:** - `*unstructured.Unstructured`: route to `(*PatchMaker).unstructuredJsonMergePatch`. - default: continue with strategic merge. 6. **`strategicMergePatcher.CreateThreeWayMergePatch(original, modified, current, currentObject)`** produces the candidate patch. 7. **If `string(patch) != "{}"`:** - Apply the patch to `current` locally: `patchCurrent, _ := strategicMergePatcher.StrategicMergePatch(current, patch, currentObject)`. - Re-diff: `patch, _ := strategicMergePatcher.CreateTwoWayMergePatch(current, patchCurrent, currentObject)`. This collapses `$setElementOrder` markers that strategic merge generates even for unchanged lists. 8. **Build and return the `PatchResult`.** The unstructured path mirrors the same trick: `CreateThreeWayJSONMergePatch` → if non-empty, `MergePatch` to apply locally, then `CreateMergePatch` to re-derive a clean diff. ### 8.3 The per-kind cleaner pattern A "cleaner" is a `CalculateOption` that knows about a specific Kubernetes kind and normalises a field that the API server mutates. The shipped cleaner is `IgnoreVolumeClaimTemplateTypeMetaAndStatus`, which targets StatefulSet `spec.volumeClaimTemplates`. The pattern: ```go func IgnoreXYZ() CalculateOption { return func(current, modified []byte) ([]byte, []byte, error) { current, err := mutateXYZ(current) if err != nil { return nil, nil, errors.Wrap(err, "...") } modified, err = mutateXYZ(modified) if err != nil { return nil, nil, errors.Wrap(err, "...") } return current, modified, nil } } func mutateXYZ(obj []byte) ([]byte, error) { var resource map[string]interface{} if err := json.Unmarshal(obj, &resource); err != nil { return nil, errors.Wrap(err, "could not unmarshal byte sequence") } // ...navigate into the structure, normalise the field... out, err := json.ConfigCompatibleWithStandardLibrary.Marshal(resource) if err != nil { return nil, errors.Wrap(err, "could not marshal byte sequence") } return out, nil } ``` `mutateXYZ` is responsible for being **defensive** — every layer of type assertion (`if spec, ok := resource["spec"].(map[string]interface{}); ok`) must guard against the field not existing. Returning an unmodified byte stream when the field isn't present is the correct behaviour. ### 8.4 Adding support for a new resource kind The high-level recipe to add an `IgnoreFooBarBaz()` option for a new kind: 1. **Identify the server-mutated fields** by `kubectl get -o json` the live object after creation and diffing it against the desired YAML you submitted. Note every key that appears in the live but not the desired. 2. **Decide the mutation strategy** — *delete* the field on both sides (use `delete()`), or *normalise* it to a fixed value (set it on both sides, like `IgnoreVolumeClaimTemplateTypeMetaAndStatus` does). 3. **Implement** following the cleaner pattern above. 4. **Add a unit test** in `patch/deletenull_test.go` (create the file if it doesn't exist) that: - Creates two byte streams with the field set. - Runs the option. - Asserts both streams come out identical and the field is gone (or normalised). - Asserts that for inputs without the field, the option is a no-op. 5. **Test against `Calculate`** end-to-end with a representative `*unstructured.Unstructured` and a typed object. 6. **Document** in this guide and the User Guide's option table. > [!NOTE] > The `tests/` directory at the repo root is a separate Go module > with `k8s.io/client-go` and `apiextensions-apiserver` as > dependencies. End-to-end tests for new options usually live there > rather than in the `patch/` unit tests, because they require a > real (or `envtest`-simulated) API server. ### 8.5 Why the second pass exists Strategic merge generates `$setElementOrder/` directives whenever a mergeable list is involved, even when the contents and order are identical. That's because strategic merge's three-way algorithm computes the order from the merge of three versions and emits a directive when the **logical** order could differ — even if in practice it doesn't. The library's response is to **apply the patch back onto current** and re-diff with two-way merge. If applying the patch yields the same `current`, the two-way diff is empty, and the patch collapses to `{}`. This is the key innovation that makes the library practical for real controllers; without it, every reconcile of any object with a list (Pods always have at least `containers`, `volumes`, etc.) would see a non-empty patch. The unstructured path uses the same trick but with `MergePatch` (apply) and `CreateMergePatch` (re-derive). JSON merge doesn't have `$setElementOrder` semantics — but it does have its own quirks around explicit-`null` values, which the second-pass round-trip also smooths over. ### 8.6 The `intstr.IntOrString` json-iterator override The `init()` block in `patch/deletenull.go` is the single largest behavioural override the library imposes on its callers' process. It registers a custom encoder for `intstr.IntOrString`: - If `IntVal == 0` and `StrVal != "" && StrVal != "0"` → write the string. - If `IntVal == 0` and `StrVal == ""` or `StrVal == "0"` → write `null`. - Otherwise → write the int. The companion `IsEmpty` predicate returns `true` only for the second case — so json-iterator omits the field entirely when serialising a struct with `omitempty`. The motivation: Kubernetes uses `intstr.IntOrString` for `targetPort`, `maxSurge`, `maxUnavailable`, `port`, etc. Default json-iterator (and stdlib `encoding/json`) marshal an unset `IntOrString` as `0`, which produces a false diff against a server representation that often serialises it as `null` or omits it. Forcing both sides to write `null` for the unset case eliminates that whole category of spurious diff. > [!IMPORTANT] > Importing `github.com/cisco-open/k8s-objectmatcher/patch` registers > this encoder process-wide. Any other code in the binary that uses > json-iterator to marshal an `IntOrString` will see the same > behaviour. This is *almost always* what you want for Kubernetes > code, but be aware of it. ### 8.7 Testing strategy | Layer | Files | Focus | |---|---|---| | Unit (annotation) | `patch/annotation_test.go` | Empty-annotation edge case (verifies `metadata.annotations` is removed when its only entry was `LastAppliedConfig`) | | Unit (patch) | `patch/patch_test.go` | The unstructured JSON-merge code path against four canonical scenarios: non-existent field not deleted, existent field deleted, existent field updated, new field added | | Integration | `tests/integration_test.go`, `tests/main_test.go` | Real API server interactions (`tests/` is a separate module) | Run the production unit tests: ```bash go test ./patch/... ``` ```text ok github.com/cisco-open/k8s-objectmatcher/patch 0.123s ``` Run the integration suite (requires `KUBECONFIG`): ```bash cd tests && go test ./... ``` ### 8.8 Code conventions From `.golangci.yml`: - gocyclo `min-complexity: 15` - linters enabled: `misspell`, `gofmt`, `goimports` - linters explicitly disabled: `golint`, `errcheck`, `gas`, `megacheck`, `varcheck`, `gochecknoglobals`, `govet` (the last due to a known false positive about non-nil ≠ nil) Errors are wrapped with `emperror.dev/errors` (not stdlib `fmt.Errorf` with `%w`) to attach contextual key/value pairs via `errors.WrapWithDetails`. Every Go file carries the Apache-2.0 license header, with `Copyright © :YEAR: Banzai Cloud`. The header template is enforced by `.licensei.toml`. ### 8.9 Potential improvements | Area | Today | Improvement | |---|---|---| | Nested-field ignore | Only top-level | Add `IgnorePath("spec.replicas")` ([#57](https://github.com/cisco-open/k8s-objectmatcher/issues/57)) | | Pod label ignore | Manual via custom option | First-class `IgnorePodLabel("key")` ([#65](https://github.com/cisco-open/k8s-objectmatcher/issues/65)) | | StatefulSet field exclusion | Cleaner pattern only | `IgnoreStatefulSetFields(...)` curated set ([#40](https://github.com/cisco-open/k8s-objectmatcher/issues/40)) | | `$setElementOrder` residuals | Second-pass mostly catches them | Investigate the Volumes/VolumeMounts edge case ([#9](https://github.com/cisco-open/k8s-objectmatcher/issues/9)) | | `IgnoreVolumeClaimTemplateTypeMetaAndStatus` edge cases | Reported broken in some scenarios ([#51](https://github.com/cisco-open/k8s-objectmatcher/issues/51)) | Add test cases for the failing inputs and harden the cleaner | | Buffer pooling | None | Optional `sync.Pool` for the `[]byte` and `map[string]interface{}` buffers in the hot path | ### 8.10 Contributing Standard fork-and-PR flow against `github.com/cisco-open/k8s-objectmatcher`. The repo enforces: 1. Apache-2.0 license header on every Go file (template in `.licensei.toml`). 2. `gofmt` + `goimports` clean. 3. `go test ./patch/...` passes. 4. The integration suite in `tests/` passes if your change touches anything that an API server interaction would observe. Open issues are catalogued at [github.com/cisco-open/k8s-objectmatcher/issues](https://github.com/cisco-open/k8s-objectmatcher/issues). ## 9. Integration Points The library's only integration with the outside world is the calling controller. This section enumerates the integration surfaces and how each maps to a public API symbol. ### 9.1 Inbound integration: what callers depend on | Caller need | API symbol | |---|---| | "Is there a real diff?" | `(*PatchMaker).Calculate` → `(*PatchResult).IsEmpty` | | "What changed?" | `(*PatchResult).Patch` | | "Stamp the desired state on a new object" | `(*Annotator).SetLastAppliedAnnotation` | | "Use a custom annotation key" | `NewAnnotator(key)` | | "Inject a fake patcher in tests" | `NewPatchMaker(annotator, smp, jmp)` and the `Maker` interface | | "Add a precondition guard" | `K8sStrategicMergePatcher.PreconditionFuncs` | | "Strip a top-level field" | `IgnoreField(field)` | | "Strip status" | `IgnoreStatusFields()` | | "Handle StatefulSet PVC templates" | `IgnoreVolumeClaimTemplateTypeMetaAndStatus()` | | "Strip nulls / zero IntOrStrings outside Calculate" | `DeleteNullInJson(jsonBytes)` | ### 9.2 Outbound integration: what the library depends on | Dependency | Use | |---|---| | `k8s.io/apimachinery/pkg/api/meta.MetadataAccessor` | Read/write annotations on any `runtime.Object` without type-asserting | | `k8s.io/apimachinery/pkg/runtime.Object` | The single argument type that lets `Calculate` accept any Kubernetes object | | `k8s.io/apimachinery/pkg/util/strategicpatch` | All three strategic-merge operations | | `k8s.io/apimachinery/pkg/util/jsonmergepatch` | `CreateThreeWayJSONMergePatch` | | `k8s.io/apimachinery/pkg/util/mergepatch` | The `PreconditionFunc` type used by `K8sStrategicMergePatcher` | | `k8s.io/apimachinery/pkg/apis/meta/v1/unstructured` | Type assertion for the JSON-merge dispatch | | `k8s.io/apimachinery/pkg/util/intstr` | `IntOrString` encoder override | | `github.com/json-iterator/go` | All JSON marshal/unmarshal | | `github.com/evanphx/json-patch` | `MergePatch`, `CreateMergePatch` for the unstructured path | | `emperror.dev/errors` | Error wrapping with context | ### 9.3 Common integration patterns | Pattern | Files / examples | |---|---| | controller-runtime `Reconcile` | See User Guide §5.5 | | client-go shared informer | Same pattern — substitute `informer.GetStore().Get` for `Get` | | Operator SDK | controller-runtime under the hood; works the same | | Cross-cluster reconciler | Each cluster's `Calculate` runs locally on the watcher — the library doesn't care about cluster topology | | Helm post-renderer style | Render desired YAML, parse to typed objects, call `Calculate` against the live cluster | ### 9.4 Compatibility matrix | Component | Compatible versions | |---|---| | Go | 1.22+ (any newer Go works) | | `k8s.io/apimachinery` | v0.19.16 (baseline in `go.mod`); MVS upgrades to whatever the calling controller needs (typically v0.28+ in current operators) | | `controller-runtime` | Any version — the library doesn't import it | | Kubernetes API server | 1.21+ (matches the test module's `client-go` v0.21.3 baseline; the strategic-merge metadata works across all currently supported versions) | | `*unstructured.Unstructured` | Always supported — JSON-merge path | | Typed objects with `patchStrategy` struct tags | Always supported — strategic-merge path | | Typed CRD objects without strategic-merge tags | Use unstructured access; see §8.2 and User Guide §5.7 | ### 9.5 What the library does **not** integrate with To set expectations and prevent looking-for-features-that-aren't-there: | Not integrated | Why | |---|---| | Server-side apply (`metadata.managedFields`) | Different model — API server handles diff calculation server-side | | `kubectl.kubernetes.io/last-applied-configuration` | Different annotation key; not interoperable with `kubectl apply` clients out of the box | | Webhooks (admission, validation) | Outside the diff layer | | Metrics (Prometheus / OpenTelemetry) | Caller's responsibility | | Logging | Caller's responsibility — library returns errors, never logs | | Tracing | Caller's responsibility | | Feature flags | None defined | > [!NOTE] > If you need interoperability with `kubectl apply`'s > `last-applied-configuration` annotation, you can swap the key with > `NewAnnotator("kubectl.kubernetes.io/last-applied-configuration")`. > The format `kubectl` writes is plain JSON (not zip+base64), and > the `Annotator` read path falls back to plain JSON when base64 > decode fails — so reading interoperates. Writing through this > library will switch the annotation to zip+base64 on the next > reconcile, which `kubectl apply --prune` does not understand. > Mixing the two is generally inadvisable. --- _Generated April 2026 from cisco-open/k8s-objectmatcher at 17b471d. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._