# k8s-objectmatcher User Guide

`k8s-objectmatcher` is a small, focused Go library that answers a single
question for Kubernetes operator authors: **"is the live object in the
cluster meaningfully different from the object I want to apply?"** It does
this by computing a strategic-merge or JSON-merge patch between a desired
object and the object the API server currently holds, then stripping the
fields the server itself fills in (`status`, defaulted values, `clusterIP`,
auto-injected metadata). What's left is a real, reconcilable diff — or an
empty patch, which means "no update needed."

The library originated at Banzai Cloud and lives under `cisco-open` today.
It is the diff engine inside several production operators and exposes a
deliberately tiny surface: one `PatchMaker.Calculate` call, a handful of
`CalculateOption` filters, and an annotation-based three-way merge so your
controller can compare current state, desired state, and the last-applied
state stored on the object.

This guide covers installation, the public API, the seven options most
controllers use, and recipes for the patterns that come up daily in
controller-runtime reconcile loops.

## 1. Quick Start

`k8s-objectmatcher` is a Go library. There is no binary, no daemon, no CLI.
You add the import, instantiate a `PatchMaker`, and call `Calculate(current,
desired, opts...)` from inside your controller's `Reconcile` method. If
`PatchResult.IsEmpty()` returns `true`, skip the update; otherwise, apply
the modified object and persist the new last-applied annotation.

**Prerequisites:** Go 1.22 or newer, a project that already imports
`k8s.io/apimachinery` (every controller-runtime project does).

**Install:**

```bash
go get github.com/cisco-open/k8s-objectmatcher/patch
```

**Minimal example — the canonical 10-line wiring:**

```go
package main

import (
	"context"
	"fmt"

	"github.com/cisco-open/k8s-objectmatcher/patch"
	appsv1 "k8s.io/api/apps/v1"
	"sigs.k8s.io/controller-runtime/pkg/client"
)

func reconcileDeployment(ctx context.Context, c client.Client, desired *appsv1.Deployment) error {
	current := &appsv1.Deployment{}
	if err := c.Get(ctx, client.ObjectKeyFromObject(desired), current); err != nil {
		return err
	}

	result, err := patch.DefaultPatchMaker.Calculate(current, desired, patch.IgnoreStatusFields())
	if err != nil {
		return err
	}
	if result.IsEmpty() {
		return nil // already in sync — skip the update
	}
	fmt.Printf("patch needed: %s\n", string(result.Patch))
	return c.Update(ctx, desired)
}
```

**Expected behaviour:**

- First reconcile of a new object: `IsEmpty()` returns `false`, you create
  it and call `patch.DefaultAnnotator.SetLastAppliedAnnotation(desired)`
  before the create so future diffs are three-way.
- Steady state: `IsEmpty()` returns `true`. No spurious updates, no fights
  with controllers that own `status`.

> [!TIP]
> Skip per-controller wiring for the common case — `patch.DefaultPatchMaker`
> and `patch.DefaultAnnotator` are package-level globals that work out of
> the box. Build your own `NewPatchMaker(...)` only when you need a custom
> annotation key or you're injecting fakes for tests.

**What to try next:** Section 4 walks through the public types and every
option. Section 5 has recipes for `Service` `clusterIP`, StatefulSet
`volumeClaimTemplates`, and a wired-in controller-runtime reconcile.

## 2. Installation

The library is distributed as a Go module. There is nothing to compile
ahead of time and no separate runtime to deploy.

### 2.1 Requirements

| Requirement | Value | Source |
|---|---|---|
| Go toolchain | 1.22 or newer | `go.mod` line `go 1.22` |
| `k8s.io/apimachinery` | v0.19.16 or compatible (the indirect kept by `go mod tidy`) | `go.mod` |
| `emperror.dev/errors` | v0.8.1 | `go.mod` |
| `github.com/evanphx/json-patch` | v5.9.0+incompatible | `go.mod` |
| `github.com/json-iterator/go` | v1.1.12 | `go.mod` |

The library has **zero CGO**, no platform-specific code, and runs anywhere
Go runs (Linux, macOS, Windows, ARM64, AMD64).

### 2.2 Add to a Go module

```bash
go get github.com/cisco-open/k8s-objectmatcher/patch
go mod tidy
```

```text
go: added github.com/cisco-open/k8s-objectmatcher v0.x.y
go: added emperror.dev/errors v0.8.1
go: added github.com/evanphx/json-patch v5.9.0+incompatible
go: added github.com/json-iterator/go v1.1.12
```

> [!IMPORTANT]
> The importable subpackage is `patch`, not the module root. Always import
> `github.com/cisco-open/k8s-objectmatcher/patch` — the bare module path
> exposes nothing useful.

### 2.3 Vendoring (optional)

If your project vendors dependencies (controller-runtime projects often
do), run a vendor refresh after `go get`:

```bash
go mod vendor
```

```text
(no output on success)
```

### 2.4 Verify the install

A one-line smoke test is enough:

```go
package main

import (
	"fmt"
	"github.com/cisco-open/k8s-objectmatcher/patch"
)

func main() {
	a := patch.NewAnnotator("example.com/last-applied")
	fmt.Println(a != nil) // true
}
```

```bash
go run ./...
```

```text
true
```

> [!NOTE]
> The default annotation key is `banzaicloud.com/last-applied` (exposed as
> `patch.LastAppliedConfig`). It is unchanged from the Banzai Cloud days
> for backward compatibility — existing operators that wrote that key
> still read correctly when their image is rebuilt against `cisco-open`.

### 2.5 Local development (for contributors)

The repo ships a `Makefile`, golangci-lint config (`.golangci.yml`), and a
license-checker config (`.licensei.toml`). To run the test suite locally:

```bash
git clone https://github.com/cisco-open/k8s-objectmatcher
cd k8s-objectmatcher
go test ./patch/...
```

```text
ok  	github.com/cisco-open/k8s-objectmatcher/patch	0.123s
```

The `tests/` subdirectory is a separate module (own `go.mod`) that runs
integration tests against `k8s.io/client-go` v0.21.3. Those tests need a
working KUBECONFIG and a real or simulated API server — `go test
./patch/...` alone covers the unit tests.

## 3. Configuration

There is no config file, no environment variables, and no global state to
tune. "Configuration" of `k8s-objectmatcher` is a matter of:

1. Picking the right annotation key.
2. Choosing which `CalculateOption`s to pass to `Calculate`.
3. Deciding whether to use `DefaultPatchMaker` / `DefaultAnnotator` or
   build your own with `NewPatchMaker(...)`.

### 3.1 The annotation key

The `Annotator` stamps a JSON snapshot of the desired object onto the
live object as an annotation, then reads it back later for the three-way
merge. The key defaults to `banzaicloud.com/last-applied`:

```go
const LastAppliedConfig = "banzaicloud.com/last-applied"
var DefaultAnnotator = NewAnnotator(LastAppliedConfig)
```

Choose your own key with `NewAnnotator`:

```go
ann := patch.NewAnnotator("mycompany.io/last-applied")
maker := patch.NewPatchMaker(ann, &patch.K8sStrategicMergePatcher{}, &patch.BaseJSONMergePatcher{})
```

> [!TIP]
> Pick a stable, unique key once per operator and never change it.
> Switching keys across releases makes existing live objects look like
> they have no last-applied snapshot, which forces a fresh three-way
> baseline (and may trigger one large reconcile).

### 3.2 Annotation storage format

The annotation value is **gzipped + base64-encoded**, with a fallback
read path for plain (non-zipped) values from older operator versions.
This matters for two reasons:

- Annotations have a **256 KiB limit per object** in Kubernetes. The
  zip step roughly halves the JSON before encoding, so most CRDs fit
  comfortably; very large objects (StatefulSets with hundreds of mounts
  or ConfigMaps with embedded JSON) can still bump the ceiling.
- You will not see human-readable JSON if you `kubectl get -o yaml` the
  object — that's by design.

> [!NOTE]
> The library transparently decodes both formats: pre-zip annotations
> (raw JSON) and modern zip+base64 annotations both round-trip through
> `GetOriginalConfiguration`. Upgrading does not require a flag day.

### 3.3 The `CalculateOption` set

Every option is a `func([]byte, []byte) ([]byte, []byte, error)` — it
takes the current and modified JSON byte streams, mutates both
symmetrically, and returns them. The shipped options are:

| Option | Purpose |
|---|---|
| `IgnoreStatusFields()` | Strips the top-level `status` from both sides before diffing. |
| `IgnoreField(field)` | Strips an arbitrary top-level field from both sides. |
| `IgnoreVolumeClaimTemplateTypeMetaAndStatus()` | For StatefulSets — zeroes `kind`, `apiVersion`, and `status` inside every entry of `spec.volumeClaimTemplates`. |

You can supply zero, one, or many options to a single `Calculate` call —
they're applied left-to-right.

> [!IMPORTANT]
> Apply `IgnoreStatusFields()` to anything with a `status` subresource.
> Otherwise the API server's `status` updates (which your controller
> probably doesn't own) will show up as a non-empty patch on every
> reconcile and you'll spin in a tight update loop.

### 3.4 Using a custom `PatchMaker`

`DefaultPatchMaker` is built once at package load with the default
annotator and the standard Kubernetes strategic-merge / JSON-merge
implementations. To swap one out (e.g. inject a fake patcher in tests
or attach `PreconditionFuncs` to the strategic-merge path):

```go
maker := patch.NewPatchMaker(
	patch.NewAnnotator("mycompany.io/last-applied"),
	&patch.K8sStrategicMergePatcher{
		PreconditionFuncs: []mergepatch.PreconditionFunc{
			mergepatch.RequireKeyUnchanged("apiVersion"),
		},
	},
	&patch.BaseJSONMergePatcher{},
)
```

See the Admin Guide for a full breakdown of the strategic-merge / JSON-
merge dispatch table and how to plug in a custom implementation.

## 4. Core Usage

The two-line mental model:

1. **Calculate** a `PatchResult` from `(currentObject, desiredObject,
   opts...)`.
2. **Branch** on `PatchResult.IsEmpty()`.

Everything else — annotation management, per-kind cleaners, the
strategic-merge versus JSON-merge dispatch — happens inside `Calculate`.

### 4.1 The `PatchResult` shape

`Calculate` returns a `*PatchResult`:

```go
type PatchResult struct {
	Patch    []byte // the diff, "{}" if no real change
	Current  []byte // current object as JSON, post-cleanup
	Modified []byte // desired object as JSON, post-cleanup
	Original []byte // last-applied snapshot from the annotation, may be nil
}

func (p *PatchResult) IsEmpty() bool      // true when string(Patch) == "{}"
func (p *PatchResult) String() string     // formatted dump for debugging
```

Treat `IsEmpty() == true` as the **only** signal that no update is needed.
Inspect `Patch` only when you need to log what changed; never make
control-flow decisions on the byte length, which can be `2` (`{}`) or
larger even when there's no semantic change.

### 4.2 Calling `Calculate`

```go
result, err := patch.DefaultPatchMaker.Calculate(currentObj, desiredObj,
	patch.IgnoreStatusFields(),
	patch.IgnoreField("metadata"),
)
if err != nil {
	return err
}
if !result.IsEmpty() {
	// real change — update or recreate
}
```

Internally, `Calculate` does the following in order:

1. JSON-marshals both objects.
2. Applies every option, left-to-right.
3. Calls `DeleteNullInJson` on both to drop nil fields and zero-valued
   `IntOrString` types (a long-standing source of false diffs).
4. Reads the last-applied annotation from `currentObject`.
5. Branches on the object's runtime type:
    - `*unstructured.Unstructured` → JSON-merge path
      (`CreateThreeWayJSONMergePatch`).
    - Everything else → strategic-merge path
      (`CreateThreeWayMergePatch`, then re-applies and re-diffs to
      collapse `$setElementOrder` directives).
6. Returns `(Patch, Current, Modified, Original)` for inspection.

> [!NOTE]
> The strategic-merge path runs the patch twice — first to compute it,
> then to re-apply it locally and compute a two-way merge against the
> current state. This second pass eliminates spurious differences caused
> by `$setElementOrder` markers that strategic merge generates for
> arrays even when the contents are identical.

### 4.3 Annotation lifecycle

Three calls cover the full lifecycle for a single object:

| When | Call | Purpose |
|---|---|---|
| Just before **creating** an object | `DefaultAnnotator.SetLastAppliedAnnotation(desired)` | Stamp the desired state on the new object so the *next* reconcile has an `Original` to diff against. |
| Just before **updating** an object | `DefaultAnnotator.SetLastAppliedAnnotation(desired)` | Refresh the snapshot so subsequent reconciles compare against the just-applied state, not the stale baseline. |
| Inside `Calculate` | (automatic) `GetOriginalConfiguration` | Pulls the snapshot off `currentObject` for the three-way merge. |

You almost never call `GetOriginalConfiguration` or
`GetModifiedConfiguration` directly — `Calculate` does that. The two
calls you wire by hand are `SetLastAppliedAnnotation` (before create
and before update).

### 4.4 The full public API

| Symbol | Kind | Notes |
|---|---|---|
| `LastAppliedConfig` | const | `"banzaicloud.com/last-applied"` — the default annotation key. |
| `DefaultAnnotator` | var | `NewAnnotator(LastAppliedConfig)` — use directly for the common case. |
| `DefaultPatchMaker` | var | `NewPatchMaker(DefaultAnnotator, &K8sStrategicMergePatcher{}, &BaseJSONMergePatcher{})`. |
| `Annotator` | type | Holds a `meta.MetadataAccessor` and the annotation key. |
| `NewAnnotator(key string) *Annotator` | func | Build an annotator with a custom key. |
| `(*Annotator).GetOriginalConfiguration(obj) ([]byte, error)` | method | Read + base64-decode + unzip the annotation. Handles legacy plain values. |
| `(*Annotator).SetOriginalConfiguration(obj, original []byte) error` | method | Zip + base64 encode and write the annotation. |
| `(*Annotator).GetModifiedConfiguration(obj, annotate bool) ([]byte, error)` | method | Marshal the object without the annotation. With `annotate=true`, embeds the snapshot inside the same payload (rare). |
| `(*Annotator).SetLastAppliedAnnotation(obj) error` | method | Convenience: get modified, strip nulls, then set. **The call you make from your controller.** |
| `Maker` | interface | Single method `Calculate(current, modified, opts...) (*PatchResult, error)`. |
| `PatchMaker` | type | Concrete `Maker`. Holds an `*Annotator`, a `StrategicMergePatcher`, and a `JSONMergePatcher`. |
| `NewPatchMaker(annotator, smp, jmp) Maker` | func | Build a custom maker. Returns the interface, not the struct. |
| `(*PatchMaker).Calculate(current, modified, opts...) (*PatchResult, error)` | method | The diff engine. |
| `PatchResult` | type | Fields: `Patch`, `Current`, `Modified`, `Original`. |
| `(*PatchResult).IsEmpty() bool` | method | `string(Patch) == "{}"`. |
| `(*PatchResult).String() string` | method | Human-readable dump for logs. |
| `CalculateOption` | type | `func([]byte, []byte) ([]byte, []byte, error)`. |
| `IgnoreStatusFields() CalculateOption` | func | Strip `status` from both sides. |
| `IgnoreField(field string) CalculateOption` | func | Strip an arbitrary top-level field. |
| `IgnoreVolumeClaimTemplateTypeMetaAndStatus() CalculateOption` | func | Zero `kind` / `apiVersion` / `status` inside every `spec.volumeClaimTemplates` entry (StatefulSets). |
| `DeleteNullInJson(jsonBytes []byte) ([]byte, map[string]interface{}, error)` | func | Walk a JSON object and drop nils / zero-valued `IntOrString`s. Used internally; also useful in tests. |
| `StrategicMergePatcher` | interface | `StrategicMergePatch` / `CreateTwoWayMergePatch` / `CreateThreeWayMergePatch`. |
| `JSONMergePatcher` | interface | `MergePatch` / `CreateMergePatch` / `CreateThreeWayJSONMergePatch`. |
| `K8sStrategicMergePatcher` | type | Default `StrategicMergePatcher`. Has a `PreconditionFuncs []mergepatch.PreconditionFunc` field for invariant guards. |
| `BaseJSONMergePatcher` | type | Default `JSONMergePatcher`. No fields. |

> [!TIP]
> Need to mock the diff engine in unit tests? Implement the `Maker`
> interface (`Calculate(current, modified, opts...) (*PatchResult,
> error)`) in your test code and inject your fake. The library never
> hardcodes `*PatchMaker` anywhere it consumes `Maker`.

## 5. Common Tasks

The seven recipes below are the patterns that come up in nearly every
controller. Each one stays inside the public API surface — no internal
types, no reflection.

### 5.1 Diff a Deployment without spinning on `status` updates

The single most common pattern: reconcile a `Deployment` and stop
fighting the deployment controller's `status.observedGeneration` /
`status.replicas` writes.

```go
import (
	"github.com/cisco-open/k8s-objectmatcher/patch"
	appsv1 "k8s.io/api/apps/v1"
	"sigs.k8s.io/controller-runtime/pkg/client"
)

func reconcileDeployment(ctx context.Context, c client.Client, desired *appsv1.Deployment) error {
	current := &appsv1.Deployment{}
	err := c.Get(ctx, client.ObjectKeyFromObject(desired), current)
	if errors.IsNotFound(err) {
		if err := patch.DefaultAnnotator.SetLastAppliedAnnotation(desired); err != nil {
			return err
		}
		return c.Create(ctx, desired)
	}
	if err != nil {
		return err
	}

	result, err := patch.DefaultPatchMaker.Calculate(current, desired, patch.IgnoreStatusFields())
	if err != nil {
		return err
	}
	if result.IsEmpty() {
		return nil
	}
	if err := patch.DefaultAnnotator.SetLastAppliedAnnotation(desired); err != nil {
		return err
	}
	return c.Update(ctx, desired)
}
```

> [!NOTE]
> `SetLastAppliedAnnotation` runs once on create and again on every
> non-empty update. Skipping it leaves the live object's last-applied
> annotation stale, which downgrades subsequent diffs from three-way to
> two-way and re-introduces the spurious-update problem.

### 5.2 Diff a Service correctly (`clusterIP`, `nodePort`, `targetPort`)

Services pick up server-side defaults — the API server picks a
`clusterIP` from the service CIDR, picks `nodePort`s on type
`NodePort`/`LoadBalancer`, and resolves `targetPort` to a concrete
value. None of those belong in your desired object, but they round-trip
into your `current` object.

```go
result, err := patch.DefaultPatchMaker.Calculate(currentSvc, desiredSvc, patch.IgnoreStatusFields())
```

`IgnoreStatusFields()` covers `status.loadBalancer.ingress` and friends.
The server-set `spec.clusterIP`, `spec.clusterIPs`, and `spec.ports[*].nodePort`
fields are *already* preserved by the strategic-merge logic the
`PatchMaker` runs — leaving them off the desired object is enough.

> [!TIP]
> If you must hardcode a `clusterIP` on a desired Service (for example,
> to pin a headless DNS service across rebuilds), set it explicitly on
> the desired object. The patch will not flip it back to empty because
> the desired side now matches the current side.

### 5.3 StatefulSet `volumeClaimTemplates` — the special case

`spec.volumeClaimTemplates` is one of the few places where the API
server permanently rewrites your desired object — it stamps `kind`,
`apiVersion`, and an initial `status.phase=Pending` on every PVC
template. Without a filter, every reconcile sees a non-empty patch
forever.

```go
result, err := patch.DefaultPatchMaker.Calculate(currentSS, desiredSS,
	patch.IgnoreStatusFields(),
	patch.IgnoreVolumeClaimTemplateTypeMetaAndStatus(),
)
```

`IgnoreVolumeClaimTemplateTypeMetaAndStatus` walks
`spec.volumeClaimTemplates[*]` on both sides and **sets** `kind=""`,
`apiVersion=""`, and `status={"phase":"Pending"}`, normalising the two
representations rather than deleting them.

> [!WARNING]
> `IgnoreVolumeClaimTemplateTypeMetaAndStatus` does not handle the
> nested `selector.matchLabels` / `resources.requests.storage` fields
> being mutated post-creation — Kubernetes treats those as immutable
> and will reject the update. If your reconciler is the one trying to
> change PVC capacity, you need a separate recreate-or-resize path; see
> issue [#51](https://github.com/cisco-open/k8s-objectmatcher/issues/51)
> for an open report on edge cases.

### 5.4 Use `IgnoreField` to opt-out a top-level key

`IgnoreField(field)` deletes a top-level key from both byte streams
before diffing. Useful for fields a third-party controller writes that
you want to leave alone:

```go
result, err := patch.DefaultPatchMaker.Calculate(curObj, desObj,
	patch.IgnoreStatusFields(),
	patch.IgnoreField("metadata"), // ignore everything in metadata (rare; usually too broad)
)
```

The argument is an exact, top-level JSON key. To ignore a nested field
(say, `spec.replicas`), `IgnoreField` is **not** sufficient — that
capability is open as
[#57](https://github.com/cisco-open/k8s-objectmatcher/issues/57). Use
the workaround in 5.7 below.

### 5.5 Wire it into a controller-runtime `Reconcile`

Every healthy controller-runtime reconciler that uses this library has
the same shape: get → calculate → branch → update + reannotate.

```go
func (r *AppReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
	desired := r.buildDesired(req) // your construction logic

	current := &appsv1.Deployment{}
	err := r.Get(ctx, req.NamespacedName, current)

	switch {
	case errors.IsNotFound(err):
		if err := patch.DefaultAnnotator.SetLastAppliedAnnotation(desired); err != nil {
			return ctrl.Result{}, err
		}
		return ctrl.Result{}, r.Create(ctx, desired)

	case err != nil:
		return ctrl.Result{}, err
	}

	result, err := patch.DefaultPatchMaker.Calculate(current, desired,
		patch.IgnoreStatusFields(),
	)
	if err != nil {
		return ctrl.Result{}, err
	}
	if result.IsEmpty() {
		return ctrl.Result{}, nil
	}

	r.Log.V(1).Info("applying patch", "patch", string(result.Patch))
	if err := patch.DefaultAnnotator.SetLastAppliedAnnotation(desired); err != nil {
		return ctrl.Result{}, err
	}
	return ctrl.Result{}, r.Update(ctx, desired)
}
```

### 5.6 Debug spurious updates

If `IsEmpty()` keeps returning `false` and you can't see why:

1. Log `result.String()` — it dumps `Patch`, `Current`, `Modified`, and
   `Original` in one block.
2. Look at `Patch`. The keys present are exactly the keys disagreeing.
3. Compare against `Original`. If `Original` is empty, the live object
   has no last-applied snapshot — call `SetLastAppliedAnnotation` on
   create / on first reconcile.
4. If `Patch` contains `$setElementOrder/...` even for unchanged
   slices, you've found the rare residual the second pass didn't
   collapse. File an issue with the reproducer; the strategic-merge
   path already handles the common cases.
5. If `Patch` references a field your controller never sets but the API
   server fills in (`spec.clusterIP`, `spec.podCIDR`, defaulted enum
   values), use `IgnoreField(...)`.

> [!TIP]
> The fastest reproducer is `result.String()` printed from inside a
> failing reconcile. Paste that into the issue. A plain
> `reflect.DeepEqual(current, desired)` is not informative — you need
> the *strategic-merge* perspective the library uses.

### 5.7 Handle CRDs / unstructured objects

When `currentObject` is a `*unstructured.Unstructured`, `Calculate`
automatically routes to the JSON-merge path. There is nothing to
configure — the type assertion in the dispatch decides:

```go
import "k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"

cur := &unstructured.Unstructured{}
cur.SetGroupVersionKind(myCRDGVK)
_ = c.Get(ctx, key, cur)

desired := &unstructured.Unstructured{Object: myDesiredMap}

result, err := patch.DefaultPatchMaker.Calculate(cur, desired)
```

The JSON-merge path also runs the same "apply locally, re-diff" trick
to suppress no-op patches. CRDs without a registered Go type — the
common case for dynamic operators — are a first-class citizen.

> [!NOTE]
> The strategic-merge path needs a typed Go struct so it can look up
> patch metadata (`patchStrategy` / `patchMergeKey` struct tags). For
> unstructured objects there's no struct, so `CreateThreeWayJSONMergePatch`
> from `k8s.io/apimachinery/pkg/util/jsonmergepatch` is used instead.
> The trade-off: list-of-objects with merge-keys won't merge as
> intelligently. Test your CRD schema against real updates.

### 5.8 Opt out of an entire subtree (workaround for nested-field ignore)

Until nested-field ignoring lands ([#57](https://github.com/cisco-open/k8s-objectmatcher/issues/57)),
the workaround is a **custom `CalculateOption`** — the type is just a
function, so you can write your own:

```go
import (
	json "github.com/json-iterator/go"
	"github.com/cisco-open/k8s-objectmatcher/patch"
)

// IgnoreSpecReplicas zeroes spec.replicas on both sides — useful when
// HPA owns replicas.
func IgnoreSpecReplicas() patch.CalculateOption {
	return func(current, modified []byte) ([]byte, []byte, error) {
		for i, b := range [][]byte{current, modified} {
			var m map[string]interface{}
			if err := json.Unmarshal(b, &m); err != nil {
				return nil, nil, err
			}
			if spec, ok := m["spec"].(map[string]interface{}); ok {
				delete(spec, "replicas")
			}
			out, err := json.ConfigCompatibleWithStandardLibrary.Marshal(m)
			if err != nil {
				return nil, nil, err
			}
			if i == 0 {
				current = out
			} else {
				modified = out
			}
		}
		return current, modified, nil
	}
}
```

You can stack any number of these alongside the shipped options.

## 6. Troubleshooting

The diagnostic decision tree:

```text
Reconcile loops forever?
├── Calculate returns no error, IsEmpty() == false on every pass
│     → 6.1   spurious diff from server-set fields
│     → 6.2   stale or missing last-applied annotation
│     → 6.3   $setElementOrder noise
├── Calculate returns an error
│     → 6.4   "Failed to lookup patch meta" (unsupported type)
│     → 6.5   "could not unmarshal" (object isn't valid JSON)
│     → 6.6   "Failed to convert current object to byte sequence"
└── Annotation never appears on the live object
      → 6.7   forgot to call SetLastAppliedAnnotation before create/update
```

### 6.1 Error: every reconcile reports a non-empty patch on `status`

**Cause:** You're not passing `IgnoreStatusFields()`. The API server
writes `status` for almost every workload type, and your reconciler
sees that as a difference between desired (`status: {}`) and current
(`status: {observedGeneration: 7, ...}`).

**Solution:**

```go
result, err := patch.DefaultPatchMaker.Calculate(cur, des, patch.IgnoreStatusFields())
```

**Prevention:** Add `IgnoreStatusFields()` to every `Calculate` call by
default. There's no downside on objects that don't have a `status`
field — the option just deletes a key that's already absent.

### 6.2 Error: `result.Original` is `nil` and patches are huge

**Cause:** The live object has no last-applied annotation, so the
three-way merge degrades into a two-way merge between `Current` and
`Modified`. Every server-defaulted field shows up.

**Solution:**

1. Call `patch.DefaultAnnotator.SetLastAppliedAnnotation(desired)`
   immediately before `c.Create(ctx, desired)`.
2. Call it again immediately before every `c.Update(ctx, desired)`.
3. For existing objects in the cluster predating your operator,
   one-time backfill: read the live object, call
   `SetLastAppliedAnnotation(live)` against it as the "desired", and
   `Update`. The next reconcile starts clean.

**Prevention:** Move the `SetLastAppliedAnnotation` call into a small
helper that wraps `Create`/`Update` so you can never forget it.

### 6.3 Error: patch contains `$setElementOrder/containers` for an unchanged Pod spec

**Cause:** Strategic merge generates `$setElementOrder` directives for
mergeable list fields. The library's second pass (apply locally, re-
diff with two-way merge) catches the common cases, but a few residuals
slip through — see issue
[#9](https://github.com/cisco-open/k8s-objectmatcher/issues/9).

**Solution:** Confirm that the `name` keys on every list element match
exactly between current and desired. Strategic merge uses those as
merge keys; even a single rename re-orders the list and produces a
real `$setElementOrder` directive — not a bug.

**Prevention:** Build desired Pod specs from a deterministic source
(template + Helm-like rendering, not random map iteration) so the list
order is stable.

### 6.4 Error: `Failed to lookup patch meta`

**Cause:** `K8sStrategicMergePatcher.CreateThreeWayMergePatch` calls
`strategicpatch.NewPatchMetaFromStruct(dataStruct)`. If `dataStruct` is
a type with no strategic-merge tags (a CRD struct that doesn't
register `patchStrategy="merge"` annotations), this errors.

**Solution:** Pass the object as `*unstructured.Unstructured` instead
— `Calculate` will route it through the JSON-merge path, which doesn't
need patch metadata.

```go
u, err := runtime.DefaultUnstructuredConverter.ToUnstructured(myCRDObj)
// ...wrap and use as *unstructured.Unstructured
```

**Prevention:** For CRDs without proper struct tags, standardize on
unstructured access in the reconciler.

### 6.5 Error: `could not unmarshal byte sequence` from a `CalculateOption`

**Cause:** Your custom option (or one of the shipped ones) was handed
bytes that aren't valid JSON. Almost always: a previous option in the
chain returned malformed bytes.

**Solution:** Inspect the option order. Each option must return
**valid** JSON for both `current` and `modified`. If you're writing
your own option, never return raw `[]byte("")` on the success path —
always round-trip through marshaling.

**Prevention:** Unit-test custom options with the `unstructured` test
helpers from `patch/patch_test.go` (`mustFromUnstructured` /
`mustToUnstructured`) as a template.

### 6.6 Error: `Failed to convert current object to byte sequence`

**Cause:** The runtime.Object you passed wraps a value that
json-iterator refuses to marshal — typically a pointer cycle or a Go
interface with no encoder. With Kubernetes types, this almost never
happens; with custom embedded structs in your own CRD types, it can.

**Solution:** Round-trip the object through
`runtime.DefaultUnstructuredConverter.ToUnstructured` and back. If
that fails too, the bug is in your CRD type definition (an
unexported field on the JSON path, a custom `MarshalJSON` that
panics, etc.).

**Prevention:** Run `json.Marshal(yourObj)` once at startup as a smoke
check.

### 6.7 Error: live object has no annotation after `Create`

**Cause:** `SetLastAppliedAnnotation` was called on the desired object
*after* `Create`. The mutation lives in memory but never reaches the
API server.

**Solution:** Move the `SetLastAppliedAnnotation` call **before**
`c.Create(ctx, desired)`. The mutation is on the in-memory struct and
will be persisted as part of the create payload.

**Prevention:** Treat the annotator as a "stamp before send"
operation, not a "stamp after send" one.

> [!WARNING]
> Calling `SetLastAppliedAnnotation` on the **current** (server-side)
> object inside a reconcile, then calling `Update`, will work — but
> only if you also reapply your other desired-state mutations to the
> live object first. Most controllers find it cleaner to mutate the
> desired object and `Update` that, never the live object.

## 7. FAQ

**Q: Is this a CLI tool?**

No — it's a Go library. There is no binary, no `kubectl` plugin, no
container image. You import `github.com/cisco-open/k8s-objectmatcher/patch`
into a Go program (typically a Kubernetes operator) and call its API.

**Q: Does it work with controller-runtime / Operator SDK / Kubebuilder?**

Yes — those all build on `k8s.io/client-go` and `k8s.io/apimachinery`,
and the only types this library cares about are `runtime.Object`
(which all of those use) and `*unstructured.Unstructured`. Drop it
into any reconciler.

**Q: What's the difference between `IsEmpty()` and `string(Patch) == "{}"`?**

`IsEmpty()` is exactly that comparison. Use the method — it's clearer
and survives a future internal change to representation.

**Q: Why isn't there a `kubectl apply --dry-run` style mode?**

Because that's the wrong abstraction for an operator. `kubectl apply`
operates on a single object at a time and prints diffs to a human; an
operator needs a programmatic answer ("should I update or not?") and a
machine-readable patch. `Calculate` returns both — the answer
(`IsEmpty()`) and the byte-level patch.

**Q: Does it handle server-side apply?**

Not directly. Server-side apply uses managed-fields tracking
(`metadata.managedFields`), which is a different model from the
client-side last-applied annotation this library implements. If your
controller uses server-side apply, the API server itself does the
"is this a real change" calculation — you don't need this library.
Many older operators use the client-side model and that's where this
library is the right fit.

**Q: Can I use this without controller-runtime?**

Yes. The library has zero dependency on controller-runtime — it only
needs `k8s.io/apimachinery` types. Plain `client-go` informers and
custom reconcile loops work fine.

**Q: What about Pods?**

Pods work fine, but most operators don't reconcile Pods directly —
they reconcile workload controllers (Deployment, StatefulSet, Job)
and let those manage their Pods. If you do diff a Pod, expect a lot
of server-defaulted fields (`securityContext`,
`terminationGracePeriodSeconds`, `dnsPolicy`, the auto-mounted
ServiceAccount token volume) so use `IgnoreField("status")` and
consider a custom option for the auto-injected volume.

**Q: How do I ignore a label on Pods specifically?**

This is open as
[#65](https://github.com/cisco-open/k8s-objectmatcher/issues/65). For
now, write a custom `CalculateOption` that walks
`spec.template.metadata.labels` and deletes the offending key on both
sides. The recipe in 5.8 is the template.

**Q: What's the deal with the `banzaicloud.com/last-applied` annotation key?**

The library was originally built at Banzai Cloud (a Hungarian Kubernetes
shop) and Cisco kept the key when they took over the repo. Customers
depend on the exact value for backward compatibility — operators
upgrading from Banzai Cloud builds to `cisco-open` builds need their
existing live objects to round-trip correctly.

**Q: Does it handle StatefulSet `volumeClaimTemplates` resize?**

No — the library answers "is this a real change?", but StatefulSet
PVC capacity is **immutable** in Kubernetes itself. Even with a real
change, the API server will reject the update. Use a separate
recreate-with-data-migration path. See issue
[#51](https://github.com/cisco-open/k8s-objectmatcher/issues/51) for
related discussion.

**Q: Where do I report a missing per-kind cleaner?**

[github.com/cisco-open/k8s-objectmatcher/issues](https://github.com/cisco-open/k8s-objectmatcher/issues).
Issue [#40](https://github.com/cisco-open/k8s-objectmatcher/issues/40)
is the canonical "exclude specific fields from a StatefulSet" thread.

**Q: Is the library safe to call from many goroutines?**

Yes — `DefaultPatchMaker` and `DefaultAnnotator` are stateless beyond
their immutable fields, and `Calculate` is a pure function over its
inputs. The only shared state is the underlying
`meta.MetadataAccessor`, which is itself goroutine-safe.

**Q: How big can the last-applied annotation get?**

Kubernetes caps annotations at **256 KiB per object**. The library
gzip-compresses the JSON before base64-encoding, so practical limits
are roughly 1–2 MiB of raw JSON. ConfigMaps with embedded large
payloads can hit the ceiling — for those, consider `IgnoreField` on
the `data` key and reconcile-by-hash instead.

---

_Generated April 2026 from cisco-open/k8s-objectmatcher at 17b471d. AI-generated documentation. The library is checked by sampling (three repositories so far); this guide was not individually reviewed. Verify against the project before relying on it. Unofficial; not affiliated with or reviewed by the project._