kubectl apply --- metadata.resourceVersion: Invalid value: 0x0: must be specified for an update

kubectl apply metadata.resourceVersion

https://feichashao.com/kubectl-apply-fail/

Root Cause
This is because there's a resourceVersion field in last-applied-configuration annotation, which is not expected. But why there's a resourceVersion field?

We can reproduce the issue like this:

## Create a CR using "kubectl apply"
$ cat origin.yml
apiVersion: cache.example.com/v1alpha1
kind: Memcached
metadata:
  name: memcached-sample
spec:
  size: 3

$ kubectl apply -f origin.yml

## Get the yaml of the resource.
$ kubectl get memcacheds memcached-sample -o yaml > current_memcached.yml

## Modify the exported yaml, eg change size 3->4, then re-apply the modified current_memcached.yml
$ kubectl apply -f current_memcached.yml
memcached.cache.example.com/memcached-sample configured

## We can see there's a resourceVersion field in last-applied-configuration.
$ kubectl get memcacheds memcached-sample -o yaml
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"cache.example.com/v1alpha1","kind":"Memcached","metadata":{"annotations":{},"creationTimestamp":"2020-08-12T12:56:46Z","generation":5,"managedFields":[{"apiVersion":"cache.example.com/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:kubectl.kubernetes.io/last-applied-configuration":{}}},"f:spec":{".":{},"f:size":{}}},"manager":"kubectl","operation":"Update","time":"2020-08-24T11:18:42Z"},{"apiVersion":"cache.example.com/v1alpha1","fieldsType":"FieldsV1","fieldsV1":{"f:status":{".":{},"f:nodes":{}}},"manager":"manager","operation":"Update","time":"2020-08-25T00:54:37Z"}],"name":"memcached-sample","namespace":"default","resourceVersion":"302294","selfLink":"/apis/cache.example.com/v1alpha1/namespaces/default/memcacheds/memcached-sample","uid":"5ae4fdaf-f962-4355-878c-eee75be558f3"},"spec":{"size":4},"status":{"nodes":["memcached-sample-9b765dfc8-kszwv","memcached-sample-9b765dfc8-krsvd","memcached-sample-9b765dfc8-9vxvb"]}}

## Apply the original file. (Sometimes we keep a copy of the CR before we create it, and we can use this copy to restore/modify the resource)
## It fails.
$ kubectl apply -f origin.yml            
The memcacheds "memcached-sample" is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update

## To fix the issue, we can remove the whole last-applied-configuration annotation in the resource
$ kubectl edit memcacheds memcached-sample

## And it should succeed this time.
$ kubectl apply -f origin.yml
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
memcached.cache.example.com/memcached-sample configured


The reason behind "kubectl apply -f origin.yml" failure is that, "kubectl apply" will try patching the existing resource. It will compare the last-applied-configuration with the current yaml, and send the patch. If there a resourceVersion in last-applied-configuration, it will send the resourceVersion value, but as there's no resourceVersion in yaml, it will send a null, which cause the error.


$ kubectl -v=8 apply -f origin.yml
I0825 15:54:46.065735    1555 request.go:1068] Request Body: {"metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{"apiVersion":"cache.example.com/v1alpha1","kind":"Memcached","metadata":{"annotations":{},"name":"memcached-sample","namespace":"default"},"spec":{"size":3}}\n"},"creationTimestamp":null,"generation":null,"managedFields":null,"resourceVersion":null,"selfLink":null,"uid":null},"spec":{"size":3},"status":null}


Resolution
- Remove the last-applied-configuration annotation in the target resource, and apply again.
- In the future, if we want to modify the configuration of a resource, modify the original yaml and apply that yaml. Or use the "kubectl edit" command against the resource. Don't export the yaml of the current resource then modify then apply.