enisoc/relnotes-1.9.md

## relnotes-1.9.md

      
    Raw
  

              relnotes-1.9.md
            
          
    master

Documentation & Examples
Changelog since v1.8.0

Action Required


Api-Machinery


NullArea


NullIssue

Graduate the admission and admissionregistration (webhook part) API to v1beta1 (#56004, @caesarxuchao)
action required: Deprecated flags --portal-net and service-node-ports of kube-apiserver are removed. (#52547, @xiangpengzhao)


Auth


NullArea


NullIssue

RBAC objects are now stored in etcd in v1 format. After completing an upgrade to 1.9, RBAC objects (Roles, RoleBindings, ClusterRoles, ClusterRoleBindings) should be migrated to ensure all persisted objects are written in v1 format, prior to v1alpha1 support being removed in a future release. (#52950, @liggitt)


unable to deploy privileged pod after 1.8 upgrade unless I set allowPrivilegeEscalation true (#53437)

PodSecurityPolicy: Fixes a compatibility issue that caused policies that previously allowed privileged pods to start forbidding them, due to an incorrect default value for allowPrivilegeEscalation. PodSecurityPolicy objects defined using a 1.8.0 client or server that intended to set allowPrivilegeEscalation to false must be reapplied after upgrading to 1.8.1. (#53443, @liggitt)


NullSig


NullArea


NullIssue

kubeadm join: Error out if CA pinning isn't used or opted out of (#55468, @yuexiao-wang)
update podtolerations admission to mutate and validate separately (#55251, @deads2k)


Storage


NullArea


Volume topology aware scheduling binding changes (#54435)

Topology aware volume scheduler and PV controller changes (#55039, @msau42)


NullIssue

action required: The storage.k8s.io/v1beta1 API  and volume.beta.kubernetes.io/storage-class annotation are deprecated. They will be removed in a future release. Please use v1 API and field v1.PersistentVolumeClaim.Spec.StorageClassName/v1.PersistentVolume.Spec.StorageClassName instead. (#53580, @xiangpengzhao)


Other notable changes


Api-Machinery


Admission-Control


NullIssue

Added mutation supports to admission webhooks. (#54892, @caesarxuchao)
the generic admission webhook is now available in the generic apiserver (#54513, @deads2k)


Authentication for webhook admission to heterogenous authentication domains (#54404)

generic webhook admission now takes a config file which describes how to authenticate to webhook servers (#54414, @deads2k)


ExternalAdmissionHookConfiguration cannot choose URL (#53826)

admission webhook registration now allows URL paths (#54145, @deads2k)


Api


Eliminate Phase and Conditions from the API (#7856)

StatefulSet status now has support for conditions, making it consistent with other core controllers in v1  (#55268, @foxish)


Api & Apiserver


Reported subresource discovery is incorrect (#54684)

Fixes discovery information for scale subresources in the apps API group (#54683, @liggitt)


Apiserver


NullIssue

Fix a bug that prevents client-go metrics from being registered in prometheus in multiple components. (#53434, @crassirostris)


Audit


Shutdown http handlers before shutting down audit backend (#50781)

Implement graceful shutdown of the kube-apiserver by waiting for open connections to finish before exiting. Moreover, the audit backend will stop dropping events on shutdown. (#53695, @hzxuzhonghu)


[audit] Always retry sending to webhook (#52909)

Webhook always retries connection reset error. (#53947, @crassirostris)


Custom-Resources


Getting CRD Validation to Beta (#53829)

Promote validation for custom resources defined through CRD to beta (#54647, @colemickens)


sample-controller example repository (#52752)

sample-controller: add example CRD controller (#52753, @munnerz)


Unable to use a fieldSelector with custom resources (#51046)

Custom resources served through CustomResourceDefinition now support field selectors for metadata.name and metadata.namespace. (#53345, @ncdc)


CRD and TPR doesn't support watching one single instance (#49424)

Custom resources served through CustomResourceDefinition now support field selectors for metadata.name and metadata.namespace. (#53345, @ncdc)


Etcd


Update to etcd 3.1.X  (#49386)

etcd: update version to 3.1.10 (#49393, @hongchaodeng)


Hw-Accelerators


NullIssue

Expose single annotation/label via downward API (#55902, @yguo0905)


Ipv6


NullIssue


Fix handling of IPv6 URLs in NO_PROXY. (#53898, @kad)


NullArea


PersistentVolumeSource should be read-only (#54562)

Validate that PersistentVolumeSource is not changed during PV Update (#54761, @ianchakeres)


[apps/v1] Change DefaultGarbageCollectionPolicy for workload controllers (#55027)

change DefaultGarbageCollectionPolicy to DeleteDependents for workloads controllers (#55148, @dixudx)


Collect metrics on admission rejections (#55030)

Metrics have been added for monitoring admission plugins, including the new dynamic (webhook-based) ones. (#55183, @jpbetz)


kube-apiserver "no --service-cluster-ip-range specified" and "Defaulting to 10.0.0.0/24". (#52695)

Fixed a bug which is causes kube-apiserver to not run without specifying service-cluster-ip-range (#52870, @jennybuckley)


apiserver proxy feature does not rewrite Location header on redirects (#51790)

Apiserver proxy rewrites URL when service returns absolute path with request's host. (#52556, @roycaihw)


kubectl attach: client-go does not respect CIDRs in NO_PROXY (#54407)

API machinery's httpstream/spdy calls now support CIDR notation for NO_PROXY (#54413, @kad)


should prevent the deletion of a PVC that is referenced by an active pod (#45143)

PVC Finalizing Controller is introduced in order to prevent deletion of a PVC that is being used by a pod. (#55824, @pospispa)


client-gen tag shortcomings when newline is omitted (#53893)

Allow code-generator tags in the 2nd closest comment block and directly above a statement. (#55233, @sttts)


controller-manager crash loops if gc controller doesn't have access to extension apis (#55022)

API discovery failures no longer crash the kube controller manager via the garbage collector. (#55259, @ironcladlou)


Update gRPC library to pick up data race fix (#53124)

update gRPC to v1.6.0 to pick up data race fix grpc/grpc-go#1316  (#53128, @dixudx)


Enhance the codegen script within the staging sample apiserver to work with multiple groups and versions (#48714)

Add generate-groups.sh and generate-internal-groups.sh to k8s.io/code-generator to easily run generators against CRD or User API Server types. (#52186, @sttts)


"kubectl explain" should be able to explain "apiservices" and "customresourcedefinition" (#49465)

kubectl explain now uses openapi rather than swagger 1.2. (#53228, @apelisse)


kubectl set commands on ReplicaSet and DaemonSet occasionally return version registration errors (#53040)

Fixes an issue with kubectl set commands encountering conversion errors for ReplicaSet and DaemonSet objects (#53158, @liggitt)


apiserver uses wrong CommonName to verify service certificates for aggregated API Server when External admission controller is enabled (#56385)

Fixes server name verification of aggregated API servers and webhook admission endpoints (#56415, @liggitt)


NullIssue

ReplicationController now shares its underlying controller implementation with ReplicaSet to reduce the maintenance burden going forward. However, they are still separate resources and there should be no externally visible effects from this change. (#49429, @enisoc)
Google KMS integration was removed from in-tree in favor of a out-of-process extension point that will be used for all KMS providers. (#54759, @sakshamsharma)
kubectl get will by default fetch large lists of resources in chunks of up to 500 items rather than requesting all resources up front from the server. This reduces the perceived latency of managing large clusters since the server returns the first set of results to the client much more quickly.  A new flag --chunk-size=SIZE may be used to alter the number of items or disable this feature when 0 is passed.  This is a beta feature. (#53768, @smarterclayton)
Add events.k8s.io api group with v1beta1 API containing redesigned Event type. (#49112, @gmarek)
kubectl apply use openapi to calculate diff be default. It will fall back to use baked-in types when openapi is not available. (#51321, @mengqiy)
The GenericAdmissionWebhook is renamed as ValidatingAdmissionWebhook. Please update you apiserver configuration file to use the new name to pass to the apiserver's --admission-control flag. (#55988, @caesarxuchao)
Admission response alt (#55829, @cheftako)
The apiserver sends external versioned object to the admission webhooks now. Please update the webhooks to expect admissionReview.spec.object.raw to be serialized external versions of objects.  (#55127, @caesarxuchao)
apiserver: --etcd-quorum-read now defaults to true, to ensure correct operation with HA etcd clusters (#53717, @liggitt)
API chunking via the limit and continue request parameters is promoted to beta in this release.  Client libraries using the Informer or ListWatch types will automatically opt in to chunking. (#52949, @smarterclayton)
DaemonSet, Deployment, ReplicaSet, and StatefulSet have been promoted to GA and are available in the apps/v1 group version. (#53679, @kow3ns)


conversion-gen --extra-peer-dirs references k8s.io/kubernetes types (#54301)

In conversion-gen removed Kubernetes core API from default extra-peer-dirs. (#54394, @sttts)


Apps


Api


Eliminate Phase and Conditions from the API (#7856)

StatefulSet status now has support for conditions, making it consistent with other core controllers in v1  (#55268, @foxish)


Batch & Workload-Api/Job


Implement scale endpoint for jobs (#38756)

RBAC PolicyRules now allow resource=*/<subresource> to cover any-resource/<subresource>.   For example, */scale covers replicationcontroller/scale. (#53722, @deads2k)


NullArea


NullIssue

StatefulSet controller will create a label for each Pod in a StatefulSet. The label is named statefulset.kubernetes.io/pod-name and it is equal to the name of the Pod. This allows users to create a Service per Pod to expose a connection to individual Pods. (#55329, @kow3ns)
ReplicationController now shares its underlying controller implementation with ReplicaSet to reduce the maintenance burden going forward. However, they are still separate resources and there should be no externally visible effects from this change. (#49429, @enisoc)
DaemonSet status now has a new field named "conditions", making it consistent with other workloads controllers. (#55272, @janetkuo)
DaemonSet, Deployment, ReplicaSet, and StatefulSet have been promoted to GA and are available in the apps/v1 group version. (#53679, @kow3ns)
Add API version apps/v1, and bump DaemonSet to apps/v1 (#53278, @janetkuo)


Remove CreatedByAnnotation in v1.9, in favor of ControllerRef (#50720)

The kubernetes.io/created-by annotation is no longer added to controller-created objects. Use the  metadata.ownerReferences item that has controller set to true to determine which controller, if any, owns an object. (#54445, @crimsonfaith91)


kubectl scale implementation for core workload controllers (#49504)

RBAC PolicyRules now allow resource=*/<subresource> to cover any-resource/<subresource>.   For example, */scale covers replicationcontroller/scale. (#53722, @deads2k)


[apps/v1] Change DefaultGarbageCollectionPolicy for workload controllers (#55027)

change DefaultGarbageCollectionPolicy to DeleteDependents for workloads controllers (#55148, @dixudx)


Architecture


Api


Eliminate Phase and Conditions from the API (#7856)

StatefulSet status now has support for conditions, making it consistent with other core controllers in v1  (#55268, @foxish)


NullArea


NullIssue

Google KMS integration was removed from in-tree in favor of a out-of-process extension point that will be used for all KMS providers. (#54759, @sakshamsharma)


Auth


Admission-Control


Authentication for webhook admission to heterogenous authentication domains (#54404)

generic webhook admission now takes a config file which describes how to authenticate to webhook servers (#54414, @deads2k)


Audit


[audit] Always retry sending to webhook (#52909)

Webhook always retries connection reset error. (#53947, @crassirostris)


[audit] Figure out timestamps in event objects (#52160)

add RequestReceivedTimestamp and StageTimestamp to audit event (#52981, @CaoShuFeng)


Kubeadm


CertificateManager blocks kubelet start if auto-approval is not enabled (#53237)

kubeadm allows the kubelets in the cluster to automatically renew their client certificates (#53252, @kad)


Kubeadm & Kubelet


TLS-bootstrapped kubelet loses client certs after reboot, node stays on NotReady status (#53288)

kubelet --cert-dir now defaults to /var/lib/kubelet/pki, in order to ensure bootstrapped and rotated certificates persist beyond a reboot. resolves an issue in kubeadm with false-positive /var/lib/kubelet is not empty message during pre-flight checks (#53317, @liggitt)


NullArea


[PodSecurityPolicy] Optimize getMatchingPolicies (#55521)

Improved PodSecurityPolicy admission latency, but validation errors are no longer limited to only errors from authorized policies. (#55643, @tallclair)


OIDC username prefix option is not working (#56169)

kube-apiserver: fixed --oidc-username-prefix and --oidc-group-prefix flags which previously weren't correctly enabled (#56175, @ericchiang)


RFE: Bootstrap Checkpointing - Modify manifest behavior slightly for self hosting.  (#49236)

Initial basic bootstrap-checkpoint support (#50984, @timothysc)


Proposal: support unequivocal DENY in union authorizer (#51862)

Add support for the webhook authorizer to make a Deny decision that short-circuits the union authorizer and immediately returns Deny.  (#53273, @mikedanese)


Split PSP defaulting and validation (#36184)

PodSecurityPolicy: when multiple policies allow a submitted pod, priority is given to ones which do not require any fields in the pod spec to be defaulted. If the pod must be defaulted, the first policy (ordered by name) that allows the pod is used. (#52849, @liggitt)


Certificate Signing Request cleaner to GC CSRs (#51550)

New controller to GC CSRs. (#51840, @jcbsmpsn)


Reconciliation adds duplicated subjects on server start (#53296)

Fixes an issue with RBAC reconciliation that could cause duplicated subjects in some bootstrapped rolebindings on each restart of the API server. (#53239, @enj)


NullIssue

RBAC ClusterRoles can now select other roles to aggregate (#54005, @deads2k)
Audit policy files without apiVersion and kind are treated as invalid. (#54267, @ericchiang)
kubeadm: Add an experimental mode to deploy CoreDNS instead of KubeDNS (#52501, @rajansandeep)
Google KMS integration was removed from in-tree in favor of a out-of-process extension point that will be used for all KMS providers. (#54759, @sakshamsharma)
RBAC: The default admin and edit roles now include read/write permissions and the view role includes read permissions on poddisruptionbudget.policy resources. (#52654, @liggitt)
Pod Security Policy can now manage access to specific FlexVolume drivers (#53179, @wanghaoran1988)
Implement kubelet side file system resizing. Also implement GCE PD resizing (#55815, @gnufied)
The RBAC bootstrapping policy now allows authenticated users to create selfsubjectrulesreviews. (#56095, @ericchiang)
Defaulting of controller-manager options for --cluster-signing-cert-file and --cluster-signing-key-file is deprecated and will be removed in a later release. (#54495, @mikedanese)
Resolves forbidden error when accessing replicasets and daemonsets via the apps API group (#54309, @liggitt)


Platform/Gce


NullIssue

Add support for PodSecurityPolicy on GCE: ENABLE_POD_SECURITY_POLICY=true enables the admission controller, and installs policies for default addons. (#52367, @tallclair)


Security


Bring PodSecurityPolicy to usable state (#23217)

PodSecurityPolicy: when multiple policies allow a submitted pod, priority is given to ones which do not require any fields in the pod spec to be defaulted. If the pod must be defaulted, the first policy (ordered by name) that allows the pod is used. (#52849, @liggitt)


Autoscaling


NullArea


HPA is still using replicationcontrollers.extensions/scale (#38810)

RBAC PolicyRules now allow resource=*/<subresource> to cover any-resource/<subresource>.   For example, */scale covers replicationcontroller/scale. (#53722, @deads2k)
Introduces a polymorphic scale client, allowing HorizontalPodAutoscalers to properly function on scalable resources in any API group. (#53743, @DirectXMan12)


NullIssue

Cluster Autoscaler 1.0.1 (#54298, @mwielgus)


kubectl scale implementation for core workload controllers (#49504)

RBAC PolicyRules now allow resource=*/<subresource> to cover any-resource/<subresource>.   For example, */scale covers replicationcontroller/scale. (#53722, @deads2k)


Update HPA tolerance to be a flag  (#18155)

Control HPA tolerance through the horizontal-pod-autoscaler-tolerance flag. (#52275, @mattjmcnaughton)


HPA scaling above spec.maxReplicas (#53670)

Address a bug which allowed the horizontal pod autoscaler to allocate desiredReplicas > maxReplicas in certain instances. (#53690, @mattjmcnaughton)


Aws


NullArea


Adding tag annotations on service manifest for ELB does not trigger update on AWS (#54642)

Ensure additional resource tags are set/updated AWS load balancers (#55731, @georgebuckerfield)


Taint a AWS node if a volume is stuck in "attaching" state for too long (#55502)

AWS: Apply taint to a node if volumes being attached to it are stuck in attaching state (#55558, @gnufied)


AWS makes high number of redundant AttachVolume and DeleteVolume calls (#55014)

Check for available volume before attach/delete operation in EBS (#55008, @gnufied)


AWS error messages printed on 2 lines (#49813)

avoid newline "\n" in the error to break log msg to 2 lines (#49826, @dixudx)


NullIssue

Add support for resizing EBS disks (#56118, @gnufied)
It is now possible to override the healthcheck parameters for AWS ELBs via annotations on the corresponding service. The new annotations are healthy-threshold, unhealthy-threshold, timeout, interval (all prefixed with service.beta.kubernetes.io/aws-load-balancer-healthcheck-) (#56024, @dimpavloff)
Support AWS ECR credentials in China (#50108, @zzq889)


Enable AWS Network Load Balancer for Services of type LoadBalancer (#52173)

Add Amazon NLB support (#53400, @micahhausler)


Azure


NullArea


there are lots of warning message due to GetMountRefs func in windows (#54670)

fix warning messages due to GetMountRefs func not implemented in windows (#52401, @andyzhangx)


Azure loadbalancer should reconcile security groups properly. (not just by name, but also by other properties) (#55733)

Kubernetes update Azure nsg rules based on not just difference in Name, but also in Protocol, SourcePortRange, DestinationPortRange, SourceAddressPrefix, DestinationAddressPrefix, Access, and Direction. (#55752, @kevinkim9264)


wrong controller-master detection (#54570)

fix azure disk storage account init issue (#55927, @andyzhangx)


Azure disk dose not work as expected (#55776)

fix azure disk storage account init issue (#55927, @andyzhangx)


Azure data disk should provision storage account on on-demand (#50883)

fix azure disk storage account init issue (#55927, @andyzhangx)


[Azure] Support setting the DNS name label for public IPs created by ingress controllers  (#44775)

New service annotation "service.beta.kubernetes.io/azure-dns-label-name" to set Azure DNS label name for public IP (#47849, @tomerf)


there is azure file mount limit issue on windows due to using drive letter (#54668)

fix azure file mount limit issue on windows due to using drive letter (#53629, @andyzhangx)


Azure disk: storage class should support the sku if the storage accout support it (#55774)

add GRS, RAGRS storage account type support for azure disk (#55931, @andyzhangx)


NullIssue

Restrict Azure NSG rules to allow external access only to load balancer IP (#54177, @itowlson)
Upgraded Azure SDK to v11.1.1. (#54971, @itowlson)
allow windows mount path (#51240, @andyzhangx)
Azure cloudprovider: Fix controller manager crash issue on a manually created k8s cluster. (#53694, @andyzhangx)


Panic in azure_dd/azure_mounter.go when syncing pod (#54149)

fix azure pv crash due to volumeSource.ReadOnly value nil (#54607, @andyzhangx)


azure_dd: managed disks don't pass "FormatAndMount" (#50150)

fix azure disk mount failure on coreos and some other distros (#54334, @andyzhangx)


Platform/Azure


azure_file volumes should allow setting of dir_mode and file_mode (#37005)

support mount options in azure file (#54674, @andyzhangx)


Cli


Federation


NullIssue

Development of Kubernetes Federation has moved to github.com/kubernetes/federation.  This move out of tree also means that Federation will begin releasing separately from Kubernetes.  The impact of this is Federation-specific behavior will no longer be included in kubectl, kubefed will no longer be released as part of Kubernetes, and the Federation servers will no longer be included in the hyperkube binary and image. (#53816, @marun)


Kubectl


Unable to use kubectl get with a fieldSelector (#14129)

add field selector for kubectl get (#50140, @dixudx)


kubectl scale should use the scale subresource (#29698)

RBAC PolicyRules now allow resource=*/<subresource> to cover any-resource/<subresource>.   For example, */scale covers replicationcontroller/scale. (#53722, @deads2k)


NullIssue

kubectl cp command supports coping remote file into local directory (#46762, @bruceauyeung)


Kubectl: Replace usages of swagger with open API (#44589)

Kubectl: Remove swagger 1.2 validation. Also removes options --use-openapi and --schema-cache-dir as these are no longer needed. (#53232, @apelisse)


NullArea


kubectl set commands on ReplicaSet and DaemonSet occasionally return version registration errors (#53040)

Fixes an issue with kubectl set commands encountering conversion errors for ReplicaSet and DaemonSet objects (#53158, @liggitt)


NullIssue

"kubectl cp" updated to honor destination names  (#51215, @juanvallejo)
Added --dry-run option to kubectl drain (#52440, @juanvallejo)
outputs <none> for columns specified by -o custom-columns but not found in object (#51750, @jianhuiz)
kubectl apply use openapi to calculate diff be default. It will fall back to use baked-in types when openapi is not available. (#51321, @mengqiy)
kubectl create pdb will no longer set the min-available field by default.  (#53047, @yuexiao-wang)
DaemonSet, Deployment, ReplicaSet, and StatefulSet have been promoted to GA and are available in the apps/v1 group version. (#53679, @kow3ns)
kubectl get will by default fetch large lists of resources in chunks of up to 500 items rather than requesting all resources up front from the server. This reduces the perceived latency of managing large clusters since the server returns the first set of results to the client much more quickly.  A new flag --chunk-size=SIZE may be used to alter the number of items or disable this feature when 0 is passed.  This is a beta feature. (#53768, @smarterclayton)
add --raw to kubectl create to POST using the normal transport (#54245, @deads2k)


Add create priorityclass sub command  (#54857)

add create priorityclass sub command (#54858, @wackxu)


kubectl scale implementation for core workload controllers (#49504)

RBAC PolicyRules now allow resource=*/<subresource> to cover any-resource/<subresource>.   For example, */scale covers replicationcontroller/scale. (#53722, @deads2k)


"kubectl explain" should be able to explain "apiservices" and "customresourcedefinition" (#49465)

kubectl explain now uses openapi rather than swagger 1.2. (#53228, @apelisse)


Usability


top pod and top node output is unstable ordered (#53513)

kubectl top pod and node should sort by namespace / name so that results don't jump around. (#53560, @dixudx)


Cluster-Lifecycle


Apiserver & Cloudprovider & Platform/Gce


NullIssue

kube-apiserver: --ssh-user and --ssh-keyfile are now deprecated and will be removed in a future release. Users of SSH tunnel functionality used in Google Container Engine for the Master -> Cluster communication should plan to transition to alternate methods for bridging master and node networks. (#54433, @dims)


Cloudprovider


NullIssue

hyperkube: add cloud-controller-manager (#54197, @colemickens)


Expose concurrent-service-syncs flag on the CCM like it is for the KCM (#55560)

Add the concurrent service sync flag to the Cloud Controller Manager to allow changing the number of workers. (--concurrent-service-syncs) (#55561, @jhorwit2)


cloud controller manager does not support configmap resource locks (#55124)

Fix support for configmap resource lock type in CCM (#55125, @jhorwit2)


Remove --cloud-provider=auto-detect (#50986)

kubelet's --cloud-provider flag no longer defaults to "auto-detect".  If you want cloud-provider support in kubelet, you must set a specific cloud-provider explicitly. (#53573, @dims)


Etcd


Update to etcd 3.1.X  (#49386)

etcd: update version to 3.1.10 (#49393, @hongchaodeng)


Hw-Accelerators


NullIssue

GCE nodes with NVIDIA GPUs attached now expose nvidia.com/gpu as a resource instead of alpha.kubernetes.io/nvidia-gpu. (#54826, @mindprince)


Ipv6


Add kubeadm config for setting kube-proxy BindAddress (#50927)

Adds kubeadm support for using ComponentConfig for the kube-proxy (#55972, @rpothier)


Use CNI v0.6.x in Kubernetes v1.9.0 (#49480)

bump CNI to v0.6.0 (#51250, @dixudx)


Kubeadm


kubeadm 1.8.0 init fails with "/var/lib/kubelet is not empty" (#53356)

kubelet --cert-dir now defaults to /var/lib/kubelet/pki, in order to ensure bootstrapped and rotated certificates persist beyond a reboot. resolves an issue in kubeadm with false-positive /var/lib/kubelet is not empty message during pre-flight checks (#53317, @liggitt)


CertificateManager blocks kubelet start if auto-approval is not enabled (#53237)

kubeadm allows the kubelets in the cluster to automatically renew their client certificates (#53252, @kad)


NullIssue

Implement individual control for kubeadm preflight checks (#56072, @kad)
kubeadm now produces error during preflight checks if swap is enabled. Users, who can setup kubelet to run in unsupported environment with enabled swap, will be able to skip that preflight check. (#55399, @kad)


kubeadm will produce error if kubelet too new for control plane (#54868, @kad)


kubeadm health checks can also be skipped with --ignore-checks-errors (#56130, @anguslees)


Kubeadm & Kubelet


TLS-bootstrapped kubelet loses client certs after reboot, node stays on NotReady status (#53288)

kubelet --cert-dir now defaults to /var/lib/kubelet/pki, in order to ensure bootstrapped and rotated certificates persist beyond a reboot. resolves an issue in kubeadm with false-positive /var/lib/kubelet is not empty message during pre-flight checks (#53317, @liggitt)


Kubelet-Api


Specifying feature gates as a string of key-value pairs in ComponentConfig structures is awkward (#53024)

The Kubelet's feature gates are now specified as a map when provided via a JSON or YAML KubeletConfiguration, rather than as a string of key-value pairs. (#53025, @mtaufen)


NullArea


RFE: Bootstrap Checkpointing - Modify manifest behavior slightly for self hosting.  (#49236)

Initial basic bootstrap-checkpoint support (#50984, @timothysc)


Refactor kube-scheduler configuration (#52428)

The kube-scheduler command now supports a --config flag which is the location of a file containing a serialized scheduler configuration. Most other kube-scheduler flags are now deprecated. (#52562, @ironcladlou)


Creation of gitRepo volume is broken in 1.8.0+ (#54129)

Add openssh-client back into the hyperkube image. This allows the gitRepo volume plugin to work properly. (#54250, @ixdy)


Update to Go 1.9 (#49484)

Upgrade to go1.9 (#51375, @cblecker)


NullIssue

kubeadm: Add CoreDNS support for kubeadm "upgrade" and "alpha phases addons". (#55952, @rajansandeep)
Kubeadm now supports for Kubelet Dynamic Configuration. (#55803, @xiangpengzhao)
Base images bumped to Debian Stretch (9) (#52744, @rphillips)
kubeadm init: fix a bug that prevented the --token-ttl flag and tokenTTL configuration value from working as expected for infinite (0) values. (#54640, @mattmoyer)
Feature gates now check minimum versions (#54539, @jamiehannaford)
Load kernel modules automatically inside a kube-proxy pod (#52003, @vfreex)
kubeadm: added --print-join-command flag for kubeadm token create. (#56185, @mattmoyer)
Adding etcd version display to kubeadm upgrade plan subcommand (#56156, @sbezverk)
Adds to kubeadm upgrade apply, a new --etcd-upgrade keyword. When this keyword is specified, etcd's static pod gets upgraded to the etcd version officially recommended for a target kubernetes release. (#55010, @sbezverk)
kubeadm: Add an experimental mode to deploy CoreDNS instead of KubeDNS (#52501, @rajansandeep)


kubeadm  will warn users if access to IP ranges for Pods or Services will be done via HTTP proxy. (#52792, @kad)


Contributor-Experience


NullArea


NullIssue

kubectl apply use openapi to calculate diff be default. It will fall back to use baked-in types when openapi is not available. (#51321, @mengqiy)


Federation (Deprecated - Do Not Use)


NullArea


Enable kubefed init support nodeSelector (#50716)

support nodeSelector in kubefed init (#50749, @dixudx)


Gcp


Apiserver & Cloudprovider & Platform/Gce


NullIssue

kube-apiserver: --ssh-user and --ssh-keyfile are now deprecated and will be removed in a future release. Users of SSH tunnel functionality used in Google Container Engine for the Master -> Cluster communication should plan to transition to alternate methods for bridging master and node networks. (#54433, @dims)


Platform/Gce


GCE should allow users to configure with what service account their nodes are created (#53603)

Allow GCE users to configure the service account made available on their nodes (#52868, @ihmccreery)


Instrumentation


Audit


[audit] Always retry sending to webhook (#52909)

Webhook always retries connection reset error. (#53947, @crassirostris)


NullIssue

Adjust batching audit webhook default parameters: increase queue size, batch size, and initial backoff. Add throttling to the batching audit webhook. Default rate limit is 10 QPS. (#53417, @crassirostris)


Hw-Accelerators


NullIssue

Kubelet now exposes metrics for NVIDIA GPUs attached to the containers. (#55188, @mindprince)


Logging


NullIssue

[fluentd-gcp addon] Fixes fluentd deployment on GCP when custom resources are set. (#55950, @crassirostris)
[fluentd-gcp addon] Fluentd now runs in its own network, not in the host one. (#54395, @crassirostris)


NullArea


Collect metrics on admission rejections (#55030)

Metrics have been added for monitoring admission plugins, including the new dynamic (webhook-based) ones. (#55183, @jpbetz)


Bring all prom-to-sd container to the same image version (#54583)

Reduce log noise produced by prometheus-to-sd, by bumping it to version 0.2.2. (#54635, @loburm)


NullIssue

Fix a typo in prometheus-to-sd configuration, that drops some stackdriver metrics. (#56473, @loburm)
[fluentd-elasticsearch addon] Elasticsearch and Kibana are updated to version 5.6.4 (#55400, @mrahbar)
A new field is added to CRI container log format to support splitting a long log line into multiple lines. (#55922, @Random-Liu)
fluentd now supports CRI log format. (#54777, @Random-Liu)


Multicluster


Federation


NullIssue

Development of Kubernetes Federation has moved to github.com/kubernetes/federation.  This move out of tree also means that Federation will begin releasing separately from Kubernetes.  The impact of this is Federation-specific behavior will no longer be included in kubectl, kubefed will no longer be released as part of Kubernetes, and the Federation servers will no longer be included in the hyperkube binary and image. (#53816, @marun)


Enable kubefed init support ImagePullSecrets and imagePullPolicy (#50718)

support imagePullSecrets and imagePullPolicy in kubefed init (#50740, @dixudx)


Get clusters --show-labels does not work in a federation context (#53729)

update cluster printer to enable --show-labels (#53771, @dixudx)


Network


Controller-Manager


Pod in graceful termination should not be on the ready address list of related Endpoints objects (#54723)

Fix a bug where pod address is not removed from endpoints object while pod is in graceful termination. (#54828, @freehan)


Ipv6


Use CNI v0.6.x in Kubernetes v1.9.0 (#49480)

bump CNI to v0.6.0 (#51250, @dixudx)


Ipvs


Need to install ipset in debian-iptables docker image (#56116)

install ipset in debian-iptables docker image (#56115, @m1093782566)


IPVS kube-proxy will flush all existing ipvs rules in its startup (#55857)

Add cleanup-ipvs flag for kube-proxy  (#56036, @m1093782566)


Ipvs & Kube-Proxy


Try ipset in kube-proxy (#54203)

Using ipset doing SNAT and packet filtering in IPVS kube-proxy (#54219, @m1093782566)


Failed to access NodePort if kube-proxy running in ipvs mode (#53393)

Using ipset doing SNAT and packet filtering in IPVS kube-proxy (#54219, @m1093782566)


Kube-Proxy


Handle nodes with iptables FORWARD DROP better (#39823)

Add iptables rules to allow Pod traffic even when default iptables policy is to reject. (#52569, @tmjd)


NullArea


Add CoreDNS in kube-up (#56439)

Add CoreDNS as an optional addon in kube-up (#55728, @rajansandeep)


Calico add-on: calico/node pod can take a long time to be restarted (#55013)

Disable the termination grace period for the calico/node add-on DaemonSet to reduce downtime during a rolling upgrade or deletion. (#55015, @fasaxc)


Service controller retries on doNotRetry service update failure (#54183)

Fix for service controller so that it won't retry on doNotRetry service update failure. (#54184, @MrHohn)


apiserver proxy feature does not rewrite Location header on redirects (#51790)

Apiserver proxy rewrites URL when service returns absolute path with request's host. (#52556, @roycaihw)


[kubelet] ignore keyword "options" define in /etc/resolv.conf, only look for nameserver and search (#42542)

Support copying "options" in resolv.conf into pod sandbox when dnsPolicy is Default (#54773, @phsiao)


NullIssue

Fixes bad conversion in host port chain name generating func which leads to some unreachable host ports. (#55153, @chenchun)
kubeadm: Add an experimental mode to deploy CoreDNS instead of KubeDNS (#52501, @rajansandeep)
Fix IPVS availability check (#51874, @vfreex)
Enhanced the network policy describer. (#46951, @aanm)
Load kernel modules automatically inside a kube-proxy pod (#52003, @vfreex)
Improve resilience by annotating kube-dns addon with podAntiAffinity to prefer scheduling on different nodes. (#52193, @StevenACoffman)
Add DNSConfig field to PodSpec and support "None" mode for DNSPolicy (Alpha). (#55848, @MrHohn)


Support annotations for AWS ELB Security Policies (#43744)

Added service annotation for AWS ELB SSL policy (#54507, @micahhausler)


kube-proxy: session affinity stops working when ESIPP=Local (#55429)

Fix session affinity issue with external load balancer traffic when ExternalTrafficPolicy=Local. (#55519, @MrHohn)


zero-value settings for kube-proxy being overwritten by default values (#50787)

Restored kube-proxy's support for 0 values for conntrack min, max, max per core, tcp close wait timeout, and tcp established timeout. (#55261, @ncdc)


Platform/Gce


NullIssue

GCE: Bump GLBC version to 0.9.7. (#53625, @nikhiljindal)


GCE: ILB sync fails for legacy networks and auto networks with unusual subnet names (#53409)

GCE: Fixes ILB sync on legacy networks and auto networks with unique subnet names (#53410, @nicksardo)


GCE: Ignore resource not found errors when deleting LB resources (#53411)

GCE: Fix issue deleting internal load balancers when the firewall resource may not exist. (#53450, @nicksardo)


Node


Cloudprovider


Remove --cloud-provider=auto-detect (#50986)

kubelet's --cloud-provider flag no longer defaults to "auto-detect".  If you want cloud-provider support in kubelet, you must set a specific cloud-provider explicitly. (#53573, @dims)


Hw-Accelerators


NullIssue

Device plugin Alpha API no longer supports returning artifacts per device as part of AllocateResponse. (#53031, @vishh)
Expose single annotation/label via downward API (#55902, @yguo0905)
Kubelet now exposes metrics for NVIDIA GPUs attached to the containers. (#55188, @mindprince)


Ipv6


Use CNI v0.6.x in Kubernetes v1.9.0 (#49480)

bump CNI to v0.6.0 (#51250, @dixudx)


Kubeadm


CertificateManager blocks kubelet start if auto-approval is not enabled (#53237)

kubeadm allows the kubelets in the cluster to automatically renew their client certificates (#53252, @kad)


Kubelet & Kubelet-Api


NullIssue

The EvictionHard, EvictionSoft, EvictionSoftGracePeriod, EvictionMinimumReclaim, SystemReserved, and KubeReserved fields in the KubeletConfiguration object (kubeletconfig/v1alpha1) are now of type map[string]string, which facilitates writing JSON and YAML files. (#54823, @mtaufen)
Relative paths in the Kubelet's local config files (--init-config-dir) will be resolved relative to the location of the containing files. (#55648, @mtaufen)
It is now possible to set multiple manifest url headers via the Kubelet's --manifest-url-header flag. Multiple headers for the same key will be added in the order provided. The ManifestURLHeader field in KubeletConfiguration object (kubeletconfig/v1alpha1) is now a map[string][]string, which facilitates writing JSON and YAML files. (#54643, @mtaufen)


Kubelet & Security


Dockershim doesn't consider seccomp profile root? (#55359)

Fix CRI localhost seccomp path in format localhost//profileRoot/profileName. (#55450, @feiskyer)


Kubelet-Api


Specifying feature gates as a string of key-value pairs in ComponentConfig structures is awkward (#53024)

The Kubelet's feature gates are now specified as a map when provided via a JSON or YAML KubeletConfiguration, rather than as a string of key-value pairs. (#53025, @mtaufen)


NullIssue

The Kubelet's --enable-custom-metrics flag is now marked deprecated. (#54154, @mtaufen)


NullArea


CRI: Debug API (#53757)

Verbose option is added to each status function in CRI. Container runtime could return extra information in status response for debugging. (#53965, @Random-Liu)


Error when using journald log driver and FallbackToLogsOnError (#52502)

Get fallback termination msg from docker when using journald log driver (#52503, @joelsmith)


kubelet cannot show Docker-CE version correctly  (#54039)

falls back to parse Docker runtime version as generic if not semver (#54040, @dixudx)


FailedSync event from kubelet provides no value (#53900)

kubelet provides more specific events when unable to sync pod (#53857, @derekwaynecarr)


Extra CRI call during processing cpu set (#53304)

Only do UpdateContainerResources when cpuset is set  (#53122, @resouer)


Consume ImageFS stats from StatsProvider in ImageGCManager (#53083)

Change ImageGCManage to consume ImageFS stats from StatsProvider (#53094, @yguo0905)


[Failing Test] [k8s.io] Summary API when querying /stats/summary should report resource usage through the stats api (#55909)

Add containerd monitoring support (#56109, @dashpole)


Cut and vendor cAdvisor v0.28.1 for the 1.9 release. (#55628)

Add containerd monitoring support (#56109, @dashpole)


Pods moving from Succeeded to Pending (#54499)

Fix an issue where pods were briefly transitioned to a "Pending" state during the deletion process. (#54593, @dashpole)


Hyperkube doesn't support --experimental-dockershim for kubelet (#54424)

Fix hyperkube kubelet --experimental-dockershim (#54508, @ivan4th)


Local pods stay around after node deletion (#48213)

don't recreate a mirror pod for static pod when node gets deleted (#48339, @dixudx)


Should be able to specific unconfined AppArmor profile (#52370)

enable to specific unconfined AppArmor profile (#52395, @dixudx)


Reduce cpumanager default logging verbosity (#54804)

set leveled logging (v=4) for 'updating container' message (#54865, @phsiao)


Remove the backward compatibility code for kubelet 1.2 in NodeController. (#48995)

The NodeController will not support kubelet 1.2. (#48996, @k82cn)


Deprecate --network-plugin-dir for kubelet (#46410)

Remove the --network-plugin-dir flag. (#53564, @supereagle)


Zone labels are removed every kubelet restart in 1.8 (#54070)

kubelet: prevent removal of default labels from Node API objects on startup (#54073, @liggitt)


NullIssue

Add pod-level CPU and memory stats from pod cgroup information (#55969, @jingxu97)
Load kernel modules automatically inside a kube-proxy pod (#52003, @vfreex)
RBAC: The default admin and edit roles now include read/write permissions and the view role includes read permissions on poddisruptionbudget.policy resources. (#52654, @liggitt)
Remove docker dependency during kubelet start up  (#54405, @resouer)
Fix overlay2 container disk metrics for Docker and CRI-O (#54827, @dashpole)
Metrics were added to network plugin to report latency of CNI operations (#53446, @sjenning)
Fix stats summary network value when multiple network interfaces are available. (#52144, @andyxning)
Kubelet can provide full summary api support except container log stats for CRI container runtime now. (#55810, @abhi)
Add pod-level local ephemeral storage metric in Summary API. Pod-level ephemeral storage reports the total filesystem usage for the containers and emptyDir volumes in the measured Pod. (#55447, @jingxu97)
A new field is added to CRI container log format to support splitting a long log line into multiple lines. (#55922, @Random-Liu)
Kubelet supports running mount utilities and final mount in a container instead running them on the host. (#53440, @jsafrane)
Fix the bug that query Kubelet's stats summary with CRI stats enabled results in error. (#53107, @Random-Liu)
BugFix: Exited containers are not Garbage Collected by the kubelet while the pod is running (#53167, @dashpole)
Base images bumped to Debian Stretch (9) (#52744, @rphillips)
fluentd now supports CRI log format. (#54777, @Random-Liu)
Add Windows support to the system verification check (#53730, @bsteciuk)
fix a bug where disk pressure could trigger prematurely when using overlay2 (#53684, @dashpole)
Don't remove extended resource capacities that are not registered with kubelet from node status. (#53353, @jiayingz)


RFE: Bootstrap Checkpointing - Modify manifest behavior slightly for self hosting.  (#49236)

Initial basic bootstrap-checkpoint support (#50984, @timothysc)


Better handling of device plugin resource deletion (#53395)

Extends deviceplugin to gracefully handle full device plugin lifecycle. (#55088, @jiayingz)


[feature] for GPU and hugepages, default must match defaultRequest in LimitRange if both are specified (#54917)

validate if default and defaultRequest match when creating LimitRange for GPU and hugepages. (#54919, @tianshapjq)


Make CRI logs parsing to a library (#55136)

Add CRI log parsing library at pkg/kubelet/apis/cri/logs (#55140, @feiskyer)


Creation of gitRepo volume is broken in 1.8.0+ (#54129)

Add openssh-client back into the hyperkube image. This allows the gitRepo volume plugin to work properly. (#54250, @ixdy)


Large kubemark performance tests failing with timeout during ns deletion (#53327)

Optimize random string generator to avoid multiple locks & use bit-masking (#53720, @shyamjvs)
Use separate client for leader election in scheduler to avoid starving leader election by regular scheduler operations. (#53793, @wojtek-t)


Platform/Gce


Remove compute-rw scope from GCE nodes (#8074)

gce: remove compute-rw, see what breaks (#53266, @mikedanese)


NullSig


Admission-Control


NullIssue

admission webhook registrations without a specific failure policy default to failing closed. (#54162, @deads2k)


Admission-Control & Security


NullIssue

The PodSecurityPolicy annotation kubernetes.io/psp on pods is only set once on create. (#55486, @sttts)


Audit


NullIssue

[advanced audit]add a policy wide omitStage (#54634, @CaoShuFeng)


NullArea


NullIssue

The dynamic admission webhook now supports a URL in addition to a service reference, to accommodate out-of-cluster webhooks. (#54889, @lavalamp)
not calculate new priority when user update other spec of a pod (#55221, @CaoShuFeng)
Fix iptables FORWARD policy for Docker 1.13 in kubernetes-worker charm (#54796, @Cynerva)
Add PodDisruptionBudget to scheduler cache. (#53914, @bsalamat)
Log when node is successfully initialized by Cloud Controller Manager (#53517, @andrewsykim)
Upgrading the kubernetes-master units now results in staged upgrades just like the kubernetes-worker nodes. Use the upgrade action in order to continue the upgrade process on each unit such as juju run-action kubernetes-master/0 upgrade (#55990, @hyperbolic2346)
Addon manager supports HA masters. (#55466, @x13n)
In PodTolerationRestriction admisson plugin, if namespace level tolerations are empty, now they override cluster level tolerations.  (#54812, @aveshagarwal)
Improve explanation of ReplicaSet (#53403, @rcorre)
kubeadm: Fix a bug on some OSes where the kubelet tried to mount a volume path that is non-existent and on a read-only filesystem  (#55320, @andrewrynhard)
Avoid unnecessary spam in kube-controller-manager log if --cluster-cidr is not specified and --allocate-node-cidrs is false. (#54934, @akosiaris)
GCI mounter is moved from the manifests tarball to the server tarball. (#47497, @mikedanese)
Ignore extended resources that are not registered with kubelet during container resource allocation. (#53547, @jiayingz)
PodSecurityPolicies for addons (#55509, @tallclair)
Correct wording of kubeadm upgrade response for missing ConfigMap. (#53337, @jmhardison)
Support completion for --clusterrole of kubectl create clusterrolebinding (#48267, @superbrothers)
GCE: provide an option to disable docker's live-restore on COS/ubuntu (#55260, @yujuhong)
secret data containing Docker registry auth objects is now generated using the config.json format (#53916, @juanvallejo)
Add support for RBAC support to Kubernetes via Juju (#53820, @ktsakalozos)
Kubelet evictions take pod priority into account (#53542, @dashpole)
Add --etcd-compaction-interval to apiserver for controlling request of compaction to etcd3 from apiserver. (#51765, @mitake)
Bugfix: master startup script on GCP no longer fails randomly due to concurrent iptables invocations. (#55945, @x13n)
Add extra-args configs for scheduler and controller-manager to kubernetes-master charm (#55185, @Cynerva)
Fix clustered datastore name to be absolute. (#54438, @pshahzeb)
Enable Priority admission control in kubeadm.  (#53175, @andrewsykim)
If you are using the cloud provider API to determine the external host address of the apiserver, set --external-hostname explicitly instead. The cloud provider detection has been deprecated and will be removed in the future (#54516, @dims)
Added support for SAN entries in the master node certificate via juju kubernetes-master config. (#54234, @hyperbolic2346)
kubectl get will now use OpenAPI schema extensions by default to select columns for custom types. (#53483, @apelisse)
Fix kubeadm reset crictl command (#55717, @runcom)
kubeadm: Add support for adding a Windows node (#53553, @bsteciuk)
Update AWS SDK to 1.12.7 (#53561, @justinsb)
Optimize Repeated registration of AlgorithmProvider when ApplyFeatureGates (#54047, @kuramal)
Upgrade fluentd-elasticsearch addon to Elasticsearch/Kibana 5.6.2 (#53307, @aknuds1)
The output of kubectl config get-contexts is now sorted alphabetically by the context name. (#46946, @kellycampbell)
kubeadm: reset: use crictl to reset containers (#54721, @runcom)
kubeadm: use the CRI for preflights checks (#55055, @runcom)
Added extra_sans config option to kubeapi-load-balancer charm. This allows the user to specify extra SAN entries on the certificate generated for the load balancer. (#54947, @hyperbolic2346)
Fix metrics API group name in audit configuration (#53493, @piosz)
default fail-swap-on to false for kubelet on kubernetes-worker charm (#53386, @wwwtyro)
Adding vishh as an reviewer/approver for hack directory (#54007, @vishh)
If a non-absolute mountPath is passed to the kubelet, prefix it with the appropriate root path. (#55665, @brendandburns)
Update kube-dns 1.14.7 (#54443, @bowei)
Horizontal pod autoscaler uses REST clients through the kube-aggregator instead of the legacy client through the API server proxy. (#53205, @kawych)
The minimum supported go version bumps to 1.9.1. (#55301, @xiangpengzhao)
Add a new feature gate for enabling an alpha annotation which, if present, excludes the annotated node from being added to a service load balancers. (#54644, @brendandburns)
Metadata concealment on GCE is now controlled by the ENABLE_METADATA_CONCEALMENT env var.  See cluster/gce/config-default.sh for more info. (#54150, @ihmccreery)
Remove the LbaasV1 of OpenStack cloud provider, currently only support LbaasV2. (#52717, @FengyunPan)
Fixes a performance issue (#51899) identified in large-scale clusters when deleting thousands of pods simultaneously across hundreds of nodes, by actively removing containers of deleted pods, rather than waiting for periodic garbage collection and batching resulting pod API deletion requests. (#53233, @dashpole)
Fix code-generators to produce correct code when GroupName, PackageName and/or GoName differ. (#55614, @sttts)
Allow HPA to read custom metrics. (#54854, @kawych)
Fix permissions for Metrics Server. (#53330, @kawych)
[fluentd-elasticsearch addon] Elasticsearch service name can be overridden via env variable ELASTICSEARCH_SERVICE_NAME (#54215, @mrahbar)
Change kubeadm create token to default to the group that almost everyone will want to use.  The group is system:bootstrappers:kubeadm:default-node-token and is the group that kubeadm sets up, via an RBAC binding, for auto-approval (system:certificates.k8s.io:certificatesigningrequests:nodeclient). (#53512, @jbeda)
Add --no-negcache flag to kube-dns to prevent caching of NXDOMAIN responses. (#53604, @cblecker)
Deprecation: The flag etcd-quorum-read  of kube-apiserver is deprecated and the ability to switch off quorum read will be removed in a future release. (#53795, @xiangpengzhao)
kubeadm: Strip bootstrap tokens from the kubeadm-config ConfigMap (#53559, @fabriziopandini)
Added integration test for TaintNodeByCondition. (#53184, @k82cn)
Added namespaceSelector to externalAdmissionWebhook configuration to allow applying webhooks only to objects in the namespaces that have matching labels. (#54727, @caesarxuchao)
Add masquerading rules by default to GCE/GKE (#55178, @dnardo)
[cluster-monitoring addon] Update monitoring-influxdb-grafana to latest version (#53319, @kairen)


Improved generation of deb and rpm packages in bazel build (#53163, @kad)


Increase waiting time (120s) for docker startup in health-monitor.sh (#54099, @dchen1107)
update podtolerations admission to mutate and validate separately (#55251, @deads2k)
Add limitrange/resourcequota/downward_api  e2e tests for local ephemeral storage (#52523, @NickrenREN)
Update fluentd-gcp DaemonSet (#54175, @tallclair)
Allow for configuring etcd hostname in the manifest (#54403, @wojtek-t)
Use multi-arch busybox image for e2e (#54034, @dixudx)
Addon manager supports HA masters. (#55782, @x13n)
Add extra-args configs to kubernetes-worker charm (#55334, @Cynerva)
Log error of failed healthz check (#53048, @mrIncompetent)
Fix kubeadm upgrade plan for offline operation: ignore errors when trying to fetch latest versions from dl.k8s.io (#54016, @praseodym)
Support completion for kubectl config rename-context (#48340, @superbrothers)
Removes Priority Admission Controller from kubeadm since it's alpha.  (#55237, @andrewsykim)


Openstack


NullArea


NullIssue

OpenStack cloud provider supports Cinder v3 API. (#52910, @FengyunPan)
Octavia v2 now supported as a LB provider (#55393, @jamiehannaford)
Make OpenStack LBaaS v2 Provider configurable (#54176, @gonzolino)
Support autoprobing node-security-group for openstack cloud provider, Support multiple Security Groups for cluster's nodes. (#50836, @FengyunPan)


OpenStack Cinder version detection fails (and is incorrect) (#50461)

Using OpenStack service catalog to do version detection (#53115, @FengyunPan)


Release


NullArea


NullIssue

Upgrade to go1.9.2 (#55420, @cblecker)


Update to Go 1.9 (#49484)

Upgrade to go1.9 (#51375, @cblecker)


Scalability


Etcd


Update to etcd 3.1.X  (#49386)

etcd: update version to 3.1.10 (#49393, @hongchaodeng)


NullArea


NullIssue

ReplicationController now shares its underlying controller implementation with ReplicaSet to reduce the maintenance burden going forward. However, they are still separate resources and there should be no externally visible effects from this change. (#49429, @enisoc)
apiserver: --etcd-quorum-read now defaults to true, to ensure correct operation with HA etcd clusters (#53717, @liggitt)
API chunking via the limit and continue request parameters is promoted to beta in this release.  Client libraries using the Informer or ListWatch types will automatically opt in to chunking. (#52949, @smarterclayton)


Update to Go 1.9 (#49484)

Upgrade to go1.9 (#51375, @cblecker)


Large kubemark performance tests failing with timeout during ns deletion (#53327)

Optimize random string generator to avoid multiple locks & use bit-masking (#53720, @shyamjvs)
Use separate client for leader election in scheduler to avoid starving leader election by regular scheduler operations. (#53793, @wojtek-t)


[PodSecurityPolicy] Optimize getMatchingPolicies (#55521)

Improved PodSecurityPolicy admission latency, but validation errors are no longer limited to only errors from authorized policies. (#55643, @tallclair)


Scheduling


Hw-Accelerators


NullIssue

Add ExtendedResourceToleration admission controller. This facilitates creation of dedicated nodes with extended resources. If operators want to create dedicated nodes with extended resources (like GPUs, FPGAs etc.), they are expected to taint the node with extended resource name as the key. This admission controller, if enabled, automatically adds tolerations for such taints to pods requesting extended resources, so users don't have to manually add these tolerations.  (#55839, @mindprince)
GCE nodes with NVIDIA GPUs attached now expose nvidia.com/gpu as a resource instead of alpha.kubernetes.io/nvidia-gpu. (#54826, @mindprince)


NullArea


Support PodDisruptionBudget during preemption (#53913)

Add PodDisruptionBudget support during pod preemption (#56178, @bsalamat)


Fix starvation problem in pod preemption (#54501)

Add support to take nominated pods into account during scheduling to avoid starvation of higher priority pods. (#55933, @bsalamat)


Refactor kube-scheduler configuration (#52428)

The kube-scheduler command now supports a --config flag which is the location of a file containing a serialized scheduler configuration. Most other kube-scheduler flags are now deprecated. (#52562, @ironcladlou)


Large kubemark performance tests failing with timeout during ns deletion (#53327)

Optimize random string generator to avoid multiple locks & use bit-masking (#53720, @shyamjvs)
Use separate client for leader election in scheduler to avoid starving leader election by regular scheduler operations. (#53793, @wojtek-t)


Scheduler should handle pod updates during scheduling more gracefully (#52914)

Change scheduler to skip pod with updates only on pod annotations (#54008, @yguo0905)


"notReady" toleration should be "not-ready" (#51246)

Use not-ready to replace notReady in node condition taint keys. (#51266, @resouer)


Remove support for opaque integer resources (deprecated in v1.8) (#55102)

Remove opaque integer resources (OIR) support (deprecated in v1.8.) (#55103, @ConnorDoyle)


Consider moving TaintNodeUnreachable out of alpha (#54198)

Move of unreachable taint key out of alpha (#54208, @resouer)


NullIssue

move getMaxVols function to predicates.go and add some NewVolumeCountPredicate funcs (#51783, @jiulongzaitian)
Apply algorithm in scheduler by feature gates. (#52723, @k82cn)
RBAC: The default admin and edit roles now include read/write permissions and the view role includes read permissions on poddisruptionbudget.policy resources. (#52654, @liggitt)
Add a new scheduling queue that helps schedule the highest priority pending pod first. (#55109, @bsalamat)
add hostIP and protocol to the original hostport predicates procedure in scheduler. (#52421, @WIZARD-CXY)
Object count quotas supported on all standard resources using count/<resource>.<group> syntax (#54320, @derekwaynecarr)


Scheduler dies with "Schedulercache is corrupted" (#50916)

Fixed 'Schedulercache is corrupted' error in kube-scheduler (#55262, @liggitt)


Service-Catalog


NullArea


PodPreset Feature Tests Consistently Causing ci-kubernetes-e2e-gci-gce-alpha-features to Fail (#53079)

Skip podpreset test if the alpha feature setttings/v1alpha1 is disabled (#53080, @jennybuckley)


Storage


NullArea


PVCs using standard StorageClass create PDs in disks in wrong zone in multi-zone GKE clusters (#50115)

Fix a bug in GCE multizonal clusters where PersistentVolumes were sometimes created in zones without nodes. (#52322, @davidz627)


Taint a AWS node if a volume is stuck in "attaching" state for too long (#55502)

AWS: Apply taint to a node if volumes being attached to it are stuck in attaching state (#55558, @gnufied)


Add support for verifying attached but desired to be detached PVCs  (#52573)

Implement correction mechanism for dangling volumes attached for deleted pods (#55491, @gnufied)


Zero capacity PVs cause pods to fail and zero capacity PVCs create zero capacity PVs (#55553)

Validate positive capacity for PVs and PVCs. (#55532, @ianchakeres)


ScaleIO - credentials could be accessed by non-admin users (#53619)

ScaleIO persistent volumes now support referencing a secret in a namespace other than the bound persistent volume claim's namespace; this is controlled during provisioning with the secretNamespace storage class parameter; StoragePool and ProtectionDomain attributes no longer defaults to the value default (#54013, @vladimirvivien)


Recycle always failed on non x86 platform (#53942)

fix PV Recycle failed on non-amd64 platfrom (#53958, @dixudx)


NullIssue

Implement kubelet side file system resizing. Also implement GCE PD resizing (#55815, @gnufied)
Implement volume resize for cinder (#51498, @NickrenREN)
Block volumes Support: CRI, volumemanager and operationexecutor changes (#51494, @mtanino)
iSCSI Persistent Volume Sources can now reference CHAP Secrets in namespaces other than the namespace of the bound Persistent Volume Claim (#51530, @rootfs)
Add resize support for ceph RBD (#52767, @NickrenREN)
Add support for resizing EBS disks (#56118, @gnufied)
Block volumes Support: FC plugin update (#51493, @mtanino)
Updating vsphere cloud provider to support k8s cluster spread across multiple vCenters (#55845, @rohitjogvmw)
Kubelet supports running mount utilities and final mount in a container instead running them on the host. (#53440, @jsafrane)
allow windows mount path (#51240, @andyzhangx)
RBD Persistent Volume Sources can now reference User's Secret in namespaces other than the namespace of the bound Persistent Volume Claim (#54302, @sbezverk)
Pod Security Policy can now manage access to specific FlexVolume drivers (#53179, @wanghaoran1988)


should prevent the deletion of a PVC that is referenced by an active pod (#45143)

PVC Finalizing Controller is introduced in order to prevent deletion of a PVC that is being used by a pod. (#55824, @pospispa)


PersistentVolumeSource should be read-only (#54562)

Validate that PersistentVolumeSource is not changed during PV Update (#54761, @ianchakeres)


Multi Attach PVC errors and events are too noisy (#53214)

Fix multi-attach error spam in logs and events (#53401, @gnufied)


wrong controller-master detection (#54570)

fix azure disk storage account init issue (#55927, @andyzhangx)


Remove ScaleIO dependency on drv_cfg binary for containerization (#54954)

ScaleIO driver completely removes dependency on drv_cfg binary so a Kubernetes cluster can easily run a containerized kubelet. (#54956, @vladimirvivien)


Platform/Gce


Remove compute-rw scope from GCE nodes (#8074)

gce: remove compute-rw, see what breaks (#53266, @mikedanese)


Testing


Federation


NullIssue

Development of Kubernetes Federation has moved to github.com/kubernetes/federation.  This move out of tree also means that Federation will begin releasing separately from Kubernetes.  The impact of this is Federation-specific behavior will no longer be included in kubectl, kubefed will no longer be released as part of Kubernetes, and the Federation servers will no longer be included in the hyperkube binary and image. (#53816, @marun)


Hw-Accelerators


NullIssue

Kubelet now exposes metrics for NVIDIA GPUs attached to the containers. (#55188, @mindprince)


NullArea


Update to Go 1.9 (#49484)

Upgrade to go1.9 (#51375, @cblecker)


NullIssue

Fix to prevent downward api change break on older versions (#53673, @timothysc)
API chunking via the limit and continue request parameters is promoted to beta in this release.  Client libraries using the Informer or ListWatch types will automatically opt in to chunking. (#52949, @smarterclayton)


Windows


NullArea


wrong controller-master detection (#54570)

fix azure disk storage account init issue (#55927, @andyzhangx)


there is azure file mount limit issue on windows due to using drive letter (#54668)

fix azure file mount limit issue on windows due to using drive letter (#53629, @andyzhangx)


there are lots of warning message due to GetMountRefs func in windows (#54670)

fix warning messages due to GetMountRefs func not implemented in windows (#52401, @andyzhangx)


NullIssue

allow windows mount path (#51240, @andyzhangx)