v1.38.0

Upgrade K8s versions to use 1.30 and Kubebuilder v4

This update has a lot of scaffolding changes due to the removal of kube-rbac-proxy, if these migrations become difficult to follow, it might be beneficial to scaffold a net new sample project to compare.

  1. [helm/v1, ansible/v1] Update the kustomize version in your Makefile

     - curl -sSLo - https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize/v5.3.0/kustomize_v5.3.0_$(OS)_$(ARCH).tar.gz | \
 + curl -sSLo - https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize/v5.4.2/kustomize_v5.4.2_$(OS)_$(ARCH).tar.gz | \

  2. [go/v4] Update your go.mod file to upgrade the dependencies and run go mod tidy to download them

     go 1.22.0
    
 github.com/onsi/ginkgo/v2 v2.17.1
 github.com/onsi/gomega v1.32.0
 k8s.io/api v0.30.1
 k8s.io/apimachinery v0.30.1
 k8s.io/client-go v0.30.1
 sigs.k8s.io/controller-runtime v0.18.4

  3. [go/v4] Update your Makefile with the below changes:

     - ENVTEST_K8S_VERSION = 1.29.0
 + ENVTEST_K8S_VERSION = 1.30.0

     - KUSTOMIZE ?= $(LOCALBIN)/kustomize-$(KUSTOMIZE_VERSION)
 - CONTROLLER_GEN ?= $(LOCALBIN)/controller-gen-$(CONTROLLER_TOOLS_VERSION)
 - ENVTEST ?= $(LOCALBIN)/setup-envtest-$(ENVTEST_VERSION)
 - GOLANGCI_LINT = $(LOCALBIN)/golangci-lint-$(GOLANGCI_LINT_VERSION)
 + KUSTOMIZE ?= $(LOCALBIN)/kustomize
 + CONTROLLER_GEN ?= $(LOCALBIN)/controller-gen
 + ENVTEST ?= $(LOCALBIN)/setup-envtest
 + GOLANGCI_LINT = $(LOCALBIN)/golangci-lint

     - KUSTOMIZE_VERSION ?= v5.3.0
 - CONTROLLER_TOOLS_VERSION ?= v0.14.0
 - ENVTEST_VERSION ?= release-0.17
 - GOLANGCI_LINT_VERSION ?= v1.57.2
 + KUSTOMIZE_VERSION ?= v5.4.2
 + CONTROLLER_TOOLS_VERSION ?= v0.15.0
 + ENVTEST_VERSION ?= release-0.18
 + GOLANGCI_LINT_VERSION ?= v1.59.1

     - $(call go-install-tool,$(GOLANGCI_LINT),github.com/golangci/golangci-lint/cmd/golangci-lint,${GOLANGCI_LINT_VERSION})
 + $(call go-install-tool,$(GOLANGCI_LINT),github.com/golangci/golangci-lint/cmd/golangci-lint,$(GOLANGCI_LINT_VERSION))

     - @[ -f $(1) ] || { \
 + @[ -f "$(1)-$(3)" ] || { \
   echo "Downloading $${package}" ;\
 + rm -f $(1) || true ;\
 - mv "$$(echo "$(1)" | sed "s/-$(3)$$//")" $(1) ;\
 - }
 + mv $(1) $(1)-$(3) ;\
 + } ;\
 + ln -sf $(1)-$(3) $(1)

  4. [go/v4] Update your .golangci.yml with the below changes:

    -  exportloopref
+     - ginkgolinter
      - prealloc
+     - revive
+ 
+ linters-settings:
+   revive:
+     rules:
+       - name: comment-spacings

  5. [go/v4] Update your Dockerfile file with the below changes:

    - FROM golang:1.21 AS builder
+ FROM golang:1.22 AS builder

  6. [go/v4] Update your main.go file with the below changes:

         "sigs.k8s.io/controller-runtime/pkg/log/zap"
+    "sigs.k8s.io/controller-runtime/pkg/metrics/filters"

     var enableHTTP2 bool
-    flag.StringVar(&metricsAddr, "metrics-bind-address", ":8080", "The address the metric endpoint binds to.")
+    var tlsOpts []func(*tls.Config)
+    flag.StringVar(&metricsAddr, "metrics-bind-address", "0", "The address the metrics endpoint binds to. "+
+        "Use :8443 for HTTPS or :8080 for HTTP, or leave as 0 to disable the metrics service.")
     flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.")
     flag.BoolVar(&enableLeaderElection, "leader-elect", false,
         "Enable leader election for controller manager. "+
             "Enabling this will ensure there is only one active controller manager.")
-    flag.BoolVar(&secureMetrics, "metrics-secure", false,
-        "If set the metrics endpoint is served securely")
+    flag.BoolVar(&secureMetrics, "metrics-secure", true,
+        "If set, the metrics endpoint is served securely via HTTPS. Use --metrics-secure=false to use HTTP instead.")

-    tlsOpts := []func(*tls.Config){}

+    // Metrics endpoint is enabled in 'config/default/kustomization.yaml'. The Metrics options configure the server.
+    // More info:
+    // - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.18.4/pkg/metrics/server
+    // - https://book.kubebuilder.io/reference/metrics.html
+    metricsServerOptions := metricsserver.Options{
+        BindAddress:   metricsAddr,
+        SecureServing: secureMetrics,
+        // TODO(user): TLSOpts is used to allow configuring the TLS config used for the server. If certificates are
+        // not provided, self-signed certificates will be generated by default. This option is not recommended for
+        // production environments as self-signed certificates do not offer the same level of trust and security
+        // as certificates issued by a trusted Certificate Authority (CA). The primary risk is potentially allowing
+        // unauthorized access to sensitive metrics data. Consider replacing with CertDir, CertName, and KeyName
+        // to provide certificates, ensuring the server communicates using trusted and secure certificates.
+        TLSOpts: tlsOpts,
+    }
+
+    if secureMetrics {
+        // FilterProvider is used to protect the metrics endpoint with authn/authz.
+        // These configurations ensure that only authorized users and service accounts
+        // can access the metrics endpoint. The RBAC are configured in 'config/rbac/kustomization.yaml'. More info:
+        // https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.18.4/pkg/metrics/filters#WithAuthenticationAndAuthorization
+        metricsServerOptions.FilterProvider = filters.WithAuthenticationAndAuthorization
+    }
+
     mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
-        Scheme: scheme,
-        Metrics: metricsserver.Options{
-            BindAddress:   metricsAddr,
-            SecureServing: secureMetrics,
-            TLSOpts:       tlsOpts,
-        },
+        Scheme:                 scheme,
+        Metrics:                metricsServerOptions,

  7. [go/v4, helm/v1, ansible/v1] Update your /config/default/kustomization.yaml file with the below changes:

      # [PROMETHEUS] To enable prometheus monitor, uncomment all sections with 'PROMETHEUS'.
  #- ../prometheus
+ # [METRICS] Expose the controller manager metrics service.
+ - metrics_service.yaml
 
+ # Uncomment the patches line if you enable Metrics, and/or are using webhooks and cert-manager
  patches:
- # Protect the /metrics endpoint by putting it behind auth.
- # If you want your controller-manager to expose the /metrics
- # endpoint w/o any authn/z, please comment the following line.
- - path: manager_auth_proxy_patch.yaml
+ # [METRICS] The following patch will enable the metrics endpoint using HTTPS and the port :8443.
+ # More info: https://book.kubebuilder.io/reference/metrics
+ - path: manager_metrics_patch.yaml
+   target:
+     kind: Deployment

  8. [go/v4, helm/v1, ansible/v1] Remove /config/default/manager_auth_proxy_patch.yaml and /config/default/manager_config_patch.yaml files.

  9. [go/v4, helm/v1, ansible/v1] Add /config/default/manager_metrics_patch.yaml file with the below changes:

    # This patch adds the args to allow exposing the metrics endpoint using HTTPS
- op: add
  path: /spec/template/spec/containers/0/args/0
  value: --metrics-bind-address=:8443

  10. [helm/v1, ansible/v1] Update /config/default/manager_metrics_patch.yaml file with the below changes:

    # This patch adds the args to allow securing the metrics endpoint
- op: add
  path: /spec/template/spec/containers/0/args/0
  value: --metrics-secure
# This patch adds the args to allow RBAC-based authn/authz the metrics endpoint
- op: add
  path: /spec/template/spec/containers/0/args/0
  value: --metrics-require-rbac

  11. [go/v4, helm/v1, ansible/v1] Add /config/default/metrics_service.yaml file with the below changes:

    apiVersion: v1
kind: Service
metadata:
  labels:
    control-plane: controller-manager
    app.kubernetes.io/name: <operator-name>
    app.kubernetes.io/managed-by: kustomize
  name: controller-manager-metrics-service
  namespace: system
spec:
  ports:
    - name: https
      port: 8443
      protocol: TCP
      targetPort: 8443
  selector:
    control-plane: controller-manager

  12. [go/v4, helm/v1, ansible/v1] Update your /config/manager/manager.yaml file with the below changes (Note: The port for ansible is 6789):

      - --leader-elect
+ - --health-probe-bind-address=:8081

  13. [go/v4, helm/v1, ansible/v1] Update your /config/prometheus/monitor/yaml file with the below changes:

         - path: /metrics
-      port: https
+      port: https # Ensure this is the name of the port that exposes HTTPS metrics
       tlsConfig:
+        # TODO(user): The option insecureSkipVerify: true is not recommended for production since it disables
+        # certificate verification. This poses a significant security risk by making the system vulnerable to
+        # man-in-the-middle attacks, where an attacker could intercept and manipulate the communication between
+        # Prometheus and the monitored services. This could lead to unauthorized access to sensitive metrics data,
+        # compromising the integrity and confidentiality of the information.
+        # Please use the following options for secure configurations:
+        # caFile: /etc/metrics-certs/ca.crt
+        # certFile: /etc/metrics-certs/tls.crt
+        # keyFile: /etc/metrics-certs/tls.key
         insecureSkipVerify: true

  14. [go/v4, helm/v1, ansible/v1] Remove the following files from /config/rbac

    - auth_proxy_client_clusterrole.yaml
- auth_proxy_role.yaml
- auth_proxy_role_binding.yaml
- auth_proxy_service.yaml

  15. [go/v4, helm/v1, ansible/v1] Update your /config/rbac/kustomization.yaml file with the below changes:

      - leader_election_role_binding.yaml
- # Comment the following 4 lines if you want to disable
- # the auth proxy (https://github.com/brancz/kube-rbac-proxy)
- # which protects your /metrics endpoint.
- - auth_proxy_service.yaml
- - auth_proxy_role.yaml
- - auth_proxy_role_binding.yaml
- - auth_proxy_client_clusterrole.yaml
+ # The following RBAC configurations are used to protect
+ # the metrics endpoint with authn/authz. These configurations
+ # ensure that only authorized users and service accounts
+ # can access the metrics endpoint. Comment the following
+ # permissions if you want to disable this protection.
+ # More info: https://book.kubebuilder.io/reference/metrics.html
+ - metrics_auth_role.yaml
+ - metrics_auth_role_binding.yaml
+ - metrics_reader_role.yaml

  16. [go/v4, helm/v1, ansible/v1] Add /config/rbac/metrics_auth_role.yaml file with the below changes:

    apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: metrics-auth-role
rules:
  - apiGroups:
      - authentication.k8s.io
    resources:
      - tokenreviews
    verbs:
      - create
  - apiGroups:
      - authorization.k8s.io
    resources:
      - subjectaccessreviews
    verbs:
      - create

  17. [go/v4, helm/v1, ansible/v1] Add /config/rbac/metrics_auth_role_binding.yaml file with the below changes:

    apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metrics-auth-rolebinding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: metrics-auth-role
subjects:
  - kind: ServiceAccount
    name: controller-manager
    namespace: system

  18. [go/v4, helm/v1, ansible/v1] Add /config/rbac/metrics_reader_role.yaml file with the below changes:

    apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: metrics-reader
rules:
- nonResourceURLs:
  - "/metrics"
  verbs:
  - get

See #6862 for more details.

Last modified November 19, 2024: Release v1.38.0 (#6866) (0735b20c)