Managing secrets in Kubernetes can be challenging, especially in production environments. Secrets, such as database passwords, API tokens, and encryption keys, are critical for applications but need careful handling to ensure security and compliance. This blog dives into best practices for managing Kubernetes Secrets, highlights modern tools, and explains their benefits with relatable examples.

What is a Kubernetes Secret?

In Kubernetes, a Secret is a resource object used to store sensitive data separate from application code. Rather than hardcoding credentials into container images or pod specifications, Secrets allow you to keep sensitive data secure and organized.

Types of Kubernetes Secrets

Kubernetes provides different types of Secrets, each designed for specific use cases:

1. Opaque: Default type for arbitrary key-value pairs.
2. kubernetes.io/service-account-token: Used to store tokens for service accounts.
3. kubernetes.io/dockerconfigjson: Stores credentials for accessing Docker registries.
4. kubernetes.io/basic-auth: Stores basic authentication credentials (username and password).
5. kubernetes.io/ssh-auth: Stores SSH private keys.
6. kubernetes.io/tls: Stores TLS certificates and private keys.
7. bootstrap.kubernetes.io/token: Used during the bootstrapping process of clusters.

Example:

A Secret can store a database username and password. Instead of embedding this information in your application, you can store it in a Secret and inject it into your pods at runtime as environment variables or mounted files.

apiVersion: v1
kind: Secret
metadata:
  name: my-database-secret
type: Opaque
data:
  username: bXl1c2Vy  # Base64 encoded "myuser"
  password: bXlwYXNzd29yZA==  # Base64 encoded "mypassword"

You can inject this data into a pod as environment variables:

apiVersion: v1
kind: Pod
metadata:
  name: my-app
spec:
  containers:
  - name: app
    image: my-app-image
    env:
    - name: DB_USERNAME
      valueFrom:
        secretKeyRef:
          name: my-database-secret
          key: username
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: my-database-secret
          key: password

However, Kubernetes Secrets are only base64-encoded, not encrypted. This is where additional security measures become essential.

Approaches to Managing Kubernetes Secrets

1. The Manual Way (Not Recommended)

This involves creating and managing secrets manually using kubectl commands or YAML files. While simple for testing, it’s unsuitable for production due to scalability and security risks.

Example:

kubectl create secret generic my-secret --from-literal=username=myuser --from-literal=password=mypassword

Or, using a YAML file:

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
data:
  username: bXl1c2Vy
  password: bXlwYXNzd29yZA==

Why Avoid It?

• Secrets stored in plain text or version control systems are highly vulnerable.
• No built-in automation for rotation or updates.

2. The GitOps Way (Encrypted Secrets)

A step up involves encrypting secrets using tools like Sealed Secrets or SOPS before committing them to Git. These tools ensure that sensitive data remains encrypted in version control and is only decrypted within the Kubernetes cluster.

How It Works:

• Encrypt secrets using CLI tools.
• Commit encrypted secrets to your Git repository.
• Use GitOps tools like ArgoCD to sync and decrypt secrets in your cluster.

Challenges:

• Requires managing encryption keys across clusters and environments.
• Onboarding new team members can be complex due to the encryption workflow.

3. Secrets Operators (The Enterprise Approach)

Secrets operators like External Secrets Operator (ESO) connect Kubernetes with external secret management systems like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault. This approach stores secrets outside Kubernetes, fetching and synchronizing them as native Kubernetes Secrets when needed.

How It Works:

• Deploy the operator in your cluster.
• Configure it to connect with your external secret manager.
• Define custom resources to map external secrets to Kubernetes Secrets.

Example Configuration:

apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
  name: my-external-secret
spec:
  backendType: vault
  data:
    - secretKey: username
      remoteRef:
        key: secret/data/my-secret
        property: username
    - secretKey: password
      remoteRef:
        key: secret/data/my-secret
        property: password

Apply the file:

kubectl apply -f my-external-secret.yaml

Advantages:

• Enhanced security through external storage.
• Centralized secret management across clusters and environments.
• Automated secret rotation and audit logging.

Challenges:

• Initial setup can be complex.
• Some operators lack automatic pod redeployment when secrets change.

4. Kubernetes External Secrets (A Flexible Alternative)

Kubernetes External Secrets offer an efficient way to manage secrets by integrating with external secret management solutions. This allows sensitive data to be stored outside the Kubernetes cluster while still making it accessible to applications running within the cluster.

How Does It Work?

• Kubernetes External Secrets act as a bridge between your cluster and external secret management systems.
• These custom resources fetch and synchronize secrets from external systems, making them available as native Kubernetes Secrets without modifying application code.

Integration with External Systems
Kubernetes External Secrets can integrate with tools like HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and Google Cloud Secret Manager. For instance, to use HashiCorp Vault:

• Deploy the Kubernetes External Secrets controller.
• Configure it with authentication details for Vault.
• Define resources linking Kubernetes Secrets to Vault-stored secrets.

Example Configuration:

apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
  name: vault-external-secret
spec:
  backendType: vault
  data:
    - secretKey: api-key
      remoteRef:
        key: secret/api
        property: key

Advantages:

• Improved security with external encryption and access control.
• Centralized management across Kubernetes clusters.
• Simplified workflows for secret updates and rotation.

Challenges in Kubernetes Secrets Management

1. Lack of Encryption by Default: Secrets are stored in etcd in plain base64 encoding. Without encryption at rest, they are vulnerable if etcd is compromised.
2. Manual Management Overhead: Rotating secrets, updating configurations, and ensuring access controls require significant effort.
3. Scaling Issues: Managing secrets across multiple clusters and environments can be cumbersome.
4. Human Errors: Developers often accidentally expose secrets by storing them in version control or logging them.

Best practices for managing secrets in Kubernetes

To ensure the security and integrity of your sensitive data, it is crucial to follow best practices for secret management in Kubernetes. In this section, we will discuss some of the most important practices to keep your secrets secure and maintain a robust Kubernetes environment.

Role-based access control (RBAC)

RBAC is essential for managing secrets securely, as it enables you to control which users and components can create, read, update, or delete secrets. By implementing fine-grained access control, you can minimize the risk of unauthorized access and potential data breaches.

To implement RBAC for secrets management, you should create roles and role bindings that define the allowed actions on secrets for each user or group. For example, you can create a role that allows read-only access to secrets within a specific namespace and bind it to a specific user or group:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: my-namespace
  name: secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "watch", "list"]

Kubernetes secrets encryption

Encrypting secrets is crucial for protecting sensitive data from unauthorized access, both when stored in etcd (at rest) and when transmitted within the cluster (in transit).

Kubernetes provides native encryption options, such as enabling etcd encryption to protect secrets at rest and using TLS for securing communications within the cluster. Ensure these options are configured and enabled to maintain the confidentiality of your secrets.

In addition to Kubernetes native encryption options, you can also integrate third-party encryption solutions, such as HashiCorp Vault or cloud-based key management services, to further enhance the security of your secrets.

Secret rotation and expiration

Regularly rotating secrets is an essential security practice that minimizes the risk of unauthorized access and potential data breaches.

Strategies for secret rotation include manual updates using kubectl or automated rotation using custom controllers or third-party secret management solutions.

Automating secret rotation can be achieved using Kubernetes operators, external secret management systems, or custom scripts that periodically update secrets based on a predefined schedule or events.

Auditing and monitoring

Auditing and monitoring are crucial for maintaining the security and integrity of your secrets, as they enable you to track and analyze secret access, usage, and modifications and detect potential security incidents.

Several tools can be used for auditing and monitoring secrets, such as Kubernetes audit logs, Prometheus and Grafana.

Configure alerts and notifications to proactively notify administrators of potential security incidents or irregular secret access patterns, enabling timely investigation and response to potential threats.

Wrapping Up

As Kubernetes evolves, secrets management remains a critical aspect of secure deployments. From manual methods to advanced operators, the tools and practices available today offer varying levels of security and convenience. By adopting modern solutions like Kubernetes External Secrets or advanced operators, you can achieve robust secrets management tailored to your needs. The key is finding a balance between security, simplicity, and scalability that empowers your team to focus on building great applications.