Skip to content

Instantly share code, notes, and snippets.

Show Gist options
  • Save brianjbayer/9c7782232327287005b8065697c1041a to your computer and use it in GitHub Desktop.
Save brianjbayer/9c7782232327287005b8065697c1041a to your computer and use it in GitHub Desktop.
A comprehensive examination of Rails Secrets, Credentials, and Secret Key Base

Unlocking the Secrets of Rails Secrets and Credentials

Interurban Tunnel at Blackhand Gorge- Brian J Bayer


If you are like many (most?) of us, you have encountered Rails Credentials/Secrets and secret_key_base and may have been left a bit (or more) confused.

This post is an attempt to remove some of that confusion by taking a comprehensive look at Rails Secrets and Credentials.

This post covers Rails Secrets and Rails Credentials, how they are different, and their history, as well as the different Rails secret_key_bases. Finally, it presents an approach for setting the secret_key_base used by Rails without using Rails Secrets or Credentials files.

🚒 Captain Obvious says that the general term "secrets" is used to describe those configuration values that are sensitive like usernames, passwords, and API keys

⏱️This post is current as of and based upon Rails 7.0.4.2


Rails Secrets and Credentials Are Two Different Things

One of the first areas of potential confusion is that Rails Credentials and Secrets are two different and separate things that can be interconnected in the case of the Rails secret_key_base (which will be covered later in this post).


History of Rails Secrets and Credentials

To better understand why Rails Credentials and Secrets both exist, consider their history and the need for some backward compatibility.

  • Rails 4.1 (2Q2014) introduced the automatic generation and use of the file config/secrets.yml which contained the application's secret_key_base by default. The secrets in this file were made accessible through Rails.application.secrets (e.g. Rails.application.secrets.secret_key_base). This file was not encrypted and should not be checked into the source code repository or made public.

  • Rails 5.1 (2Q2017) introduced Encrypted Secrets (based on the sekrets gem). Running bin/rails secrets:setup created the encrypted secrets file config/secrets.yml.enc and the encryption key file config/secrets.yml.key (instead of "always going [to] the RAILS_MASTER_KEY"). This is also when the requirement for the specification of a master key was added in order to start the Rails server. The encrypted config/secrets.yml.enc file could now be checked into the source code repository, but the plain text config/secrets.yml.key file should not be. For more information, see the PR.

  • Rails 5.2 (2Q2018) introduced (encrypted) Credentials and the automatically generated files config/credentials.yml.enc for the encrypted credentials and config/master.key for the encryption key. Per the release notes, "[t]his will eventually replace Rails.application.secrets and the encrypted secrets introduced in Rails 5.1". Like the similar config/secrets.yml.enc, the encrypted config/credentials.yml.enc could be checked into the source code repository (but not the unencrypted config/master.key). For more information, see the PR.

  • Rails 6.0 (3Q2019) introduced support for Multi-Environment Credentials with environment-specific files config/credentials/<rails-env>.yml.enc for the encrypted credentials and config/credentials/<rails-env>.key for the encryption key (e.g. config/credentials/production.yml.enc and config/credentials/production.key). Also introduced were the configuration settings config.credentials.content_path and config.credentials.key_path for overriding the default locations of the credentials and key files. For more information, see the PR.


Rails Secrets

🚫 Rails Secrets are deprecated as of Rails 5.2 and are only supported for backwards compatibility. If you want Rails to manage your sensitive configuration, you should use Rails Credentials.

Rails Secrets were introduced in Rails 4.1 and Encrypted Secrets were introduced in Rails 5.1. As of Rails 7.0.4.2 both are still (somewhat) supported.

Secrets Files and Environment Variable

  • Unencrypted secrets use the file config/secrets.yml to store the secrets

    development:
    secret_key_base: 49d3f3de9ed86c74b94ad6bd0...
    my_custom_secret: some-value
    
    test:
    secret_key_base: 707db01ac0f0d50d62c14a3ce...
    my_custom_secret: some-value
    
    production:
    secret_key_base: 4795c8bffcdf8e54b327f117d...
    my_custom_secret: some-value

    Note that the same file is used to define the secrets for all environments and that you can define your own application-specific secrets in it.

    Since it is not encrypted, this file should never be checked into a source code repository or made public.

  • Encrypted secrets use the two files...

    1. config/secrets.yml.enc to store the encrypted secrets

      Note that like config/secrets.yml, this same file is used to define the secrets for all environments and that you can define your own application-specific secrets in it.

      Since this secrets file is encrypted, you could check it into a source code repository.

    2. config/secrets.yml.key to store the unencrypted (i.e. plain text) key used to encrypt/decrypt the encrypted secrets in config/secrets.yml.enc

      Since this key file is not encrypted, this file should never be checked into a source code repository or made public.

    The environment variable RAILS_MASTER_KEY can also be used to specify the encryption key instead of the config/secrets.yml.key file.

Encrypted Secrets Configuration

Encrypted secrets are not active by default in Rails 5.1. To use encrypted secrets in Rails 5.1, add the following configuration to your application...

config.read_encrypted_secrets = true

Creating and Editing Encrypted Secrets

To create your encrypted secrets in Rails 5.1, run the following command...

bundle exec bin/rails secrets:setup

This will generate the two files config/secrets.yml.enc and config/secrets.yml.key.

The generated config/secrets.yml.enc will be empty, so you must edit it to add your secrets.

πŸ‘‰ Note that with the Rails 5.2 introduction of encrypted credentials the bin/rails secrets:setup command is deprecated and can no longer be used. It returns the message Encrypted secrets is deprecated in favor of credentials. Run:rails credentials:help

In order to edit your encrypted secrets you must have...

  1. The key used to create the encrypted secrets file either in file config/secrets.yml.key or in the environment variable RAILS_MASTER_KEY
  2. An editor set in the system environment variable EDITOR (e.g. EDITOR=vim)

To edit your encrypted secrets file config/secrets.yml.enc, run the following command...

bundle exec bin/rails secrets:edit

Merged Secrets

You can use both unencrypted secrets in config/secrets.yml and encrypted secrets in config/secrets.yml.enc and Rails will merge them together during initialization.

Accessing Secrets

Secrets can be accessed inside of your application (or in the Rails console) at...

Rails.application.secrets

For example, Rails.application.secrets.secret_key_base

Rails.application.secrets contains the merged secrets from config/secrets.yml and config/secrets.yml.enc.

References and More Information

For more information on Rails Secrets...


Rails Credentials

Rails Credentials were introduced in Rails 5.2 and multi-environment credentials were introduced in Rails 6.

Rails Credentials are the preferred and fully-supported mechanism if you want Rails to manage your sensitive configuration.

Rails Credentials are always encrypted and are generated automatically when creating a rails project with rails new.

Credentials Files and Environment Variable

Like Rails Encrypted Secrets, Rails Credentials use two types of files...

  1. The encrypted credential file(s):

    • config/credentials.yml.enc which is the generated default shared across all rails environments

      # aws:
      #   access_key_id: 123
      #   secret_access_key: 345
      
      # Used as the base secret for all MessageVerifiers in Rails, including the one protecting cookies.
      secret_key_base: 14b6462b8dc4366e1266baab0...
    • config/credentials/<rails-env>.yml.enc which are for Multi-Environment Credentials and are specific to a given Rails environment (e.g. config/credentials/production.yml.enc)

    Like Rails Secrets, you can define your own application-specific credentials in this file type and check this file type into your source code repository.

    πŸ‘‰ Note that Embedded RuBy (ERB) is not supported in the Rails encrypted credential files

  2. The encryption key file(s) containing the unencrypted (i.e. plain text) encryption key(s) used to encrypt (and decrypt) the credential file(s):

    • config/master.key which is the generated default shared across all rails environments

    • config/credentials/<rails-env>.key which are for multi-environment credentials and are specific to a given Rails environment (e.g. config/credentials/production.key)

      Like Rails Encrypted Secrets, this key file is not encrypted and should never be checked into a source code repository or made public.

Like Rails Encrypted Secrets, the environment variable RAILS_MASTER_KEY can also be used to specify the encryption key instead of the config/master.key file.

πŸ‘‰ Note that the RAILS_MASTER_KEY environment variable takes precedence over the credentials key file

Credentials Configuration

Encrypted credentials are active and generated by default in Rails versions 5.2 and later and no additional configuration is needed.

However, for Rails version 6.0 and later, you can configure the locations of the credential and key files...

  • config.credentials.content_path specifies the location of the encrypted credentials file (e.g. config.credentials.content_path = config/devtest.credentials.yml.enc)

  • config.credentials.key_path specifies the location of the credential's encryption key file (e.g. config.credentials.key_path = config/devtest.key)

You can require the use of a master key with the configuration...

config.require_master_key = true

This master key requirement is the default in production and does not seem to be able to be disabled.

Creating and Editing Credentials (Lost Key)

As mentioned in Rails versions 5.2 and later, the Rails credential files config/credentials.yml.enc and config/master.key are created automatically by rails new.

The rails credentials:edit command is used to both edit existing credentials and create new ones. When editing existing credentials, you must supply the encryption key either in the appropriate key file or with the RAILS_MASTER_KEY environment variable.

Handling Lost Encryption Keys

If you lose your encryption key, you can regenerate your credentials. However, you will have to delete your encrypted credential file and readd your application-specific credentials.

To regenerate your credential files...

  1. Delete config/master.key if it exists
    rm -f config/master.key
    
  2. Delete config/credentials.yml.enc
    rm -f config/credentials.yml.enc
    
  3. Ensure that the EDITOR system environment variable is set (e.g. EDITOR="code --wait") or specify it on the command line
  4. Run the rails credentials:edit command to generate the new credential files
    bundle exec bin/rails credentials:edit
    
    or to specify the editor to use (for example vim) on the command line
     EDITOR=vim bundle exec bin/rails credentials:edit
    

πŸ‘€ For more information on the rails credentials:edit command run the command...

bundle exec bin/rails credentials:edit -h

Multi-Environment Credentials

To create and edit credentials for a specific environment use the rails credentials:edit command with the --environment option to specify the desired environment.

Again, you will need to either have the EDITOR system environment variable already set or specify it on the command line.

For example to generate credentials for the production environment...

EDITOR=vim bundle exec bin/rails credentials:edit --environment production

This will generate the two credential files config/credentials/production.yml.enc and config/credentials/production.key.

🀷 Per Pull Request 47107 it appears that the Rails Credentials commands (e.g. rails credentials:edit) use the config.credentials.content_path and config.credentials.key_path configuration for the specified environment

Credentials Precedence

Multi-environment credentials take precedence over the default credentials and there is no "fall back" or merging.

If you want to use Multi-environment credentials and use the default credential files for an environment, you will need to use the config.credentials.content_path and config.credentials.key_path configuration to specify them for that environment. For example in config/environments/development.rb

Rails.application.configure do
...

  config.credentials.content_path = 'config/credentials.yml.enc'
  config.credentials.key_path = 'config/master.key'
...

Accessing Credentials

Credentials can be accessed inside of your application (or in the Rails console) at...

Rails.application.credentials

For example, Rails.application.credentials.secret_key_base

To display the contents of your Rails Credentials file(s), use the rails credentials:show command...

bundle exec bin/rails credentials:show

References and More Information

For more information on Rails Credentials...


Rails Secret Key Base

There is always at least one sensitive configuration key and value which is automatically generated by Rails when it generates Secrets or Credentials. This sensitive configuration is the Rails secret_key_base.

According to the Ruby on Rails API, the Rails secret_key_base "is used as the input secret to the application's key generator, which in turn is used to create all ActiveSupport::MessageVerifier and ActiveSupport::MessageEncryptor instances, including the ones that sign and encrypt cookies."

So Many secret_key_bases

What can be very confusing is that there are five different secret_key_bases:

  1. There is the Rails.application.secret_key_base method which is the one used by Rails for its key generator (ActiveSupport::MessageVerifier and ActiveSupport::MessageEncryptor instances)

  2. There is a secret_key_base attribute for Rails Credentials (i.e. Rails.application.credentials.secret_key_base)

  3. There is a secret_key_base attribute for Rails Secrets (i.e. Rails.application.secrets.secret_key_base)

  4. There is the Rails.application.config.secret_key_base configuration

  5. Finally, there is the Rails environment variable SECRET_KEY_BASE (i.e. ENV['SECRET_KEY_BASE'])

Some of these secret_key_bases are interconnected and some are completely separate.

The following analysis of these different secret_key_bases is based upon inspecting the Rails source code and actual structured exploratory testing as of Rails 7.0.4.2 and "edge" (pre Rails 7.1) circa 1Q2023.

Rails.application.secret_key_base Method

Rails.application.secret_key_base is the method actually used by Rails itself for its encryption of things like cookies. It is described in the Ruby on Rails API.

Generally, this is the one that you should use in your application. It is also the most complicated and its value depends directly on the...

  • Rails environment
  • Value in the automatically generated temporary file tmp/development_secret.txt
  • Value in ENV['SECRET_KEY_BASE']
  • Value in Rails.application.credentials.secret_key_base
  • Value in Rails.application.secrets.secret_key_base

Given its direct dependencies, the value of Rails.application.secret_key_base also indirectly depends on the value of the Rails.application.config.secret_key_base configuration.

Specifically and in order of precedence, the value of Rails.application.secret_key_base is determined as follows...

  • In the Rails development or test environments, Rails.application.secret_key_base returns...

    1. The value of Rails.application.secrets.secret_key_base if it is set for that environment in the file config/secrets.yml.*, for example...
      development:
        secret_key_base: from-secrets-yml-development
    2. Otherwise, the value in the temporary file tmp/development_secret.txt and if that file does not exist, Rails automatically creates it with a value

    🀦 As a side-effect of initialization, if there is not a value for the environment in config/secrets.yml.*, Rails sets Rails.application.secrets.secret_key_base to the value in tmp/development_secret.txt.

    πŸ‘‰ Note that the Rails.application.secret_key_base method does not use Rails Credential at all in the development or test environments.

  • In any other Rails environment other than development or test (e.g production), Rails.application.secret_key_base returns...

    1. The value in the Rails SECRET_KEY_BASE environment variable (i.e. ENV['SECRET_KEY_BASE']) if it is set
    2. Otherwise, the value in Rails.application.credentials.secret_key_base if set from the Rails Credentials files (with any multi-environment credential file taking precedence)
    3. Otherwise, the value in Rails.application.secrets.secret_key_base if set from the Rails Secrets files

    πŸ’₯ If none of these sources have a value, then Rails raises the following error...

    Missing `secret_key_base` for 'production' environment, set this string with `bin/rails credentials:edit` (ArgumentError)
    

Rails.application.credentials.secret_key_base Attribute

The Rails.application.credentials.secret_key_base attribute is set from the Rails Credentials files described previously in this post.

Specifically and in order of precedence, the value of Rails.application.credentials.secret_key_base is determined as follows...

  1. If configured, from the encrypted credentials file specified in config.credentials.content_path
  2. Otherwise, from the encrypted credentials file config/credentials/<rails-env>.yml.enc if it exists
  3. Otherwise, from the default encrypted credentials file config/credentials.yml.enc if it exists
  4. Otherwise it is set to nil

The key used to decrypt the value for the Rails.application.credentials.secret_key_base just described is determined as follows in order of precedence...

  1. From the value in the Rails RAILS_MASTER_KEY environment variable (i.e. ENV['RAILS_MASTER_KEY']) if set
  2. Otherwise, if configured, from the key file specified in config.credentials.key_path
  3. Otherwise, from the key file config/credentials/<rails_env>.key if it exists
  4. Otherwise, from the default credentials key file config/master.key if it exists

πŸ’₯ If using Rails Credentials for the value of the Rails.application.secret_key_base method, you must have a master key value otherwise Rails will raise a MissingKeyError. You may encounter this issue in the Rails production environment during Asset Precompile.

Rails.application.secrets.secret_key_base Attribute

🀷 Since I was using Rails 7.0.4.2 which no longer supports creating Rails Encrypted Secrets, I was unable to test specific behavior associated with them versus unencrypted secrets

Unlike the Rails.application.credentials.secret_key_base attribute, the Rails.application.secrets.secret_key_base attribute is not set from just the Rails Secrets files. Thus, Rails.application.secrets.secret_key_base is more complex and configurable.

As mentioned previously in this section, in the Rails development and test environments, the value of the Rails.application.secrets.secret_key_base is interconnected with the value of the Rails.application.secret_key_base method.

Specifically and in order of precedence, the value of Rails.application.secrets.secret_key_base is determined as follows...

  • In the Rails development or test environments...

    1. If present, from the environment configuration in the secrets file(s) config/secrets.yml.*, for example...
      development:
        secret_key_base: from-secrets-yml-development
    2. Otherwise, if present, from the value specified in Rails.application.config.secret_key_base configuration
    3. Otherwise, from the value in the Rails-generated temporary file tmp/development_secret.txt
  • In any other Rails environment other than development or test (e.g production)...

    1. If present, from the environment configuration in the secrets file(s) config/secrets.yml.*
    2. Otherwise, if present, from the value specified in Rails.application.config.secret_key_base configuration
    3. Otherwise it is set to nil

πŸ‘‰ Recall that with Rails Secrets files, the encrypted secrets file takes precedence and is merged with the unencrypted secrets file

Rails.application.config.secret_key_base Configuration

The Rails.application.config.secret_key_base configuration is simply a setting in the Rails application configuration.

πŸ‘€ For more information, see Configuring Rails Applications in the Rails Guide

SECRET_KEY_BASE Environment Variable

This is simply the Rails-specific environment variable for specifying a secret_key_base value referenced inside of a Rails application by ENV['SECRET_KEY_BASE'].


How to NOT Use Rails Secrets or Credentials

You may have one or more good reasons for not using Rails Secrets or Credentials for your application's sensitive configuration, for example...

  • Your application's sensitive configuration is managed by another team and/or secrets manager such as Hashicorp Vault, AWS Secrets Manager, Azure Key Vault, or Kubernetes Secrets

  • Your applications follows the twelve-factor methodology as described in the Twelve-Factor App where configuration is stored in environment variables

  • You want less complexity and similar behavior across all Rails environments

While you have complete control over your application-specific "secrets", you will have to accommodate and address the Rails.application.secret_key_base method in order to even start your application (i.e. Rails server) in the Rails production environment and/or precompile assets.

Here is an approach for addressing Rails.application.secret_key_base consistently in all Rails environments without using or needing Rails Secrets or Credentials. Here you will use the Rails SECRET_KEY_BASE environment variable to specify the value.

  1. Remove any custom config.credentials.content_path and config.credentials.key_path configuration being sure to note the custom locations of your credential files
  2. Delete all Rails Credentials files, for example...
    rm -rf config/credentials*
    rm config/master.key
    
  3. Delete all Rails Secrets files
    rm config/secrets.yml*
    
  4. Configure your Rails application for all environments to set Rails.application.config.secret_key_base from ENV['SECRET_KEY_BASE'], for example in file config/application.rb
    class Application < Rails::Application
    ...
       # Do not use Rails Secrets or Credentials for secret_key_base
       # Set the secret_key_base used by Rails key generators from ENV
       config.secret_key_base = ENV.fetch('SECRET_KEY_BASE')
       ...
    end
    Here you are using ENV.fetch without a default value or block so that an obvious error occurs if the SECRET_KEY_BASE environment variable is not set.

Here are some Rails console sessions, showing the different secret_key_bases' values using this approach and how they are the same for all environments...

  • For the Rails development (and test) environment...

    $ RAILS_ENV=development SECRET_KEY_BASE='from-env-var'  bundle exec bin/rails c
    Loading development environment (Rails 7.0.4.2)
    >> Rails.application.secret_key_base
    => "from-env-var"
    >> Rails.application.credentials.secret_key_base
    => nil
    >> Rails.application.secrets.secret_key_base
    => "from-env-var"
    >> Rails.application.config.secret_key_base
    => "from-env-var"
    >> ENV['SECRET_KEY_BASE']
    => "from-env-var"
  • For the Rails production environment...

    $ RAILS_ENV=production SECRET_KEY_BASE='from-env-var'  bundle exec bin/rails c
    Loading production environment (Rails 7.0.4.2)
    >> Rails.application.secret_key_base
    => "from-env-var"
    >> Rails.application.credentials.secret_key_base
    => nil
    >> Rails.application.secrets.secret_key_base
    => "from-env-var"
    >> Rails.application.config.secret_key_base
    => "from-env-var"
    >> ENV['SECRET_KEY_BASE']
    => "from-env-var"

Generating an Encryption Key

Finally, if you need to generate an encryption key like or for secret_key_base, you can use the rails secret command...

bundle exec bin/rails secret

🀦 Unfortunately, it appears through testing that the rails secret command requires secret_key_base. So if you are using the approach presented in How to NOT Use Rails Secrets or Credentials, you will need to supply a valid dummy value for the SECRET_KEY_BASE environment variable.

Here is a janky workaround if you want to dynamically generate an encryption key for the SECRET_KEY_BASE environment variable on the command line...

export SECRET_KEY_BASE="a-chicken-to-lay-the-first-egg"; SECRET_KEY_BASE=$(bundle exec bin/rails secret) bundle exec bin/rails c

@andreslucena
Copy link

This brings a lot of light in all the details of this API. Thanks for the outstanding explanation!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment