brianjbayer/gist-unlocking-the-secrets-of-rails-secrets-and-credentials.md

## gist-unlocking-the-secrets-of-rails-secrets-and-credentials.md

      
    Raw
  

              gist-unlocking-the-secrets-of-rails-secrets-and-credentials.md
            
          
    Unlocking the Secrets of Rails Secrets and Credentials


If you are like many (most?) of us, you have encountered Rails
Credentials/Secrets and secret_key_base and may have been
left a bit (or more) confused.
This post is an attempt to remove some of that confusion by
taking a comprehensive look at Rails Secrets and Credentials.
This post covers Rails Secrets and Rails Credentials, how
they are different, and their history, as well as the different
Rails secret_key_bases.  Finally, it presents an approach
for setting the secret_key_base used by Rails without using
Rails Secrets or Credentials files.

🚢 Captain Obvious says that the general term "secrets"
is used to describe those configuration values that are
sensitive like usernames, passwords, and API keys


⏱️This post is current as of and based upon Rails 7.0.4.2


Rails Secrets and Credentials Are Two Different Things

One of the first areas of potential confusion is that Rails
Credentials and Secrets are two different and separate things
that can be interconnected in the case of the Rails
secret_key_base (which will be covered later in this post).

History of Rails Secrets and Credentials

To better understand why Rails Credentials and Secrets both
exist, consider their history and the need for some backward
compatibility.


Rails 4.1
(2Q2014) introduced the automatic generation and use of the
file config/secrets.yml which contained the application's
secret_key_base by default.  The secrets in this file
were made accessible through Rails.application.secrets
(e.g. Rails.application.secrets.secret_key_base).
This file was not encrypted and should not be
checked into the source code repository or made public.


Rails 5.1
(2Q2017) introduced Encrypted Secrets (based on the
sekrets gem).  Running bin/rails secrets:setup created
the encrypted secrets file config/secrets.yml.enc and the
encryption key file config/secrets.yml.key (instead of
"always going [to] the RAILS_MASTER_KEY").  This is also
when the requirement for the specification of a master key
was added in order to start the Rails server.  The encrypted
config/secrets.yml.enc file could now be checked into the
source code repository, but the plain text
config/secrets.yml.key file should not be.  For more
information, see the
PR.


Rails 5.2
(2Q2018) introduced (encrypted) Credentials and the
automatically generated files config/credentials.yml.enc
for the encrypted credentials and config/master.key for
the encryption key. Per the release notes, "[t]his will
eventually replace Rails.application.secrets and the
encrypted secrets introduced in Rails 5.1".  Like the
similar config/secrets.yml.enc, the encrypted
config/credentials.yml.enc could be checked into the
source code repository (but not the unencrypted
config/master.key).  For more information, see the
PR.


Rails 6.0
(3Q2019) introduced support for Multi-Environment
Credentials with environment-specific files
config/credentials/<rails-env>.yml.enc for the encrypted
credentials and config/credentials/<rails-env>.key for
the encryption key (e.g.
config/credentials/production.yml.enc and
config/credentials/production.key). Also introduced
were the configuration settings
config.credentials.content_path and
config.credentials.key_path for overriding the default
locations of the credentials and key files. For more
information, see the
PR.


Rails Secrets


🚫 Rails Secrets are deprecated as of Rails 5.2
and are only supported for backwards compatibility.  If you
want Rails to manage your sensitive configuration, you
should use Rails Credentials.

Rails Secrets were introduced in Rails 4.1 and Encrypted
Secrets were introduced in Rails 5.1.  As of Rails 7.0.4.2
both are still (somewhat) supported.
Secrets Files and Environment Variable


Unencrypted secrets use the file config/secrets.yml
to store the secrets
development:
secret_key_base: 49d3f3de9ed86c74b94ad6bd0...
my_custom_secret: some-value

test:
secret_key_base: 707db01ac0f0d50d62c14a3ce...
my_custom_secret: some-value

production:
secret_key_base: 4795c8bffcdf8e54b327f117d...
my_custom_secret: some-value
Note that the same file is used to define the secrets
for all environments and that you can define your own
application-specific secrets in it.
Since it is not encrypted, this file should never be
checked into a source code repository or made public.


Encrypted secrets use the two files...


config/secrets.yml.enc to store the encrypted secrets
Note that like config/secrets.yml, this same file
is used to define the secrets for all environments
and that you can define your own application-specific
secrets in it.
Since this secrets file is encrypted, you could check
it into a source code repository.


config/secrets.yml.key to store the unencrypted
(i.e. plain text) key used to encrypt/decrypt the
encrypted secrets in config/secrets.yml.enc
Since this key file is not encrypted, this file should
never be checked into a source code repository or made
public.


The environment variable RAILS_MASTER_KEY can also be
used to specify the encryption key instead of the
config/secrets.yml.key file.


Encrypted Secrets Configuration

Encrypted secrets are not active by default in Rails 5.1.
To use encrypted secrets in Rails 5.1, add the following
configuration to your application...
config.read_encrypted_secrets = true
Creating and Editing Encrypted Secrets

To create your encrypted secrets in Rails 5.1, run the
following command...
bundle exec bin/rails secrets:setup

This will generate the two files config/secrets.yml.enc and
config/secrets.yml.key.
The generated config/secrets.yml.enc will be empty, so you
must edit it to add your secrets.

👉 Note that with the Rails 5.2 introduction of
encrypted credentials the bin/rails secrets:setup command
is deprecated and can no longer be used.  It returns the
message Encrypted secrets is deprecated in favor of credentials. Run:rails credentials:help

In order to edit your encrypted secrets you must have...

The key used to create the encrypted secrets file either
in file config/secrets.yml.key or in the environment
variable RAILS_MASTER_KEY
An editor set in the system environment variable
EDITOR (e.g. EDITOR=vim)

To edit your encrypted secrets file config/secrets.yml.enc,
run the following command...
bundle exec bin/rails secrets:edit

Merged Secrets

You can use both unencrypted secrets in config/secrets.yml
and encrypted secrets in config/secrets.yml.enc and Rails
will merge them together during initialization.
Accessing Secrets

Secrets can be accessed inside of your application (or in the
Rails console) at...
Rails.application.secrets
For example, Rails.application.secrets.secret_key_base
Rails.application.secrets contains the merged secrets from
config/secrets.yml and config/secrets.yml.enc.
References and More Information

For more information on Rails Secrets...

Source Code
Rails 4.1 with Manual Generation of Secrets Release Notes
Encrypted Secrets Pull Request
Rails 5.1 with Encrypted Secrets Release Notes
Engine Yard article on Encrypted Rails Secrets on Rails 5.1


Rails Credentials

Rails Credentials were introduced in Rails 5.2 and
multi-environment credentials were introduced in Rails 6.
Rails Credentials are the preferred and fully-supported
mechanism if you want Rails to manage your sensitive
configuration.
Rails Credentials are always encrypted and are generated
automatically when creating a rails project with rails new.
Credentials Files and Environment Variable

Like Rails Encrypted Secrets, Rails Credentials use two types
of files...


The encrypted credential file(s):


config/credentials.yml.enc which is the generated
default shared across all rails environments
# aws:
#   access_key_id: 123
#   secret_access_key: 345

# Used as the base secret for all MessageVerifiers in Rails, including the one protecting cookies.
secret_key_base: 14b6462b8dc4366e1266baab0...


config/credentials/<rails-env>.yml.enc which are
for Multi-Environment Credentials and are specific
to a given Rails environment (e.g.
config/credentials/production.yml.enc)


Like Rails Secrets, you can define your own
application-specific credentials in this file type and
check this file type into your source code repository.

👉 Note that Embedded RuBy (ERB) is not
supported in the Rails encrypted credential files


The encryption key file(s) containing the unencrypted
(i.e. plain text) encryption key(s) used to encrypt (and
decrypt) the credential file(s):


config/master.key which is the generated default
shared across all rails environments


config/credentials/<rails-env>.key which are
for multi-environment credentials and are specific
to a given Rails environment (e.g.
config/credentials/production.key)
Like Rails Encrypted Secrets, this key file is not
encrypted and should never be checked into a source code
repository or made public.


Like Rails Encrypted Secrets, the environment variable
RAILS_MASTER_KEY can also be used to specify the
encryption key instead of the config/master.key file.

👉 Note that the RAILS_MASTER_KEY environment
variable takes precedence over the credentials key file

Credentials Configuration

Encrypted credentials are active and generated by default
in Rails versions 5.2 and later and no additional
configuration is needed.
However, for Rails version 6.0 and later, you can configure
the locations of the credential and key files...


config.credentials.content_path specifies the location of
the encrypted credentials file (e.g.
config.credentials.content_path = config/devtest.credentials.yml.enc)


config.credentials.key_path specifies the location of
the credential's encryption key file (e.g.
config.credentials.key_path = config/devtest.key)


You can require the use of a master key with the
configuration...
config.require_master_key = true
This master key requirement is the default in production and
does not seem to be able to be disabled.
Creating and Editing Credentials (Lost Key)

As mentioned in Rails versions 5.2 and later, the Rails
credential files config/credentials.yml.enc and
config/master.key are created automatically by rails new.
The rails credentials:edit command is used to both
edit existing credentials and create new ones.  When
editing existing credentials, you must supply the encryption
key either in the appropriate key file or with the
RAILS_MASTER_KEY environment variable.
Handling Lost Encryption Keys

If you lose your encryption key, you can regenerate your
credentials.  However, you will have to delete your encrypted
credential file and readd your application-specific
credentials.
To regenerate your credential files...

Delete config/master.key if it exists
rm -f config/master.key


Delete config/credentials.yml.enc
rm -f config/credentials.yml.enc


Ensure that the EDITOR system environment variable is
set (e.g. EDITOR="code --wait") or specify it on the
command line
Run the rails credentials:edit command to generate the
new credential files
bundle exec bin/rails credentials:edit

or to specify the editor to use (for example vim)
on the command line
 EDITOR=vim bundle exec bin/rails credentials:edit


👀 For more information on the rails credentials:edit
command run the command...
bundle exec bin/rails credentials:edit -h


Multi-Environment Credentials

To create and edit credentials for a specific environment
use the  rails credentials:edit command with the
--environment option to specify the desired environment.
Again, you will need to either have the EDITOR system
environment variable already set or specify it on the command
line.
For example to generate credentials for the production
environment...
EDITOR=vim bundle exec bin/rails credentials:edit --environment production

This will generate the two credential files
config/credentials/production.yml.enc and
config/credentials/production.key.

🤷 Per Pull Request 47107
it appears that the Rails Credentials commands
(e.g. rails credentials:edit) use the
config.credentials.content_path and
config.credentials.key_path configuration for the
specified environment

Credentials Precedence

Multi-environment credentials take precedence over the
default credentials and there is no "fall back" or merging.
If you want to use Multi-environment credentials and use the
default credential files for an environment, you will need
to use the config.credentials.content_path and
config.credentials.key_path configuration to specify them
for that environment.  For example in
config/environments/development.rb
Rails.application.configure do
...

  config.credentials.content_path = 'config/credentials.yml.enc'
  config.credentials.key_path = 'config/master.key'
...
Accessing Credentials

Credentials can be accessed inside of your application (or in the
Rails console) at...
Rails.application.credentials
For example, Rails.application.credentials.secret_key_base
To display the contents of your Rails Credentials file(s),
use the rails credentials:show command...
bundle exec bin/rails credentials:show

References and More Information

For more information on Rails Credentials...

Source Code
Rails Credentials Pull Request
Rails 5.2 with Rails Credentials Release Notes
Rails Multi-Environment Credentials Pull Request
Rails 6.0 with Multi-Environment Credentials Release Notes
Rails Guide on Securing Rails Applications
Big Binary article on Rails 6 adds support for Multi Environment credentials
Ruby on Rails discussion on
The status of simultaneous support for secrets, encrypted credentials and multi-environment encrypted credentials is confusing


Rails Secret Key Base

There is always at least one sensitive configuration key and
value which is automatically generated by Rails when it
generates Secrets or Credentials.  This sensitive
configuration is the Rails secret_key_base.
According to the
Ruby on Rails API,
the Rails secret_key_base "is used as the input secret to
the application's key generator, which in turn is used to
create all ActiveSupport::MessageVerifier and
ActiveSupport::MessageEncryptor instances, including the
ones that sign and encrypt cookies."
So Many secret_key_bases

What can be very confusing is that there are five
different secret_key_bases:


There is the Rails.application.secret_key_base method
which is the one used by Rails for its key
generator (ActiveSupport::MessageVerifier and
ActiveSupport::MessageEncryptor instances)


There is a secret_key_base attribute for Rails Credentials
(i.e. Rails.application.credentials.secret_key_base)


There is a secret_key_base attribute for Rails Secrets
(i.e. Rails.application.secrets.secret_key_base)


There is the Rails.application.config.secret_key_base
configuration


Finally, there is the Rails environment variable
SECRET_KEY_BASE (i.e. ENV['SECRET_KEY_BASE'])


Some of these secret_key_bases are interconnected and some
are completely separate.
The following analysis of these different secret_key_bases
is based upon inspecting the
Rails source code
and actual structured exploratory testing as of Rails 7.0.4.2
and "edge" (pre Rails 7.1) circa 1Q2023.
Rails.application.secret_key_base Method

Rails.application.secret_key_base is the method actually
used by Rails itself for its encryption of things like
cookies.  It is described in the
Ruby on Rails API.
Generally, this is the one that you should use in your
application.  It is also the most complicated and its value
depends directly on the...

Rails environment
Value in the automatically generated temporary file
tmp/development_secret.txt
Value in ENV['SECRET_KEY_BASE']
Value in Rails.application.credentials.secret_key_base
Value in Rails.application.secrets.secret_key_base

Given its direct dependencies, the value of
Rails.application.secret_key_base also indirectly depends
on the value of the Rails.application.config.secret_key_base
configuration.
Specifically and in order of precedence, the value of
Rails.application.secret_key_base is determined as
follows...


In the Rails development or test environments,
Rails.application.secret_key_base returns...

The value of Rails.application.secrets.secret_key_base
if it is set for that environment in the file
config/secrets.yml.*, for example...
development:
  secret_key_base: from-secrets-yml-development

Otherwise, the value in the temporary file
tmp/development_secret.txt and if that file does not
exist, Rails automatically creates it with a value

🤦 As a side-effect of initialization, if there is
not a value for the environment in config/secrets.yml.*,
Rails sets Rails.application.secrets.secret_key_base to
the value in tmp/development_secret.txt.
👉 Note that the
Rails.application.secret_key_base method does not use
Rails Credential at all in the development or test
environments.


In any other Rails environment other than development or
test (e.g production), Rails.application.secret_key_base
returns...

The value in the Rails SECRET_KEY_BASE environment
variable (i.e. ENV['SECRET_KEY_BASE']) if it is
set
Otherwise, the value in
Rails.application.credentials.secret_key_base if
set from the Rails Credentials files (with any
multi-environment credential file taking precedence)
Otherwise, the value in
Rails.application.secrets.secret_key_base if set
from the Rails Secrets files

💥 If none of these sources have a value, then Rails
raises the following error...
Missing `secret_key_base` for 'production' environment, set this string with `bin/rails credentials:edit` (ArgumentError)


Rails.application.credentials.secret_key_base Attribute

The Rails.application.credentials.secret_key_base attribute
is set from the Rails Credentials files described previously
in this post.
Specifically and in order of precedence, the value of
Rails.application.credentials.secret_key_base is determined
as follows...

If configured, from the encrypted credentials file specified
in config.credentials.content_path
Otherwise, from the encrypted credentials file
config/credentials/<rails-env>.yml.enc if it exists
Otherwise, from the default encrypted credentials file
config/credentials.yml.enc if it exists
Otherwise it is set to nil

The key used to decrypt the value for the
Rails.application.credentials.secret_key_base just described
is determined as follows in order of precedence...

From the value in the Rails RAILS_MASTER_KEY environment
variable (i.e. ENV['RAILS_MASTER_KEY']) if set
Otherwise, if configured, from the key file specified in
config.credentials.key_path
Otherwise, from the key file
config/credentials/<rails_env>.key if it exists
Otherwise, from the default credentials key file
config/master.key if it exists

💥 If using Rails Credentials for the value of
the Rails.application.secret_key_base method, you must
have a master key value otherwise Rails will raise a
MissingKeyError.  You may encounter this issue in the
Rails production environment during Asset Precompile.
Rails.application.secrets.secret_key_base Attribute


🤷 Since I was using Rails 7.0.4.2 which no longer
supports creating Rails Encrypted Secrets, I was unable
to test specific behavior associated with them versus
unencrypted secrets

Unlike the Rails.application.credentials.secret_key_base
attribute, the Rails.application.secrets.secret_key_base
attribute is not set from just the Rails Secrets files.
Thus, Rails.application.secrets.secret_key_base is more
complex and configurable.
As mentioned previously in this section, in the Rails
development and test environments, the value of
the Rails.application.secrets.secret_key_base is
interconnected with the value of the
Rails.application.secret_key_base method.
Specifically and in order of precedence, the value of
Rails.application.secrets.secret_key_base is determined as
follows...


In the Rails development or test environments...

If present, from the environment configuration in the
secrets file(s) config/secrets.yml.*, for example...
development:
  secret_key_base: from-secrets-yml-development

Otherwise, if present, from the value specified in
Rails.application.config.secret_key_base configuration
Otherwise, from the value in the Rails-generated
temporary file tmp/development_secret.txt


In any other Rails environment other than development or
test (e.g production)...

If present, from the environment configuration in the
secrets file(s) config/secrets.yml.*
Otherwise, if present, from the value specified in
Rails.application.config.secret_key_base configuration
Otherwise it is set to nil


👉 Recall that with Rails Secrets files, the
encrypted secrets file takes precedence and is merged with
the unencrypted secrets file

Rails.application.config.secret_key_base Configuration

The Rails.application.config.secret_key_base configuration
is simply a setting in the Rails application configuration.

👀 For more information, see
Configuring Rails Applications
in the Rails Guide

SECRET_KEY_BASE Environment Variable

This is simply the Rails-specific environment variable
for specifying a secret_key_base value referenced
inside of a Rails application by ENV['SECRET_KEY_BASE'].

How to NOT Use Rails Secrets or Credentials

You may have one or more good reasons for not using Rails
Secrets or Credentials for your application's sensitive
configuration, for example...


Your application's sensitive configuration is managed by
another team and/or secrets manager such as Hashicorp Vault,
AWS Secrets Manager, Azure Key Vault, or Kubernetes Secrets


Your applications follows the twelve-factor methodology as
described in the Twelve-Factor App
where configuration is stored in environment variables


You want less complexity and similar behavior across all
Rails environments


While you have complete control over your application-specific
"secrets", you will have to accommodate and address the
Rails.application.secret_key_base method in order to even
start your application (i.e. Rails server) in the Rails
production environment and/or precompile assets.
Here is an approach for addressing
Rails.application.secret_key_base consistently in all
Rails environments without using or needing Rails Secrets
or Credentials.  Here you will use the Rails SECRET_KEY_BASE
environment variable to specify the value.

Remove any custom config.credentials.content_path and
config.credentials.key_path configuration being sure
to note the custom locations of your credential files
Delete all Rails Credentials files, for example...
rm -rf config/credentials*
rm config/master.key


Delete all Rails Secrets files
rm config/secrets.yml*


Configure your Rails application for all environments to
set Rails.application.config.secret_key_base from
ENV['SECRET_KEY_BASE'], for example in
file config/application.rb
class Application < Rails::Application
...
   # Do not use Rails Secrets or Credentials for secret_key_base
   # Set the secret_key_base used by Rails key generators from ENV
   config.secret_key_base = ENV.fetch('SECRET_KEY_BASE')
   ...
end
Here you are using ENV.fetch without a default value or
block so that an obvious error occurs if the
SECRET_KEY_BASE environment variable is not set.

Here are some Rails console sessions, showing the different
secret_key_bases' values using this approach and how they
are the same for all environments...


For the Rails development (and test) environment...
$ RAILS_ENV=development SECRET_KEY_BASE='from-env-var'  bundle exec bin/rails c
Loading development environment (Rails 7.0.4.2)
>> Rails.application.secret_key_base
=> "from-env-var"
>> Rails.application.credentials.secret_key_base
=> nil
>> Rails.application.secrets.secret_key_base
=> "from-env-var"
>> Rails.application.config.secret_key_base
=> "from-env-var"
>> ENV['SECRET_KEY_BASE']
=> "from-env-var"


For the Rails production environment...
$ RAILS_ENV=production SECRET_KEY_BASE='from-env-var'  bundle exec bin/rails c
Loading production environment (Rails 7.0.4.2)
>> Rails.application.secret_key_base
=> "from-env-var"
>> Rails.application.credentials.secret_key_base
=> nil
>> Rails.application.secrets.secret_key_base
=> "from-env-var"
>> Rails.application.config.secret_key_base
=> "from-env-var"
>> ENV['SECRET_KEY_BASE']
=> "from-env-var"


Generating an Encryption Key

Finally, if you need to generate an encryption key like or for
secret_key_base, you can use the rails secret command...
bundle exec bin/rails secret

🤦 Unfortunately, it appears through testing that the
rails secret command requires secret_key_base.  So if you
are using the approach presented in
How to NOT Use Rails Secrets or Credentials, you will need
to supply a valid dummy value for the SECRET_KEY_BASE
environment variable.
Here is a janky workaround if you want to dynamically generate
an encryption key for the SECRET_KEY_BASE environment
variable on the command line...
export SECRET_KEY_BASE="a-chicken-to-lay-the-first-egg"; SECRET_KEY_BASE=$(bundle exec bin/rails secret) bundle exec bin/rails c