A rocky but encouraging start with Rails credentials
For the biggest part of this week, I’ve been working on upgrading the Rails versions of our managed hosting dashboard application. I started with Rails 6.0, and now I’m at Rails 7.1. Today, I struggled for a long time with deploying to the staging instance on Heroku. At the centre of my issues was that I had started using Rails Encrypted Credentials, which led me to create a new Rails environment staging.
Why is there now a staging Rails environment?
Our staging instance on Heroku used to get deployed with the Rails environment production (more on that later). We inject those configuration details that differ between staging and production instances via environment variables defined on the hosting platform.
The reason that I created a new Rails environment staging for it was that I decided to switch from the outdated secrets.yml file to Rails Encrypted Credentials. The secrets file in our dashboard application pulled in production values from environment variables, but for development and testing, the values were hard-coded in plain sight. (There’s also some level of inconsistency. For most of the credentials, I never used the secrets file; instead, the code references environment variables directly.) The task of migrating from environment variables to Rails credentials meant introducing environment-specific credentials files, which in turn required making staging a proper Rails environment.
What are Rails credentials?
Rails originally introduced credentials in version 5.2 as a safe way to distribute sensitive configuration details with the application code. The approach is simple: Credentials are stored in an encrypted YAML file, by default in config/credentials.yml.enc. There’s also the option of using environment-specific files, such as config/credentials/development.yml.enc. Access to these files obviously requires an encryption key. This key can be stored on disk, by default in config/master.key; this file must, for obvious reasons, not be added to the version control repository. If the key file does not exist locally, Rails will fall back on the value of the environment variable RAILS_MASTER_KEY.
Values stored in these credentials files can be accessed in the application via method calls like Rails.application.credentials.MY_SECRET_NAME.
For a more detailed description, I recommend “The Complete Guide to Ruby on Rails Encrypted Credentials“.
Why switch to Rails credentials, anyway?
Firstly, it’s been the official way to store the secrets of a Rails application for a while now. For that reason, Rubocop and/or RailsBestPractices started to complain about my use of Rails.application.secrets. I tend to follow those recommendations, even (or especially?) when I don’t yet fully understand the reasoning.
What I already do understand is that Rails credentials make it easier to share secrets within a team. Instead of making everyone collect this data from different places such as a password manager into locally stored .env files (or copy them from a colleague hoping that they’re up-to-date), they’re now stored safely in the code repository. The only secret we have to share is the encryption key. And by using individual encryption keys for different environments, I was already able to improve access control. While the encryption keys for the development and test environments are available to the whole engineering team, access to the keys for staging and production is restricted. In the rare occasion that we need to touch those, only team leads will be able to do so.
And should we come to the conclusion that it is too problematic to store those most sensitive encryption keys on disk even just locally, people will be able to use the RAILS_MASTER_KEY environment variable on an on-demand basis instead.
The reason the staging deployment failed
In the CI configuration, I had set RAILS_ENV to staging and RAILS_MASTER_KEY to the staging key. On Heroku, the encryption key for staging was already set up, too, but I had not yet switched RAILS_ENV from production to staging. I knew I had to fix this soon. But I thought it could wait until after deploying the new credentials code for the first time.
What I did not know is that, apparently, the dpl deployment tool we’re using in CI overrides Gitlab variables with Heroku variables. This caused the asset precompilation step to fail every time I tried a new deployment. The obvious reason being that the encryption key did not match the target environment. Finding this out took me an annoyingly large number of hours. Unfortunately, Gitlab CI issues in general are notoriously difficult to debug.
Credentials vs environment variables
When the deployment finally worked, I noticed a build process warning in which Heroku discourages using a Rails environment other than production for deployment. The reason is the risk of configuration drift. It can severely diminish the value of having a staging instance in the first place. Heroku recommends always using production, just with different sets of environment variables. I’m not sure yet where exactly I land between credentials and variables. But what this insight made me do was to reduce environments/staging.rb to a simple require_relative "production". Rails.config settings really should be identical between the two environments.
Conclusion
I’m definitely going to research the advantages and pitfalls of Rails credentials in everyday Rails operations further. Should I make any interesting new findings, I’ll certainly write another article. So if you found this interesting, subscribe to my RSS feed and newsletter! Please also consider following my social media accounts.
