If you're working on any software projects that require talking to other services, chances are that you probably have to make use of secrets. The most common type of secrets you may have run into -- are passwords and API keys.

There are others as well, and Cyberark does a good job of defining a secret as a:

[...] private piece of information that acts as a key to unlock protected resources or sensitive information in tools, applications, containers, DevOps and cloud-native environments.

Some of the most common types of secrets include:

Privileged account credentials

Passwords

Certificates

SSH keys

API keys

Encryption keys

As you may have learned, perhaps the hard way through a security incident, secrets can't be treated like just any other type of data. Because of their ability to access resources; they must be handled with care and with security in mind.

🧐 How Do You Store Secrets During Development?

If you've googled this question, you may have run into a couple of articles that suggest the following options:

🐚 Provide Secrets as Shell Variables.

The assumption is that your code (i.e. run-server.sh) is set up to look for a key as a shell variable. For example:

cat run-server.sh
#! /bin/bash
echo "OUTPUT: API Key is ${API_KEY}"

In this approach, you can provide the secret to your program at run-time by running: API_KEY=<my-api-key> run-server.sh which provides the following output:

API_KEY=my-api-key ./run-server.sh                                                                                                      
OUTPUT: API Key is my-api-key

❗️The drawback of this approach is that your key has now been included in plaintext, within your shell history. If an attacker were to ever compromise your machine, the secrets would be available by running a simple history command.

1815  API_KEY=my-api-key ./run-server.sh

🌲 Provide Secrets as Environment Variables

For this approach, we do a bit better setting the secret as an environment variable, which our program can read from.

Setting the actual environment variable can be done in a number of ways, such as reading from a file on the system. (i.e. api-key-secret.txt)

export API_KEY=$(cat api-key-secret.txt)
./run-server.sh
OUTPUT: API Key is my-env-var-api-key

While the secret is no longer printed out to shell history ✅ , the secret is contained within a file that must now be secured appropriately ❗️.
You need to ensure that the file containing the secret key isn't acidentally checked into Verison Control (i.e. Github)❗️.

📁 Use A .env File

Some programming languages recommend the pattern of using a dotenv file, where programs can be instructed to look up values from a specific file (i.e. development.env) However, this approach has the same drawback as above.

🤔 What's A Better Way To Store Secrets

A decent number of moons ago when I worked as a Security Engineer doing ✨ security ✨ things, one of the items in our portfolio was a Secret Management system named Conjur (coincidentally, also built by Cyberark).

As an enterprise-grade secret management solution, it had a lot of features such as audit logging and access control policies, but one of the coolest features was the ability to do dynamic secret retrieval.

Specifically, it meant that after engineers checked secrets in to Conjur, they could simply (1) refer to their secrets by their secret reference paths and (2) use the Conjur CLI to "wrap" the invocation of their programs, to pull those secrets at run-time.

I remember thinking that this was super cool, and wished I could do that for personal projects with my existing secret manager (1PW). Unfortunately, the main disqualifier for that at the time, outside of licensing costs, was the requirement to run and secure your own secrets server which wasn't really feasible for local development.

Well, fast forward about 5 years and things have now changed!

🤨 What's A Better Way To Store Secrets in Development?

For me, the "development phase" in personal projects is the phase where I want to rapidly prototype my ideas. I don't want to get bogged down by tedious tasks, which is what secret management used to be. I could move fast by using env or shell variables, but I was only a single .gitignore file mistake away from checking in the contents to version control.

I'd always hoped for a better way and thankfully a couple of weeks ago - I came across this fantastic blog post by 1Password when looking into how to access secrets programmatically.

https://blog.1password.com/1password-cli-2_0/

This blog post goes into using the 1Password CLI to essentially achieve what I wanted to do those years ago. It enables developers to:

Add secrets to their password vault
Get the references to those secrets
Fetch secrets dynamically at runtime.

How does it work?

Download & install the 1Password CLI + 1Pasword8
Add your secret(s) into 1Password. In my case my secret value for my-api-key is: secret-key

Grab the references for the secrets you're interested in.

op item get my-api-key --format json                                                                                                  
{
  "id": "<<REDACTED>>",
  "title": "my-api-key",
  "version": 1,
  "vault": {
    "id": "<<REDACTED>>",
    "name": "Blog-post-development"
  },
  "category": "API_CREDENTIAL",
  "last_edited_by": "[...]",
  "created_at": "2023-02-26T23:14:43Z",
  "updated_at": "2023-02-26T23:14:43Z",
  "fields": [
    [...]
    {
      "id": "credential",
      "type": "CONCEALED",
      "label": "credential",
      "value": "secret-key",
      "reference": "op://Blog-post-development/my-api-key/credential"
    },

Which in this case is: op://Blog-post-development/my-api-key/credential

Include the secret reference and wrap your execution command with op run

Before, I used to have a Makefile to run my server:

cat Makefile                                                                                                                            
run:
    ./run-server.sh

Now I simply wrap the run target with op run and my secret reference as follows:

cat Makefile                                                                                                                            
run:
    API_KEY="op://Blog-post-development/my-api-key/credential" \
    op run ./run-server.sh

When I execute with make run will first:

Prompt me for a fingerprint (via touchID on Mac, or master password on all others); and

Retrieve the secret for use in the command.

make run                                                                                                                                
API_KEY="op://Blog-post-development/my-api-key/credential" \
    op run ./run-server.sh
OUTPUT: API Key is <concealed by 1Password>

Note: that 1PW's CLI is smart / safe enough to conceal the password. If you want to override this option and display the secret, you can run the same command with the --no-masking flag.

For example:

cat Makefile                                                                                                                            
run:
    API_KEY="op://Blog-post-development/my-api-key/credential" \
    op run --no-masking ./run-server.sh

make run                                                                                                                                
API_KEY="op://Blog-post-development/my-api-key/credential" \
    op run --no-masking ./run-server.sh
OUTPUT: API Key is secret-key

Why is this better?

Secrets are stored and retrieved from a single place. ✅ The single source of truth for your secret's value is in 1Password. This means that if you need to read this secret from multiple places, you only need to update it once. It also means that if your secret gets "popped", you only need to change or "rotate" it in one place.
Uses a password or biometric to authenticate access. ✅ If you're using a mac, you'll be prompted for TouchID. Else, you'll be prompted for your master password anywhere else.
Secrets are not stored in code. ✅ You don't have to worry about accidentally checking in secrets to version control (i.e. Github).
You don't have to worry about the security of where your secrets are stored. ✅That's managed by an organization that specializes in secret storage and has a vested interest security of your secrets (1PW).
You don't need to do any other lift ✅ to get this working. No need to set up IAM with AWS or GCP, uses 1Password.

Further Extensibility

1Password can group secrets into vaults which means you can also extend this to your colleagues & partners. When you add a secret to a vault and add them as members to your vault, they can also use secrets in the same manner from their 1password accounts.
1Password also has other shell-plugins to securely authenticate with services right from your shell.
There's also a way to integrate CI/CD as well.

Conclusion

Overall, I am extremely excited to see that these sorts of features are slowly trickling down and becoming more accessible for more usecases. The more we do to make security "easier", the better that software becomes for all.

🔐 How-To Securely Work With Secrets During Development

Comments

More from this blog

From Conversations to Code: "I'm Good at Interacting with People, Can't You See That?"

A misadventure with Terraform Sets & PagerDuty Schedules

Common CI Pipeline Considerations: Ordering and Caching

Lessons Learned - Sharpening my Bash Skills

Understanding Helm Templates and Utilizing YQ for YAML Parsing Mastery

🧐 How Do You Store Secrets During Development?

🐚 Provide Secrets as Shell Variables.

🌲 Provide Secrets as Environment Variables

📁 Use A `.env` File

🤔 What's A Better Way To Store Secrets

🤨 What's A Better Way To Store Secrets in Development?

How does it work?

Why is this better?

Further Extensibility

Conclusion

Command Palette

Comments

More from this blog

🧐 How Do You Store Secrets During Development?

🐚 Provide Secrets as Shell Variables.

🌲 Provide Secrets as Environment Variables

📁 Use A .env File

🤔 What's A Better Way To Store Secrets

🤨 What's A Better Way To Store Secrets in Development?

How does it work?

Why is this better?

Further Extensibility

Conclusion

📁 Use A `.env` File