Lately I've been working on an interesting project that's required me to learn how to use Helm to include all files within a directory as entries in a Kubernetes (K8s) configmap - which is not as straight forward as one might think.

What Am I Trying To Do?

Run a container in a K8s cluster whose entire job is to spin up and execute a binary file with a specific configuration file as a parameter.

Note: The binary is set up to use a config file that should be mounted in a specific location (i.e. /etc/config/config.yaml`)

1st Pass - Mount a single config file, contents pasted in-line.

In K8s, we can make files available to a resource by making use of a configMap.

In regular manifests (plain ol' YAML), you can do the following to add a file to a configMap which can then be mounted as a volume within a container.

1. Specify the ConfigMap with the contents of a single config file.

This is actually exactly the example that's specified in the K8s docs for Add ConfigMap data to a Volume

apiVersion: v1
kind: ConfigMap
metadata:
  name: special-config
data:
  config.yaml: |-
    lorem impsum dolor things.
    foo = bang
    things.

2. Update the pod spec to make use of the configmap and mount it to the desired location.

apiVersion: v1
kind: Pod
metadata:
  name: my-test-pod
spec:
  containers:
    - name: test-container
      image: registry.k8s.io/busybox
      command: [ "/bin/sh", "-c", "ls /etc/config/" ]
      volumeMounts:
      - name: config-volume
        mountPath: /etc/config
  volumes:
    - name: config-volume
      configMap:
        name: special-config
  restartPolicy: Never

Note that the volumeMount means that all the keys in the ConfigMap will be mounted as their own files - so in our example, since there is only one element in the data field of the configmap - the file that this will get mounted as is config.yaml within the /etc/config directory.

However, I thought this is kind of tacky to have to expect the contents of the configmap to be updated in-line everytime. It would. be nice if we could decouple this (i.e. if config files could be loaded from elsewhere and be handled differently).

2nd Pass - Mount a single config file - contents inserted from a separate file.

Since it turns out I'm able to make use of Helm (a K8s configuration management and templating tool) for this project, an optimization we did next was simply read in the contents of a file dynamically (for mounting) at deploy-time instead of including the contents in-line. This means as long as a deployer had the required file in the right place, no updates would be required to the configmap's contents.

The magic snippet being: config.yaml: {{ tpl (.Files.Get "<filename>.yaml") . | quote }}

apiVersion: v1
kind: ConfigMap
metadata:
  name: special-config
data:
  config.yaml: {{ tpl (.Files.Get "super_duper_sweet_config.yaml") . | quote }}

Even better!

Sweet, we're done right? Nope. While there might only be a one configuration file today - the implementation needs to support having multiple configuration files available to be fed into the binary at run-time.

Assume that there is now a directory named /config_files in the top level directory that contain special configuration files. All of these need to be present at start-up for the container, of which one can be provided to the binary.

Some Possible Solutions:

1. Modify the app code (binary) to fetch config files during first-run.

This was the first solution that jumped into our minds, but we decided against it because forcing a change on the binary (instead of making changes in the way a container was deployed) is pretty antithetical to the principles of K8s. It would also mean having to ask the engineering-teams to change the way the program runs which might cause more problems to solve this one.

2. Use an initContainer

[Init Containers are] specialized containers that run before app containers in a Pod. Init containers can contain utilities or setup scripts not present in an app image.

Where the setup script could be git clone / download all the config files into a volume mount to be used by the main container(s).

While this was a possiblity, we decided against it in the interest of time - specifically beacuse the person I was pairing with is a Helm master who knew about the proper snippet to use. See below.

3. Use Helm To Include All-Files From a Directory At Deploy-Time.

While my partner was a helmMaster and knew that this could be done - this is the StackOverflow post that confirmed it and helped refine the necessary break-through.

Essentially, what we needed Helm to do was to:

• Range over a list of YAML files in a directory
• Create a new element in the config map per file with the key being the filename and the value (data) being the contents of that file.

# Create a new config map for every Deployment.
apiVersion: v1
kind: ConfigMap
metadata:
  name: special-config
data:
  {{- range $path, $_ :=  .Files.Glob  "config_files/**.yaml" }}    
  {{ $path | trimPrefix "config_files/" }}: |- 
{{ $.Files.Get $path | indent 4 }}
  {{ end }}

Specifically:

{{- range $path, $_ :=  .Files.Glob  "config_files/**.yaml" }}    
  {{ $path | trimPrefix "config_files/" }}: |- 
{{ $.Files.Get $path | indent 4 }}
  {{ end }}

This helm snippet ranges over the config_files directory for all .yaml files and then creates key-value pairs where the key is the name of the file (minus .yaml) and the value is the contents of the file.

i.e. If the directory looked like:

/config_files
---> config_a.yaml
---> config_b.yaml

Then the templated configMap (post-templating) would be:

# Create a new config map for every Deployment.
apiVersion: v1
kind: ConfigMap
metadata:
  name: special-config
data:
  config_a: |-
    <contents>
  config_b: |-
    <contents>

This means, that when used in conjunction with our previous podSpec which had the following line:

     volumeMounts:
      - name: config-volume
        mountPath: /etc/config
  volumes:
    - name: config-volume
      configMap:
        name: special-config

The files will be present as:

• /etc/config/config_a.yaml
• /etc/config/config_b.yaml

Well that's great T, but how are you going to make each container choose a different config file? Stay tuned because that's the next bridge to cross!

Anyways, quick post - but that's how we learned how to use Helm to fetch all files and their contents from a directory and include them in-line. Note that a limitation of this approach is that there's a max limit of 1MB worth of data that can be sent through as as ConfigMap.

A ConfigMap is not designed to hold large chunks of data. The data stored in a ConfigMap cannot exceed 1 MiB. If you need to store settings that are larger than this limit, you may want to consider mounting a volume or use a separate database or file service.