Search for a command to run...
A Hacky Helm Play To Keep The Pods At Bay
I'm a developer from Vancouver, BC who's had an interesting journey in tech starting from support, through cloud infrastructure and project management. Currently I work as an SRE at lightstep helping build and "operationalize" things that helps to guide others towards better o11y :)
No comments yet. Be the first to comment.
How a quotable movie from the 90's drove me to build our internal support & success tooling.
How Terraform's setunion() disregards ordering.
How using a table during the planning step could have saved a lot of headache.
A practical guide in using Bash to solve a real-world problem with examples.
This week I've been finishing up a task that involves updating our CI/CD pipeline to make use of a feature available in the newest release of the Open Telemetry Collector to validate configuration files. In doing so, I got a chance to get up close an...
Lately I've been working on an interesting project that's required me to learn how to use Helm to include all files within a directory as entries in a Kubernetes (K8s) configmap - which is not as straight forward as one might think.
Run a container in a K8s cluster whose entire job is to spin up and execute a binary file with a specific configuration file as a parameter.
Note: The binary is set up to use a config file that should be mounted in a specific location (i.e. /etc/config/config.yaml`)
In K8s, we can make files available to a resource by making use of a configMap.
In regular manifests (plain ol' YAML), you can do the following to add a file to a configMap which can then be mounted as a volume within a container.
This is actually exactly the example that's specified in the K8s docs for Add ConfigMap data to a Volume
apiVersion: v1
kind: ConfigMap
metadata:
name: special-config
data:
config.yaml: |-
lorem impsum dolor things.
foo = bang
things.
apiVersion: v1
kind: Pod
metadata:
name: my-test-pod
spec:
containers:
- name: test-container
image: registry.k8s.io/busybox
command: [ "/bin/sh", "-c", "ls /etc/config/" ]
volumeMounts:
- name: config-volume
mountPath: /etc/config
volumes:
- name: config-volume
configMap:
name: special-config
restartPolicy: Never
Note that the volumeMount means that all the keys in the ConfigMap will be mounted as their own files - so in our example, since there is only one element in the data field of the configmap - the file that this will get mounted as is config.yaml within the /etc/config directory.
However, I thought this is kind of tacky to have to expect the contents of the configmap to be updated in-line everytime. It would. be nice if we could decouple this (i.e. if config files could be loaded from elsewhere and be handled differently).
Since it turns out I'm able to make use of Helm (a K8s configuration management and templating tool) for this project, an optimization we did next was simply read in the contents of a file dynamically (for mounting) at deploy-time instead of including the contents in-line. This means as long as a deployer had the required file in the right place, no updates would be required to the configmap's contents.
The magic snippet being: config.yaml: {{ tpl (.Files.Get "<filename>.yaml") . | quote }}
apiVersion: v1
kind: ConfigMap
metadata:
name: special-config
data:
config.yaml: {{ tpl (.Files.Get "super_duper_sweet_config.yaml") . | quote }}
Even better!
Sweet, we're done right? Nope. While there might only be a one configuration file today - the implementation needs to support having multiple configuration files available to be fed into the binary at run-time.
Assume that there is now a directory named /config_files in the top level directory that contain special configuration files. All of these need to be present at start-up for the container, of which one can be provided to the binary.
This was the first solution that jumped into our minds, but we decided against it because forcing a change on the binary (instead of making changes in the way a container was deployed) is pretty antithetical to the principles of K8s. It would also mean having to ask the engineering-teams to change the way the program runs which might cause more problems to solve this one.
initContainer[Init Containers are] specialized containers that run before app containers in a Pod. Init containers can contain utilities or setup scripts not present in an app image.
Where the setup script could be git clone / download all the config files into a volume mount to be used by the main container(s).
While this was a possiblity, we decided against it in the interest of time - specifically beacuse the person I was pairing with is a Helm master who knew about the proper snippet to use. See below.
While my partner was a helmMaster and knew that this could be done - this is the StackOverflow post that confirmed it and helped refine the necessary break-through.
Essentially, what we needed Helm to do was to:
key being the filename and the value (data) being the contents of that file.# Create a new config map for every Deployment.
apiVersion: v1
kind: ConfigMap
metadata:
name: special-config
data:
{{- range $path, $_ := .Files.Glob "config_files/**.yaml" }}
{{ $path | trimPrefix "config_files/" }}: |-
{{ $.Files.Get $path | indent 4 }}
{{ end }}
Specifically:
{{- range $path, $_ := .Files.Glob "config_files/**.yaml" }}
{{ $path | trimPrefix "config_files/" }}: |-
{{ $.Files.Get $path | indent 4 }}
{{ end }}
This helm snippet ranges over the config_files directory for all .yaml files and then creates key-value pairs where the key is the name of the file (minus .yaml) and the value is the contents of the file.
i.e. If the directory looked like:
/config_files
---> config_a.yaml
---> config_b.yaml
Then the templated configMap (post-templating) would be:
# Create a new config map for every Deployment.
apiVersion: v1
kind: ConfigMap
metadata:
name: special-config
data:
config_a: |-
<contents>
config_b: |-
<contents>
This means, that when used in conjunction with our previous podSpec which had the following line:
volumeMounts:
- name: config-volume
mountPath: /etc/config
volumes:
- name: config-volume
configMap:
name: special-config
The files will be present as:
/etc/config/config_a.yaml/etc/config/config_b.yamlWell that's great T, but how are you going to make each container choose a different config file? Stay tuned because that's the next bridge to cross!
Anyways, quick post - but that's how we learned how to use Helm to fetch all files and their contents from a directory and include them in-line. Note that a limitation of this approach is that there's a max limit of 1MB worth of data that can be sent through as as ConfigMap.
A ConfigMap is not designed to hold large chunks of data. The data stored in a ConfigMap cannot exceed 1 MiB. If you need to store settings that are larger than this limit, you may want to consider mounting a volume or use a separate database or file service.