Mapping Kubernetes ConfigMap to Read/Write folders and files
Handling Configurations in Kubernetes is not straight forward, especially if you are dealing with Stateful applications with a broad set of files and folders as configs. Kubernetes itself provided ConfigMap API resource which helps to inject containers with configuration data while keeping containers agnostic of Kubernetes.
However, things are not straightforward if you want to map a folder in your container to a set of configurations or map to multiple folders and files, especially with different permissions and ownerships. Also, Mapping Kubernetes volumes to ConfigMap will delete the files that are already existing in that folder and overwrite everything in it with the mapped ConfigMap folder or file. Also, it will set the config folder or file as read-only to make the configs immutable. The reason why it is immutable is a malicious container running in a pod with a secret, configMap, downwardAPI or projected volume mounted can cause the Kubelet to remove any file or directory on the host filesystem.
If it is just a change of permissions on files, like making a shell script executable we can easily achieve that by
volumeMounts:
- name: cm-test-app-script
mountPath: /dev/folder1/myscript.sh
subPath: myscript.sh
- name: cm-test-app-script
configMap:
name: cm-test-app-script
defaultMode: 0777But what if, you want your config to be with different flexible ownerships and you want to delete or modify the scripts whenever you want due to different reasons? You can overcome this restriction by disabling the feature gate — ReadOnlyAPIDataVolumes. However we don’t want to disable a security feature just because the configs are immutable.
In these situations, we can leverage InitContainers with emptyDir volume. And if you are using Helm package manager, things will be much easier.
What we have to do is to map the configs to the InitContainer. Also, map some emptyDir volumes which shared across both InitContainer and actual container. Then copy it across to respective folder of the emptyDir volume via InitContainer. All those files will be available inside the actual container with all flexible permissions.
Step 1 — Define the configs in values.yaml for helm
Step 2 — Define ConfigMap
All the above configuration files are sitting in an environment specific configuration folder ‘dev’. i.e.
dev/folder1/file1–1.conf
dev/folder1/file1–2.conf
dev/folder1/file1–3.conf
dev/folder1/file1–4.conf
dev/folder2/file2–1.conf
dev/folder2/file2–2.conf
dev/folder2/file2–3.conf
So we have to read all the files from both the folders to define the ConfigMap ‘cm-test-app’. So we loop through the list defined in Values.yaml and read all the files inside dev/folder1/ and dev/folder2/ folder and load it in the ConfigMap. The tpl function will helps us to refer values inside the config files directly.
apiVersion: v1
kind: ConfigMap
metadata:
name: cm-test-app
labels:
app: test-appdata:
{{- range $key, $value := .Values.conf.configmaps.sharedConfigInfo }}
{{- (tpl ($.Files.Glob (printf "conf/%s/%s/*" 'dev' .configInfo.configMapFolder ) ).AsConfig $ ) | nindent 2 }}
{{- end }}
Step 3 — Refer this in your manifest eg;- deployment.yaml
Now we define shared volumes between InitContainer ‘setup-configs’ and the actual container ‘test-app’. We setup ConfigMap ‘config-props’ to the InitContainer and use InitContainer ‘commands’ to copy all the configurations to the required mapped shared volumeMounts which is named as config-emptydir-<<configMapName>>. So when the actual container ‘test-app’ starts up it will have all configurations available ready in its emptyDir volumeMount which is in read write format. The following logic will work for ’n’ number of config folders and files.
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: test-app
spec:
selector:
matchLabels:
app: test-app
template:
metadata:
labels:
app: test-app
spec:
volumes:
- name: config-props
configMap:
name: cm-test-app
{{- range $key, $value := .Values.conf.configmaps.sharedConfigInfo }}
- name: config-emptydir-{{ .configInfo.configMapName }}
emptyDir: {}
{{- end }}
initContainers:
- name: setup-configs
image: busybox:latest
command:
- sh
- "-c"
args:
- " echo 'Setting configmaps'
{{- $tmpFolderName := .Values.conf.configmaps.mainConfigInfo.configMapMount -}}
{{- $tmpSpacer := .Values.conf.configmaps.mainConfigInfo.configMapSpacer -}}
{{- range $key, $value := .Values.conf.configmaps.sharedConfigInfo -}}
{{- $folderName := .configInfo.configMapMount -}}
{{- range $keyfile, $valuefile := .configInfo.filenames -}}
{{ $tmpSpacer }} && cp {{ $tmpFolderName }}{{ . }} {{ $folderName }}/{{ . }}
{{- end -}}
{{- end -}}"
volumeMounts:
- name: config-props
mountPath {{.Values.conf.configmaps.mainConfigInfo.configMapMount}}
{{- range $key, $value := .Values.conf.configmaps.sharedConfigInfo}}
- name: config-emptydir-{{ .configInfo.configMapName }}
mountPath: {{ .configInfo.configMapMount }}
{{- end }}
containers:
- name: test-app
image: "test-app-image:test-app-tag"
volumeMounts:
{{- range $key, $value := .Values.conf.configmaps.sharedConfigInfo}}
- name: config-emptydir-{{ .configInfo.configMapName }}
mountPath: {{ .configInfo.configMapMount }}
{{- end }}Once this is deployed and containers are started we can list the folders and change the ownership of files or even delete the files. You can generate the deployment using helm for validation
---helm 2:
helm template ./test-chart/ -x ./templates/deployment.yaml -f ./test-chart/values.yaml---helm 3:
helm template ./test-chart --show-only templates/deployment.yaml -f ./test-chart/values.yaml --debug
or you can check deployment manifest via kubectl after it is applied. This will looks like the following:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
app: test-app
name: test-app
spec:
selector:
matchLabels:
app: test-app
spec:
containers:
- image: "test-app-image:test-app-tag"
imagePullPolicy: Always
name: test-app
volumeMounts:
- mountPath: /etc/actualfolder/folder1
name: config-emptydir-folder1
- mountPath: /etc/actualfolder/folder2
name: config-emptydir-folder1
initContainers:
- args:
- ' echo ''Setting configmaps'' && cp /tmp/configmapfolder/file1-1.conf /etc/httpd/folder1/file1-1.conf
&& cp /tmp/configmapfolder/file1-2.conf /etc/httpd/folder1/file1-2.conf && cp /tmp/configmapfolder/file1-3.conf /etc/httpd/folder1/file1-3.conf
&& cp /tmp/configmapfolder/file1-4.conf /etc/httpd/folder1/file1-4.conf && cp /tmp/configmapfolder/file2-1.conf /etc/httpd/folder2/file2-1.conf
&& cp /tmp/configmapfolder/file2-2.conf /etc/httpd/folder2/file2-2.conf && cp /tmp/configmapfolder/file2-3.conf /etc/httpd/folder2/file2-3.conf'
command:
- sh
- -c
image: busybox:latest
imagePullPolicy: Always
name: setup-configs
volumeMounts:
- mountPath: /tmp/configmapfolder/
name: config-props
- mountPath: /etc/actualfolder/folder1
name: config-emptydir-folder1
- mountPath: /etc/actualfolder/folder2
name: config-emptydir-folder2
volumes:
- configMap:
name: cm-test-app
name: config-props
- emptyDir: {}
name: config-emptydir-folder1
- emptyDir: {}
name: config-emptydir-folder2Note that we are not mapping the ConfigMap volume to actual container i.e. ‘config-props’. Also we are defining separate volumes to each folder. This is to keep each volumes seperated and not to mix each other. So now if we do ls -lart on one of the the folder say /etc/actualfolder/folder1 by kubectl exec we can see that the files are having normal file permissions.
kubectl exec -it mypod — ls -lart /etc/actualfolder/folder1
If we do a chown we can now change the ownership to anyuser say ‘testuser’ as normal files. This action is not possible if we mount ConfigMap directly.
kubectl exec -it mypod — chown -R testuser:users /etc/actualfolder/folder1 && ls -lart /etc/actualfolder/folder1
So basically we have setup a read/write flexible config using ConfigMap which will help in applications where configs needs to be RW and need to be under different user.
