Out of all the 12 Factor App principles I struggle with Store config in the environment the most and I can see the principle is giving other teams issues as well.
I think the principle over simplifies the subject and ignores how libraries use config. On the flipside however having config as code can cause config pain, resulting in config being stored in multiple places with developers unsure where to amend or append changes.
That’s why I think config management is a good engineering practise that is often overlooked and if not managed correctly can have a negative impact on a teams ability to deliever.
So what are the issues that I see when trying to just use env vars for config?
- Version Control. Config can change the behaviour of an application very easily. By changing env variables directly you have no audit history and if there is an issue in the environment it isn’t obvious to someone diagnosing the issue at first. There is also the issue of rollback which isn’t simple when we are trying to get a one click deployment culture.
- Complexity. Some config is complex and not a simple matter of key value pairs. For example look at your logging config. I once was trying to apply the env principal for a config in Serilog that sinked to a centralised instance of Splunk. The config for this is a json with a depth of 3 nodes so to override it I had a key of Serilog__WriteTo__Args[0]__splunkHost. Trying to make this an env key makes it difficult to read and takes it out of context.
- Different Types of Config. There are different types of config and they should be applied differently. For a simple app there is normally 4 types
- Common Config – Often used by libraries with in the App and enables behaviour you want to see in all deployments. Examples of this could be logging behaviour or in the case of Spring, enabling behvaiour in spring actuator such as health and metric endpoints.
- Environment Specific – Often used when speaking to middleware. This is the database or the message broker you connect to.
- Secrets – Your passwords, secrets, etc should be stored in a secure place and not be checked in with your code.
- Runtime variables – There are some config that needs to be passed into your startup process, for example JVM parameters.
- Portability. When you adopt Cloud services such as Azure or AWS you tend to go all in with their CI/CD and store this data in their platform. However some industries are being made to mitigate the risk of going all in with one provider so may deploy to Azure and AWS. Some teams are on a journey and are migrating from VMs or Tin to the cloud. Storing Config as Code makes this migration easier and manageable
The Hierarchy of Config
So we have established there are different types of config and they should be stored differently but how do apply them to our application? Not all config is equal and some should be able to override the behaviour of others.
- Common Config as our baseline. This can give a lot of context around what the config is for (espcially if it’s yaml or json) and we can store local development properties in here.
- Environment Config. This is environment specific such as the name of servers, topics, databases etc. We overlay the common with this. For example we may have different logging behaviour locally that we store in common that is changed when deployed to an environment.
- Secrets. This overwrites anything in config. If someone accidently checked in their secrets for a local instance then this can overwrite it.
- Runtime. When we talk about runtime we often think JVM Heap tuning. However runtime should be able to overwride common and environment config when trying to diagnose issues or patching a fix that will later be promoted to config. If the common and environment config is shared between services (something which I don’t condone but understand it can be a necessary evil) then runtime can be used for specificity such as a service name.
This concept is supported by different libraries
ASP.NET Core Configuration Fundementals