We are a software development company and we have a problem. We develop and maintain a number of systems, each composed of multiple applications. Each application has multiple configuration files, and each configuration file has environment and instance overrides. For cloud deployments with auto scaling, we either go with a configuration table or server. For the rest of our applications, we like the simplicity of configuration files. They work well with small applications, but as the application grows, managing configuration files become a problem.
The following are our pain points:
Multiple Configurations. We have multiple systems for different clients, each composed of multiple applications. It’s not uncommon for one application to have multiple configuration files. Clients also prefer very configurable applications to minimize code changes after project completion. And don’t forget that fact that there are many configuration formats out there, depending on the programmer, library or language.
Multiple Environments. Usually we have local, Dev, QA and Prod environments. Larger setups have an additional Training environment. QA and Prod will be load balanced, so they belong in a cluster with multiple instances. Some Prod set ups are sharded. Have you ever tried adding a new configuration to all these environments?
Configuration in Artifact. What is easier than a configuration file? That will be constants. Some clients prefer a very configurable application, like SQL in configuration files. So it’s a balancing act between simplicity and configurability. When things do go into configuration files, they should not be part of the build artifact. If not, how can you make them vary per environment?
Version Control. Configuration files are version controlled like source code. In reality, it’s not that simple. Local configuration is not typically source controlled. The more environments and instances, the more likely only the base copy is source controlled. There is a one to many relationship between code and configuration, so storing and retrieving configuration in version control is not as straightforward as doing it for source code. Configuration should be separate from source code, and highly configurable applications should have separate code and configuration version numbers. In case you store configuration files with source code, how is that working out with your continuous integration and deployment? What about cases where configuration files are edited directly on the server and is now out of sync with source control?
Too Technical. Configuration files are meant to be easy to change, but in reality, it’s too technical for people who are not programmers or system administrators. We usually get called to make simple changes to configuration files. It’s great as we can bill this time, but it shouldn’t be this way. And this is when we have access to the environment in question. We need VPN, SSH, RDP, etc., after taking the required training classes of course. Some clients don’t give us Prod access, which means we have to work closely with system administrators to make this configuration change for locked down environments. Some companies use automation tools like Ansible, but since we are on the topic of Too Technical, let’s not go there.
Change Review. With a big team working on multiple applications, it’s sometimes hard to keep track of all the changes done to configuration files. We need to keep track of changes to make sure our local environment are up to date, but more importantly, to make sure we have the right configuration for Prod deployment or rollback. In locked down environments, system administrators need to review or make the changes themselves. It’s not unusual for us not to know what’s in the Prod configuration files. For example, we don’t know the database details for Prod.
Runtime Errors. Have you ever made configuration changes that broke Prod? It could have been a simple typo in vi or Notepad. One way of testing configuration changes is running some read-only unit test against the configuration, assuming you have the necessary permissions. Another way is triple checking changes or asking a peer to review. Would be nice to have a configuration compiler or lint, to validate configuration and detect errors early.
Security. Configuration files can contain sensitive information like passwords and keys. It’s important that sensitive information is encrypted in some way, especially since this is data at rest. Some people don’t bother, and we are among the guilty. It adds a layer of complexity since the data needs to be encrypted in the configuration and decrypted in the code. Manually encrypting data for configuration files is a process in itself. Decrypting them for troubleshooting purposes doesn’t make life easier as well.
Documentation. Like source code, configuration needs to be documented. Like applications, configuration needs technical documentation. Easier said than done. Not a lot of thought is normally given to configuration documentation. If they are, they are either in a separate document that can easily get out of sync, or can be part of the configuration file itself, where multiple copies become a maintenance problem. And by the way, some configuration files don’t allow comments. Many popular open source applications have very detailed configuration documentation as part of the configuration file. This is great to start, but can get overwhelming. And we are only talking about documentation for the configuration file contents. What about configuration file locations? And what about searching for configuration across all environments, or doing a diff between environments?
Configuration Server. Some, not all, of our pain points can be solved be using a configuration server. ZooKeeper quickly comes to mind, but we are fond of Redis. The main problem is this no longer is an application concern but a company concern. This approach needs buy-in, system administration and code changes. It’s not just one server as we don’t want a single point of failure. So it’s between our lowly configuration file versus a fancy cluster of configuration servers. There must be a better way.
So we have listed all the issues we have managing our configuration files. We tried Google and Stack Exchange but found no solution. We did come across partial solutions. Some of the solutions solve only parts of the problem. Some are library or platform dependent. So what will a team of software developers to do?
We now have a product we want to build. A solution to the problem of how to best manage configuration files. Just like our previous projects, this will be generic enough so others can benefit. But why do we want to build it? We didn’t want to build it for the money. We already have a couple of projects that we released for free to the public so this can be no different. But there’s a better approach. We have been giving a few scholarships to less fortunate kids who wants to go to school but don’t have the means to do so. It’s great news whenever you hear that these kids now have good careers, helping their families improve their quality of life. So we decided to build Config, monetize it, and create a continuing educational scholarship program for less fortunate kids. Can we do it? I bet we can.