> Another approach to config is the use of config files which are not checked into revision control, such as config/database.yml in Rails. This is a huge improvement over using constants which are checked into the code repo, but still has weaknesses: it’s easy to mistakenly check in a config file to the repo; there is a tendency for config files to be scattered about in different places and different formats, making it hard to see and manage all the config in one place. Further, these formats tend to be language- or framework-specific.

> The twelve-factor app stores config in environment variables (often shortened to env vars or env).

yeah the reason they argue for this is that the people that wrote this worked on Heroku, and the way that Heroku worked is you populated the environment variables from some fields in a web app. If you want your config history tracked in version control, or you do gitops, or you have k8s configmaps, or you want to have your configuration files on a mounted volume ... those things are all broadly fine; they keep the configuration state separate from the app deploy state. This document really confuses the forest with the trees and recommends things based less on actual engineering principles and more on the product capabilities of the corporation that produced it. It is an actively harmful set of guidelines.

If you want secure config storage on Kubernetes, you end up using Secrets, which ends up being key-value like env vars anyway. If you want similar security with Git you need a layer of encryption, which breaks diffs and requires additional tooling. This all leads back to why high-level deployment tools like Heroku were created.

You can load the secret file directly into the app, no need to load it as env vars or keep it strictly as key-value pairs.

> If you want similar security with Git you need a layer of encryption, which breaks diffs and requires additional tooling. This all leads back to why high-level deployment tools like Heroku were created.

You can use tools like ejson[1] or sops[2] to get encrypted files checked into Git that have key level granularity on diffs.

There is definitely a place for higher level abstractions than Kubernetes. Mostly it gives operators a standard platform to build from when teams outgrow the PaaS sandbox.

[1] https://github.com/Shopify/ejson [2] https://github.com/getsops/sops

Now you're getting away from the spirit of 12factor and hard-coupling them again. The intent is for the app to consume the secrets but have no knowledge or care where they came from.

Edit: misread as "load the secrets directly into the app." Yeah, this is just env vars but different.

1. create a Secret that defines the file as a key

2. mount that Secret into the container (.secret.secretName in the volume definition) under an arbitrary conventional path, e.g. /secrets

3. define an env-var that gives the full path to the file; where the fallback when this env-var isn't defined is to find the file in the working directory with a conventional filename.

If your own code uses the file, it can read the path from this env-var; while if the core of your container is some opaque tool that doesn't know about the env-var but does take the file's path on the command-line, then you can wrap the tool in an entrypoint script that reads the env-var and passes it as a command-line arg to the tool.

IMHO, this approach is flexible but principle-of-least-surprise obeying for both development and production deployment.

A .env file allows you to go from knowing/assuming where a particular file is, to having environment variables.

What I'm describing is that you can go from being passed environment variables directly, to knowing where certain files are; which therefore allows those files to be mounted into a container-image in arbitrary places, rather than encoding brittle assumptions about config-file locations into the binary within the container-image — assumptions that usually get in the way during development, when you're not running the binary inside a container.

We use a custom setup in which the config values are loaded from multiple sources actually, so we could put everything into secrets manager, or load them all from env vars or HOCON files.

If we had to set every config value in ECS / lambda using env vars, it would be a major pain in the ass, and error prone.

And how your apps authenticate to Secrets Manager ? Did you ever call `env` on a pod that has IRSA configured ?

This is just a middle step to do exactly the same thing but instead of using envFromSecret you use envFromSM

That if you dont want to pull in k8s dependencies to the code, at the end every approach ends up as a mounted file or env vars.

The entire point here is that 12-factor isn't the be-all, end-all, so this is irrelevant.

> The intent is for the app to consume the secrets but have no knowledge or care where they came from.

This doesn't really make sense, and is an impossible requirement. The app will always have to know where configuration or secrets come from. Environment variables is just one option of "knowing where they come from". Choosing file storage is just as valid.

The Twelve-Factor App is marketing content produced by the former Founder & CTO of Heroku. It's a great piece of content no doubt, but it should be viewed with that lens.

Not all marketing is that straightforward.

Doesn't break diffs, doesn’t directly couple anything.

I'm fine reading a few passwords and tokens from a secure key-value store. But the rest of my config can be a bit more expressive.

- Logs as a stream: yep, just write your logs to STDOUT instead of a file. Let the orchestrator deal with reading and storing it.

- Configuration in the environment. Apps tend to pull config from different sources depending on the deploy - maybe a .env locally and a secret store in production.

- Port binding - your app listens on a port and you put nginx or whatever in front of it and set up a reverse proxy. K8S services and ingress do that.

I think the biggest criticism you can lay down on 12 Factor these days is that it's not actually written very well and assumes you know exactly what it is talking about.

This is one that always feels odd to me: what happened to rsyslogd?

The normal output of a non-service command (eg. grep) sometimes/often goes to STDOUT, which is why that command will write errors to STDERR -- even "info" messages that look like logging.

Apache for my personal projects, in front of the docker containers.

If you remember the world before 12 factor you know that even if motivated by self serving incentives, Heroku making this mainstream has been an overall win for the entire industry and we are generally better and more secure for them having done so. We are still benefitting from all of this today, and Heroku who championed it is making death rattles.

Here is some obvious case observed many times.

A small and very fast pace team builds a server application having limited to no external dependencies. No db even.

Just a server doing stuff.

It does have some config yes, many many key/value pairs even, a config file ended up with a dozen properties just to control Req per sec limits and such.

Now just to deploy the thing, since of course env defined configurations is where things belong in best practice, the dev team setup a process to peer review those env values and despite all of that, fail to get the app running.

Human errors happen but since more mistakes and release delays were caused by misconfiguration than any other reasons, the keen and bright enginnering team all agreed it would be better to have a config repo. One genius in the group argued that having a config service would be even neater than have the app read from a git repo! The whole got so hyped and of course spend the whole next sprint building and testing that. Brilliant, it worked!

Except that of course outages happened, of course. Outages happen.

Bringing down all environments one by one, hence blocking pretty much everyone. Everything went down because hey, since we made that effort which took for longer than we thought to build a config service and have our app reads from it, one of the dev minds thought it was a genius idea to have the app reload the config every 5 mins at runtime so that if a configuration change was made, no need to even restart anything. Of course error handling wasn't robust and well tested since the boss reminded the team we've got to ship a product by the end of the quarter and solving the universe can maybe wait 5 or 10 years after the IPO.

Team resorted to simply track the config in the repo. Kept the runtime reload of the config and things worked just fine.

Things can be done right and impacts could have been avoided. The example may seem like a twisted case since these were all configurations, not environment specific values. But I've just seen so many occurences of env variables being separated from the application repo costing so much and distracting contributors for no benefit that I have grown very cautious of it.

If you store the database hostname/credentials, as an example, in the "code" (either literally in the app code, or in a config.ini that lives in the repo and gets baked into a release), then the ops people can't quickly repoint the app at a hot-spare database during an incident: they'll need to edit the repo, push it, cut a new release, etc. If there is a long build step (as for a C++ app, for example), then this isn't really tenable during an outage. If this is in an environment variable, their normal deployment scripts can make the change for them.

I agree that this is probably not necessary anymore if your team is all wearing both hats (dev and ops), and its just as quick to make an edit to the repo and cut/deploy a new release as it would have been for the Ops team to run their deployment script in the old days.

Even when ops are perfectly comfortable with git, they are rarely keen on touching dev repos even if deployment times were tenable. If code changes need peer reviewed, how so configuration values don't get treated at least the same way?

Secrets in clear are obviously a no go, I feel though that encrypted secrets in repos has become reasonably simple with mature tooling in this area. Ansible's fast growth got cut short once people started to hear k8s at every meet ups perhaps.

Can I ask what the modern solution for this is?

Also:

> If code changes need peer reviewed, how so configuration values don't get treated at least the same way?

The easy answer to this is a CODEOWNERS file, with the app code being owned by the devs, and the config being owned by ops. But again, not sure how to do this for secrets without their being plaintext in the repo. Do any code review tools support encrypted content?

This way, the same source deployed in, i.e. your AKS cluster in Azure EU north will take the configuration you set for that cluster while the one you run out of an Docker Swarm of RPi Zero's in an Ikea photo frame (I have one cluster like that, called Gibson), will take the configuration from that cluster.

> The twelve-factor app stores config in environment variables (often shortened to env vars or env).

I think that's fine. 12-factor is a set of guidelines about what you do in production. There's a whole universe of things that are smart ideas for development and bad ideas for production. Leaving debug symbols in your binaries, leaving ports for connecting a debugger open on your containers, etc.

You just set the variables at startup in prod, and in the file on development (probably mostly because you want to change them all the time).

There's an enormous difference between "I'm going to use a dev-friendly way to manage and swap out all the different configurations that I might want to plug into the app while I'm testing it, but still plug them in using the same mechanism", which is basically what you get with dotenv files, and, "I'm going to configure with environment variables in production and JSON files in development," which is another thing I often see people doing.

... your secrets get checked into git.

Configure your secrets in Azure Key Vault (AWS Secrets Manager, GCP-something-I-forget) or whatever the non-cloud Vault software is called.

Devs only set client_id, client_secret, and env defaulted to dev. On startup, the app fetches configs for the secret store. The needed secrets are in source control. If the needed secret is missing from the secret store, there is a clear error on startup.

You are welcome, I just saved you weeks of onboarding, confusion, and devs passing around .envs on Slack.

My company open sourced the .NET library we use for this: https://github.com/Lokad/Secrets

Don't get it. At one company I worked, .env files for the DEV environment were open (the file itself was stored on Vault, so only engineers had access to it), and one could only access DEV resources via VPN. So, starting a service using the DEV .env file was rather easy an uncomplicated.

My proposal:

1. Key Vault | HashiCorp Vault | whatever is the source of truth for secrets.

2. When a developer adds a key, they add it to Key Vault and update code to use it.

3. On app startup, code checks the dev's local for 2 env vars ONLY, which allow access to key vault.

4. The app then tries to pull in all the env vars it needs by secret name.

5. If any are missing, it fails.

This keeps devs from having to pass around .env files outside of source control. Btw, Azure (and all other cloud) builds can be configured to pull env vars directly from Key Vault, so there's that benefit too.

That's literally all you need to solve the scenario you described with .env files.

So with .env file:

- developer adds to .env

- pushes code

- all developers see failures in dev env due to missing env var

- all developers search slack/email to figure out what went wrong

- all developers update their .env file and things are working again

————- In gp’s comment ———-

- one person works on feature that introduces new config

- failure happens in their local environment

- they update config backend and push code

- all other developers get updates with no changes needed

I do and have done both and a blend. Secrets for external services needed in dev usually in parameter store. Kinda depends on the existing work flow. There is so much to do you should be tackling problems that move the needle and chip away at small stuff, pet peeve stuff in smaller time-boxed increments.

It depends, always it depends.

We use SOPS and the encrypted .env file is stored in VCS. The key is stored in 1password with instructions on where to stuff it on your machine. Then you just decrypt the .env file and you're off to the races.

Then in the case of a kubernetes setup. The application is wrapped inside a mechanism that gets the configuration from some where else.

Why are they fine? This seems to be an argument grounded in "it's a newer norm therefore it's better" but I don't see the justification. I've never used Heroku but ConfigMaps are a bane to manage. Even as a general programming principle within application code, colocality is a well-known, long-discussed topic around improving maintainability. If you're not using monorepos, that's currently not a reasonable option with current orchestration conventions.

Even if you are using monorepos, you're highly likely to either get into writing & automating DSLs to feed into your configmaps or do weird things like using Jinja to do file includes in your mounts. That's all before you run into the etcd 1MB size limit & realise that actually this whole system is a hack.

TL;DR newer ain't intrinsically better

1. One app - one repo. There is nothing fundamentally wrong with multiple apps (as in separate pods/containers/processes) being developed in a single repo, as long as they are tightly coupled functionally, share the release cycle, but still need to be deployed separately to get advantages of separate processes and independent scaling. The first example that comes to mind is separation of public api and worker processes, e.g. ruby's sidekiq, python's celery or a generic kafka consumer.

2. > A twelve-factor app never relies on implicit existence of system-wide packages.

This is incredibly hard to achieve in practice if you're not running something nix, and even then there are leaks of dependencies on kernel syscall APIs. Every Rust app implicitly depends on a glibc (except musl ones), which is present in major Linux distros except slim ones such as Alpine. In fact, the whole Docker crutch IMO is needed specifically to alleviate this problem.

3. > The twelve-factor app stores config in environment variables

It seems to me much more brittle, as it forces you to store secrets in envs, which could be less secure, and to abandon structured file configs, which may enjoy type safety, IDE autocomplete and automatic parsing. With envs you are forced to implement parses for non-trivial and non-string config vars yourself. In fact, envs are often committed into repos as well in form of dot-env files, so the point about commit-safety is moot as well.

You're misinterpreting that guideline. The text says (my emphasis):

> Most programming languages offer a packaging system for distributing support libraries, such as CPAN for Perl or Rubygems for Ruby. Libraries installed through a packaging system can be installed system-wide (known as “site packages”)

It's not saying not to depend on the operating system (of which glibc is part), it's saying not to require particular language package manager packages to be installed on the machine.

Agreed that the other ones you mention are silly, though. Especially the one about environment variables - that isn't actually reasoned advice, it was just the authors writing down what was most common in Ruby development at the time, on the assumption that that must be the best thing to do.

Personally, i would pick a config file over environment variables any day: easier to manually inspect and tweak, easier to give structure to, doesn't get implicitly passed down to subprocesses, can be secured with familiar file permissions, etc.

Environment variables exist on just about EVERY runtime, changed without requiring a pipeline run.

However, subprocess is interesting but I've never run into subprocess that is not sharing a config. Where I worked, if you have subprocess that can't see those configs, you create a new service.

If your point is that existing managers support setting environment variables but not creating config files, then sure. But that isn't a fundamental truth, it's just working with the tools you have.

It just seems far easier to have your kube config inject the env variables from your kube file and that way you don't have to worry during deployment how to add that file and you don't check it into the source. This applies to basically any way you deploy your app, especially containerised ones. Unless you're running your app in an EC2 machine where you can go in and easily edit the file (not that you can't do it in a container), it just seems like extra steps. I've found the happy medium to be sourcing default configs from a file that can all be overridden through env variables

But there's no fundamental reason for that to be the case. You could imagine a very nearby parallel universe where Kubernetes config files don't let you set environment variables, but do let you inject files. In this universe, you can see other deployment approaches that make config files easier (Puppet and suchlike). And if you're building something from scratch, you can make the choice.

Nevertheless, glibc is not the only system dependency which is often used by apps, another such example is JVM of a specific version, which is not tracked by any of the popular JVM build tools and package managers; and OpenSSL, which is too installed by the system package manager.

Dockerfiles exists to fix this by defining all the necessary installation steps as a sequence of instructions, but this is far from a proper package manager.

And not just libc - on a typical unix, there are a small number of other libraries which constitute part of the operating system, like libm, libdl, and libpthread. They are part of the means by which the operating system delivers the POSIX environment.

You're right that the JVM is a dependency, and that there is very poor versioning support for it in production. Everywhere i've worked, we've come up with ad-hoc solutions for it, and none of them have been very good.

If they're deployed separately, they don't share the release cycle. Sooner or later you'll end up with mismatched versions of them running (e.g. perhaps one piece fails to deploy). So you need to be prepared for that, so you want them in separate repos so you can easily test with different versions.

> This is incredibly hard to achieve in practice if you're not running something nix, and even then there are leaks of dependencies on kernel syscall APIs. Every Rust app implicitly depends on a glibc (except musl ones), which is present in major Linux distros except slim ones such as Alpine.

Most higher-level languages avoid depending on a specific glibc; as long as you have a working runtime for that language then your app will work with it. And sure in some cases you resort to docker or something. It can indeed be hard but that doesn't mean it's not worth doing.

In that case I'll just rollback the first one. I see no reason to logically decouple two processes into two services when the separation is technically driven. They share the queues, the database, the domain, further division will just make things more complex with no obvious advantage.

At that point you're not deploying separately any more. They explicitly say you can have multiple processes and process types as part of the same app (see #8). The point is that "repo of its own" should align with "atomic unit of release/deployment".

We disagree on fundamentals then and I think more of the world agrees with me and with the 12 factors.

Dynamic or managed languages can often ignore most of the hassle with system packages.

There is a U-shaped utility to monorepos. Most projects fall in the middle somewhere, largely ones managed by independent engineering teams that have no centralized platform/devops/whatever-we-are-calling-it-these-days team.

Where did you see that being expressed?

They do mean that a repo = a codebase. A remark about "set of repos" is just a technicality of distributed VCSs, meaning that each copy cloned from a single source (or from other copies) is a single repo.

Ignoring it completely on principal is just as bad as the people who treat it like mandatory dogma.

I’ve also had over-eager junior engineers or wannabe architects use the 12-factor article as mandatory requirements for any launch. It takes a firm and unified explanation that these are good goals to strive for, but in the real world we need to make compromises for launch and choose which aspects to delay or defer.

I have moved on a long way from the days that 12 factor was my bible, but those days completely redefined my way of developing software. I'm not exaggerating here - it completely shifted my approach.

I guess it's similar to Scrum in that way - it isn't a great end point, but if you're stuck in the old ways (i.e. the methods used before it was even a thing) then it's at least a good approach to sink your teeth into, try it out, shape your approach, and move on from there.

Basically, I'd rather see a replacement that improves on 12 factor than throwing out the book entirely.

In 2023, yes. A lot of the 12 factor methodologies became table stakes in the modern cloud era. Back when this was written "cloud" was not nearly as ubiquitous as it is now, although we were certainly headed in that direction. A lot of the comments here are viewing this through a 2023 lens and forgetting it was written in 2011.

I want to read and write the language I'm developing with, not the helper methods of some overeager developer who wanted to leave his mark on the codebase.

When you discover it for the first time, you think, "huh, neat, why not use it". Before you know it, nobody can reason about your code and if you leave, nobody wants to maintain it (esp. if your test suite is not extensive).

[0] https://groovy-lang.org/metaprogramming.html

With the 12 factor nonsense, I see the guidance of "Dump unformatted logs into stdout and make it someone else's problem; they'll hadoop or splunk it and everyone wins." There is no magic splunk that absolve you (the developer) of generating useful data to logs, if you expect useful data out of your logs. You can't reverse "cow -> burger -> sewage" and shouldn't expect some magic hadoop to reverse entropy without a ton of work.

So -- I guess the 12 factor app dictum is fine for some trivial "I rewrote twitter in 80 lines of erlang" app, but in my opinion it is a "if we can land a man on the moon we can land a man on the sun" misstatement of complexity in actually getting things done in the real world.

[1] https://hypercritical.co/2023/08/18/the-plumber-problem

Yes, if you want useful data from logs you'll need to log useful data, but this is a case where 12 factor is so successful / dominant today I don't think you even know what it's arguing against anymore.

Thank you! This is the same thing people miss when they read things like the Agile Manifesto, or about DevOps, or even architectural choices like React's original pitch for one-way data binding - or, dare I say, microservices.

You can read these things today, and see flaws - holes in the argument, mistaken assumptions, or misread them as claiming to be superior to things that have been developed since. No - to understand what they are talking about you need to know what was current common knowledge or accepted wisdom in the world when they were written.

If the logs matter -- write them to an API that retains log structure by serializing it.

If the logs don't matter, then they don't matter, but typically the logs are used for all sorts of analytics processes after the fact.

Asking someone else to parse your data is rude.

That guidance is deeply foolish; I have to assume the rest of the guidance is just as silly for anything but a toy application.

You write your logs in whatever structure you want and have something in the runtime environment (k8s, systemd, file redirection in bash whatever) forward them to the eventual storage location. I don't see the issue.

I'm supposed to just load it into memory hoping it's json or avro or whatever? Nope, errors happen; there's a limit to the number of bytes that can be sent from one place to another before the kernel may interrupt it (you're writing an application log and your runtime emits it's own debugging message; they get intermingled because they're both being written to a fifo).

Maybe this is fine (usually it's fine) but eventually it isn't fine. Use a logging library, categories your logs, maybe start off with stdout but eventually put on some big person pants and use a logging api and use the stderr/stdout for high cardinality exceptions, not "all my logs"

So now, when I see a reference to "12 factor app" I assume "ah, a hello world app from someone who's going to leave and let someone else figure out how to make it work in the real world at real loads with real regulatory or compliance needs"

In other words, for me, it's a shibboleth for "toy" or "unserious piece of shit"

There may, in fact, be parts of the guidance that aren't deeply incorrect. But anyone who points at it as a way to do things, without caveats, is suspect.

The idea of 12 factor is to define a reasonable interface for your application. The environment doesn't need to worry about providing interfaces to logging services or collecting files from a custom directory, or making sure the application isn't going to fill the disk with log output - if the app is 12 factor, the only thing the environment needs to do is collect stdout.

As a 12 factor app, write such that what you put on stdout is useful for that purpose.

As a system hosting 12 factor apps, collect stdout and put it somewhere useful.

As a developer of a 12 factor app, when you want to see your log output, go wherever the environment puts it and look there.

That's a useful contract that works at scale. Not sure why you think it doesn't.

I'd love to know what other things you saw one part of and then dismissed fully without further investigation.

Are iPhones unacceptable because they suggest you set a 4 digit pin for access? Aeroplane safety demonstrations? Languages with gendered verbs?

Use a logging library. Configure it to output well formed and categorized log messages to stdout. Let the runtime environment take that stream of data from there.

it says: A twelve-factor app never concerns itself with routing or storage of its output stream. It should not attempt to write to or manage logfiles. Instead, each running process writes its event stream, unbuffered, to stdout. During local development, the developer will view this stream in the foreground of their terminal to observe the app’s behavior.

In staging or production deploys, each process’ stream will be captured by the execution environment, collated together with all other streams from the app, and routed to one or more final destinations for viewing and long-term archival. These archival destinations are not visible to or configurable by the app, and instead are completely managed by the execution environment. Open-source log routers (such as Logplex and Fluentd) are available for this purpose.

The event stream for an app can be routed to a file, or watched via realtime tail in a terminal. Most significantly, the stream can be sent to a log indexing and analysis system such as Splunk, or a general-purpose data warehousing system such as Hadoop/Hive. These systems allow for great power and flexibility for introspecting an app’s behavior over time, including:

This is, at scale, utter horseshit. If it said "do what's expedient, but use a library and be flexible in how you generate logs, but for goodness sake use any of the publicly available libraries to be expedient now and reliable and flexible later" I'd shrug and move on, but it's just absolute garbage. "Do a shitty job and someone else will fix it for you" is not good advice, even if it lets you get to market quickly.

A more reasonable reading of 12 factor is to see that since it suggests that the output streams should be gathered up from multiple running instances of the application, and collated together in a system like splunk, that it behooves you as a 12 factor app developer to build your app such that your output stream is amenable to that approach.

I.e., that your output is a stream of structured serialized data.

You haven't clarified what you think is a better practice than this.

The words don't suggest anything else. Maybe your experience suggests maybe you should use a library and reserve the ability to serialize that data to a proper api if necessary, but that's you not the words in the 12 factor app, and some large subset of people just do the minimum necessary to get the ticket off their queue, and if that means "printf log" that's what this advice in this suggests they'll do. It's splunk's problem now!

And my advice is "use a damned logging library; understand the distinction between using a networking stack and API that manages abstractions like event size and retrying and such and stdout, which offers none of that."

Right. It's not telling you anything about whether to use a library or not. All it's talking about is where you output your logging streams to, not about how you produce or format those streams. Is it bad advice because it doesn't tell you you should wear sunscreen when you go outside in summer?

(FWIW I haven't seen any logging library add enough value to be worthwhile. Eventually graduating to structured event logs is a good idea, but having started with a logging library doesn't actually get you any closer to that).

It is the least controversial and most broadly beneficial of the 12 factors.

Why do you think it’s foolish?

As that someone else, who often has bug reports like "why aren't my log events all in one record in my thing? Because you filled your stdout with newlines!" "Why did my message vanish? It was important!" "because it's malformed json because your runtime had an error; it's over there instead of over here."

Programming's a stack of details; all of which matter. Log using a logging API where available; use stdout/stderr for exception logging only, if at all possible. If you want your logging to be useful, think carefully about how and what you log.

All it means is to let someone else (other than the program itself) figure out how to get the log messages (whatever you decide they should be) from your process (running wherever it is) to wherever you want to have those logs for inspection.

Do not hardcode into your application the assumption that they are written to a file or sent to a port.

In your application, write them to stdout.

In local dev, stdout will probably go to an interactive console. In a container running in a cloud hosted cluster it will probably go to a networked log aggregator that collects it all into an indexed time oriented data store along with other logs from other running instances.

Because you used stdout, that’s easy to do.

If you wrote them to a file whose name your application manages and changes itself to manage log rotation… either of those is hard to do.

I can give an example. The part about sticky sessions is good generic advice for someone building generic web services, but there are situations where sticky session-like functionality or sharding-like behavior might be a better choice. Anyone adhering literally to the 12-factor guide would get hung up on trying to move that data to redis or memcached:

> Some web systems rely on “sticky sessions” – that is, caching user session data in memory of the app’s process and expecting future requests from the same visitor to be routed to the same process. Sticky sessions are a violation of twelve-factor and should never be used or relied upon. Session state data is a good candidate for a datastore that offers time-expiration, such as Memcached or Redis.

The guide also has a strong preference for languages like Ruby, Perl, and Python. There’s even a quip about how 12-factor prefers languages with a REPL, which in the wrong hands becomes an argument against using Rust or other languages.

People who think the entire world is simple CRUD web apps tend to think that guides like this are completely universal. It’s a good guide for people making common web apps, but it’s not a perfect methodology for every backend service.

That's the very first point they make:

> In the modern era, software is commonly delivered as a service: called web apps, or software-as-a-service. The twelve-factor app is a methodology for building software-as-a-service apps that...

Anything not web app/SaaS they express no opinion on.

You didn't actually give an complete example. When are sticky sessions a better choice than persistence?

12FA is not a single box that is to be checked. It can and in most cases probably should be implemented over time as the product matures, taking each point, point by point (and maybe even breaking those up) and adding that into the product. If you've done good engineering up front (abstractions and interfaces where it makes sense, not hard coding everything) this should not be an issue.

inb4 YAGNI: this is just as abused as 12FA.

EDIT: the biggest thing 12FA is missing are concrete examples that junior engineers can reference.

I've learned to take issues like that in good faith. Why are they raising an issue? Is the existing process not clear? Is there a deficiency in it? Is there a lack of reliability? If reasonable concerns are found, the process can change (and the documentation is to be updated).

This has its own drawbacks. I worked for a company, in which Process Was God, and Thou Shalt Do Only That Which God Has Proclaimed Good.

It worked most of the time, but made it very, very difficult to introduce flexibility and orthogonal approaches.

There's really no way to duplicate the results of experienced, cohesive teams of high-functioning engineers, with process. We can get similar results, in many ways, but there's always a price to be paid.

T. A. N. S. T. A. A. F. L.

Yeah, I work in ops^W platform.

I've found it pretty useful in practice. I don't see why it's much more useful to those people than it has been for me.

Being able to point management at this can be really useful.

For most of us here, it is 99% common sense.

There are gold nuggets in many of this style of "manifesto". It just takes a handful of people treating it as dogma to stink them up.

Hard to believe that treating best practices as strict rules would not turn out to be best practice, isn't it?

Docker: Don't use environment for config, it's insecure

I like and use many of the patterns of 12 factor, but indeed some of these were written in the context of VPS, where environment is stable, secure and more constant (unlike containers, where environment might end up shipped in one of the layers). In the age of containers, this particular point has been a bit of a struggle for me. Docker secrets don't always play well, and you need to do some gymnastics to make it work.

Just because a secret is injected into the application as an environment variable does not mean it's handled insecurely elsewhere. For example, AWS ECS containers have built-in support for fetching secrets from Secret Manager at startup and passing them as environment variables: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/...

This fetches the secret at the time the container starts up (IIRC), and uses the IAM credentials of the running application to fetch them from Secret Manager (so it must have permission to the secret).

The merit of using environment variables seems to depend entirely on how they are, well, configured. With a mechanism like this I don't see much disadvantage. (The main disadvantage I see is that generic malware that attempts to dump environment variables might capture them, but defending against that threat is largely obfuscation, unless you aren't storing the secret at all permanently in memory.)

- env variables are prone to be dumped in various contexts and might leak secrets to e.g. logs

- env variables might leak, e.g. in context of docker build stages etc. (but then using building a docker image to build both the image and the software is often anyway not a good idea)

- env variables are often easily accessible by other processes

the method you mentioned of ad-hoc injecting a secret as env by some form of security manager for containers avoid many such problems

A process would need to have the same permission (same user) or root to read /proc/$PID/environ no? or what mechanism is there for a process to read another process env vars (not children processes, that was already mentioned)? unless of course, a process dumped them in some world-readable file (like logs as mentioned) or some other silly way

still works if your system is a single application container

otherwise it does not

This is (part of) the reason programs accept secrets like passwords normally only (or strongly recommended) over stdin or similar instead of env variable or cmdline.

Or why it's not rare (or was in the past, probably is today) to have a encrypted cert file and inject the password to it through a side channel.

It's also not rare for programs to dump environment variables in various situations, including to logs. Or e.g. an intrusion detection program might snapshot all programs running + their cmd arguments + their (creation) env and then run analysis on it.

And while you can use e.g. selinux and similar to add a ton of security and prevent such issues, or use systemd to run the program under ad-hoc users with minimal permissions and various isolation it's very often not done.

EDIT:

To be clear I'm not saying you can't use secrets in environment variables safely.

I'm saying by default by their design environment variables are not "secure for secret passing". But many things for which stuff like that is true can, under the right circumstances, still be used securely.

Since the rule describes runtime configuration, it shouldn't be involved with the build process at all

It's similar to the reason that you're not supposed to supply passwords on the command-line (/proc/$PID/cmdline exists), but instead either read them from stdin, or do the big dance of

1. creating a file that contains the password, is owned by root, and is 0400;

2. having your daemon start off running as root for just long enough to read the password into process memory;

3. then having your daemon drop privileges before doing any work, so that someone exploiting the daemon won't have the privilege as the "do the work" user to read the file with the password in it.

I remember reading that it is possible to end up in situations where your environments can end up in the layers of the docker image. Also, you can read the secrets of a container with inspect, and apparently linked containers might be able to access other containers's secrets.

I think it was this article: https://snyk.io/blog/keeping-docker-secrets-secure/

There's really nothing wrong with secrets mounted on the file system via k8s. But, of course, all of this depends on your deployment environment.

Sometimes env vars are the least bad option, because secrets really are always hard.

Huh? If the app can read the secret, then the app can... read the secret. Maybe I'm misunderstanding what you call a "debug console", but if this is anything running in the app, it can ... read the secrets. Regardless of their source.

Or, to put it differently: it matters nothing if your secrets at rest are stored in a quantum-resistent-encrypted vault, secrets.yml or ENV vars, if your logging, backtracing or debug consoles dump them somewhere, you are compromised regardless.

I've got this problem to debug in prod, I know, I'll embed the env vars in the source code of the error page. No big deal and it might help me debug that! For a dev used to not using secrets in env vars (for example, when testing locally), this is an easy mistake. He may even use a third party library that embeds features like this without knowing it! Even things like the jmx console can be risky, since access to that may not relate 1:1 with access to secrets.

I've never seen anyone make a similar mistake and accidentally embed all_my_passwords.ini in their output.

I must admit that some of what I've said above has become outdated, since it is now a much more common practice to embed passwords in env vars than it used to be.

I've yet to come across any git repo that didn't, at some point, commit the all_mypasswords.ini (or, just as bad, the .env.backup). I've only once ever seen a repo that had ENV vatlrs committed (where some moron committed the entire ./dist directory which included a Json that held the copy of ENV at compile time)

My point is not that "one is obviously better", but that none is obviously better. All solutions have downsides. Implementing 12 factor means understanding those.

From a security perspective it is always more secure to have something only in memory of the running process than it is to have a file on disk regardless of file permissions

File are not necessarily written to disk. e.g. the Secrets CSI Driver loads secrets directly from a secrets store as virtual files within ephemeral volumes.

You could totally spin this as agreeing with 12factor… the virtual files are attached resources that are part of a deployment. The configuration, expressed as env vars, configures where to look for those secrets (ie. their filesystem path) and where they’re used (e.g. as a templatized database URL.)

12factor says config should be env vars, it doesn’t say secrets should be. It’s unfortunate that the site lists credentials as an example of configuration, but… it’d be great IMO if we could sidestep a lot of contention by just considering this to be errata of the example, and that secrets are left as an exercise to the user.

I guess you avoid the risk of accidentally logging secrets with other env variables but otherwise it seems to be just as secure/insecure.

It's a pretty fine distinction and I don't know how many people actually bother doing SELinux etc. in practice, but theoretically it's marginally better.

Bonus: you can watch the file for changes. Which means your app can pick up rotated secrets without a process restart, whereas if you inject secrets via the environment they're fixed for process lifetime.

putting any form of secrets into a env var is deeply insecure

using env is the most simple uniformly available way to have configs

through I would take it a step further, anything which must not be leaked probably should not be in any config, weather env or file or similar, it should be injected through other means if viable

For people that don't already understand the why behind the rules, it took more in-depth explanation. For that purpose, I made the video: "What are 12 Factor Apps and Why Should You Care?"[1]. I know of a couple of companies that use that for new-hire orientation for engineers/devops that have said it's been really helpful.

Regardless where you get the information, it's worth taking an hour or two to learn about 12-factor apps. Most of these "rules" are just things that take awareness and aren't immediately obvious unless you've learned them all the hard way by making the mistakes and feeling the pain.

[1] https://youtu.be/REbM4BDeua0

But the reason I didn't just do it myself before Kent Beck published JUnit was because the code I worked on didn't lend itself well to being controlled by other code. Global variables everywhere, widely distributed state, dependence on external systems and specific file system layouts and general disregard for modularity made it impossible to run anything outside of the context it was designed in. All of this was "bad design", but it met deadlines, so everybody did it anyway. I was _hoping_ that if unit testing gained some traction, programmers would get away from monolithic design.

After 25 years of observing unit tests for getters and setters and then one big unit test that creates an in-memory database because every function in the app needs a live database just to run and then gets commented out because it fails, I've lost any faith that unit testing will ever be more than a meaningless checkmark that everybody fills out because it's a "best practice" but nobody actually stops to think why.

Many people see the value of unit testing. And see how TDD actually shapes your code to be testable and (accidentally, well, not really) therefore also cleaner and better organized.

I've always said, and will continue to do so: good, maintainable code, is accidentally also very well testable code. And vice-versa. And it's not a real accident, if you think about it.

defaults for dev or defaults for prod :)

If you say dev, you will eventually break prod. And prod defaults may not make _any_ sense for dev.

They are actually rather dangerous even. You certainly don't want your dev (or test CI) to reset the database if you are accidentally connected to prod. Same for email-endpoints, CDN, a message-bus, etc.

I've had my share of near-heart-attacks where I send out 10k emails or thousands of payments on actual production services because of a spagetti config loaders with stuff that boils down to `if (getHostname() === "localhost") { include("dev.json") } else { include("prod.json") }`.

I'd prefer a hundred .envs and whatnots to avoid just one of these scares.

- defaults are sensible (safe and secure) defaults for prod

- prod secrets are only available in the prod environment

So if you accidentally end up trying to run with the prod settings in a nonproduction environment it might try and access the prod database server... but it won't get anywhere because it doesn't have those secrets.

The backdrop of when the 12factor app was written is that you can toggle modes like that with RAILS_ENV=test (an environment variable).

One way I think of the config section is “does your config strategy play well with containers?” Once you build an image you have a static disk state that doesn’t persist changes unless you make a new image. If your config is file based (only), then to toggle between test and production behavior you would have to build a whole new image.

By allowing config to change without the underlying disk, it helps to isolate changes. Did the app break because something in the deploy (image generation) broke? Or is it a bad config? If you decouple image generation from configuration changes it eliminates that question.

You could mount a volume with the config file. The nice thing is that this lets you change the config at runtime (assuming the app watches for file changes). Env vars cannot be changed at runtime.

You can specify env vars any time you boot up an image. I see this as a benefit. I use containers for production and don’t want wild ssh yolo modifications. I want to have very clear immutable inputs.

> mount a volume with the config file

Sure, but where? And what if you need multiple files for multiple different things to configure? What’s to prevent someone from accidentally putting non config files into that volume or accidentally changing the permissions on the file or any other host of things that we just don’t have to reinvent if we use env vars.

Configuration should be version controlled, but separately from source code because it describes deployments, not images used for deployments.

Money is not data, it is a complex social construct. Moving money requires interacting with that social construct, i.e. government regulations.

Yes, crypto was supposed to get around that. No, government didn't agree.

It traditionally hasn't behaved as one – which sort of makes sense, because WordPress grew-up in a world where long-lived servers with writable and persistent local disk storage was commonplace.

I'm sure things have moved on since those days. This was back in 2016. But it sure was a fun challenge!

Of course it made things harder in other ways like having a separate DB for sessions.

"Twelve-factor app anno 2022"

Article: https://xenitab.github.io/blog/2022/02/23/12factor/

Discussion: https://news.ycombinator.com/item?id=31225921

https://smallbatches.fm/5/transcript

[00:03:52] Joe Kuttner: But today the infrastructure that we run on is disposable, right? It's much more like livestock or cattle where we can buy new ones when we need one at market and we can dispose of, you know, something's not working. Right. We get. Yeah. So it's a very, it's a very different model.

[00:04:07] Joe Kuttner: And that was just emerging when the 12 factor app was first launched. I think it did a great job of helping bring people into that state of mind.

https://smallbatches.fm/20/transcript

https://developer.ibm.com/articles/15-factor-applications/

Many complications of config management will go away then. Every app instances knows just its ID, then everything else (region, cluster, stage, current experiment, maintenance, anything else) is discovered by "configd" service and config suitable for that very instance is returned. Not unlike feature flags, but for backend services.

OPA (Open Policy Agent) comes close: it has data sources (so it can fetch inventory from cloud providers, kubernetes, any other internal system) and Rego language to program config generation, but it doesn't have means to notify instances of config updates, so they have to resort to polling.

edit: I am correct, it is much more old. https://github.com/heroku/12factor/graphs/contributors

https://www.joelonsoftware.com/2000/08/09/the-joel-test-12-s...

Of course even then added layers were often considered "burden", but in general, we need some degree of order lest we create an unmanageable mess of index.old.3 and files edited directly on production.