Conversation with josevalim about exrm and runtime environment-based configuration for releases

convo

[02:34:43] <bitwalker> josevalim: got a sec to talk about https://github.com/phoenixframework/phoenix/issues/354 real quick?
[02:36:10] <josevalim> bitwalker:  sure, even minutes!
[02:36:26] <bitwalker> awesome :)
[02:38:56] <josevalim> bitwalker:  how can i help?
[02:38:57] <bitwalker> josevalim: I'm wondering if you can clarify your earlier comment about a solution...it seems to me that referencing external configs doesn't really solve the problem, it still requires manual configuration
[02:39:36] <josevalim> bitwalker: yeah, basically we would need a way to tell exrm that a given config is coming from the system environment
[02:40:04] <josevalim> because it seems the issue is that people need to read it from the system environment
[02:40:47] <bitwalker> josevalim: Any ideas of how to flag that for exrm?
[02:41:08] <bitwalker> Since compilation will turn System.get_env to whatever the value is, we don't have the context at boot time
[02:41:16] <josevalim> one idea I had, not sure it is good
[02:41:20] <josevalim> is to have a way to say
[02:41:43] <josevalim> system_env: [phoenix: [port: "PORT"], foo: [bar: "BAR"]]
[02:42:30] <bitwalker> Oh I gotcha, so rather than calling the function System.get_env, you'd use that configuration convention to pull stuff from the environment at runtime?
[02:42:47] <bitwalker> Or exrm would I suppose
[02:42:58] <josevalim> bitwalker:  that's my suggestion because I think that's the only way exrm could make it work
[02:43:00] <josevalim> but i could be wrong
[02:44:14] <josevalim> bitwalker:  we could add that to mix too
[02:44:16] <bitwalker> I wonder if it would make more sense for exrm to read config.exs as AST, and convert function calls like that to a more appropriate format (where applicable, System.get_env being the only one transformed at first)
[02:44:55] <josevalim> hard because what if someone has a conditional based on something else?
[02:45:02] <josevalim> how would you evaluate it?
[02:45:14] <josevalim> it can be tricky really quick
[02:45:42] <bitwalker> Have an example of a situation where transforming System.get_env to the format you described would fail?
[02:46:23] <josevalim> well, something like
[02:46:37] <josevalim> if Mix.env == :foo, do: import_config "something_special.exs"
[02:46:46] <josevalim> err
[02:47:04] <josevalim> System.get_env("FOO") == "BAR"
[02:47:23] <josevalim> also, will you also traverse the import_configs recursively to you also transform the imported ones?
[02:47:32] <josevalim> it feels like a dangerous game to play :P
[02:48:19] <josevalim> anyway, we could add system_env config directly to Mix, but we need to make sure it works and solves exrm issues before
[02:48:31] <bitwalker> Yeah, I think you'd have to traverse the configs, or it doesn't work. I wonder though, why would someone use System.get_env in config.exs for compile-time configuration logic?
[02:48:32] <bitwalker> My main fear is, let's say someone sees the configuration format you described, but uses it without exrm in the mix, they'd get unexpected results. I'd like to avoid requiring anything special just for exrm if possible.
[02:48:50] <bitwalker> Ah, your last comment maybe answers that question then
[02:49:02] <josevalim> yeah, exactly
[02:49:11] <josevalim> it is a general problem for when we compile configs and people want system values
[02:49:13] <bitwalker> Although, would that work for erlang deps?
[02:49:48] <josevalim> you will need to invoke something that sets the proper apps flags on boot
[02:50:33] <josevalim> if make it work at the vm boot via flags, i think it will work for whatever dependency
[02:50:39] <josevalim> as we don't really add anything special for elixir deps
[02:51:11] <josevalim> bitwalker:  if you want to avoid users from missing system_env what you could do is, when reading the config.exs file, check if there is a System.get_env call, you could even use String.contains?
[02:51:53] <bitwalker> Yeah can definitely warn users about System.get_env calls
[02:52:12] <bitwalker> I think this could work, I'll try and draft up a test case and see how it goes
[02:52:20] <josevalim> bitwalker:  if there is such a call and system_env is not specified, you could emit a warning like: "There is a System.get_env file in config/prod.exs but system environment variables are evaluted when the config file is compiled. If you want to use the system environment variable at runtime, please set system_env: [app: [key: value]]"
[02:52:34] <josevalim> bitwalker: yeah, let me know if i can help with anything
[02:52:40] <bitwalker> josevalim: perfect
[02:52:45] <josevalim> i think the biggest issue is that you will have to deep merge
[02:52:47] <josevalim> if someone does
[02:52:58] <bitwalker> I'll follow up in that thread with notes from our convo here, just for posterity
[02:53:07] <bitwalker> Yeah, I have code for that already
[02:53:13] <josevalim> system_env: [phoenix: [foo: [bar: "BAT]]]
[02:53:32] <josevalim> bitwalker: you deep merge on boot though, i don't think you ship with any exrm code to be used on boot?
[02:54:07] <bitwalker> josevalim: If users are using the conform plugin with exrm, I do, if not, then no, there is no code executed by exrm on boot
[02:54:47] <bitwalker> I could easily add a plugin for non-conform releases though that does something similar
[03:02:40] <josevalim> cool