gistfile1.txt

2:02 <lifeless> Worse case, if you use pystache directly at least the templates will be in the same language.
2:03 <hub_cap> does pystache let me do things like {{ X*some_var}}
2:03 <hub_cap> sry using jinja template
2:03 <lifeless> What does that mean in jinja?
2:03 <hub_cap> so if i have a value in X itll multiply it
2:03 <hub_cap> so i dont have to put all that stuff into the code, cuz i know that the flavor will change a bunch of values
2:04 <hub_cap> i assume apply-config does this too?
2:04 <lifeless> Fairly sure not; it aims for complete separation of code and template.
2:04 <hub_cap> https://gist.github.com/hub-cap/5952484
2:05 <lifeless> os-apply-config takes one (today) or more (very soon) json files containing the data to render, and then renders all the templates in a template tree for you.
2:05 <hub_cap> i dont care to know what each value is in server code for our template
2:05 <hub_cap> even more, people can customize this to whatever they want w/o extra code
2:09 <lifeless> it doesn't do that; preprocessing should be done before, for this sort of templating
2:09 <lifeless> uhm
2:10 <lifeless> not sure of the best tradeoff here TBH; I can see the convenience of pushing that out of the python and into something you would be happy with users editing
2:11 <lifeless> but will non-deployers have access to that template anyway?
2:11 <hub_cap> not that one
2:11 <hub_cap> but it checks /etc/trove first
2:11 <lifeless> cause if its only deployers, then its moot, code and template are both equally hard to get at
2:11 <hub_cap> if it cant find, it falls back to a package
2:11 <hub_cap> https://review.openstack.org/#/c/35530/5/trove/common/template.py
2:11 <hub_cap> see line 29~30
2:12 <lifeless> same question though: end users won't have access to that will they?
2:12 <lifeless> they just get a mysql endpoint
2:12 <hub_cap> end users will get the modify_configuration stuff thats coming down the pipeline
2:13 <hub_cap> its for deployers, and im pretty durn sure that hp wont have the same defaults as us
2:13 <hub_cap> and they wil both be sittin in /etc/trove/mysql.template on our systems
2:13 <lifeless> so another way to do it
2:13 <lifeless> is to make the template really just a template
2:14 <lifeless> and have a pluggable translator layer to do transforms on the data model
2:14 <lifeless> that lets hp and rackspace have the same template
2:14 <lifeless> and the same core code
2:14 <lifeless> and just a little shim that encodes the differing defaults
2:14 <hub_cap> sure but we may not want the same template... u could not want a value we want alltogether
2:15 <hub_cap> its not just about sub'ing the stuf after the = sign for some values
2:15 <lifeless> one template can handle that.
2:15 <hub_cap> im confused. let me propose an example, a stupid one, nonetheless
2:15 <hub_cap> if rax wants bind-address=localhost, and hp doesnt want bind-address present in the template, how can 1 template handle that?
2:16 <lifeless> {{#max_connections}}
2:16 <lifeless> max_connections = {{max_connections}}
2:16 <lifeless> {{/max_connections
2:16 <lifeless> for instance
2:16 <lifeless> I was just putting that on your gist
2:16 <lifeless> that says 'if max_connections is truthy put it in the output'
2:16 <hub_cap> sure but that means that you then need to put another config file (not the template) for each value u want right?
2:16 <hub_cap> and to me i see that as just putting a new template w/ what u want
2:16 <lifeless> huh
2:17 <hub_cap> max_connection=True in some server config file
2:17 <lifeless> why would you need a new config file?
2:17 <hub_cap> not a new, persay
2:17 <lifeless> well you'd set it to a value
2:17 <hub_cap> but u need to program your config file to build the templated config file
2:17 <hub_cap> right
2:17 <hub_cap> here i say, heres a template, go put what u want in it
2:17 <lifeless> if its not set, its not in the output, if its set it is. If you're talking about defaults, they would go in the HP plugin I mentioned.
2:18 <hub_cap> if 3rd company comes along, and says, gosh i want X, do we want to make sure this new value is now in the template
2:18 <hub_cap> and so on
2:18 <lifeless> so thats a fair point
2:18 <hub_cap> rahter than saying, heres our dev template, go make your own
2:18 <lifeless> let me tell you something we have as an issue in tripleo
2:18 <lifeless> we have a bunch of config files from nova neutron etc
2:19 <lifeless> they are all copied out of git, so when they change, we're manually fixing them up
2:19 <lifeless> and they are templates
2:19 <lifeless> so that we can inject appropriate values
2:19 <lifeless> this is really annoying, because we have a burden - we have to track these things against updates to the projects.
2:20 <hub_cap> most def
2:20 <lifeless> I am concerned that what you're proposing will have the same pattern
2:20 <hub_cap> well, its mysql / redis / etc
2:20 <hub_cap> im not sure it changes as often as nova :)
2:20 <lifeless> if folk are expected to edit them, folk will have to edit them and merge deltas from trove updates into their prod variants.
2:20 <lifeless> What we're planning to do in tripleo to make this better is to separate out:
2:20 <lifeless> - things we change
2:21 <lifeless> - the original file from nova/etc (btw we have mysql too, for obvious reasons)
2:21 <hub_cap> hehe ya
2:21 <lifeless> then we'll not need to merge in changes that are outside the stuff we needed to care about, we'll just get latest file as we install
2:21 <lifeless> not exactly sure what this will look like in code yet.
2:22 <hub_cap> i actually dont expect anyone to use the template we have provided. its for dev only. id expect anyone who wants to use the system to provide a real, decent, my.cnf
2:22 <hub_cap> so im not worried about merge
2:22 <lifeless> hub_cap: so, I think thats a pretty big barrier to entry.
2:22 <hub_cap> agreed
2:22 <hub_cap> this one works well enough
2:22 <lifeless> hub_cap: as a deployer, I /want/ a production my.cnf coming from $someone else
2:22 <hub_cap> but i dont expect anyone who watns to run mysql to _never_ tweak it
2:22 <hub_cap> do u?
2:23 <hub_cap> we have to tweak our connections based on internal junk w/ our hardware, for instance
2:23 <hub_cap> our ib_logfile space will differ based on whatever local space our flavors have too
2:24 <lifeless> so I agree that local overrides are often needed.
2:24 <lifeless> but say that there are 100 settings.
2:24 <lifeless> How many have just one right value for prod - maybe 80 of them?
2:24 <hub_cap> sure a good bit of em
2:24 <lifeless> Maybe 15 more are 'scale to size of hw' - like buffer sizes.
2:24 <hub_cap> based on flavor size
2:24 <lifeless> And then 5 are local quirks.
2:25 <lifeless> replacing the entire file just so that 5 local quirks can be set makes knowing what the other 95 are, and tracking them as best practice evolves, pretty hard.
2:25 <hub_cap> i can see that point
2:26 <hub_cap> so id say that eventually, what youre saying w/ the shim makes sense
2:26 <lifeless> so things like nova let you have a reference config, and then a local config that overrides, and so forth
2:27 <hub_cap> right it seems like the shim is that notion
2:27 <lifeless> mysql might not permit that, but the same pattern might work well; and for that pattern with templates I'd do one master template that encodes /everything/
2:27 <hub_cap> yup
2:27 <lifeless> and yeah then shims or something to right - we're in sync
2:27 <hub_cap> correct
2:27 <hub_cap> so tbh, i needed to remove the apt repo dependency, and this is doing that
2:28 <hub_cap> i will admit, especially after our talk, that its not the best solution
2:28 <hub_cap> but in order for cern to be able to eval it, they need this code
2:28 <lifeless> heh
2:28 <hub_cap> so id like to put that as a blueprint on the future stuff
2:28 <lifeless> so, let me sketch another tripleo thing
2:28 <hub_cap> sure
2:28 <lifeless> Spamaps is putting the finishing touches on it at the moment
2:29 <lifeless> right now we have one metadata source (heat) which delivers a json file to /var/lib/cloud/data/cfn-init-data
2:29 <lifeless> and os-apply-config defaults to using that as its data source.
2:29 <lifeless> but we need to gather data from many places - the EC2 metadata server, local rules, etc.
2:29 <lifeless> so what we've got is 'os-config-collector' which takes an ordered list of metadata sources
2:30 <lifeless> and polls em all to create local JSON files one per source.
2:30 <lifeless> We're then going to change os-apply-config to accept an environment variables (OS_CONFIG_SOURCES or something)
2:30 <lifeless> which os-config-collector will export
2:30 <!> pdmars_ ~pdmars@of1-nat2.aus1.rackspace.com has quit Remote host closed the connection
2:30 <lifeless> and os-apply-config will read, in order, and merge together, all these sources.
2:30 <lifeless> then render the templates
2:30 <lifeless> ^- this is our implementation of the pattern we just discussed
2:31 <hub_cap> yes definitely
2:31 <lifeless> might save some wheel re-doing when you get a chance to redo this bit.
2:31 <hub_cap> and i assume os-apply-config wont require env vars? sice ill be using it all in pure python :)
2:31 <hub_cap> i assume yall are scripting it all together on the cmd line
2:32 <lifeless> we are indeed; small tools (its the sysadmin domain)
2:32 <lifeless> it has a python API of course, the CLI is just entry points.
2:33 <hub_cap> okey. i ahvent actually looked @ the source but it has to if its on pypi eh? :)
2:33 <hub_cap> so ill put up the blueprint to make it better w/ a link to this chat