gistfile1.txt

Ryan_Lane		3:11    ottomata: yo
ottomata		3:12    hiyaaaa
Ryan_Lane		3:12    so, yeah, let's delete those two gerrit repos
ottomata		3:12    oh ok!
Ryan_Lane		3:12    git-deploy and sartoris
ottomata		3:12    awesome
ottomata		3:12    our git-deploy is the python port of the perl git-deploy, and sartoris is the salt backend
Ryan_Lane		3:12    it'll help reduce the confusion factor
Ryan_Lane		3:12    nop
ottomata		3:12    and we want to rename them both to trebuchet stuff
Ryan_Lane		3:12    *nope
ottomata		3:12    no?
Ryan_Lane		3:12    we use the perl frontend
ottomata		3:13    but our repository i mean
Ryan_Lane		3:13    sartoris was a python port of that
Ryan_Lane		3:13    we never used it
Ryan_Lane		3:13    the backend of the system has always been in the deployment module in puppet
Ryan_Lane		3:13    it's really never had a name
ottomata		3:13    hm
Ryan_Lane		3:13    we decided we'd name everything sartoris, then there was an issue with that that I'll ignore
Ryan_Lane		3:13    so, I decided I'd name everything trebuchet
Ryan_Lane		3:14    and then I wrote the frontend from scratch and called it trebuchet-trigger
Ryan_Lane		3:14    now there's trebuchet (the salt backend), trebuchet-trigger (the git frontend) and trebuchet-ricochet (the web interface)
Ryan_Lane		3:15    trebuchet is most up to date in puppet in the deployment module
Ryan_Lane		3:15    I'm working on switching that upstream to git-hub, rather than wikimedia's puppet repo
Ryan_Lane		3:15    here's all the stuff that's in github: https://github.com/trebuchet-deploy/
Ryan_Lane		3:16    for trigger and ricochet, github is the proper upstream
Ryan_Lane		3:16    trigger has debianization
Ryan_Lane		3:16    ricochet and trebuchet will soon too
ottomata		3:16    ok cool, i think i'm mostly interested in trigger, right?
Ryan_Lane		3:16    trigger and trebuchet
ottomata		3:16    i want to build in support for grabbing jars from urls
ottomata		3:16    and verifying hashes
Ryan_Lane		3:16    trigger, as the name implies, just triggers a deployment
Ryan_Lane		3:17    it could generate the jars, grab them, etc
Ryan_Lane		3:17    and place them somewhere to be deployed
Ryan_Lane		3:17    but to deploy it, you'll need to use trebuchet
ottomata		3:17    hmm, interesting
ottomata		3:17    hm
ottomata		3:17    i'm thinking that this can acutally be done very simply, maybe not even dependent on java stuff
ottomata		3:17    just enabling individual files to be deployed based on urls and hashes
Ryan_Lane		3:18    yeah, that's sane
Ryan_Lane		3:18    trigger could generate them
ottomata		3:18    so, maintain a list of urls to .jar or whatever files in a config file somewhere
Ryan_Lane		3:18    and trebuchet could grab the hashes
ottomata		3:18    and then when a deploy happens, those files are dled, hashes computed and compared to what is in the file
ottomata		3:18    config file
Ryan_Lane		3:18    it may be a good idea to make a directory for each deployment
Ryan_Lane		3:18    with all the necessary files
ottomata		3:19    could this just be a feature of a normal git-deploy maybe?
Ryan_Lane		3:19    on the minion side its ideal to be able to quickly revert
Ryan_Lane		3:19    ignore git-deploy
ottomata		3:19    ok
Ryan_Lane		3:19    you don't need to use git
ottomata		3:19    hmmm
Ryan_Lane		3:19    (for deployment)
Ryan_Lane		3:19    you can write any style of deployment you want
ottomata		3:19    ok, i am about 25% familiar with this system, so appreciate the handholding
ottomata		3:19    
grrrit-wm		(PS1) Aude: Have each site group use own cache key for wikibase [operations/mediawiki-config] - https://gerrit.wikimedia.org/r/110224 			3:19
Ryan_Lane		3:19    trigger (git deploy blah) is necessary
Ryan_Lane		3:19    but for the backend it can be anything you want
Ryan_Lane		3:20    you could use git annex
Ryan_Lane		3:21    in fact, git annex may be a sane way to handle this, in fact
Ryan_Lane		3:21    s/, in fact$//
ottomata		3:21    oh hm
Ryan_Lane		3:21    I'm assuming you've seen this? https://wikitech.wikimedia.org/wiki/Trebuchet/Design
Ryan_Lane		3:21    ugh, this image is still named sartoris: https://wikitech.wikimedia.org/wiki/File:Sartoris.png
Ryan_Lane		3:22    renames
ottomata		3:22    ah yes i have seen that
Ryan_Lane		3:23    \o/ https://wikitech.wikimedia.org/wiki/File:Trebuchet.png
Ryan_Lane		3:24    so, in that diagram, everything except for git-deploy is trebuchet
Ryan_Lane		3:24    (and git-deploy is now trigger)
ottomata		3:24    Ryan_Lane: the fetch is done against the local repository where git-deploy is run, right?
ottomata		3:24    so
Ryan_Lane		3:24    yeah
ottomata		3:25    running git deploy sync triggers salt on the deploy hosts to fetch against the local repo there
ottomata		3:25    ok
Ryan_Lane		3:25    trebuchet modifies the origin and all submodules to point to the deployment server
Ryan_Lane		3:25    yeah. technically it calls a runner, which calls the salt modules on the minions
Ryan_Lane		3:26    but that's easy enough to ignore
ottomata		3:26    hhmm git annex actually looks good
Ryan_Lane		3:26    indeed
Ryan_Lane		3:27    ottomata: what you want to look at is this: puppet/modules/deployment/files/modules
Ryan_Lane		3:27    mostly deploy.py
Ryan_Lane		3:27    right now it's written to be very git specific
ottomata		3:27    deploy_server.pp?
Ryan_Lane		3:27    we'd want to split this into multiple modules: deploy_git.py, deploy_<yourmethod>.py, etc
Ryan_Lane		3:28    then from deploy, you'd want to call the method based on the type of deployment from the config
ottomata		3:28    oh files sorry
Ryan_Lane		3:28    __salt__['deploy_<method>.<this_function>'](args, kwargs)
Ryan_Lane		3:29    or something along those lines
Ryan_Lane		3:29    that way we can have multiple methods that implement the same interface
Ryan_Lane		3:30    this may take a bit of refactoring, of course
ottomata		3:30    yeah, grokking...
Ryan_Lane		3:30    so, I have a development environment for this in labs
Ryan_Lane		3:30    which is terribly enough the "sartoris" project
Ryan_Lane		3:30    let me add you to that project
Ryan_Lane		3:30    I have a new trebuchet project I'll be moving everything into
Ryan_Lane		3:31    What's your username on wikitech?
ottomata		3:31    ok cool
ottomata		3:31    ottomata
ottomata		3:32    Ryan_Lane: if we were to use git annex, could we just add support for that, like _update_gitmodules?
Ryan_Lane		3:32    yes
ottomata		3:32    that woudln't need a new deployment method then
Ryan_Lane		3:32    that would be a pretty simple way to do this without refactoring
Ryan_Lane		3:32    I'm all for that if you'd like to do it
ottomata		3:32    jars and other not-in-get content could be managed with just git annex commands then
ottomata		3:32    hmm
Ryan_Lane		3:33    so, in the sartoris project are the following instances: sartoris-server, sartoris-deploy, sartoris-target, sartoris-target4
ottomata		3:33    i think…that would be ok, right, hm
Ryan_Lane		3:33    server is a puppet/salt master
Ryan_Lane		3:33    deploy is a deployment server
Ryan_Lane		3:33    target and target4 are targets 
ottomata		3:33    ah ok cool
ottomata		3:33    awesome
Ryan_Lane		3:33    they are all pointing to the salt and puppet master that's on server
ottomata		3:33    are the deployment projects puppetized like they are in productoin?
Ryan_Lane		3:34    so you can do local dev there for testing
Ryan_Lane		3:34    yeah
Ryan_Lane		3:34    there's basically nothing custom in the dev environment right now
Ryan_Lane		3:34    you can check out their puppet config via the "configure" action in wikitech
ottomata		3:35    role::deployment::test?
ottomata		3:35    ah, parsoid::production too
ottomata		3:35    awesome
ottomata		3:35    ok
Ryan_Lane		3:35    yeah
ottomata		3:35    coool
Ryan_Lane		3:35    so, you can set up anything you want for targeting
ottomata		3:35    awesome
Ryan_Lane		3:35    it should work just like it does in production
ottomata		3:35    ok great
Ryan_Lane		3:36    I spent quite a bit of time setting this up during my last sprint 
ottomata		3:36    ok, so, since i'm really trying to get this working sooner rather than later, would you be ok with me adding the annex support (if that is what I do) to the python files in puppet?
Ryan_Lane		3:36    yep
ottomata		3:36    or would you rather me help you refactor the final stuff and get us up to date on that
Ryan_Lane		3:36    do it in puppet
ottomata		3:36    ok cool
Ryan_Lane		3:36    we can work the changes upstream later
ottomata		3:36    ok awesom, thanks
ottomata		3:36    hopefully the anex thing will make this way easier, i don't fully understand it yet
ottomata		3:36    but it looks like it should work
Ryan_Lane		3:36    the downside is that it won't show your changes in github
Ryan_Lane		3:36    unless you upstream the changes 
ottomata		3:36    heh, yeah 
ottomata		3:37    well, if the python is basically the same in upstream trebuchet
Ryan_Lane		3:37    we'll do that when I'm ready to move things
Ryan_Lane		3:37    so you get proper credit
ottomata		3:37    then that shouldn't be hard?
Ryan_Lane		3:37    yep
Ryan_Lane		3:37    exactly
ottomata		3:37    ok cool
Ryan_Lane		3:38    so, a kind of pain in the ass way of doing dev for this is to develop inside of the puppet module
Ryan_Lane		3:38    and to restart the puppetmaster, and to run puppet on the server
Ryan_Lane		3:38    it'll sync the modules to the targets
Ryan_Lane		3:38    oh...
ottomata		3:38    hm ,heh, yeah
ottomata		3:38    hm
Ryan_Lane		3:38    I have trigger in use in labs, rather than the perl frontend
ottomata		3:38    i could edit the synced modules on the targets?
Ryan_Lane		3:39    so changes to that should go to github
Ryan_Lane		3:39    you could, but I wouldn't recommend it
ottomata		3:39    heheh
Ryan_Lane		3:39    you could disable puppet
Ryan_Lane		3:39    one sec
ottomata		3:39    well, if annex works like I think it will, i don't think i will need to mess with the front end
Ryan_Lane		3:39    true
ottomata		    if config['checkout_annex']: blablalb
ottomata		3:40    or whatever
Ryan_Lane		3:40    so...
Ryan_Lane		3:40    on server: /srv/salt/_modules
Ryan_Lane		3:40    that's where modules are sync'd from
Ryan_Lane		3:41    if you disable puppet and edit that directly, you can just run: salt '*' saltutil.sync_modules
Ryan_Lane		3:41    if you run puppet it's going to overwrite your work though!
Ryan_Lane		3:41    so make sure to back that up
Ryan_Lane		3:41    or to make it into a repo
ottomata		3:41    ah ok awesome
ottomata		3:41    ok cool, i'll probably editi locally
ottomata		3:41    so ja
Ryan_Lane		3:41    editing the modules directly on the minions is hard
Ryan_Lane		3:42    because it does stuff with caching and such
ottomata		3:42    actually, when I need to test things like this in labs, i have a really hacky rsync setup so I can edit locally
Ryan_Lane		3:42    heh
ottomata		3:42    so i will probalby edit locally, rsync to sartoris-server (rigth?) and then run saltutil.sync
Ryan_Lane		3:42    I usually edit in git, then do a git push
ottomata		3:42    that would be fine, right?
Ryan_Lane		3:42    yep
ottomata		3:42    k cool
Ryan_Lane		3:42    I'm definitely interested in having this support 
ottomata		3:43    yeah, this will be simple and generic and awesome if it works this way
Ryan_Lane		3:43    git sucks at large objects
Ryan_Lane		3:43    annex is good with it, though
ottomata		3:43    i've been playing with archiva, and it will let me curl whatever i want from it
Ryan_Lane		3:43    and it works with s3/swift, etc
ottomata		3:43    and annex looks like it verifies checksums, maybe?  
Ryan_Lane		3:43    yes
ottomata		3:43    hmm, ok still ahve lots of research to do
ottomata		3:44    basically, all we want is a way to maintain a list of files and checksums
Ryan_Lane		3:44    let me know if you have any issues, I'm around to help
ottomata		3:44    and change thme (in git?)
ottomata		3:44    that way the list goes through git review
ottomata		3:44    awesooome
ottomata		3:44    ok thanks Ryan_Lane
Ryan_Lane		yw