gistfile1.txt

ehazlett      sjl: hey have a sec?
sjl           ehazlett: sure
ehazlett      sjl: i'm trying to migrate the old testdata instance to the latest production
sjl           ok
ehazlett      sjl: when i try to migrate i'm getting the "ghost migrations" error
sjl           fun
ehazlett      sjl: http://d.pr/i/k7N
ehazlett      sjl: sorry for the terrible formatting
sjl           ehazlett: what exactly are you trying to do again?
sjl           the production branch doesn't have team migrations 16+
ehazlett      sjl: i'm trying to update a database from rev 646137e to the latest production (maggie's test instance)
sjl           it looks like you're trying to take an instance that was running on the dev branch and migrate it to the production branch
ehazlett      sjl: well shit
ehazlett      sjl: figures
sjl           what you'd need to do is migrate *backwards* to the last migration that's present in production first, *before* you change branches
sjl           because after you switch the branch, south doesn't know how to undo those migrations, since they're not on disk any more
sjl           (this is all assuming those migrations actually do contain the backwards migrations, but I'm usually pretty good at making sure we do that)
sjl           databases are bullshit
ehazlett      sjl: ugh
sjl           yeah that's an accurate depiction of my thoughts too
nicksergeant  let's just all save everything to plain text files
ehazlett      LOL
nicksergeant  need to login? upload your identity file and we'll try and find it on-disk
ehazlett      ha
nicksergeant  log in*
sjl           ugh, stupid rochester, stop snowing
nicksergeant  sjl: nevar!
ehazlett      sjl: ok so i'm a south newb -- could you spare 5 min to educate me?
sjl           ehazlett: sure
sjl           ehazlett: so the basic idea is that a single "migration" describes a change in the schema of the database
sjl           ehazlett: so let's say I add a "foo" field to a Django model
ehazlett      sjl: yep
sjl           ehazlett: the migration actually contains two parts, a "forward" part that says "add column foo to table whatever"
sjl           ehazlett: and a "backward" part that says "remove column foo from table whatever"
ehazlett      sjl: got it
sjl           so it describes how to go from the original DB schema to the new one, and how to go from the new one back to the original
sjl           each migration is stored in its own file on disk.  that's those 0115_add_blah_blah.py files in the migrations/ folder for each app
ehazlett      sjl: yep i've seen those
sjl           there are also things called data migrations.  in general you use them to just modify data, but conceptually they're not any different than the normal ones
ehazlett      sjl: ok
sjl           all migrations take the DB from one state to another
sjl           whether that means adding/removing a field, or capitalizing some data or something, doesn't matter
sjl           DB + migration = new DB
sjl           make sense so far?
ehazlett      sjl: yep
sjl           ok
sjl           so normally this is pretty straightforward to work with.  You have your local database, and you need to add a field to a model
sjl           You do that by creating the migration.  How exactly that happens isn't important right now.
ehazlett      sjl: got it
sjl           But you end up with a new file like 0116_add_field_foo.py
sjl           and that file contains two parts: the forward part (that adds the field) and the backwards part
ehazlett      sjl: ok
sjl           Now, south itself also has a table in your database for its own data
sjl           It stores the current "state" for each app in there
ehazlett      sjl: ok
sjl           so something like "the teams app is currently at migration 0114_..."
ehazlett      sjl: gotcha
sjl           when you tell south to migrate an app, it look at the current state of the DB, and it looks at the files in your migrations folder for that app
ehazlett      sjl: ah and that's how it decides where it needs to be?
sjl           so in our example, if you started out at migration 0115 and created 0116 to add your field, south would know that it needed to run the SQL for the forwards part of 0116
ehazlett      sjl: got it
sjl           and once that finished it would update its own table to record that the teams app is now at 0116
sjl           now, you can also tell south to migrate to a specific target, instead of just "whatever is the newest"
ehazlett      sjl: ok
sjl           so you can say "manage.py migrate teams 0112" for example
sjl           south will look in its table to see where it currently is (which is 0116 in our example)
sjl           so it says okay, I'm at 116 and need to get to 112, so I need to go backwards
sjl           so first it runs the backwards part of 0116
ehazlett      sjl: ah ok -- i wondered how it went back
sjl           then it runs the backwards part of 0115
sjl           then it runs the backwards part of 0114
sjl           then it runs the backwards part of 0113
sjl           then it's done
ehazlett      sjl: ah ok
sjl           so far, so good
sjl           right?
ehazlett      sjl: yep
sjl           ehazlett: ok, so this still is pretty straightforward.  you do your local development, maybe creating a few migrations along the way
sjl           let's say 0116 0117 and 0118
sjl           now you push those to github, and as part of your deployment process you tell south to migrate to the latest point
sjl           so south migrates your server or whatever up to 0118 and you're all good
sjl           that's the way things are *supposed* to work
ehazlett      :)
sjl           ok, so where this gets tricky is when you have concurrent development happening
sjl           i.e.: more that one line of development happening at the same time
sjl           which may or may not involve git branching
sjl           Let's use a single branch as an example for now
ehazlett      ok
sjl           let's say we're in a repo with just one master branch and that's it
sjl           and nick and I are working on two different things
sjl           let's say that in the morning when we start, there are 10 migrations for the app we're going to work on 
sjl           so our server knows it's at migration 10, and our local servers are at the same place
sjl           so now we start working.  nick creates a migration 11 to add a field foo on his local machine
sjl           while he's doing that, I create a migration (also 11, because nick hasn't pushed his yet) to add field bar on my machine
sjl           each of us tells south to migrate, and in both cases there's no issue (yet).  south runs the sql for the forwards part of my 11 on my machine, and his 11 on his machine
ehazlett      yep
sjl           now let's say nick finishes first and pushes to github
sjl           so the master branch on github now contains the first 10 migrations we started with, plus 11N(ick)
ehazlett      start the chaos
sjl           now when I pull from github, I get 11N
sjl           so now my migrations folder contains 11S and 11N (plus the first 10, of course)
sjl           understand so far?
ehazlett      yep
sjl           ok, so at this point there are two things that can happen with I try to run "python manage.py migrate myapp"
sjl           the key point here is that south runs migrations in filename order
ehazlett      yeah
sjl           which normally is sane, because there's a number at the beginning of the filename
sjl           0001_... 0002_... etc
sjl           but now that we have two with the same number, it's going to look at whatever the rest of the name happens to be to decide the order
sjl           If I'm lucky, at this point South will explode when it tries to migrate.  This would happen when Nick's file comes *before* mine
sjl           Because my DB has ... 8 9 10 11S applied
ehazlett      yeah
sjl           and South looks at the files and sees that it should be ... 8 9 10 11N 11S
sjl           it can't just apply the 11N on top of 11S because that would be in the wrong order
ehazlett      but it would for 11S on nick's right?
sjl           so at that point South will just throw up its hands and say "fuck it, I'm out, good luck"
sjl           that make sense?
ehazlett      yeah
sjl           ok, so let's think about the opposite case: my filename happens to come before nick's
sjl           so the order South thinks it needs is 8 9 10 11S 11N
sjl           and my current database is at 8 9 10 11S
sjl           in this case, South will happily just apply nick's migration on top of mine
ehazlett      ok
sjl           (just like any other new migration)
sjl           this is the case that will make you drink
ehazlett      lol
sjl           because now everything seems fine
sjl           but what happens when you push to the server?
ehazlett      wouldn't it work also if it was at 10?
ehazlett      9 10 11S 11N
sjl           yes
sjl           but remember that nick pushed, and maybe deployed
ehazlett      ah
sjl           what would happen if he deployed?
ehazlett      so it's fubar'd
sjl           right
ehazlett      9 10 11N - no 11S
sjl           now we're at the first case again, only this time it's the *server* that's borked
sjl           (and Nick's machine once he pulls down the new migration)
ehazlett      got it
ehazlett      so does he have to migrate 10, then migrate?
sjl           that's a way to manually fix things -- tell south to migrate back to 10 (which unapplies one of the 11's) and then tell it to migrate forwards to the latest, which will apply the 11's in the correct order
ehazlett      ok
nicksergeant  I disprove of the use of my name in this horrific story
nicksergeant  disapprove*
ehazlett      haha
sjl           lol
sjl           be glad you're not the one typing it out :P
nicksergeant  :)
sjl           though this keyboard sounds so awesome I don't really mind at all
nicksergeant  lol
nicksergeant  don't forget to add the part where Nick drinks a fifth of vodka and runs his car into a ditch intentionally
sjl           ehazlett: ok, 2m break and then we'll dive further into the rabbit hole
* ehazlett thx sjl for typing and apologizes to his neighbors for the extra machine gun
ehazlett      sjl, sure thx!
nicksergeant  lol
nicksergeant  sjl: what keyboard did you switch to this week?
sjl           nicksergeant: a unicomp one
sjl           (buckling spring like the original model Ms)
nicksergeant  sjl: one of these? http://pckeyboard.com/page/category/UKBD
sjl           yeah, the spacesaver m in white with blank keycaps
nicksergeant  sjl: I like the feel of the mechanicals but my office is right next to the bedroom and it's insanely loud when the wife is sleeping
sjl           heh
sjl           my cat doesn't seem to mind
nicksergeant  lol
nicksergeant  the profit margins on those kb's must be insanely high
nicksergeant  they look like they were constructed in the 40s
nicksergeant  and they're selling for $100
sjl           I dunno, the keys are pretty complicated
sjl           there's an individual spring in each key
nicksergeant  ah
sjl           ehazlett: ok, ready to continue?
ehazlett      sjl: yep
sjl           ok, so the next important thing to understand is that South migrations aren't quite as simple as I've been making them out to be
ehazlett      ok
sjl           there are actually three parts to each migration
sjl           the first two I mentioned
sjl           1. a description of how to go "forwards" from the previous DB state to the new one
sjl           2. a description of how to go "backwards" from the new state to the previous one
sjl           but there's one more thing that's included in the file
sjl           3. a "frozen" state definition
sjl           this is a Python dictionary that describes the models as they stand *after* the forwards migration
ehazlett      ok
sjl           so each migration not only includes forwards/backwards definitions, but also a full description of what the tables look like after that forwards part takes effect
sjl           you can think of it kind of like a checksum -- "this is what the DB should look like after the forwards part is done"
sjl           does that make sense?
ehazlett      yep
sjl           ok, so now think back to our 9 10 11S 11N example
sjl           let's assume we migrate backwards to 10 on the server and then migrate forwards to get all the stuff applied in the correct order
ehazlett      ok
sjl           is there anything wrong with this scenario?
ehazlett      doesn't seem to be but i'm not sure what south uses those "checksum dicts" for
ehazlett      if it compares the state before/after it would fail
sjl           let's assume my migration added a field "foo" and nick's added a field "bar"
sjl           and let's say the model had fields "a" and "b" to start with
ehazlett      ok
sjl           so 10's frozen state would say something like "model: a b"
sjl           11S's frozen state would say "model: a b foo"
sjl           what would 11N say?
ehazlett      model: a b bar
sjl           right
sjl           and what does the database *actually* look like?
ehazlett      model: a b -- right? at 10?
sjl           yep
sjl           and when you migrate forwards to apply 11s and 11n?
sjl           assuming 11s gets ordered before 11n alphabetically
ehazlett      assuming model: a b foo bar
sjl           right
sjl           but now the frozen state in the current migration says a b bar
ehazlett      ugh
sjl           in this particular case, we could probably slip this by without too much trouble, since we're just adding fields
sjl           but things get a lot more complicated when you're removing or renaming columns
ehazlett      sjl: ok
sjl           ehazlett: ok, so we've now hit a roadblock and are in a place where we don't want to be.  to fix this we need to rewind time to the point where Nick has just pushed his migration to github and deployed it to the server
ehazlett      sjl: ok
sjl           ehazlett: so now the server has 9 10 11n and my machine has 9 10 11s
ehazlett      ok
sjl           now I pull from GitHub and notice that I've got two 11's: 11N and 11S
sjl           at this point it's my responsibility to stop and fix this mess before moving on, otherwise we're going to hit the brokenness
sjl           any guesses on how I should do that?
ehazlett      hmm no -- renaming the file wouldn't have the right frozen state, right?
ehazlett      re-do the migration?
sjl           renaming the file would still have the incorrect frozen state, that's right
sjl           re-do how?
ehazlett      or i guess you could manually edit the frozen state
ehazlett      if it's just a dict
sjl           ok, but *which* one would you manually edit?
sjl           remember that I currently have 9 10 11S applied, and now I just pulled in 11N
ehazlett      11N since it's been deployed?
sjl           if it's been deployed, you probably don't want to change it
ehazlett      ah yeah
sjl           11S only exists on my laptop, so it's the safest one to modify, right?
ehazlett      yep
sjl           ok, so let's say you can manually edit the dict to contain the correct state (you usually don't want to do this but for now that's fine)
sjl           what exactly do you need to do?
sjl           1. edit the state dict
sjl           what's the next step?
* ehazlett blank stare
sjl           heh
sjl           let's assume we want to avoid having migrations with the same number, because it's confusing as hell
sjl           having each migration have its own unique number makes it really easy to tell what goes where
ehazlett      yeah
sjl           so the obvious next step is to rename 11S to 12, right?
ehazlett      yeah
sjl           except that's going to break something
ehazlett      dammit
sjl           heh
sjl           think about what south thinks is applied on my local database
sjl           9 10 11S
sjl           if I rename the file with 11S to 12, what happens when I try to migrate?
ehazlett      it would try to re-run it?
sjl           in theory, yes, it would try to run 11N (which is fine) and then 12, but 12 has already been applied
ehazlett      yep
sjl           but in practice south says "wait a second, there's this 11S thing in the database, and I can't find a file on disk for that.  wtf?"
sjl           so what do we need to do?
sjl           (note: at this point "sqlreset the whole fucking thing and get on with your life" is a valid answer, but let's go through the "right" way just for the exercise)
ehazlett      lol
ehazlett      migrate 10, then migrate?
sjl           OK, so what does south do when you say "migrate 10"
ehazlett      shit
sjl           it looks at the current stack of "applied" migrations: 9 10 11S
sjl           then what?
ehazlett      it would try to go backwards from 11s right?
sjl           yes
ehazlett      dammit
sjl           and where is the information on how to go backwards stored?
ehazlett      so migrate back 10, then rename?
sjl           ding ding ding
ehazlett      what a CF
sjl           you have to migrate backwards *first* so that the backwards instructions are in the correct locations
sjl           *then* rename your file to sit on top of the ones you've pulled
sjl           *then* migrate forwards again
ehazlett      got it
sjl           it has to be exactly in that order, as you've seen
ehazlett      yeah
sjl           also you need to make sure the frozen state is correct for every case
ehazlett      this is a nightmare
sjl           sometimes it's easy to manually edit the dicts, other times it might be easier to just recreate the migration from scratch
ehazlett      got it
sjl           heh, we're still not all the way down the rabbit hole
ehazlett      ha
sjl           the final bit of pain is when you throw git branches into the mix
sjl           let's assume that staging is at migration 10, and we want to add a new migration on dev
sjl           no problem, we add migration 11, push to github, etc etc
ehazlett      oh in which they exist at a certain point but then disappear when switching ?
ehazlett      k
sjl           now say we need to fix a ticket on staging.  how do you get your local machine back in the "staging" state?
ehazlett      well i would nuke the db and re-sync from staging
sjl           haha
ehazlett      but that's probably not what you were looking for ;)
sjl           yeah, let's do it the right way for now
ehazlett      migrate back to the common migration?
sjl           before or after you git checkout staging?
ehazlett      before to get before the new migrations
sjl           right
sjl           those migrations need to exist because you need the backwards parts of them
sjl           now
ehazlett      yep
sjl           what happens when, in order to solve your staging ticket, you need to create a migration?
ehazlett      well shit -- vodka?
ehazlett      jk
ehazlett      create the migration on staging with next number, push, merge to dev and re-do your local migration as before?
ehazlett      er wait, that wouldn't work if it's been pushed / deployed
sjl           heh
sjl           good, good, let the hate flow through you
ehazlett      ffs
honza       --vodka?
ehazlett      i'm full of hate now
sjl           heh
ehazlett      i don't know -- so dev is now at 11S and pushed ; staging is at 12 and pushed
ehazlett      wtf
sjl           at this point there's no simple solution
sjl           what you need to do is rename/modify the migrations that are on the dev branch to sit on top of the new one from the staging branch, as before
ehazlett      yeah because if you create a new migration on dev it would presumably fuck up staging
ehazlett      but what if they've been pushed?
sjl           but you're also going to need to tell everyone else about it, and fix it on the server, by migrating back to the common point and then forwards like you need to do locally
ehazlett      got it
sjl           but remember
sjl           in order to migrate backwards...?
ehazlett      you have to do it before touching the current migrations
sjl           right
sjl           so they'll need to migrate backwards to a common point (which you should probably tell them instead of making them look it up), THEN pull your changes, THEN migrate forwards
sjl           in that order
sjl           and the same thing needs to happen on the server
sjl           BUT, when you merge in the other direction, dev -> staging, everything will go cleanly and smoothly
sjl           and the same for staging -> prod
ehazlett      got it
ehazlett      and you still have to rename as before right?
sjl           yes
ehazlett      in order to keep staging happy
ehazlett      got it
sjl           exactly
sjl           so there you have it
ehazlett      awesome
sjl           there are some other painful edge cases having to do with data migrations, but you probably don't need to know about those (yet)
sjl           so let's stop there
sjl           untangle your brain
ehazlett      i'm sure i'll have questions, but thanks a ton!
ehazlett      exactly
sjl           so now you should be able to understand your original problem

gistfile2.md