Create a gist now

Instantly share code, notes, and snippets.

What would you like to do?
ehazlett sjl: hey have a sec?
sjl ehazlett: sure
ehazlett sjl: i'm trying to migrate the old testdata instance to the latest production
sjl ok
ehazlett sjl: when i try to migrate i'm getting the "ghost migrations" error
sjl fun
ehazlett sjl:
ehazlett sjl: sorry for the terrible formatting
sjl ehazlett: what exactly are you trying to do again?
sjl the production branch doesn't have team migrations 16+
ehazlett sjl: i'm trying to update a database from rev 646137e to the latest production (maggie's test instance)
sjl it looks like you're trying to take an instance that was running on the dev branch and migrate it to the production branch
ehazlett sjl: well shit
ehazlett sjl: figures
sjl what you'd need to do is migrate *backwards* to the last migration that's present in production first, *before* you change branches
sjl because after you switch the branch, south doesn't know how to undo those migrations, since they're not on disk any more
sjl (this is all assuming those migrations actually do contain the backwards migrations, but I'm usually pretty good at making sure we do that)
sjl databases are bullshit
ehazlett sjl: ugh
sjl yeah that's an accurate depiction of my thoughts too
nicksergeant let's just all save everything to plain text files
ehazlett LOL
nicksergeant need to login? upload your identity file and we'll try and find it on-disk
ehazlett ha
nicksergeant log in*
sjl ugh, stupid rochester, stop snowing
nicksergeant sjl: nevar!
ehazlett sjl: ok so i'm a south newb -- could you spare 5 min to educate me?
sjl ehazlett: sure
sjl ehazlett: so the basic idea is that a single "migration" describes a change in the schema of the database
sjl ehazlett: so let's say I add a "foo" field to a Django model
ehazlett sjl: yep
sjl ehazlett: the migration actually contains two parts, a "forward" part that says "add column foo to table whatever"
sjl ehazlett: and a "backward" part that says "remove column foo from table whatever"
ehazlett sjl: got it
sjl so it describes how to go from the original DB schema to the new one, and how to go from the new one back to the original
sjl each migration is stored in its own file on disk. that's those files in the migrations/ folder for each app
ehazlett sjl: yep i've seen those
sjl there are also things called data migrations. in general you use them to just modify data, but conceptually they're not any different than the normal ones
ehazlett sjl: ok
sjl all migrations take the DB from one state to another
sjl whether that means adding/removing a field, or capitalizing some data or something, doesn't matter
sjl DB + migration = new DB
sjl make sense so far?
ehazlett sjl: yep
sjl ok
sjl so normally this is pretty straightforward to work with. You have your local database, and you need to add a field to a model
sjl You do that by creating the migration. How exactly that happens isn't important right now.
ehazlett sjl: got it
sjl But you end up with a new file like
sjl and that file contains two parts: the forward part (that adds the field) and the backwards part
ehazlett sjl: ok
sjl Now, south itself also has a table in your database for its own data
sjl It stores the current "state" for each app in there
ehazlett sjl: ok
sjl so something like "the teams app is currently at migration 0114_..."
ehazlett sjl: gotcha
sjl when you tell south to migrate an app, it look at the current state of the DB, and it looks at the files in your migrations folder for that app
ehazlett sjl: ah and that's how it decides where it needs to be?
sjl so in our example, if you started out at migration 0115 and created 0116 to add your field, south would know that it needed to run the SQL for the forwards part of 0116
ehazlett sjl: got it
sjl and once that finished it would update its own table to record that the teams app is now at 0116
sjl now, you can also tell south to migrate to a specific target, instead of just "whatever is the newest"
ehazlett sjl: ok
sjl so you can say " migrate teams 0112" for example
sjl south will look in its table to see where it currently is (which is 0116 in our example)
sjl so it says okay, I'm at 116 and need to get to 112, so I need to go backwards
sjl so first it runs the backwards part of 0116
ehazlett sjl: ah ok -- i wondered how it went back
sjl then it runs the backwards part of 0115
sjl then it runs the backwards part of 0114
sjl then it runs the backwards part of 0113
sjl then it's done
ehazlett sjl: ah ok
sjl so far, so good
sjl right?
ehazlett sjl: yep
sjl ehazlett: ok, so this still is pretty straightforward. you do your local development, maybe creating a few migrations along the way
sjl let's say 0116 0117 and 0118
sjl now you push those to github, and as part of your deployment process you tell south to migrate to the latest point
sjl so south migrates your server or whatever up to 0118 and you're all good
sjl that's the way things are *supposed* to work
ehazlett :)
sjl ok, so where this gets tricky is when you have concurrent development happening
sjl i.e.: more that one line of development happening at the same time
sjl which may or may not involve git branching
sjl Let's use a single branch as an example for now
ehazlett ok
sjl let's say we're in a repo with just one master branch and that's it
sjl and nick and I are working on two different things
sjl let's say that in the morning when we start, there are 10 migrations for the app we're going to work on
sjl so our server knows it's at migration 10, and our local servers are at the same place
sjl so now we start working. nick creates a migration 11 to add a field foo on his local machine
sjl while he's doing that, I create a migration (also 11, because nick hasn't pushed his yet) to add field bar on my machine
sjl each of us tells south to migrate, and in both cases there's no issue (yet). south runs the sql for the forwards part of my 11 on my machine, and his 11 on his machine
ehazlett yep
sjl now let's say nick finishes first and pushes to github
sjl so the master branch on github now contains the first 10 migrations we started with, plus 11N(ick)
ehazlett start the chaos
sjl now when I pull from github, I get 11N
sjl so now my migrations folder contains 11S and 11N (plus the first 10, of course)
sjl understand so far?
ehazlett yep
sjl ok, so at this point there are two things that can happen with I try to run "python migrate myapp"
sjl the key point here is that south runs migrations in filename order
ehazlett yeah
sjl which normally is sane, because there's a number at the beginning of the filename
sjl 0001_... 0002_... etc
sjl but now that we have two with the same number, it's going to look at whatever the rest of the name happens to be to decide the order
sjl If I'm lucky, at this point South will explode when it tries to migrate. This would happen when Nick's file comes *before* mine
sjl Because my DB has ... 8 9 10 11S applied
ehazlett yeah
sjl and South looks at the files and sees that it should be ... 8 9 10 11N 11S
sjl it can't just apply the 11N on top of 11S because that would be in the wrong order
ehazlett but it would for 11S on nick's right?
sjl so at that point South will just throw up its hands and say "fuck it, I'm out, good luck"
sjl that make sense?
ehazlett yeah
sjl ok, so let's think about the opposite case: my filename happens to come before nick's
sjl so the order South thinks it needs is 8 9 10 11S 11N
sjl and my current database is at 8 9 10 11S
sjl in this case, South will happily just apply nick's migration on top of mine
ehazlett ok
sjl (just like any other new migration)
sjl this is the case that will make you drink
ehazlett lol
sjl because now everything seems fine
sjl but what happens when you push to the server?
ehazlett wouldn't it work also if it was at 10?
ehazlett 9 10 11S 11N
sjl yes
sjl but remember that nick pushed, and maybe deployed
ehazlett ah
sjl what would happen if he deployed?
ehazlett so it's fubar'd
sjl right
ehazlett 9 10 11N - no 11S
sjl now we're at the first case again, only this time it's the *server* that's borked
sjl (and Nick's machine once he pulls down the new migration)
ehazlett got it
ehazlett so does he have to migrate 10, then migrate?
sjl that's a way to manually fix things -- tell south to migrate back to 10 (which unapplies one of the 11's) and then tell it to migrate forwards to the latest, which will apply the 11's in the correct order
ehazlett ok
nicksergeant I disprove of the use of my name in this horrific story
nicksergeant disapprove*
ehazlett haha
sjl lol
sjl be glad you're not the one typing it out :P
nicksergeant :)
sjl though this keyboard sounds so awesome I don't really mind at all
nicksergeant lol
nicksergeant don't forget to add the part where Nick drinks a fifth of vodka and runs his car into a ditch intentionally
sjl ehazlett: ok, 2m break and then we'll dive further into the rabbit hole
* ehazlett thx sjl for typing and apologizes to his neighbors for the extra machine gun
ehazlett sjl, sure thx!
nicksergeant lol
nicksergeant sjl: what keyboard did you switch to this week?
sjl nicksergeant: a unicomp one
sjl (buckling spring like the original model Ms)
nicksergeant sjl: one of these?
sjl yeah, the spacesaver m in white with blank keycaps
nicksergeant sjl: I like the feel of the mechanicals but my office is right next to the bedroom and it's insanely loud when the wife is sleeping
sjl heh
sjl my cat doesn't seem to mind
nicksergeant lol
nicksergeant the profit margins on those kb's must be insanely high
nicksergeant they look like they were constructed in the 40s
nicksergeant and they're selling for $100
sjl I dunno, the keys are pretty complicated
sjl there's an individual spring in each key
nicksergeant ah
sjl ehazlett: ok, ready to continue?
ehazlett sjl: yep
sjl ok, so the next important thing to understand is that South migrations aren't quite as simple as I've been making them out to be
ehazlett ok
sjl there are actually three parts to each migration
sjl the first two I mentioned
sjl 1. a description of how to go "forwards" from the previous DB state to the new one
sjl 2. a description of how to go "backwards" from the new state to the previous one
sjl but there's one more thing that's included in the file
sjl 3. a "frozen" state definition
sjl this is a Python dictionary that describes the models as they stand *after* the forwards migration
ehazlett ok
sjl so each migration not only includes forwards/backwards definitions, but also a full description of what the tables look like after that forwards part takes effect
sjl you can think of it kind of like a checksum -- "this is what the DB should look like after the forwards part is done"
sjl does that make sense?
ehazlett yep
sjl ok, so now think back to our 9 10 11S 11N example
sjl let's assume we migrate backwards to 10 on the server and then migrate forwards to get all the stuff applied in the correct order
ehazlett ok
sjl is there anything wrong with this scenario?
ehazlett doesn't seem to be but i'm not sure what south uses those "checksum dicts" for
ehazlett if it compares the state before/after it would fail
sjl let's assume my migration added a field "foo" and nick's added a field "bar"
sjl and let's say the model had fields "a" and "b" to start with
ehazlett ok
sjl so 10's frozen state would say something like "model: a b"
sjl 11S's frozen state would say "model: a b foo"
sjl what would 11N say?
ehazlett model: a b bar
sjl right
sjl and what does the database *actually* look like?
ehazlett model: a b -- right? at 10?
sjl yep
sjl and when you migrate forwards to apply 11s and 11n?
sjl assuming 11s gets ordered before 11n alphabetically
ehazlett assuming model: a b foo bar
sjl right
sjl but now the frozen state in the current migration says a b bar
ehazlett ugh
sjl in this particular case, we could probably slip this by without too much trouble, since we're just adding fields
sjl but things get a lot more complicated when you're removing or renaming columns
ehazlett sjl: ok
sjl ehazlett: ok, so we've now hit a roadblock and are in a place where we don't want to be. to fix this we need to rewind time to the point where Nick has just pushed his migration to github and deployed it to the server
ehazlett sjl: ok
sjl ehazlett: so now the server has 9 10 11n and my machine has 9 10 11s
ehazlett ok
sjl now I pull from GitHub and notice that I've got two 11's: 11N and 11S
sjl at this point it's my responsibility to stop and fix this mess before moving on, otherwise we're going to hit the brokenness
sjl any guesses on how I should do that?
ehazlett hmm no -- renaming the file wouldn't have the right frozen state, right?
ehazlett re-do the migration?
sjl renaming the file would still have the incorrect frozen state, that's right
sjl re-do how?
ehazlett or i guess you could manually edit the frozen state
ehazlett if it's just a dict
sjl ok, but *which* one would you manually edit?
sjl remember that I currently have 9 10 11S applied, and now I just pulled in 11N
ehazlett 11N since it's been deployed?
sjl if it's been deployed, you probably don't want to change it
ehazlett ah yeah
sjl 11S only exists on my laptop, so it's the safest one to modify, right?
ehazlett yep
sjl ok, so let's say you can manually edit the dict to contain the correct state (you usually don't want to do this but for now that's fine)
sjl what exactly do you need to do?
sjl 1. edit the state dict
sjl what's the next step?
* ehazlett blank stare
sjl heh
sjl let's assume we want to avoid having migrations with the same number, because it's confusing as hell
sjl having each migration have its own unique number makes it really easy to tell what goes where
ehazlett yeah
sjl so the obvious next step is to rename 11S to 12, right?
ehazlett yeah
sjl except that's going to break something
ehazlett dammit
sjl heh
sjl think about what south thinks is applied on my local database
sjl 9 10 11S
sjl if I rename the file with 11S to 12, what happens when I try to migrate?
ehazlett it would try to re-run it?
sjl in theory, yes, it would try to run 11N (which is fine) and then 12, but 12 has already been applied
ehazlett yep
sjl but in practice south says "wait a second, there's this 11S thing in the database, and I can't find a file on disk for that. wtf?"
sjl so what do we need to do?
sjl (note: at this point "sqlreset the whole fucking thing and get on with your life" is a valid answer, but let's go through the "right" way just for the exercise)
ehazlett lol
ehazlett migrate 10, then migrate?
sjl OK, so what does south do when you say "migrate 10"
ehazlett shit
sjl it looks at the current stack of "applied" migrations: 9 10 11S
sjl then what?
ehazlett it would try to go backwards from 11s right?
sjl yes
ehazlett dammit
sjl and where is the information on how to go backwards stored?
ehazlett so migrate back 10, then rename?
sjl ding ding ding
ehazlett what a CF
sjl you have to migrate backwards *first* so that the backwards instructions are in the correct locations
sjl *then* rename your file to sit on top of the ones you've pulled
sjl *then* migrate forwards again
ehazlett got it
sjl it has to be exactly in that order, as you've seen
ehazlett yeah
sjl also you need to make sure the frozen state is correct for every case
ehazlett this is a nightmare
sjl sometimes it's easy to manually edit the dicts, other times it might be easier to just recreate the migration from scratch
ehazlett got it
sjl heh, we're still not all the way down the rabbit hole
ehazlett ha
sjl the final bit of pain is when you throw git branches into the mix
sjl let's assume that staging is at migration 10, and we want to add a new migration on dev
sjl no problem, we add migration 11, push to github, etc etc
ehazlett oh in which they exist at a certain point but then disappear when switching ?
ehazlett k
sjl now say we need to fix a ticket on staging. how do you get your local machine back in the "staging" state?
ehazlett well i would nuke the db and re-sync from staging
sjl haha
ehazlett but that's probably not what you were looking for ;)
sjl yeah, let's do it the right way for now
ehazlett migrate back to the common migration?
sjl before or after you git checkout staging?
ehazlett before to get before the new migrations
sjl right
sjl those migrations need to exist because you need the backwards parts of them
sjl now
ehazlett yep
sjl what happens when, in order to solve your staging ticket, you need to create a migration?
ehazlett well shit -- vodka?
ehazlett jk
ehazlett create the migration on staging with next number, push, merge to dev and re-do your local migration as before?
ehazlett er wait, that wouldn't work if it's been pushed / deployed
sjl heh
sjl good, good, let the hate flow through you
ehazlett ffs
honza --vodka?
ehazlett i'm full of hate now
sjl heh
ehazlett i don't know -- so dev is now at 11S and pushed ; staging is at 12 and pushed
ehazlett wtf
sjl at this point there's no simple solution
sjl what you need to do is rename/modify the migrations that are on the dev branch to sit on top of the new one from the staging branch, as before
ehazlett yeah because if you create a new migration on dev it would presumably fuck up staging
ehazlett but what if they've been pushed?
sjl but you're also going to need to tell everyone else about it, and fix it on the server, by migrating back to the common point and then forwards like you need to do locally
ehazlett got it
sjl but remember
sjl in order to migrate backwards...?
ehazlett you have to do it before touching the current migrations
sjl right
sjl so they'll need to migrate backwards to a common point (which you should probably tell them instead of making them look it up), THEN pull your changes, THEN migrate forwards
sjl in that order
sjl and the same thing needs to happen on the server
sjl BUT, when you merge in the other direction, dev -> staging, everything will go cleanly and smoothly
sjl and the same for staging -> prod
ehazlett got it
ehazlett and you still have to rename as before right?
sjl yes
ehazlett in order to keep staging happy
ehazlett got it
sjl exactly
sjl so there you have it
ehazlett awesome
sjl there are some other painful edge cases having to do with data migrations, but you probably don't need to know about those (yet)
sjl so let's stop there
sjl untangle your brain
ehazlett i'm sure i'll have questions, but thanks a ton!
ehazlett exactly
sjl so now you should be able to understand your original problem
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment