Created
June 13, 2016 12:27
-
-
Save bjeanes/808926b096b38e30f026bff40e8cd355 to your computer and use it in GitHub Desktop.
Incomplete script to generate schema hints for 18F's rdbms-subsetter tool. However, when running their tool, I kept getting Python exceptions that implied the dependent libraries couldn't deal with constraints in our DB schema, so I tabled it for now.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#!/usr/bin/env ruby | |
############################################################################### | |
# | |
# Uses https://github.com/18F/rdbms-subsetter to pull a subset of data from the | |
# production database. Because Rails databases rarely define foreign keys, this | |
# inspects the models for in-Ruby declared associations and generates a config | |
# file for `rdbms-subsetter` to use for understanding the schema. | |
# | |
############################################################################### | |
abort "Only use this in development mode" if ENV.fetch('RAILS_ENV', 'development') != 'development' | |
abort "`rdbms-subsetter` required; install with `pip install rdbms-subsetter`" unless system("which rdbms-subsetter >/dev/null") | |
require_relative "../config/environment" | |
# Hash of table names to column hashes | |
# | |
SCHEMA = Hash.new do |tables, table| | |
# Each column hash is of a column name (foreign key) to a target table+column | |
# names tuple. | |
tables[table] = {} | |
end | |
# Iterate over all known models configuration and define association column | |
# relationships based on it. | |
Rails.application.eager_load! # make sure all models are loaded | |
ActiveRecord::Base.descendants.each do |model_class| | |
model_class.reflections.each do |name, reflection| | |
next unless ActiveRecord::Reflection::AssociationReflection === reflection | |
next unless reflection.macro == :belongs_to | |
next if reflection.options[:polymorphic] | |
target = [reflection.table_name, reflection.association_primary_key] | |
SCHEMA[model_class.table_name][reflection.foreign_key] = target | |
end | |
end | |
subsetter_config = {} | |
subsetter_config[:constraints] = Hash[SCHEMA.map do |table, columns| | |
[table, columns.flat_map do |col, target| | |
{ | |
referred_schema: nil, | |
referred_table: target[0], | |
referred_columns: [target[1]], | |
constrained_columns: [col] | |
} | |
end] | |
end] | |
config_file = Rails.root.join("config/rdbms-subsetter.json") | |
File.open(config_file, "w") do |f| | |
f.write subsetter_config.to_json | |
end | |
source = `heroku config:get DATABASE_URL`.chomp | |
destination = "postgresql:///prod" | |
exec(["rdbms-subsetter", "-l", "--config", config_file, source, destination, "0.4"]) |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Exception I'm getting when
rdbms-subsetter
actually runs:I haven't looked into what table/index/column is tripping it up yet, though.