public
Last active

Pull in US Economic data from IMF database, pull out some columns and save as tab-delimited file. Updating to clarify the extraction algo and adding the write data algo.

  • Download Gist
munge_data.rb
Ruby
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36
require 'csv'
 
module MungeData
 
# Open a tab-delimited data file converting it into a CSV structure,
# expecting the first row to be headers,
# passing the table to map each row of the table,
# converting each row to a hash and then selecting only
# those items in the array that match the characteristics in the block
def self.extract_years(censusdata, y_start, y_end)
CSV.table(censusdata, col_sep: "\t").map do |r|
r.to_hash.select do |k,v|
true if (k =~ /^#{y_start}$/ .. k =~ /^#{y_end}/) || # a flip-flop that pulls keys from y_start to y_end
k =~ /^subject_descriptor$/ || # the subject descriptor field
k =~ /^weo_subject_code$/ # and the weo_subject_code
end
end
end
 
# Open a tab-delimited data file for writing,
# Creating the header row from the keys of the first row of data.
# Write all the rows from the data table using just the values.
def self.write_tdv(outfile,data)
CSV.open(outfile,'wb', col_sep: "\t") do |csv_file|
csv_file << data.first.keys # write the header row
data.each do |row|
csv_file << row.values # write the data row
end
end
end
 
end
 
 
us_eco = MungeData.extract_years("weoreptc-us-all.xls","1980","2010")
MungeData.write_tdv("outfile.txt",us_eco)
outfile.txt
1 2 3 4 5 6 7 8 9 10
weo_subject_code subject_descriptor 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
NGDP_R Gross domestic product, constant prices 5,833.975 5,982.075 5,865.925 6,130.925 6,571.525 6,843.400 7,080.500 7,307.050 7,607.400 7,879.175 8,027.025 8,008.325 8,280.025 8,516.175 8,863.125 9,085.975 9,425.850 9,845.925 10,274.750 10,770.625 11,216.425 11,337.475 11,543.100 11,836.425 12,246.925 12,622.950 12,958.475 13,206.375 13,161.925 12,757.950 13,062.975
NGDP_RPCH Gross domestic product, constant prices -0.275 2.539 -1.942 4.518 7.187 4.137 3.465 3.2 4.11 3.573 1.876 -0.233 3.393 2.852 4.074 2.514 3.741 4.457 4.355 4.826 4.139 1.079 1.814 2.541 3.468 3.07 2.658 1.913 -0.337 -3.069 2.391
NGDP Gross domestic product, current prices 2,788.150 3,126.850 3,253.175 3,534.600 3,930.925 4,217.475 4,460.050 4,736.350 5,100.425 5,482.125 5,800.525 5,992.100 6,342.300 6,667.325 7,085.150 7,414.625 7,838.475 8,332.350 8,793.475 9,353.500 9,951.475 10,286.175 10,642.300 11,142.225 11,853.250 12,622.950 13,377.200 14,028.675 14,291.550 13,973.650 14,498.925
NGDPD Gross domestic product, current prices 2,788.150 3,126.850 3,253.175 3,534.600 3,930.925 4,217.475 4,460.050 4,736.350 5,100.425 5,482.125 5,800.525 5,992.100 6,342.300 6,667.325 7,085.150 7,414.625 7,838.475 8,332.350 8,793.475 9,353.500 9,951.475 10,286.175 10,642.300 11,142.225 11,853.250 12,622.950 13,377.200 14,028.675 14,291.550 13,973.650 14,498.925
NGDP_D Gross domestic product, deflator 47.792 52.27 55.459 57.652 59.818 61.628 62.991 64.819 67.046 69.577 72.262 74.823 76.598 78.29 79.94 81.605 83.159 84.627 85.583 86.843 88.722 90.727 92.196 94.135 96.786 100.0 103.231 106.227 108.583 109.529 110.993
NGDPRPC Gross domestic product per capita, constant prices 25,630.111 26,018.562 25,270.235 26,172.963 27,809.240 28,703.671 29,429.409 30,101.040 31,054.732 31,862.611 32,102.011 31,604.511 32,245.955 32,736.084 33,658.555 34,099.067 34,964.859 36,089.179 37,224.343 38,577.439 39,732.863 39,750.499 40,077.564 40,712.308 41,743.018 42,628.552 43,349.542 43,743.745 43,193.789 41,506.310 42,175.090
NGDPPC Gross domestic product per capita, current prices 12,249.040 13,599.987 14,014.584 15,089.233 16,634.805 17,689.601 18,537.764 19,511.165 20,820.823 22,169.176 23,197.700 23,647.566 24,699.626 25,629.125 26,906.527 27,826.601 29,076.547 30,541.333 31,857.839 33,501.684 35,251.927 36,064.520 36,949.993 38,324.553 40,401.197 42,628.552 44,750.288 46,467.466 46,900.905 45,461.429 46,811.195
NGDPDPC Gross domestic product per capita, current prices 12,249.040 13,599.987 14,014.584 15,089.233 16,634.805 17,689.601 18,537.764 19,511.165 20,820.823 22,169.176 23,197.700 23,647.566 24,699.626 25,629.125 26,906.527 27,826.601 29,076.547 30,541.333 31,857.839 33,501.684 35,251.927 36,064.520 36,949.993 38,324.553 40,401.197 42,628.552 44,750.288 46,467.466 46,900.905 45,461.429 46,811.195
NGAP_NPGDP Output gap in percent of potential GDP -1.997 -1.117 -5.788 -4.386 -0.514 0.158 0.216 0.295 1.304 1.866 0.747 -2.16 -1.37 -1.156 0.067 -0.292 0.347 1.607 2.494 3.753 4.356 1.591 0.138 -0.078 0.546 1.348 1.787 1.409 -0.939 -5.559 -4.475
weoreptc-us-all.xls
1 2 3 4 5 6 7 8 9 10
WEO Country Code ISO WEO Subject Code Country Subject Descriptor Subject Notes Units Scale Country/Series-specific Notes 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 Estimates Start After
111 USA NGDP_R United States Gross domestic product, constant prices Expressed in billions of national currency units; the base year is country-specific. Expenditure-based GDP is total final expenditures at purchasers? prices (including the f.o.b. value of exports of goods and services), less the f.o.b. value of imports of goods and services. [SNA 1993] National currency Billions Source: Haver Analytics Latest actual data: 2011 GDP valuation: Real Gross Domestic Product determined by chained Fisher quantity growth rates. Start/end months of reporting year: January/December Base year: 2005 Chain-weighted: Yes, from 1980 Primary domestic currency: U.S. dollars Data last updated: 09/2012 5,833.975 5,982.075 5,865.925 6,130.925 6,571.525 6,843.400 7,080.500 7,307.050 7,607.400 7,879.175 8,027.025 8,008.325 8,280.025 8,516.175 8,863.125 9,085.975 9,425.850 9,845.925 10,274.750 10,770.625 11,216.425 11,337.475 11,543.100 11,836.425 12,246.925 12,622.950 12,958.475 13,206.375 13,161.925 12,757.950 13,062.975 13,299.100 13,587.652 13,875.222 14,282.756 14,762.231 15,265.684 15,773.899 2011
111 USA NGDP_RPCH United States Gross domestic product, constant prices Annual percentages of constant price GDP are year-on-year changes; the base year is country-specific . Expenditure-based GDP is total final expenditures at purchasers? prices (including the f.o.b. value of exports of goods and services), less the f.o.b. value of imports of goods and services. [SNA 1993] Percent change See notes for: Gross domestic product, constant prices (National currency). -0.275 2.539 -1.942 4.518 7.187 4.137 3.465 3.200 4.110 3.573 1.876 -0.233 3.393 2.852 4.074 2.514 3.741 4.457 4.355 4.826 4.139 1.079 1.814 2.541 3.468 3.070 2.658 1.913 -0.337 -3.069 2.391 1.808 2.170 2.116 2.937 3.357 3.410 3.329 2011
111 USA NGDP United States Gross domestic product, current prices Expressed in billions of national currency units . Expenditure-based GDP is total final expenditures at purchasers? prices (including the f.o.b. value of exports of goods and services), less the f.o.b. value of imports of goods and services. [SNA 1993] National currency Billions Source: Haver Analytics Latest actual data: 2011 GDP valuation: Real Gross Domestic Product determined by chained Fisher quantity growth rates. Start/end months of reporting year: January/December Base year: 2005 Chain-weighted: Yes, from 1980 Primary domestic currency: U.S. dollars Data last updated: 09/2012 2,788.150 3,126.850 3,253.175 3,534.600 3,930.925 4,217.475 4,460.050 4,736.350 5,100.425 5,482.125 5,800.525 5,992.100 6,342.300 6,667.325 7,085.150 7,414.625 7,838.475 8,332.350 8,793.475 9,353.500 9,951.475 10,286.175 10,642.300 11,142.225 11,853.250 12,622.950 13,377.200 14,028.675 14,291.550 13,973.650 14,498.925 15,075.675 15,653.366 16,197.956 16,912.540 17,768.397 18,717.045 19,745.323 2011
111 USA NGDPD United States Gross domestic product, current prices Values are based upon GDP in national currency converted to U.S. dollars using market exchange rates (yearly average). Exchange rate projections are provided by country economists for the group of other emerging market and developing countries. Exchanges rates for advanced economies are established in the WEO assumptions for each WEO exercise. Expenditure-based GDP is total final expenditures at purchasers? prices (including the f.o.b. value of exports of goods and services), less the f.o.b. value of imports of goods and services. [SNA 1993] U.S. dollars Billions See notes for: Gross domestic product, current prices (National currency). 2,788.150 3,126.850 3,253.175 3,534.600 3,930.925 4,217.475 4,460.050 4,736.350 5,100.425 5,482.125 5,800.525 5,992.100 6,342.300 6,667.325 7,085.150 7,414.625 7,838.475 8,332.350 8,793.475 9,353.500 9,951.475 10,286.175 10,642.300 11,142.225 11,853.250 12,622.950 13,377.200 14,028.675 14,291.550 13,973.650 14,498.925 15,075.675 15,653.366 16,197.956 16,912.540 17,768.397 18,717.045 19,745.323 2011
111 USA NGDP_D United States Gross domestic product, deflator The GDP deflator is derived by dividing current price GDP by constant price GDP and is considered to be an alternate measure of inflation. Data are expressed in the base year of each country's national accounts. Index See notes for: Gross domestic product, constant prices (National currency) Gross domestic product, current prices (National currency). 47.792 52.270 55.459 57.652 59.818 61.628 62.991 64.819 67.046 69.577 72.262 74.823 76.598 78.290 79.940 81.605 83.159 84.627 85.583 86.843 88.722 90.727 92.196 94.135 96.786 100.000 103.231 106.227 108.583 109.529 110.993 113.359 115.203 116.740 118.412 120.364 122.609 125.177 2011
111 USA NGDPRPC United States Gross domestic product per capita, constant prices GDP is expressed in constant national currency per person. Data are derived by dividing constant price GDP by total population. National currency Units See notes for: Gross domestic product, constant prices (National currency) Population (Persons). 25,630.111 26,018.562 25,270.235 26,172.963 27,809.240 28,703.671 29,429.409 30,101.040 31,054.732 31,862.611 32,102.011 31,604.511 32,245.955 32,736.084 33,658.555 34,099.067 34,964.859 36,089.179 37,224.343 38,577.439 39,732.863 39,750.499 40,077.564 40,712.308 41,743.018 42,628.552 43,349.542 43,743.745 43,193.789 41,506.310 42,175.090 42,632.722 43,229.954 43,735.131 44,593.896 45,654.996 46,765.495 47,865.349 2011
111 USA NGDPPC United States Gross domestic product per capita, current prices GDP is expressed in current national currency per person. Data are derived by dividing current price GDP by total population. National currency Units See notes for: Gross domestic product, current prices (National currency) Population (Persons). 12,249.040 13,599.987 14,014.584 15,089.233 16,634.805 17,689.601 18,537.764 19,511.165 20,820.823 22,169.176 23,197.700 23,647.566 24,699.626 25,629.125 26,906.527 27,826.601 29,076.547 30,541.333 31,857.839 33,501.684 35,251.927 36,064.520 36,949.993 38,324.553 40,401.197 42,628.552 44,750.288 46,467.466 46,900.905 45,461.429 46,811.195 48,327.861 49,802.148 51,056.461 52,804.658 54,952.135 57,338.527 59,916.496 2011
111 USA NGDPDPC United States Gross domestic product per capita, current prices GDP is expressed in current U.S. dollars per person. Data are derived by first converting GDP in national currency to U.S. dollars and then dividing it by total population. U.S. dollars Units See notes for: Gross domestic product, current prices (National currency) Population (Persons). 12,249.040 13,599.987 14,014.584 15,089.233 16,634.805 17,689.601 18,537.764 19,511.165 20,820.823 22,169.176 23,197.700 23,647.566 24,699.626 25,629.125 26,906.527 27,826.601 29,076.547 30,541.333 31,857.839 33,501.684 35,251.927 36,064.520 36,949.993 38,324.553 40,401.197 42,628.552 44,750.288 46,467.466 46,900.905 45,461.429 46,811.195 48,327.861 49,802.148 51,056.461 52,804.658 54,952.135 57,338.527 59,916.496 2011
111 USA NGAP_NPGDP United States Output gap in percent of potential GDP Output gaps for advanced economies are calculated as actual GDP less potential GDP as a percent of potential GDP. Estimates of output gaps are subject to a significant margin of uncertainty. For a discussion of approaches to calculating potential output, see Paula R. De Masi, IMF Estimates of Potential Output: Theory and Practice, in Staff Studies for the World Economic Outlook (Washington: IMF, December 1997), pp. 40-46. Percent of potential GDP See notes for: Gross domestic product, constant prices (National currency). -1.997 -1.117 -5.788 -4.386 -0.514 0.158 0.216 0.295 1.304 1.866 0.747 -2.160 -1.370 -1.156 0.067 -0.292 0.347 1.607 2.494 3.753 4.356 1.591 0.138 -0.078 0.546 1.348 1.787 1.409 -0.939 -5.559 -4.475 -4.377 -4.059 -3.967 -3.192 -2.170 -1.153 -0.209 2011

Please sign in to comment on this gist.

Something went wrong with that request. Please try again.