Skip to content

Instantly share code, notes, and snippets.

@MaxGhenis

MaxGhenis/bruenig_poverty_gap.ipynb

Created September 18, 2019 18:52
Show Gist options
  • Star 0 You must be signed in to star a gist
  • Fork 0 You must be signed in to fork a gist
  • Save MaxGhenis/211301ddd9d64c8cce91c35dd0778a2a to your computer and use it in GitHub Desktop.
Save MaxGhenis/211301ddd9d64c8cce91c35dd0778a2a to your computer and use it in GitHub Desktop.
Download ZIP
bruenig_poverty_gap
Display the source blob
Display the rendered blob
Raw
bruenig_poverty_gap.ipynb
{
"nbformat": 4,
"nbformat_minor": 0,
"metadata": {
"colab": {
"name": "bruenig_poverty_gap",
"provenance": [],
"include_colab_link": true
},
"kernelspec": {
"name": "python3",
"display_name": "Python 3"
}
},
"cells": [
{
"cell_type": "markdown",
"metadata": {
"id": "view-in-github",
"colab_type": "text"
},
"source": [
"<a href=\"https://colab.research.google.com/gist/MaxGhenis/211301ddd9d64c8cce91c35dd0778a2a/bruenig_poverty_gap.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "rErT-j0wkbqr",
"colab_type": "text"
},
"source": [
"# Calculate SPM poverty gap\n",
"\n",
"Reproduce top-level poverty gap per Matt Bruenig's analysis, [\"The US Welfare State Cut Poverty by Two-Thirds in 2018\"](https://www.peoplespolicyproject.org/2019/09/16/the-us-welfare-state-cut-poverty-by-two-thirds-in-2018/).\n",
"\n",
"Guidance from Bruenig on calculating market income:\n",
"1. For each SPM Unit, sum up the SS_VAL, SSI_VAL, UC_VAL, VET_VAL, WC_VAL, and PAW_VAL for that unit. This means going through each person in the SPM unit and adding up those values to get the total amount for each SPM Unit. You'll notice these are variables that are not in the SPM_* genre. This is because these are the cash benefits that were historically counted towards the official poverty metric before SPM was introduced in 2009. SPM was thus supplemental to that (indeed it used to have its own separate file), which is why you'll see the SPM_* benefits only include non-cash benefits and tax credits.\n",
"\n",
"2. From there, do market_income = SPM_TOTVAL - SS_VAL - SSI_VAL - UC_VAL - VET_VAL - WC_VAL - PAW_VAL - SPM_CAPWKCCXPNS - SPM_MEDXPNS - SPM_CHILDSUPPD\n",
"\n",
"3. Now you can use market_income instead of SPM_RESOURCES in the poverty gap calculation."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "vUOUImwKk6tS",
"colab_type": "text"
},
"source": [
"## Download data\n",
"\n",
"From https://thedataweb.rm.census.gov/ftp/cps_ftp.html\n",
"\n",
"\"2019 Data File (CSV format, includes Household, Family, Person, Replicate Weight)\" row\n",
"\n",
"Uncheck if data is already downloaded and unzipped from a prior run."
]
},
{
"cell_type": "code",
"metadata": {
"id": "uRs0H4M6sTIQ",
"colab_type": "code",
"colab": {}
},
"source": [
"#@title Download data from Census?\n",
"download = False #@param {type:\"boolean\"}"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "zaHMmJRgk8PK",
"colab_type": "code",
"colab": {}
},
"source": [
"if download:\n",
" !wget http://thedataweb.rm.census.gov/pub/cps/march/asecpub19csv.zip\n",
" !unzip asecpub19csv.zip"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "OHVD1XZSkh73",
"colab_type": "text"
},
"source": [
"## Imports"
]
},
{
"cell_type": "code",
"metadata": {
"id": "FNajviAqkl9N",
"colab_type": "code",
"colab": {}
},
"source": [
"import pandas as pd\n",
"import numpy as np"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "kYdj27PFkngE",
"colab_type": "text"
},
"source": [
"## Load data"
]
},
{
"cell_type": "code",
"metadata": {
"id": "7BUWZdCQobxt",
"colab_type": "code",
"colab": {}
},
"source": [
"SPM_BASE_COLS = ['SPM_ID', 'SPM_POVTHRESHOLD', 'SPM_RESOURCES', 'SPM_WEIGHT',\n",
" 'SPM_TOTVAL']\n",
"SPM_TOTVAL_SUBTRACTIONS = ['SPM_CAPWKCCXPNS', 'SPM_MEDXPNS', 'SPM_CHILDSUPPD']\n",
"CASH_BENS = ['SS_VAL', 'SSI_VAL', 'UC_VAL', 'VET_VAL', 'WC_VAL', 'PAW_VAL']\n",
"\n",
"spm_cols = SPM_BASE_COLS + SPM_TOTVAL_SUBTRACTIONS"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "u6o_tO36oecx",
"colab_type": "code",
"colab": {}
},
"source": [
"p = pd.read_csv('pppub19.csv',\n",
" usecols=spm_cols + CASH_BENS)"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "6XUCRJxVowRm",
"colab_type": "text"
},
"source": [
"## Preprocess\n",
"\n",
"Create SPM unit dataset. Copy to avoid `SettingWithCopyWarning` later."
]
},
{
"cell_type": "code",
"metadata": {
"id": "P7LbPuUBoxgN",
"colab_type": "code",
"colab": {}
},
"source": [
"spmu_base = p[spm_cols].drop_duplicates().copy()"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "lADVkGsdpagD",
"colab_type": "text"
},
"source": [
"Sum up cash benefits by SPM unit."
]
},
{
"cell_type": "code",
"metadata": {
"id": "jWV2nZbQpusS",
"colab_type": "code",
"colab": {}
},
"source": [
"spmu_cash_bens = p.groupby('SPM_ID')[CASH_BENS].sum().reset_index()"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "L94vmMVQp2hb",
"colab_type": "text"
},
"source": [
"Merge back to `spmu`."
]
},
{
"cell_type": "code",
"metadata": {
"id": "0jsML27-p53F",
"colab_type": "code",
"colab": {}
},
"source": [
"spmu = spmu_base.merge(spmu_cash_bens, on='SPM_ID')"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "I20ptIkvqOrC",
"colab_type": "text"
},
"source": [
"Calculate market income."
]
},
{
"cell_type": "code",
"metadata": {
"id": "srSlIiXWqQ7P",
"colab_type": "code",
"colab": {}
},
"source": [
"spmu['market_income'] = (spmu.SPM_TOTVAL - \n",
" spmu[CASH_BENS + SPM_TOTVAL_SUBTRACTIONS].sum(axis=1))"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "usdZM2vd3l4K",
"colab_type": "text"
},
"source": [
"## Checks\n",
"\n",
"Is this unique per SPM unit?"
]
},
{
"cell_type": "code",
"metadata": {
"id": "r7cM9XTS339O",
"colab_type": "code",
"colab": {}
},
"source": [
"assert spmu.SPM_ID.nunique() == spmu.shape[0]"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "0zP8B4UksptT",
"colab_type": "text"
},
"source": [
"Are there fewer rows than in the per-person table?"
]
},
{
"cell_type": "code",
"metadata": {
"id": "EUuY3viqpbYM",
"colab_type": "code",
"colab": {}
},
"source": [
"assert spmu.shape[0] < p.shape[0]"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "lu0Njx_6mpcG",
"colab_type": "text"
},
"source": [
"## Calculate poverty gap\n",
"\n",
"Based on market and disposable income."
]
},
{
"cell_type": "code",
"metadata": {
"id": "WB1wmVSbtRsH",
"colab_type": "code",
"colab": {}
},
"source": [
"def poverty_gap(df, col):\n",
" return np.maximum(df.SPM_POVTHRESHOLD - df[col], 0)"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "VSD_gaSVrCtv",
"colab_type": "code",
"colab": {}
},
"source": [
"spmu['market_gap'] = poverty_gap(spmu, 'market_income')\n",
"spmu['disposable_gap'] = poverty_gap(spmu, 'SPM_RESOURCES')\n",
"# Weight is multiplied by 100.\n",
"spmu['market_gap_m'] = spmu.market_gap * (spmu.SPM_WEIGHT / 100) / 1e6\n",
"spmu['disposable_gap_m'] = spmu.disposable_gap * (spmu.SPM_WEIGHT / 100) / 1e6"
],
"execution_count": 0,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "j0TExKFep-WZ",
"colab_type": "text"
},
"source": [
"Total gap in billions."
]
},
{
"cell_type": "code",
"metadata": {
"id": "o78Rtl9xpqho",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
},
"outputId": "98afae64-3a42-44da-ac9e-2826add23eab"
},
"source": [
"total_market_pov_gap_m = spmu.market_gap_m.sum()\n",
"print('Total market income poverty gap: $' + \n",
" str((total_market_pov_gap_m / 1e3).round(1)) + 'B.')"
],
"execution_count": 14,
"outputs": [
{
"output_type": "stream",
"text": [
"Total market income poverty gap: $511.7B.\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "YMHkQJJVrJgz",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 35
},
"outputId": "c1b7a3fe-1fe0-47e1-e562-1ca15176073c"
},
"source": [
"total_disposable_pov_gap_m = spmu.disposable_gap_m.sum()\n",
"print('Total disposable income poverty gap: $' + \n",
" str((total_disposable_pov_gap_m / 1e3).round(1)) + 'B.')"
],
"execution_count": 15,
"outputs": [
{
"output_type": "stream",
"text": [
"Total disposable income poverty gap: $172.6B.\n"
],
"name": "stdout"
}
]
}
]
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment