Created
July 17, 2020 14:06
-
-
Save sspboyd/25167f9f3e08a5bee71f70e7fe3ea303 to your computer and use it in GitHub Desktop.
UNArcTest Jupyter Notebook
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
{ | |
"cells": [ | |
{ | |
"cell_type": "markdown", | |
"metadata": {}, | |
"source": [ | |
"# List the unique names for the UN Agencies\n", | |
"This is an example of using pandas and jupyter notebooks for doing some simple data prep and cleaning. The script reads in several csv files of UN Agency Funding data and outputs a list of all the unique names used to name the agencies. " | |
] | |
}, | |
{ | |
"cell_type": "code", | |
"execution_count": 1, | |
"metadata": {}, | |
"outputs": [], | |
"source": [ | |
"import pandas as pd\n", | |
"# Set ipython's max row display\n", | |
"pd.set_option('display.max_row', 1000)\n", | |
"# Set iPython's max column width to 50\n", | |
"pd.set_option('display.max_columns', 20)" | |
] | |
}, | |
{ | |
"cell_type": "markdown", | |
"metadata": {}, | |
"source": [ | |
"### Load the csv files.\n", | |
"Thankfully, in this case all the data files came from the same source (UN) which means that the files have the same number of columns and they're named the exact same. " | |
] | |
}, | |
{ | |
"cell_type": "code", | |
"execution_count": 2, | |
"metadata": {}, | |
"outputs": [], | |
"source": [ | |
"df15 = pd.read_csv(r'TotalExpenditureByAgency-2015.csv')\n", | |
"df16 = pd.read_csv(r'TotalExpenditureByAgency-2016.csv')\n", | |
"df17 = pd.read_csv(r'TotalExpenditureByAgency-2017.csv')\n", | |
"df18 = pd.read_csv(r'TotalExpenditureByAgency-2018.csv')" | |
] | |
}, | |
{ | |
"cell_type": "markdown", | |
"metadata": {}, | |
"source": [ | |
"Create a single data frame with all the csv files." | |
] | |
}, | |
{ | |
"cell_type": "code", | |
"execution_count": 3, | |
"metadata": {}, | |
"outputs": [ | |
{ | |
"output_type": "execute_result", | |
"data": { | |
"text/plain": "(151, 3)" | |
}, | |
"metadata": {}, | |
"execution_count": 3 | |
} | |
], | |
"source": [ | |
"frames = [df15, df16, df17, df18]\n", | |
"df = pd.concat(frames, ignore_index=True)\n", | |
"df.shape" | |
] | |
}, | |
{ | |
"cell_type": "code", | |
"execution_count": 4, | |
"metadata": {}, | |
"outputs": [ | |
{ | |
"output_type": "execute_result", | |
"data": { | |
"text/plain": "43" | |
}, | |
"metadata": {}, | |
"execution_count": 4 | |
} | |
], | |
"source": [ | |
"agencies = df['Agency description'].unique()\n", | |
"len(agencies)" | |
] | |
}, | |
{ | |
"cell_type": "markdown", | |
"metadata": {}, | |
"source": [ | |
"### Sort the list alphabetically and then print out one agency per line." | |
] | |
}, | |
{ | |
"cell_type": "code", | |
"execution_count": 9, | |
"metadata": { | |
"tags": [] | |
}, | |
"outputs": [ | |
{ | |
"output_type": "stream", | |
"name": "stdout", | |
"text": "Comprehensive Nuclear-Test-Ban Treaty Organization\nDepartment of Peacekeeping Operations\nFood and Agriculture Organization of the United Nations\nIARC\nInternational Atomic Energy Agency\nInternational Civil Aviation Organization\nInternational Criminal Court\nInternational Fund for Agricultural Development\nInternational Labour Organization\nInternational Maritime Organization\nInternational Organization for Migration\nInternational Telecommunication Union\nInternational Trade Centre UNCTAD/WTO\nJoint United Nations Programme on HIV/AIDS\nOffice of the United Nations High Commissioner for Refugees\nOrganisation for the Prohibition of Chemical Weapons\nPan American Health Organization\nThe United Nations Framework Convention on Climate Change\nUN Relief and Works Agency for Palestine Refugees\nUN System Staff College\nUN Women\nUNITAID\nUnited Nations\nUnited Nations Capital Development Fund\nUnited Nations Children’s Fund\nUnited Nations Development Programme\nUnited Nations Educational, Scientific and Cultural Organization\nUnited Nations Environment Programme\nUnited Nations Human Settlements Programme\nUnited Nations Industrial Development Organization\nUnited Nations Institute for Training and Research\nUnited Nations Office for Project Services\nUnited Nations Office on Drugs and Crime\nUnited Nations Population Fund\nUnited Nations Research Institute for Social Development\nUnited Nations University\nUniversal Postal Union\nWorld Food Programme\nWorld Health Organization\nWorld Meteorological Organization\nWorld Tourism Organization\nWorld Trade Organization\nWorld intellectual Property Organization\n" | |
} | |
], | |
"source": [ | |
"agencies.sort()\n", | |
"print(*agencies, sep=\"\\n\")" | |
] | |
} | |
], | |
"metadata": { | |
"language_info": { | |
"codemirror_mode": { | |
"name": "ipython", | |
"version": 3 | |
}, | |
"file_extension": ".py", | |
"mimetype": "text/x-python", | |
"name": "python", | |
"nbconvert_exporter": "python", | |
"pygments_lexer": "ipython3", | |
"version": "3.7.3-final" | |
}, | |
"orig_nbformat": 2, | |
"kernelspec": { | |
"name": "python37364bitvenvunarctestvenv9005a6cfa1da44599a9e8877520e5746", | |
"display_name": "Python 3.7.3 64-bit ('.venv_UNArcTest': venv)" | |
} | |
}, | |
"nbformat": 4, | |
"nbformat_minor": 2 | |
} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
appnope==0.1.0 | |
backcall==0.2.0 | |
decorator==4.4.2 | |
ipykernel==5.3.2 | |
ipython==7.16.1 | |
ipython-genutils==0.2.0 | |
jedi==0.17.1 | |
jupyter-client==6.1.5 | |
jupyter-core==4.6.3 | |
numpy==1.19.0 | |
pandas==1.0.5 | |
parso==0.7.0 | |
pexpect==4.8.0 | |
pickleshare==0.7.5 | |
prompt-toolkit==3.0.5 | |
ptyprocess==0.6.0 | |
Pygments==2.6.1 | |
python-dateutil==2.8.1 | |
pytz==2020.1 | |
pyzmq==19.0.1 | |
six==1.15.0 | |
tornado==6.0.4 | |
traitlets==4.3.3 | |
wcwidth==0.2.5 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Agency description | Year | Expenditure | |
---|---|---|---|
Department of Peacekeeping Operations | 2015 | 8759159000 | |
United Nations | 2015 | 5613139904 | |
United Nations Children’s Fund | 2015 | 5077601656 | |
United Nations Development Programme | 2015 | 5057413898 | |
World Food Programme | 2015 | 4893472393 | |
Office of the United Nations High Commissioner for Refugees | 2015 | 3278871762 | |
World Health Organization | 2015 | 2738660315 | |
International Organization for Migration | 2015 | 1594121668 | |
Pan American Health Organization | 2015 | 1379321927 | |
UN Relief and Works Agency for Palestine Refugees | 2015 | 1333774841 | |
Food and Agriculture Organization of the United Nations | 2015 | 1219235388 | |
United Nations Population Fund | 2015 | 977375687 | |
United Nations Educational, Scientific and Cultural Organization | 2015 | 762491041 | |
United Nations Office for Project Services | 2015 | 671526000 | |
International Labour Organization | 2015 | 659746643 | |
International Atomic Energy Agency | 2015 | 570543764 | |
United Nations Environment Programme | 2015 | 559703000 | |
World intellectual Property Organization | 2015 | 351839000 | |
UN Women | 2015 | 314974000 | |
Joint United Nations Programme on HIV/AIDS | 2015 | 293936807 | |
United Nations Office on Drugs and Crime | 2015 | 278919255 | |
World Trade Organization | 2015 | 247027182 | |
United Nations Industrial Development Organization | 2015 | 244140762 | |
International Civil Aviation Organization | 2015 | 194803848 | |
International Telecommunication Union | 2015 | 191833044 | |
International Fund for Agricultural Development | 2015 | 168226000 | |
United Nations Human Settlements Programme | 2015 | 167062016 | |
International Trade Centre UNCTAD/WTO | 2015 | 102654000 | |
World Meteorological Organization | 2015 | 102471241 | |
Universal Postal Union | 2015 | 79265758 | |
United Nations University | 2015 | 74632000 | |
International Maritime Organization | 2015 | 68054803 | |
World Tourism Organization | 2015 | 27013663 | |
United Nations Institute for Training and Research | 2015 | 23473000 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Agency description | Year | Expenditure | |
---|---|---|---|
Department of Peacekeeping Operations | 2016 | 8,876,176,000 | |
United Nations | 2016 | 5,713,034,000 | |
United Nations Children’s Fund | 2016 | 5,427,255,034 | |
World Food Programme | 2016 | 5,355,409,043 | |
United Nations Development Programme | 2016 | 4,659,525,828 | |
Office of the United Nations High Commissioner for Refugees | 2016 | 3,846,924,119 | |
World Health Organization | 2016 | 2,471,062,278 | |
International Organization for Migration | 2016 | 1,602,307,417 | |
Pan American Health Organization | 2016 | 1,363,470,773 | |
UN Relief and Works Agency for Palestine Refugees | 2016 | 1,316,762,306 | |
Food and Agriculture Organization of the United Nations | 2016 | 1,201,818,140 | |
United Nations Population Fund | 2016 | 922,517,792 | |
United Nations Office for Project Services | 2016 | 769,873,000 | |
International Labour Organization | 2016 | 674,937,000 | |
United Nations Educational, Scientific and Cultural Organization | 2016 | 663,683,714 | |
United Nations Environment Programme | 2016 | 561,342,000 | |
International Atomic Energy Agency | 2016 | 549,644,351 | |
World intellectual Property Organization | 2016 | 347,037,073 | |
UN Women | 2016 | 339,801,000 | |
World Trade Organization | 2016 | 249,234,707 | |
United Nations Office on Drugs and Crime | 2016 | 241,906,000 | |
United Nations Industrial Development Organization | 2016 | 235,511,425 | |
International Civil Aviation Organization | 2016 | 192,346,640 | |
United Nations Human Settlements Programme | 2016 | 186,433,318 | |
International Telecommunication Union | 2016 | 183,604,878 | |
Joint United Nations Programme on HIV/AIDS | 2016 | 181,750,055 | |
International Fund for Agricultural Development | 2016 | 169,727,000 | |
World Meteorological Organization | 2016 | 98,226,341 | |
International Trade Centre UNCTAD/WTO | 2016 | 91,197,000 | |
United Nations University | 2016 | 89,986,553 | |
Universal Postal Union | 2016 | 77,403,536 | |
International Maritime Organization | 2016 | 57,821,457 | |
United Nations Institute for Training and Research | 2016 | 23,854,000 | |
World Tourism Organization | 2016 | 23,171,332 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Agency description | Year | Expenditure | |
---|---|---|---|
Department of Peacekeeping Operations | 2017 | 8,263,932,000 | |
World Food Programme | 2017 | 6,224,010,883 | |
United Nations Children’s Fund | 2017 | 5,843,812,313 | |
United Nations | 2017 | 5,788,872,170 | |
United Nations Development Programme | 2017 | 5,094,775,419 | |
Office of the United Nations High Commissioner for Refugees | 2017 | 3,942,839,988 | |
World Health Organization | 2017 | 2,680,744,796 | |
International Organization for Migration | 2017 | 1,605,352,909 | |
Food and Agriculture Organization of the United Nations | 2017 | 1,531,753,306 | |
Pan American Health Organization | 2017 | 1,435,096,541 | |
UN Relief and Works Agency for Palestine Refugees | 2017 | 1,310,443,631 | |
United Nations Population Fund | 2017 | 926,869,317 | |
United Nations Office for Project Services | 2017 | 815,855,396 | |
United Nations Educational, Scientific and Cultural Organization | 2017 | 688,169,870 | |
International Atomic Energy Agency | 2017 | 642,585,424 | |
International Labour Organization | 2017 | 640,794,735 | |
United Nations Environment Programme | 2017 | 562,234,566 | |
World intellectual Property Organization | 2017 | 403,729,039 | |
UN Women | 2017 | 338,615,000 | |
United Nations Office on Drugs and Crime | 2017 | 308,703,000 | |
United Nations Industrial Development Organization | 2017 | 299,180,198 | |
World Trade Organization | 2017 | 257,713,701 | |
International Civil Aviation Organization | 2017 | 215,225,968 | |
International Telecommunication Union | 2017 | 200,015,337 | |
United Nations Human Settlements Programme | 2017 | 197,483,176 | |
International Fund for Agricultural Development | 2017 | 188,817,000 | |
International Criminal Court | 2017 | 186,698,415 | |
Joint United Nations Programme on HIV/AIDS | 2017 | 173,371,100 | |
Comprehensive Nuclear-Test-Ban Treaty Organization | 2017 | 124,931,880 | |
World Meteorological Organization | 2017 | 107,891,616 | |
United Nations University | 2017 | 107,838,582 | |
The United Nations Framework Convention on Climate Change | 2017 | 95,353,834 | |
International Trade Centre UNCTAD/WTO | 2017 | 88,210,000 | |
Universal Postal Union | 2017 | 82,854,833 | |
International Maritime Organization | 2017 | 71,238,649 | |
United Nations Capital Development Fund | 2017 | 64,585,908 | |
United Nations Institute for Training and Research | 2017 | 28,129,041 | |
World Tourism Organization | 2017 | 27,201,861 | |
UN System Staff College | 2017 | 10,027,597 | |
United Nations Research Institute for Social Development | 2017 | 2,046,362 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Agency description | Year | Expenditure | |
---|---|---|---|
Department of Peacekeeping Operations | 2018 | 7,987,831,961 | |
World Food Programme | 2018 | 6,789,054,399 | |
United Nations | 2018 | 6,236,300,357 | |
United Nations Children’s Fund | 2018 | 5,918,587,378 | |
United Nations Development Programme | 2018 | 5,096,825,461 | |
Office of the United Nations High Commissioner for Refugees | 2018 | 4,063,979,955 | |
World Health Organization | 2018 | 2,500,048,502 | |
International Organization for Migration | 2018 | 1,841,571,956 | |
Food and Agriculture Organization of the United Nations | 2018 | 1,454,768,694 | |
Pan American Health Organization | 2018 | 1,299,479,539 | |
UN Relief and Works Agency for Palestine Refugees | 2018 | 1,190,222,772 | |
United Nations Population Fund | 2018 | 1,086,021,580 | |
United Nations Office for Project Services | 2018 | 923,666,350 | |
United Nations Educational, Scientific and Cultural Organization | 2018 | 682,012,889 | |
International Atomic Energy Agency | 2018 | 641,027,397 | |
International Labour Organization | 2018 | 624,826,077 | |
United Nations Environment Programme | 2018 | 558,532,206 | |
UN Women | 2018 | 380,260,000 | |
World intellectual Property Organization | 2018 | 355,663,225 | |
United Nations Office on Drugs and Crime | 2018 | 332,270,000 | |
United Nations Industrial Development Organization | 2018 | 279,463,696 | |
World Trade Organization | 2018 | 263,483,283 | |
International Civil Aviation Organization | 2018 | 229,964,201 | |
UNITAID | 2018 | 216,329,000 | |
International Fund for Agricultural Development | 2018 | 193,082,000 | |
International Telecommunication Union | 2018 | 186,793,313 | |
United Nations Human Settlements Programme | 2018 | 185,747,457 | |
Joint United Nations Programme on HIV/AIDS | 2018 | 184,336,897 | |
International Criminal Court | 2018 | 176,976,518 | |
Comprehensive Nuclear-Test-Ban Treaty Organization | 2018 | 127,564,070 | |
International Trade Centre UNCTAD/WTO | 2018 | 98,687,491 | |
World Meteorological Organization | 2018 | 96,634,245 | |
The United Nations Framework Convention on Climate Change | 2018 | 91,157,673 | |
United Nations University | 2018 | 90,538,190 | |
Organisation for the Prohibition of Chemical Weapons | 2018 | 84,194,436 | |
Universal Postal Union | 2018 | 75,172,438 | |
International Maritime Organization | 2018 | 63,760,456 | |
United Nations Capital Development Fund | 2018 | 60,853,361 | |
IARC | 2018 | 46,483,338 | |
United Nations Institute for Training and Research | 2018 | 28,584,000 | |
World Tourism Organization | 2018 | 21,123,148 | |
UN System Staff College | 2018 | 9,741,572 | |
United Nations Research Institute for Social Development | 2018 | 2,207,230 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This is a quick example of how Pandas can help do some fast filtering and preparation of data.
There's many ways of doing this type of filtering with other tools (eg Excel). The point of this example is to show the tip of the proverbial iceberg for what can be done readily with Pandas.