Created
June 23, 2022 10:15
-
-
Save fanny/ee35f34ff19fe2e95daca67b952b82d7 to your computer and use it in GitHub Desktop.
We can't make this file beautiful and searchable because it's too large.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Title,Subtitle,Owner,Votes,Versions,Tags,Data Type,Size,License,Views,Download,Kernels,Topics,URL,Description | |
| Credit Card Fraud Detection,Anonymized credit card transactions labeled as fraudulent or genuine,Machine Learning Group - ULB,1241,"Version 2,2016-11-05|Version 1,2016-11-03","crime | |
| finance",CSV,144 MB,ODbL,"442,136 views","53,128 downloads","1,782 kernels",26 topics,https://www.kaggle.com/mlg-ulb/creditcardfraud,"The datasets contains transactions made by credit cards in September 2013 by european cardholders. This dataset presents transactions that occurred in two days, where we have 492 frauds out of 284,807 transactions. The dataset is highly unbalanced, the positive class (frauds) account for 0.172% of all transactions. | |
| It contains only numerical input variables which are the result of a PCA transformation. Unfortunately, due to confidentiality issues, we cannot provide the original features and more background information about the data. Features V1, V2, ... V28 are the principal components obtained with PCA, the only features which have not been transformed with PCA are 'Time' and 'Amount'. Feature 'Time' contains the seconds elapsed between each transaction and the first transaction in the dataset. The feature 'Amount' is the transaction Amount, this feature can be used for example-dependant cost-senstive learning. Feature 'Class' is the response variable and it takes value 1 in case of fraud and 0 otherwise. | |
| Given the class imbalance ratio, we recommend measuring the accuracy using the Area Under the Precision-Recall Curve (AUPRC). Confusion matrix accuracy is not meaningful for unbalanced classification. | |
| The dataset has been collected and analysed during a research collaboration of Worldline and the Machine Learning Group (http://mlg.ulb.ac.be) of ULB (Université Libre de Bruxelles) on big data mining and fraud detection. More details on current and past projects on related topics are available on http://mlg.ulb.ac.be/BruFence and http://mlg.ulb.ac.be/ARTML | |
| Please cite: Andrea Dal Pozzolo, Olivier Caelen, Reid A. Johnson and Gianluca Bontempi. Calibrating Probability with Undersampling for Unbalanced Classification. In Symposium on Computational Intelligence and Data Mining (CIDM), IEEE, 2015" | |
| European Soccer Database,"25k+ matches, players & teams attributes for European Professional Football",Hugo Mathien,1046,"Version 10,2016-10-24|Version 9,2016-10-24|Version 8,2016-10-17|Version 7,2016-10-16|Version 6,2016-07-20|Version 5,2016-07-15|Version 4,2016-07-15|Version 3,2016-07-13|Version 2,2016-07-12|Version 1,2016-07-10","association football | |
| europe",SQLite,299 MB,ODbL,"396,214 views","46,367 downloads","1,459 kernels",75 topics,https://www.kaggle.com/hugomathien/soccer,"The ultimate Soccer database for data analysis and machine learning | |
| What you get: | |
| +25,000 matches | |
| +10,000 players | |
| 11 European Countries with their lead championship | |
| Seasons 2008 to 2016 | |
| Players and Teams' attributes* sourced from EA Sports' FIFA video game series, including the weekly updates | |
| Team line up with squad formation (X, Y coordinates) | |
| Betting odds from up to 10 providers | |
| Detailed match events (goal types, possession, corner, cross, fouls, cards etc...) for +10,000 matches | |
| *16th Oct 2016: New table containing teams' attributes from FIFA ! | |
| Original Data Source: | |
| You can easily find data about soccer matches but they are usually scattered across different websites. A thorough data collection and processing has been done to make your life easier. I must insist that you do not make any commercial use of the data. The data was sourced from: | |
| http://football-data.mx-api.enetscores.com/ : scores, lineup, team formation and events | |
| http://www.football-data.co.uk/ : betting odds. Click here to understand the column naming system for betting odds: | |
| http://sofifa.com/ : players and teams attributes from EA Sports FIFA games. FIFA series and all FIFA assets property of EA Sports. | |
| When you have a look at the database, you will notice foreign keys for players and matches are the same as the original data sources. I have called those foreign keys ""api_id"". | |
| Improving the dataset: | |
| You will notice that some players are missing from the lineup (NULL values). This is because I have not been able to source their attributes from FIFA. This will be fixed overtime as the crawling algorithm is being improved. The dataset will also be expanded to include international games, national cups, Champion's League and Europa League. Please ask me if you're after a specific tournament. | |
| Please get in touch with me if you want to help improve this dataset. | |
| CLICK HERE TO ACCESS THE PROJECT GITHUB | |
| Important note for people interested in using the crawlers: since I first wrote the crawling scripts (in python), it appears sofifa.com has changed its design and with it comes new requirements for the scripts. The existing script to crawl players ('Player Spider') will not work until i've updated it. | |
| Exploring the data: | |
| Now that's the fun part, there is a lot you can do with this dataset. I will be adding visuals and insights to this overview page but please have a look at the kernels and give it a try yourself ! Here are some ideas for you: | |
| The Holy Grail... ... is obviously to predict the outcome of the game. The bookies use 3 classes (Home Win, Draw, Away Win). They get it right about 53% of the time. This is also what I've achieved so far using my own SVM. Though it may sound high for such a random sport game, you've got to know that the home team wins about 46% of the time. So the base case (constantly predicting Home Win) has indeed 46% precision. | |
| Probabilities vs Odds | |
| When running a multi-class classifier like SVM you could also output a probability estimate and compare it to the betting odds. Have a look at your variance vs odds and see for what games you had very different predictions. | |
| Explore and visualize features | |
| With access to players and teams attributes, team formations and in-game events you should be able to produce some interesting insights into The Beautiful Game . Who knows, Guardiola himself may hire one of you some day!" | |
| TMDB 5000 Movie Dataset,"Metadata on ~5,000 movies from TMDb",The Movie Database (TMDb),1024,"Version 2,2017-09-28",film,CSV,44 MB,Other,"446,255 views","62,002 downloads","1,394 kernels",46 topics,https://www.kaggle.com/tmdb/tmdb-movie-metadata,"Background | |
| What can we say about the success of a movie before it is released? Are there certain companies (Pixar?) that have found a consistent formula? Given that major films costing over $100 million to produce can still flop, this question is more important than ever to the industry. Film aficionados might have different interests. Can we predict which films will be highly rated, whether or not they are a commercial success? | |
| This is a great place to start digging in to those questions, with data on the plot, cast, crew, budget, and revenues of several thousand films. | |
| Data Source Transfer Summary | |
| We (Kaggle) have removed the original version of this dataset per a DMCA takedown request from IMDB. In order to minimize the impact, we're replacing it with a similar set of films and data fields from The Movie Database (TMDb) in accordance with their terms of use. The bad news is that kernels built on the old dataset will most likely no longer work. | |
| The good news is that: | |
| You can port your existing kernels over with a bit of editing. This kernel offers functions and examples for doing so. You can also find a general introduction to the new format here. | |
| The new dataset contains full credits for both the cast and the crew, rather than just the first three actors. | |
| Actor and actresses are now listed in the order they appear in the credits. It's unclear what ordering the original dataset used; for the movies I spot checked it didn't line up with either the credits order or IMDB's stars order. | |
| The revenues appear to be more current. For example, IMDB's figures for Avatar seem to be from 2010 and understate the film's global revenues by over $2 billion. | |
| Some of the movies that we weren't able to port over (a couple of hundred) were just bad entries. For example, this IMDB entry has basically no accurate information at all. It lists Star Wars Episode VII as a documentary. | |
| Data Source Transfer Details | |
| Several of the new columns contain json. You can save a bit of time by porting the load data functions from this kernel. | |
| Even in simple fields like runtime may not be consistent across versions. For example, previous dataset shows the duration for Avatar's extended cut while TMDB shows the time for the original version. | |
| There's now a separate file containing the full credits for both the cast and crew. | |
| All fields are filled out by users so don't expect them to agree on keywords, genres, ratings, or the like. | |
| Your existing kernels will continue to render normally until they are re-run. | |
| If you are curious about how this dataset was prepared, the code to access TMDb's API is posted here. | |
| New columns: | |
| homepage | |
| id | |
| original_title | |
| overview | |
| popularity | |
| production_companies | |
| production_countries | |
| release_date | |
| spoken_languages | |
| status | |
| tagline | |
| vote_average | |
| Lost columns: | |
| actor_1_facebook_likes | |
| actor_2_facebook_likes | |
| actor_3_facebook_likes | |
| aspect_ratio | |
| cast_total_facebook_likes | |
| color | |
| content_rating | |
| director_facebook_likes | |
| facenumber_in_poster | |
| movie_facebook_likes | |
| movie_imdb_link | |
| num_critic_for_reviews | |
| num_user_for_reviews | |
| Open Questions About the Data | |
| There are some things we haven't had a chance to confirm about the new dataset. If you have any insights, please let us know in the forums! | |
| Are the budgets and revenues all in US dollars? Do they consistently show the global revenues? | |
| This dataset hasn't yet gone through a data quality analysis. Can you find any obvious corrections? For example, in the IMDb version it was necessary to treat values of zero in the budget field as missing. Similar findings would be very helpful to your fellow Kagglers! (It's probably a good idea to keep treating zeros as missing, with the caveat that missing budgets much more likely to have been from small budget films in the first place). | |
| Inspiration | |
| Can you categorize the films by type, such as animated or not? We don't have explicit labels for this, but it should be possible to build them from the crew's job titles. | |
| How sharp is the divide between major film studios and the independents? Do those two groups fall naturally out of a clustering analysis or is something more complicated going on? | |
| Acknowledgements | |
| This dataset was generated from The Movie Database API. This product uses the TMDb API but is not endorsed or certified by TMDb. Their API also provides access to data on many additional movies, actors and actresses, crew members, and TV shows. You can try it for yourself here." | |
| Global Terrorism Database,"More than 170,000 terrorist attacks worldwide, 1970-2016",START Consortium,789,"Version 2,2017-07-19|Version 1,2016-12-08","crime | |
| terrorism | |
| international relations",CSV,144 MB,Other,"187,877 views","26,309 downloads",608 kernels,11 topics,https://www.kaggle.com/START-UMD/gtd,"Context | |
| Information on more than 170,000 Terrorist Attacks | |
| The Global Terrorism Database (GTD) is an open-source database including information on terrorist attacks around the world from 1970 through 2016 (with annual updates planned for the future). The GTD includes systematic data on domestic as well as international terrorist incidents that have occurred during this time period and now includes more than 170,000 cases. The database is maintained by researchers at the National Consortium for the Study of Terrorism and Responses to Terrorism (START), headquartered at the University of Maryland. More Information | |
| Content | |
| Geography: Worldwide | |
| Time period: 1970-2016, except 1993 (2017 in progress, publication expected June 2018) | |
| Unit of analysis: Attack | |
| Variables: >100 variables on location, tactics, perpetrators, targets, and outcomes | |
| Sources: Unclassified media articles (Note: Please interpret changes over time with caution. Global patterns are driven by diverse trends in particular regions, and data collection is influenced by fluctuations in access to media coverage over both time and place.) | |
| Definition of terrorism: | |
| ""The threatened or actual use of illegal force and violence by a non-state actor to attain a political, economic, religious, or social goal through fear, coercion, or intimidation."" | |
| See the GTD Codebook for important details on data collection methodology, definitions, and coding schema. | |
| Acknowledgements | |
| The Global Terrorism Database is funded through START, by the US Department of State (Contract Number: SAQMMA12M1292) and the US Department of Homeland Security Science and Technology Directorate’s Office of University Programs (Award Number 2012-ST-061-CS0001, CSTAB 3.1). The coding decisions and classifications contained in the database are determined independently by START researchers and should not be interpreted as necessarily representing the official views or policies of the United States Government. | |
| GTD Team | |
| Publications | |
| The GTD has been leveraged extensively in scholarly publications, reports, and media articles. Putting Terrorism in Context: Lessons from the Global Terrorism Database, by GTD principal investigators LaFree, Dugan, and Miller investigates patterns of terrorism and provides perspective on the challenges of data collection and analysis. The GTD's data collection manager, Michael Jensen, discusses important Benefits and Drawbacks of Methodological Advancements in Data Collection and Coding. | |
| Terms of Use | |
| Use of the data signifies your agreement to the following terms and conditions. | |
| Definitions: Within this section: ""GTD"" will refer to the Global Terrorism Database produced by the National Consortium for the Study of Terrorism and Responses to Terrorism. This includes the data and codebook, any auxiliary materials present, and the World Wide Web interface by which the data are presented. ""START"" will refer to the National Consortium for the Study of Terrorism and Responses to Terrorism, a United States Department of Homeland Security Center of Excellence based at the University of Maryland. ""USER"" denotes the individual or set of individuals who access the GTD, i.e. the data, codebook, any auxiliary materials, and the World Wide Web interface by which the data are presented. ""GTD representatives"" denotes any senior management staff of START, and any employee or representative of said organization whom senior management staff designate to represent START in dealings with the USER. | |
| Usage Rights: Pursuant to this agreement, START grants the USER the non-exclusive, non-guaranteed right to search, browse, and view all contents of the GTD World Wide Web interface. | |
| Authorship: All contents of the GTD were assembled by representatives of START and do not purport to reflect the official position or data collections of the Department of Homeland Security or any other agency of the United States government. | |
| Acknowledgement: All information sourced from the GTD should be acknowledged by the USER and cited as follows: ""National Consortium for the Study of Terrorism and Responses to Terrorism (START). (2017). Global Terrorism Database [Data file]. Retrieved from https://www.kaggle.com/START-UMD/gtd"" | |
| Unauthorized Publication of the Data: No part of the GTD may be republished on any website or accessible for public download in any format without the express permission of a GTD staff member. In addition, no part of the GTD may be distributed for any commercial purpose, nor with the intent that the data be used in any commercial enterprise, without the express permission of a GTD staff member. START reserves the right to withhold this permission. | |
| Penalties: Penalties for failure to comply with the terms of this agreement may result in loss of access to the GTD and the forfeiture of user privileges, in addition to any other appropriate legal remedies. | |
| Limitation of Liability: Although every reasonable effort has been made to check sources and verify facts, START cannot guarantee that accounts reported in the open literature are complete and accurate. START shall not be held liable for any loss or damage caused by errors or omissions or resulting from any use, misuse, or alteration of GTD data by the USER. The USER should not infer any additional actions or results beyond what is presented in a GTD entry and specifically, the USER should not infer an individual associated with a particular incident was tried and convicted of terrorism or any other criminal offense. If new documentation about an event becomes available, an entry may be modified, as necessary and appropriate. | |
| Termination of Rights: The GTD developers reserve the right to remove access to the GTD website from any particular IP address or set of IP addresses, or to remove the database entirely from public access, at their discretion. In such an event, all USER rights granted in this document are terminated. | |
| Training | |
| START has released the first in a series of training modules designed to equip GTD users with the knowledge and tools to best leverage the database. This training module provides a general overview of the GTD, including the data collection process, uses of the GTD, and patterns of global terrorism. Participants will learn basic data handling and how to generate summary statistics from the GTD using PivotTables in Microsoft Excel. | |
| Questions? | |
| Find answers to Frequently Asked Questions. | |
| Contact the GTD staff at gtd@start.umd.edu." | |
| Bitcoin Historical Data,"Bitcoin data at 1-min intervals from select exchanges, Jan 2012 to Jan 2018",Zielak,618,"Version 11,2018-01-11|Version 10,2017-11-17|Version 9,2017-10-24|Version 8,2017-10-22|Version 7,2017-10-22|Version 6,2017-06-12|Version 5,2017-06-08|Version 4,2017-06-06|Version 3,2017-06-03|Version 2,2017-06-02|Version 1,2017-06-02","history | |
| finance",CSV,119 MB,CC4,"146,734 views","16,868 downloads",68 kernels,13 topics,https://www.kaggle.com/mczielinski/bitcoin-historical-data,"Context | |
| Bitcoin is the longest running and most well known cryptocurrency, first released as open source in 2009 by the anonymous Satoshi Nakamoto. Bitcoin serves as a decentralized medium of digital exchange, with transactions verified and recorded in a public distributed ledger (the blockchain) without the need for a trusted record keeping authority or central intermediary. Transaction blocks contain a SHA-256 cryptographic hash of previous transaction blocks, and are thus ""chained"" together, serving as an immutable record of all transactions that have ever occurred. As with any currency/commodity on the market, bitcoin trading and financial instruments soon followed public adoption of bitcoin and continue to grow. Included here is historical bitcoin market data at 1-min intervals for select bitcoin exchanges where trading takes place. Happy (data) mining! | |
| Content | |
| coincheckJPY_1-min_data_2014-10-31_to_2018-01-08.csv | |
| bitflyerJPY_1-min_data_2017-07-04_to_2018-01-08.csv | |
| coinbaseUSD_1-min_data_2014-12-01_to_2018-01-08.csv | |
| bitstampUSD_1-min_data_2012-01-01_to_2018-01-08.csv | |
| CSV files for select bitcoin exchanges for the time period of Jan 2012 to Jan 2018, with minute to minute updates of OHLC (Open, High, Low, Close), Volume in BTC and indicated currency, and weighted bitcoin price. Timestamps are in Unix time. Timestamps without any trades or activity have their data fields populated with NaNs. If a timestamp is missing, or if there are jumps, this may be because the exchange (or its API) was down, the exchange (or its API) did not exist, or some other unforseen technical error in data reporting or gathering. All effort has been made to deduplicate entries and verify the contents are correct and complete to the best of my ability, but obviously trust at your own risk. | |
| Acknowledgements and Inspiration | |
| The various exchange APIs, for making it difficult or unintuitive enough to get OHLC and volume data at 1-min intervals that I set out on this data scraping project. Satoshi Nakamoto and the novel core concept of the blockchain, as well as its first execution via the bitcoin protocol. I'd also like to thank viewers like you! Can't wait to see what code or insights you all have to share. | |
| I am a lowly Ph.D. student who did this for fun in my meager spare time. If you find this data interesting and you can spare a coffee to fuel my science, send it my way and I'd be immensely grateful! | |
| 1kmWmcQa8qN9ZrdGfdkw8EHKBgugKBRcF" | |
| "Kaggle ML and Data Science Survey, 2017",A big picture view of the state of data science and machine learning.,Kaggle,574,"Version 4,2017-10-28|Version 3,2017-10-03|Version 2,2017-10-03|Version 1,2017-09-29","employment | |
| sociology | |
| artificial intelligence",CSV,28 MB,ODbL,"95,587 views","9,390 downloads",244 kernels,10 topics,https://www.kaggle.com/kaggle/kaggle-survey-2017,"Context | |
| For the first time, Kaggle conducted an industry-wide survey to establish a comprehensive view of the state of data science and machine learning. The survey received over 16,000 responses and we learned a ton about who is working with data, what’s happening at the cutting edge of machine learning across industries, and how new data scientists can best break into the field. | |
| To share some of the initial insights from the survey, we’ve worked with the folks from The Pudding to put together this interactive report. They’ve shared all of the kernels used in the report here. | |
| Content | |
| The data includes 5 files: | |
| schema.csv: a CSV file with survey schema. This schema includes the questions that correspond to each column name in both the multipleChoiceResponses.csv and freeformResponses.csv. | |
| multipleChoiceResponses.csv: Respondents' answers to multiple choice and ranking questions. These are non-randomized and thus a single row does correspond to all of a single user's answers. -freeformResponses.csv: Respondents' freeform answers to Kaggle's survey questions. These responses are randomized within a column, so that reading across a single row does not give a single user's answers. | |
| conversionRates.csv: Currency conversion rates (to USD) as accessed from the R package ""quantmod"" on September 14, 2017 | |
| RespondentTypeREADME.txt: This is a schema for decoding the responses in the ""Asked"" column of the schema.csv file. | |
| Kernel Awards in November | |
| In the month of November, we’re awarding $1000 a week for code and analyses shared on this dataset via Kaggle Kernels. Read more about this month’s Kaggle Kernels Awards and help us advance the state of machine learning and data science by exploring this one of a kind dataset. | |
| Methodology | |
| This survey received 16,716 usable respondents from 171 countries and territories. If a country or territory received less than 50 respondents, we grouped them into a group named “Other” for anonymity. | |
| We excluded respondents who were flagged by our survey system as “Spam” or who did not answer the question regarding their employment status (this question was the first required question, so not answering it indicates that the respondent did not proceed past the 5th question in our survey). | |
| Most of our respondents were found primarily through Kaggle channels, like our email list, discussion forums and social media channels. | |
| The survey was live from August 7th to August 25th. The median response time for those who participated in the survey was 16.4 minutes. We allowed respondents to complete the survey at any time during that window. | |
| We received salary data by first asking respondents for their day-to-day currency, and then asking them to write in either their total compensation. | |
| We’ve provided a csv with an exchange rate to USD for you to calculate the salary in US dollars on your own. | |
| The question was optional | |
| Not every question was shown to every respondent. In an attempt to ask relevant questions to each respondent, we generally asked work related questions to employed data scientists and learning related questions to students. There is a column in the schema.csv file called ""Asked"" that describes who saw each question. You can learn more about the different segments we used in the schema.csv file and RespondentTypeREADME.txt in the data tab. | |
| To protect the respondents’ identity, the answers to multiple choice questions have been separated into a separate data file from the open-ended responses. We do not provide a key to match up the multiple choice and free form responses. Further, the free form responses have been randomized column-wise such that the responses that appear on the same row did not necessarily come from the same survey-taker." | |
| Iris Species,Classify iris plants into three species in this classic dataset,UCI Machine Learning,512,"Version 2,2016-09-27|Version 1,2016-01-12",botany,SQLite,15 KB,CC0,"162,706 views","24,361 downloads","3,394 kernels",14 topics,https://www.kaggle.com/uciml/iris,"The Iris dataset was used in R.A. Fisher's classic 1936 paper, The Use of Multiple Measurements in Taxonomic Problems, and can also be found on the UCI Machine Learning Repository. | |
| It includes three iris species with 50 samples each as well as some properties about each flower. One flower species is linearly separable from the other two, but the other two are not linearly separable from each other. | |
| The columns in this dataset are: | |
| Id | |
| SepalLengthCm | |
| SepalWidthCm | |
| PetalLengthCm | |
| PetalWidthCm | |
| Species" | |
| World Development Indicators,Explore country development indicators from around the world,World Bank,468,"Version 2,2017-05-02|Version 1,2016-01-28","economics | |
| international relations",CSV,2 GB,Other,"134,038 views","20,364 downloads",389 kernels,5 topics,https://www.kaggle.com/worldbank/world-development-indicators,"The World Development Indicators from the World Bank contain over a thousand annual indicators of economic development from hundreds of countries around the world. | |
| Here's a list of the available indicators along with a list of the available countries. | |
| For example, this data includes the life expectancy at birth from many countries around the world: | |
| The dataset hosted here is a slightly transformed verion of the raw files available here to facilitate analytics." | |
| Daily News for Stock Market Prediction,Using 8 years daily news headlines to predict stock market movement,Aaron7sun,438,"Version 1,2016-08-25","news agencies | |
| finance",CSV,14 MB,CC4,"95,770 views","10,820 downloads",293 kernels,8 topics,https://www.kaggle.com/aaron7sun/stocknews,"Actually, I prepare this dataset for students on my Deep Learning and NLP course. | |
| But I am also very happy to see kagglers play around with it. | |
| Have fun! | |
| Description: | |
| There are two channels of data provided in this dataset: | |
| News data: I crawled historical news headlines from Reddit WorldNews Channel (/r/worldnews). They are ranked by reddit users' votes, and only the top 25 headlines are considered for a single date. (Range: 2008-06-08 to 2016-07-01) | |
| Stock data: Dow Jones Industrial Average (DJIA) is used to ""prove the concept"". (Range: 2008-08-08 to 2016-07-01) | |
| I provided three data files in .csv format: | |
| RedditNews.csv: two columns The first column is the ""date"", and second column is the ""news headlines"". All news are ranked from top to bottom based on how hot they are. Hence, there are 25 lines for each date. | |
| DJIA_table.csv: Downloaded directly from Yahoo Finance: check out the web page for more info. | |
| Combined_News_DJIA.csv: To make things easier for my students, I provide this combined dataset with 27 columns. The first column is ""Date"", the second is ""Label"", and the following ones are news headlines ranging from ""Top1"" to ""Top25"". | |
| ========================================= | |
| To my students: | |
| I made this a binary classification task. Hence, there are only two labels: | |
| ""1"" when DJIA Adj Close value rose or stayed as the same; | |
| ""0"" when DJIA Adj Close value decreased. | |
| For task evaluation, please use data from 2008-08-08 to 2014-12-31 as Training Set, and Test Set is then the following two years data (from 2015-01-02 to 2016-07-01). This is roughly a 80%/20% split. | |
| And, of course, use AUC as the evaluation metric. | |
| ========================================= | |
| +++++++++++++++++++++++++++++++++++++++++ | |
| To all kagglers: | |
| Please upvote this dataset if you like this idea for market prediction. | |
| If you think you coded an amazing trading algorithm, | |
| friendly advice | |
| do play safe with your own money :) | |
| +++++++++++++++++++++++++++++++++++++++++ | |
| Feel free to contact me if there is any question~ | |
| And, remember me when you become a millionaire :P" | |
| Pokemon with stats,721 Pokemon with stats and types,Alberto Barradas,428,"Version 2,2016-08-29|Version 1,2016-08-23","popular culture | |
| games and toys | |
| video games",CSV,43 KB,CC0,"133,256 views","16,610 downloads",706 kernels,13 topics,https://www.kaggle.com/abcsds/pokemon,"This data set includes 721 Pokemon, including their number, name, first and second type, and basic stats: HP, Attack, Defense, Special Attack, Special Defense, and Speed. It has been of great use when teaching statistics to kids. With certain types you can also give a geeky introduction to machine learning. | |
| This are the raw attributes that are used for calculating how much damage an attack will do in the games. This dataset is about the pokemon games (NOT pokemon cards or Pokemon Go). | |
| The data as described by Myles O'Neill is: | |
| #: ID for each pokemon | |
| Name: Name of each pokemon | |
| Type 1: Each pokemon has a type, this determines weakness/resistance to attacks | |
| Type 2: Some pokemon are dual type and have 2 | |
| Total: sum of all stats that come after this, a general guide to how strong a pokemon is | |
| HP: hit points, or health, defines how much damage a pokemon can withstand before fainting | |
| Attack: the base modifier for normal attacks (eg. Scratch, Punch) | |
| Defense: the base damage resistance against normal attacks | |
| SP Atk: special attack, the base modifier for special attacks (e.g. fire blast, bubble beam) | |
| SP Def: the base damage resistance against special attacks | |
| Speed: determines which pokemon attacks first each round | |
| The data for this table has been acquired from several different sites, including: | |
| pokemon.com | |
| pokemondb | |
| bulbapeida | |
| One question has been answered with this database: The type of a pokemon cannot be inferred only by it's Attack and Deffence. It would be worthy to find which two variables can define the type of a pokemon, if any. Two variables can be plotted in a 2D space, and used as an example for machine learning. This could mean the creation of a visual example any geeky Machine Learning class would love." | |
| Lending Club Loan Data,Analyze Lending Club's issued loans,Wendy Kan,406,"Version 1,2016-05-03",finance,SQLite,913 MB,Other,"151,714 views","26,758 downloads",521 kernels,25 topics,https://www.kaggle.com/wendykan/lending-club-loan-data,"These files contain complete loan data for all loans issued through the 2007-2015, including the current loan status (Current, Late, Fully Paid, etc.) and latest payment information. The file containing loan data through the ""present"" contains complete loan data for all loans issued through the previous completed calendar quarter. Additional features include credit scores, number of finance inquiries, address including zip codes, and state, and collections among others. The file is a matrix of about 890 thousand observations and 75 variables. A data dictionary is provided in a separate file. k" | |
| Wine Reviews,"130k wine reviews with variety, location, winery, price, and description",zackthoutt,397,"Version 4,2017-11-28|Version 3,2017-11-25|Version 2,2017-06-22|Version 1,2017-06-22","critical theory | |
| food and drink",CSV,51 MB,CC4,"65,098 views","10,070 downloads",87 kernels,12 topics,https://www.kaggle.com/zynicide/wine-reviews,"Context | |
| After watching Somm (a documentary on master sommeliers) I wondered how I could create a predictive model to identify wines through blind tasting like a master sommelier would. The first step in this journey was gathering some data to train a model. I plan to use deep learning to predict the wine variety using words in the description/review. The model still won't be able to taste the wine, but theoretically it could identify the wine based on a description that a sommelier could give. If anyone has any ideas on how to accomplish this, please post them! | |
| Content | |
| The data consists of 10 fields: | |
| Points: the number of points WineEnthusiast rated the wine on a scale of 1-100 (though they say they only post reviews for wines that score >=80) | |
| Title: the title of the wine review, which often contains the vintage if you're interested in extracting that feature | |
| Variety: the type of grapes used to make the wine (ie Pinot Noir) | |
| Description: a few sentences from a sommelier describing the wine's taste, smell, look, feel, etc. | |
| Country: the country that the wine is from | |
| Province: the province or state that the wine is from | |
| Region 1: the wine growing area in a province or state (ie Napa) | |
| Region 2: sometimes there are more specific regions specified within a wine growing area (ie Rutherford inside the Napa Valley), but this value can sometimes be blank | |
| Winery: the winery that made the wine | |
| Designation: the vineyard within the winery where the grapes that made the wine are from | |
| Price: the cost for a bottle of the wine | |
| Taster Name: name of the person who tasted and reviewed the wine | |
| Taster Twitter Handle: Twitter handle for the person who tasted and reviewed the wine | |
| Acknowledgements | |
| The data was scraped from WineEnthusiast during the week of June 15th, 2017. The code for the scraper can be found here if you have any more specific questions about data collection that I didn't address. | |
| UPDATE 11/24/2017 After feedback from users of the dataset I scraped the reviews again on November 22nd, 2017. This time around I collected the title of each review, which you can parse the year out of, the tasters name, and the taster's Twitter handle. This should also fix the duplicate entry issue. | |
| Inspiration | |
| I think that this dataset offers some great opportunities for sentiment analysis and other text related predictive models. My overall goal is to create a model that can identify the variety, winery, and location of a wine based on a description. If anyone has any ideas, breakthroughs, or other interesting insights/models please post them." | |
| Climate Change: Earth Surface Temperature Data,Exploring global temperatures since 1750,Berkeley Earth,392,"Version 2,2017-05-02|Version 1,2016-03-11","environment | |
| climate",CSV,573 MB,CC4,"134,742 views","22,989 downloads",431 kernels,7 topics,https://www.kaggle.com/berkeleyearth/climate-change-earth-surface-temperature-data,"Some say climate change is the biggest threat of our age while others say it’s a myth based on dodgy science. We are turning some of the data over to you so you can form your own view. | |
| Even more than with other data sets that Kaggle has featured, there’s a huge amount of data cleaning and preparation that goes into putting together a long-time study of climate trends. Early data was collected by technicians using mercury thermometers, where any variation in the visit time impacted measurements. In the 1940s, the construction of airports caused many weather stations to be moved. In the 1980s, there was a move to electronic thermometers that are said to have a cooling bias. | |
| Given this complexity, there are a range of organizations that collate climate trends data. The three most cited land and ocean temperature data sets are NOAA’s MLOST, NASA’s GISTEMP and the UK’s HadCrut. | |
| We have repackaged the data from a newer compilation put together by the Berkeley Earth, which is affiliated with Lawrence Berkeley National Laboratory. The Berkeley Earth Surface Temperature Study combines 1.6 billion temperature reports from 16 pre-existing archives. It is nicely packaged and allows for slicing into interesting subsets (for example by country). They publish the source data and the code for the transformations they applied. They also use methods that allow weather observations from shorter time series to be included, meaning fewer observations need to be thrown away. | |
| In this dataset, we have include several files: | |
| Global Land and Ocean-and-Land Temperatures (GlobalTemperatures.csv): | |
| Date: starts in 1750 for average land temperature and 1850 for max and min land temperatures and global ocean and land temperatures | |
| LandAverageTemperature: global average land temperature in celsius | |
| LandAverageTemperatureUncertainty: the 95% confidence interval around the average | |
| LandMaxTemperature: global average maximum land temperature in celsius | |
| LandMaxTemperatureUncertainty: the 95% confidence interval around the maximum land temperature | |
| LandMinTemperature: global average minimum land temperature in celsius | |
| LandMinTemperatureUncertainty: the 95% confidence interval around the minimum land temperature | |
| LandAndOceanAverageTemperature: global average land and ocean temperature in celsius | |
| LandAndOceanAverageTemperatureUncertainty: the 95% confidence interval around the global average land and ocean temperature | |
| Other files include: | |
| Global Average Land Temperature by Country (GlobalLandTemperaturesByCountry.csv) | |
| Global Average Land Temperature by State (GlobalLandTemperaturesByState.csv) | |
| Global Land Temperatures By Major City (GlobalLandTemperaturesByMajorCity.csv) | |
| Global Land Temperatures By City (GlobalLandTemperaturesByCity.csv) | |
| The raw data comes from the Berkeley Earth data page." | |
| Open Food Facts,Explore nutrition facts from foods around the world,Open Food Facts,361,"Version 5,2017-09-18|Version 4,2017-06-15|Version 3,2017-02-22|Version 2,2017-02-22|Version 1,2016-01-13","food and drink | |
| nutrition",Other,963 MB,ODbL,"131,482 views","18,829 downloads",483 kernels,6 topics,https://www.kaggle.com/openfoodfacts/world-food-facts,"A food products database | |
| Open Food Facts is a free, open, collbarative database of food products from around the world, with ingredients, allergens, nutrition facts and all the tidbits of information we can find on product labels. | |
| Made by everyone | |
| Open Food Facts is a non-profit association of volunteers. 5000+ contributors like you have added 100 000+ products from 150 countries using our Android, iPhone or Windows Phone app or their camera to scan barcodes and upload pictures of products and their labels. | |
| For everyone | |
| Data about food is of public interest and has to be open. The complete database is published as open data and can be reused by anyone and for any use. Check-out the cool reuses or make your own! | |
| Dataset structure | |
| The dataset contains a single table, FoodFacts, in CSV form in FoodFacts.csv and in SQLite form in database.sqlite. | |
| The columns in Open Food Facts are as follows: | |
| code (text) | |
| url (text) | |
| creator (text) | |
| created_t (text) | |
| created_datetime (text) | |
| last_modified_t (text) | |
| last_modified_datetime (text) | |
| product_name (text) | |
| generic_name (text) | |
| quantity (text) | |
| packaging (text) | |
| packaging_tags (text) | |
| brands (text) | |
| brands_tags (text) | |
| categories (text) | |
| categories_tags (text) | |
| categories_en (text) | |
| origins (text) | |
| origins_tags (text) | |
| manufacturing_places (text) | |
| manufacturing_places_tags (text) | |
| labels (text) | |
| labels_tags (text) | |
| labels_en (text) | |
| emb_codes (text) | |
| emb_codes_tags (text) | |
| first_packaging_code_geo (text) | |
| cities (text) | |
| cities_tags (text) | |
| purchase_places (text) | |
| stores (text) | |
| countries (text) | |
| countries_tags (text) | |
| countries_en (text) | |
| ingredients_text (text) | |
| allergens (text) | |
| allergens_en (text) | |
| traces (text) | |
| traces_tags (text) | |
| traces_en (text) | |
| serving_size (text) | |
| no_nutriments (numeric) | |
| additives_n (numeric) | |
| additives (text) | |
| additives_tags (text) | |
| additives_en (text) | |
| ingredients_from_palm_oil_n (numeric) | |
| ingredients_from_palm_oil (numeric) | |
| ingredients_from_palm_oil_tags (text) | |
| ingredients_that_may_be_from_palm_oil_n (numeric) | |
| ingredients_that_may_be_from_palm_oil (numeric) | |
| ingredients_that_may_be_from_palm_oil_tags (text) | |
| nutrition_grade_uk (numeric) | |
| nutrition_grade_fr (text) | |
| pnns_groups_1 (text) | |
| pnns_groups_2 (text) | |
| states (text) | |
| states_tags (text) | |
| states_en (text) | |
| main_category (text) | |
| main_category_en (text) | |
| image_url (text) | |
| image_small_url (text) | |
| energy_100g (numeric) | |
| energy_from_fat_100g (numeric) | |
| fat_100g (numeric) | |
| saturated_fat_100g (numeric) | |
| butyric_acid_100g (numeric) | |
| caproic_acid_100g (numeric) | |
| caprylic_acid_100g (numeric) | |
| capric_acid_100g (numeric) | |
| lauric_acid_100g (numeric) | |
| myristic_acid_100g (numeric) | |
| palmitic_acid_100g (numeric) | |
| stearic_acid_100g (numeric) | |
| arachidic_acid_100g (numeric) | |
| behenic_acid_100g (numeric) | |
| lignoceric_acid_100g (numeric) | |
| cerotic_acid_100g (numeric) | |
| montanic_acid_100g (numeric) | |
| melissic_acid_100g (numeric) | |
| monounsaturated_fat_100g (numeric) | |
| polyunsaturated_fat_100g (numeric) | |
| omega_3_fat_100g (numeric) | |
| alpha_linolenic_acid_100g (numeric) | |
| eicosapentaenoic_acid_100g (numeric) | |
| docosahexaenoic_acid_100g (numeric) | |
| omega_6_fat_100g (numeric) | |
| linoleic_acid_100g (numeric) | |
| arachidonic_acid_100g (numeric) | |
| gamma_linolenic_acid_100g (numeric) | |
| dihomo_gamma_linolenic_acid_100g (numeric) | |
| omega_9_fat_100g (numeric) | |
| oleic_acid_100g (numeric) | |
| elaidic_acid_100g (numeric) | |
| gondoic_acid_100g (numeric) | |
| mead_acid_100g (numeric) | |
| erucic_acid_100g (numeric) | |
| nervonic_acid_100g (numeric) | |
| trans_fat_100g (numeric) | |
| cholesterol_100g (numeric) | |
| carbohydrates_100g (numeric) | |
| sugars_100g (numeric) | |
| sucrose_100g (numeric) | |
| glucose_100g (numeric) | |
| fructose_100g (numeric) | |
| lactose_100g (numeric) | |
| maltose_100g (numeric) | |
| maltodextrins_100g (numeric) | |
| starch_100g (numeric) | |
| polyols_100g (numeric) | |
| fiber_100g (numeric) | |
| proteins_100g (numeric) | |
| casein_100g (numeric) | |
| serum_proteins_100g (numeric) | |
| nucleotides_100g (numeric) | |
| salt_100g (numeric) | |
| sodium_100g (numeric) | |
| alcohol_100g (numeric) | |
| vitamin_a_100g (numeric) | |
| beta_carotene_100g (numeric) | |
| vitamin_d_100g (numeric) | |
| vitamin_e_100g (numeric) | |
| vitamin_k_100g (numeric) | |
| vitamin_c_100g (numeric) | |
| vitamin_b1_100g (numeric) | |
| vitamin_b2_100g (numeric) | |
| vitamin_pp_100g (numeric) | |
| vitamin_b6_100g (numeric) | |
| vitamin_b9_100g (numeric) | |
| vitamin_b12_100g (numeric) | |
| biotin_100g (numeric) | |
| pantothenic_acid_100g (numeric) | |
| silica_100g (numeric) | |
| bicarbonate_100g (numeric) | |
| potassium_100g (numeric) | |
| chloride_100g (numeric) | |
| calcium_100g (numeric) | |
| phosphorus_100g (numeric) | |
| iron_100g (numeric) | |
| magnesium_100g (numeric) | |
| zinc_100g (numeric) | |
| copper_100g (numeric) | |
| manganese_100g (numeric) | |
| fluoride_100g (numeric) | |
| selenium_100g (numeric) | |
| chromium_100g (numeric) | |
| molybdenum_100g (numeric) | |
| iodine_100g (numeric) | |
| caffeine_100g (numeric) | |
| taurine_100g (numeric) | |
| ph_100g (numeric) | |
| fruits_vegetables_nuts_100g (numeric) | |
| collagen_meat_protein_ratio_100g (numeric) | |
| cocoa_100g (numeric) | |
| chlorophyl_100g (numeric) | |
| carbon_footprint_100g (numeric) | |
| nutrition_score_fr_100g (numeric) | |
| nutrition_score_uk_100g (numeric)" | |
| Where it Pays to Attend College,"Salaries by college, region, and academic major",The Wall Street Journal,347,"Version 1,2017-04-29","news agencies | |
| universities and colleges | |
| employment | |
| education",CSV,72 KB,Other,"50,598 views","9,473 downloads",43 kernels,3 topics,https://www.kaggle.com/wsj/college-salaries,"Salary Increase By Type of College | |
| Party school? Liberal Arts college? State School? You already know your starting salary will be different depending on what type of school you attend. But, increased earning power shows less disparity. Ten years out, graduates of Ivy League schools earned 99% more than they did at graduation. Party school graduates saw an 85% increase. Engineering school graduates fared worst, earning 76% more 10 years out of school. See where your school ranks. | |
| Salaries By Region | |
| Attending college in the Midwest leads to the lowest salary both at graduation and at mid-career, according to the PayScale Inc. survey. Graduates of schools in the Northeast and California fared best. | |
| Salary Increase By Major | |
| Your parents might have worried when you chose Philosophy or International Relations as a major. But a year-long survey of 1.2 million people with only a bachelor's degree by PayScale Inc. shows that graduates in these subjects earned 103.5% and 97.8% more, respectively, about 10 years post-commencement. Majors that didn't show as much salary growth include Nursing and Information Technology. | |
| All data was obtained from the Wall Street Journal based on data from Payscale, Inc: | |
| Salaries for Colleges by Type | |
| Salaries for Colleges by Region | |
| Degrees that Pay you Back" | |
| Mushroom Classification,Safe to eat or deadly poison?,UCI Machine Learning,341,"Version 1,2016-12-02",food and drink,CSV,365 KB,CC0,"103,535 views","14,777 downloads",449 kernels,4 topics,https://www.kaggle.com/uciml/mushroom-classification,"Context | |
| Although this dataset was originally contributed to the UCI Machine Learning repository nearly 30 years ago, mushroom hunting (otherwise known as ""shrooming"") is enjoying new peaks in popularity. Learn which features spell certain death and which are most palatable in this dataset of mushroom characteristics. And how certain can your model be? | |
| Content | |
| This dataset includes descriptions of hypothetical samples corresponding to 23 species of gilled mushrooms in the Agaricus and Lepiota Family Mushroom drawn from The Audubon Society Field Guide to North American Mushrooms (1981). Each species is identified as definitely edible, definitely poisonous, or of unknown edibility and not recommended. This latter class was combined with the poisonous one. The Guide clearly states that there is no simple rule for determining the edibility of a mushroom; no rule like ""leaflets three, let it be'' for Poisonous Oak and Ivy. | |
| Time period: Donated to UCI ML 27 April 1987 | |
| Inspiration | |
| What types of machine learning models perform best on this dataset? | |
| Which features are most indicative of a poisonous mushroom? | |
| Acknowledgements | |
| This dataset was originally donated to the UCI Machine Learning repository. You can learn more about past research using the data here. | |
| Start a new kernel" | |
| Breast Cancer Wisconsin (Diagnostic) Data Set,Predict whether the cancer is benign or malignant,UCI Machine Learning,302,"Version 2,2016-09-25|Version 1,2016-09-20",healthcare,CSV,122 KB,CC4,"127,057 views","17,488 downloads",573 kernels,11 topics,https://www.kaggle.com/uciml/breast-cancer-wisconsin-data,"Features are computed from a digitized image of a fine needle aspirate (FNA) of a breast mass. They describe characteristics of the cell nuclei present in the image. n the 3-dimensional space is that described in: [K. P. Bennett and O. L. Mangasarian: ""Robust Linear Programming Discrimination of Two Linearly Inseparable Sets"", Optimization Methods and Software 1, 1992, 23-34]. | |
| This database is also available through the UW CS ftp server: ftp ftp.cs.wisc.edu cd math-prog/cpo-dataset/machine-learn/WDBC/ | |
| Also can be found on UCI Machine Learning Repository: https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+%28Diagnostic%29 | |
| Attribute Information: | |
| 1) ID number 2) Diagnosis (M = malignant, B = benign) 3-32) | |
| Ten real-valued features are computed for each cell nucleus: | |
| a) radius (mean of distances from center to points on the perimeter) b) texture (standard deviation of gray-scale values) c) perimeter d) area e) smoothness (local variation in radius lengths) f) compactness (perimeter^2 / area - 1.0) g) concavity (severity of concave portions of the contour) h) concave points (number of concave portions of the contour) i) symmetry j) fractal dimension (""coastline approximation"" - 1) | |
| The mean, standard error and ""worst"" or largest (mean of the three largest values) of these features were computed for each image, resulting in 30 features. For instance, field 3 is Mean Radius, field 13 is Radius SE, field 23 is Worst Radius. | |
| All feature values are recoded with four significant digits. | |
| Missing attribute values: none | |
| Class distribution: 357 benign, 212 malignant" | |
| Amazon Fine Food Reviews,"Analyze ~500,000 food reviews from Amazon",Stanford Network Analysis Project,281,"Version 2,2017-05-02|Version 1,2016-01-09","food and drink | |
| linguistics | |
| internet",SQLite,642 MB,CC0,"119,362 views","16,079 downloads",327 kernels,9 topics,https://www.kaggle.com/snap/amazon-fine-food-reviews,"Context | |
| This dataset consists of reviews of fine foods from amazon. The data span a period of more than 10 years, including all ~500,000 reviews up to October 2012. Reviews include product and user information, ratings, and a plain text review. It also includes reviews from all other Amazon categories. | |
| Contents | |
| Reviews.csv: Pulled from the corresponding SQLite table named Reviews in database.sqlite | |
| database.sqlite: Contains the table 'Reviews' | |
| Data includes: | |
| - Reviews from Oct 1999 - Oct 2012 | |
| - 568,454 reviews | |
| - 256,059 users | |
| - 74,258 products | |
| - 260 users with > 50 reviews | |
| Acknowledgements | |
| See this SQLite query for a quick sample of the dataset. | |
| If you publish articles based on this dataset, please cite the following paper: | |
| J. McAuley and J. Leskovec. From amateurs to connoisseurs: modeling the evolution of user expertise through online reviews. WWW, 2013." | |
| World Happiness Report,"Happiness scored according to economic production, social support, etc.",Sustainable Development Solutions Network,272,"Version 2,2017-06-15|Version 1,2017-02-28","emotion | |
| social sciences | |
| economics",CSV,62 KB,CC0,"90,642 views","19,316 downloads",234 kernels,6 topics,https://www.kaggle.com/unsdsn/world-happiness,"Context | |
| The World Happiness Report is a landmark survey of the state of global happiness. The first report was published in 2012, the second in 2013, the third in 2015, and the fourth in the 2016 Update. The World Happiness 2017, which ranks 155 countries by their happiness levels, was released at the United Nations at an event celebrating International Day of Happiness on March 20th. The report continues to gain global recognition as governments, organizations and civil society increasingly use happiness indicators to inform their policy-making decisions. Leading experts across fields – economics, psychology, survey analysis, national statistics, health, public policy and more – describe how measurements of well-being can be used effectively to assess the progress of nations. The reports review the state of happiness in the world today and show how the new science of happiness explains personal and national variations in happiness. | |
| Content | |
| The happiness scores and rankings use data from the Gallup World Poll. The scores are based on answers to the main life evaluation question asked in the poll. This question, known as the Cantril ladder, asks respondents to think of a ladder with the best possible life for them being a 10 and the worst possible life being a 0 and to rate their own current lives on that scale. The scores are from nationally representative samples for the years 2013-2016 and use the Gallup weights to make the estimates representative. The columns following the happiness score estimate the extent to which each of six factors – economic production, social support, life expectancy, freedom, absence of corruption, and generosity – contribute to making life evaluations higher in each country than they are in Dystopia, a hypothetical country that has values equal to the world’s lowest national averages for each of the six factors. They have no impact on the total score reported for each country, but they do explain why some countries rank higher than others. | |
| Inspiration | |
| What countries or regions rank the highest in overall happiness and each of the six factors contributing to happiness? How did country ranks or scores change between the 2015 and 2016 as well as the 2016 and 2017 reports? Did any country experience a significant increase or decrease in happiness? | |
| What is Dystopia? | |
| Dystopia is an imaginary country that has the world’s least-happy people. The purpose in establishing Dystopia is to have a benchmark against which all countries can be favorably compared (no country performs more poorly than Dystopia) in terms of each of the six key variables, thus allowing each sub-bar to be of positive width. The lowest scores observed for the six key variables, therefore, characterize Dystopia. Since life would be very unpleasant in a country with the world’s lowest incomes, lowest life expectancy, lowest generosity, most corruption, least freedom and least social support, it is referred to as “Dystopia,” in contrast to Utopia. | |
| What are the residuals? | |
| The residuals, or unexplained components, differ for each country, reflecting the extent to which the six variables either over- or under-explain average 2014-2016 life evaluations. These residuals have an average value of approximately zero over the whole set of countries. Figure 2.2 shows the average residual for each country when the equation in Table 2.1 is applied to average 2014- 2016 data for the six variables in that country. We combine these residuals with the estimate for life evaluations in Dystopia so that the combined bar will always have positive values. As can be seen in Figure 2.2, although some life evaluation residuals are quite large, occasionally exceeding one point on the scale from 0 to 10, they are always much smaller than the calculated value in Dystopia, where the average life is rated at 1.85 on the 0 to 10 scale. | |
| What do the columns succeeding the Happiness Score(like Family, Generosity, etc.) describe? | |
| The following columns: GDP per Capita, Family, Life Expectancy, Freedom, Generosity, Trust Government Corruption describe the extent to which these factors contribute in evaluating the happiness in each country. The Dystopia Residual metric actually is the Dystopia Happiness Score(1.85) + the Residual value or the unexplained value for each country as stated in the previous answer. | |
| If you add all these factors up, you get the happiness score so it might be un-reliable to model them to predict Happiness Scores. | |
| Start a new kernel" | |
| Fashion MNIST,"An MNIST-like dataset of 70,000 28x28 labeled fashion images",Zalando Research,268,"Version 4,2017-12-07|Version 3,2017-08-31|Version 2,2017-08-30|Version 1,2017-08-29","clothing | |
| image data | |
| multiclass classification | |
| object identification",Other,69 MB,Other,"53,821 views","6,059 downloads",119 kernels,6 topics,https://www.kaggle.com/zalando-research/fashionmnist,"Context | |
| Fashion-MNIST is a dataset of Zalando's article images—consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is a 28x28 grayscale image, associated with a label from 10 classes. Zalando intends Fashion-MNIST to serve as a direct drop-in replacement for the original MNIST dataset for benchmarking machine learning algorithms. It shares the same image size and structure of training and testing splits. | |
| The original MNIST dataset contains a lot of handwritten digits. Members of the AI/ML/Data Science community love this dataset and use it as a benchmark to validate their algorithms. In fact, MNIST is often the first dataset researchers try. ""If it doesn't work on MNIST, it won't work at all"", they said. ""Well, if it does work on MNIST, it may still fail on others."" | |
| Zalando seeks to replace the original MNIST dataset | |
| Content | |
| Each image is 28 pixels in height and 28 pixels in width, for a total of 784 pixels in total. Each pixel has a single pixel-value associated with it, indicating the lightness or darkness of that pixel, with higher numbers meaning darker. This pixel-value is an integer between 0 and 255. The training and test data sets have 785 columns. The first column consists of the class labels (see above), and represents the article of clothing. The rest of the columns contain the pixel-values of the associated image. | |
| To locate a pixel on the image, suppose that we have decomposed x as x = i * 28 + j, where i and j are integers between 0 and 27. The pixel is located on row i and column j of a 28 x 28 matrix. | |
| For example, pixel31 indicates the pixel that is in the fourth column from the left, and the second row from the top, as in the ascii-diagram below. | |
| Labels | |
| Each training and test example is assigned to one of the following labels: | |
| 0 T-shirt/top | |
| 1 Trouser | |
| 2 Pullover | |
| 3 Dress | |
| 4 Coat | |
| 5 Sandal | |
| 6 Shirt | |
| 7 Sneaker | |
| 8 Bag | |
| 9 Ankle boot | |
| TL;DR | |
| Each row is a separate image | |
| Column 1 is the class label. | |
| Remaining columns are pixel numbers (784 total). | |
| Each value is the darkness of the pixel (1 to 255) | |
| Acknowledgements | |
| Original dataset was downloaded from https://github.com/zalandoresearch/fashion-mnist | |
| Dataset was converted to CSV with this script: https://pjreddie.com/projects/mnist-in-csv/ | |
| License | |
| The MIT License (MIT) Copyright © [2017] Zalando SE, https://tech.zalando.com | |
| Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: | |
| The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. | |
| THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE." | |
| 2016 US Election,Explore data related to the 2016 US Election,Ben Hamner,261,"Version 8,2016-07-01|Version 7,2016-07-01|Version 6,2016-07-01|Version 5,2016-03-26|Version 4,2016-02-29|Version 3,2016-02-29|Version 2,2016-02-26|Version 1,2016-02-19",politics,CSV,48 MB,CC4,"140,095 views","17,273 downloads",260 kernels,17 topics,https://www.kaggle.com/benhamner/2016-us-election,"This contains data relevant for the 2016 US Presidential Election, including up-to-date primary results. | |
| Exploration Ideas | |
| What candidates within the Republican party have results that are the most anti-correlated? | |
| Which Republican candidate is Hillary Clinton most correlated with based on county voting patterns? What about Bernie Sanders? | |
| What insights can you discover by mapping this data? | |
| Do you have answers or other exploration ideas? Add your ideas to this forum post and share your insights through Kaggle Scripts! | |
| Do you think that we should augment this dataset with more data sources? Submit a pull request to this repo, or let us know here! | |
| Data Description | |
| The 2016 US Election dataset contains several main files and folders at the moment. You may download the entire archive via the ""Download Data"" link at the top of the page, or interact with the data in Kaggle Scripts through the ../input directory. | |
| Original Data Sources | |
| Primary Results from CNN | |
| New Hampshire County-Level Results | |
| County Shapefiles | |
| County QuickFacts" | |
| Game of Thrones,Explore deaths and battles from this fantasy world,Myles O'Neill,258,"Version 1,2016-05-20","literature | |
| social groups | |
| war",CSV,257 KB,CC0,"113,897 views","13,525 downloads",206 kernels,5 topics,https://www.kaggle.com/mylesoneill/game-of-thrones,"Overview | |
| Game of Thrones is a hit fantasy tv show based on the equally famous book series ""A Song of Fire and Ice"" by George RR Martin. The show is well known for its vastly complicated political landscape, large number of characters, and its frequent character deaths. | |
| Data Sources | |
| This dataset combines three sources of data, all of which are based on information from the book series. | |
| Firstly, there is battles.csv which contains Chris Albon's ""The War of the Five Kings"" Dataset, which can be found here: https://github.com/chrisalbon/war_of_the_five_kings_dataset . Its a great collection of all of the battles in the series. | |
| Secondly we have character-deaths.csv from Erin Pierce and Ben Kahle. This dataset was created as a part of their Bayesian Survival Analysis which can be found here: http://allendowney.blogspot.com/2015/03/bayesian-survival-analysis-for-game-of.html | |
| Finally we have a more comprehensive character dataset with character-predictions.csv. This comes from the team at A Song of Ice and Data who scraped it from http://awoiaf.westeros.org/ . It also includes their predictions on which character will die, the methodology of which can be found here: https://got.show/machine-learning-algorithm-predicts-death-game-of-thrones | |
| What insights about the complicated political landscape of this fantasy world can you find in this data? | |
| Of course, it goes without saying that this dataset contains spoilers ;)" | |
| Speed Dating Experiment,What attributes influence the selection of a romantic partner?,Anna Montoya,256,"Version 1,2016-05-09","love | |
| sociology",Other,5 MB,Other,"119,295 views","24,348 downloads",240 kernels,7 topics,https://www.kaggle.com/annavictoria/speed-dating-experiment,"What influences love at first sight? (Or, at least, love in the first four minutes?) This dataset was compiled by Columbia Business School professors Ray Fisman and Sheena Iyengar for their paper Gender Differences in Mate Selection: Evidence From a Speed Dating Experiment. | |
| Data was gathered from participants in experimental speed dating events from 2002-2004. During the events, the attendees would have a four minute ""first date"" with every other participant of the opposite sex. At the end of their four minutes, participants were asked if they would like to see their date again. They were also asked to rate their date on six attributes: Attractiveness, Sincerity, Intelligence, Fun, Ambition, and Shared Interests. | |
| The dataset also includes questionnaire data gathered from participants at different points in the process. These fields include: demographics, dating habits, self-perception across key attributes, beliefs on what others find valuable in a mate, and lifestyle information. See the Speed Dating Data Key document below for details. | |
| For more analysis from Iyengar and Fisman, read Racial Preferences in Dating. | |
| Data Exploration Ideas | |
| What are the least desirable attributes in a male partner? Does this differ for female partners? | |
| How important do people think attractiveness is in potential mate selection vs. its real impact? | |
| Are shared interests more important than a shared racial background? | |
| Can people accurately predict their own perceived value in the dating market? | |
| In terms of getting a second date, is it better to be someone's first speed date of the night or their last?" | |
| Football Events,"More than 900,000 events from 9,074 football games across Europe",Alin Secareanu,256,"Version 1,2017-01-25",association football,CSV,174 MB,Other,"49,559 views","9,488 downloads",89 kernels,5 topics,https://www.kaggle.com/secareanualin/football-events,"Context | |
| Most publicly available football (soccer) statistics are limited to aggregated data such as Goals, Shots, Fouls, Cards. When assessing performance or building predictive models, this simple aggregation, without any context, can be misleading. For example, a team that produced 10 shots on target from long range has a lower chance of scoring than a club that produced the same amount of shots from inside the box. However, metrics derived from this simple count of shots will similarly asses the two teams. | |
| A football game generates much more events and it is very important and interesting to take into account the context in which those events were generated. This dataset should keep sports analytics enthusiasts awake for long hours as the number of questions that can be asked is huge. | |
| Content | |
| This dataset is a result of a very tiresome effort of webscraping and integrating different data sources. The central element is the text commentary. All the events were derived by reverse engineering the text commentary, using regex. Using this, I was able to derive 11 types of events, as well as the main player and secondary player involved in those events and many other statistics. In case I've missed extracting some useful information, you are gladly invited to do so and share your findings. The dataset provides a granular view of 9,074 games, totaling 941,009 events from the biggest 5 European football (soccer) leagues: England, Spain, Germany, Italy, France from 2011/2012 season to 2016/2017 season as of 25.01.2017. There are games that have been played during these seasons for which I could not collect detailed data. Overall, over 90% of the played games during these seasons have event data. | |
| The dataset is organized in 3 files: | |
| events.csv contains event data about each game. Text commentary was scraped from: bbc.com, espn.com and onefootball.com | |
| ginf.csv - contains metadata and market odds about each game. odds were collected from oddsportal.com | |
| dictionary.txt contains a dictionary with the textual description of each categorical variable coded with integers | |
| Past Research | |
| I have used this data to: | |
| create predictive models for football games in order to bet on football outcomes. | |
| make visualizations about upcoming games | |
| build expected goals models and compare players | |
| Inspiration | |
| There are tons of interesting questions a sports enthusiast can answer with this dataset. For example: | |
| What is the value of a shot? Or what is the probability of a shot being a goal given it's location, shooter, league, assist method, gamestate, number of players on the pitch, time - known as expected goals (xG) models | |
| When are teams more likely to score? | |
| Which teams are the best or sloppiest at holding the lead? | |
| Which teams or players make the best use of set pieces? | |
| In which leagues is the referee more likely to give a card? | |
| How do players compare when they shoot with their week foot versus strong foot? Or which players are ambidextrous? | |
| Identify different styles of plays (shooting from long range vs shooting from the box, crossing the ball vs passing the ball, use of headers) | |
| Which teams have a bias for attacking on a particular flank? | |
| And many many more..." | |
| Zillow Economics Data,Turning on the lights in housing research.,Zillow,253,"Version 6,2018-01-25|Version 5,2018-01-13|Version 4,2017-12-08|Version 3,2017-12-08|Version 2,2017-11-14|Version 1,2017-10-25","housing | |
| business | |
| demographics | |
| economics",CSV,511 MB,Other,"39,924 views","4,014 downloads",27 kernels,5 topics,https://www.kaggle.com/zillow/zecon,"Context | |
| Zillow's Economic Research Team collects, cleans and publishes housing and economic data from a variety of public and proprietary sources. Public property record data filed with local municipalities -- including deeds, property facts, parcel information and transactional histories -- forms the backbone of our data products, and is fleshed out with proprietary data derived from property listings and user behavior on Zillow. | |
| The large majority of Zillow's aggregated housing market and economic data is made available for free download at zillow.com/data. | |
| Content | |
| Variable Availability: | |
| Zillow Home Value Index (ZHVI): A smoothed seasonally adjusted measure of the median estimated home value across a given region and housing type. A dollar denominated alternative to repeat-sales indices. Find a more detailed methodology here: http://www.zillow.com/research/zhvi-methodology-6032/ | |
| Zillow Rent Index (ZRI): A smoothed seasonally adjusted measure of the median estimated market rate rent across a given region and housing type. A dollar denominated alternative to repeat-rent indices. Find a more detailed methodology here: http://www.zillow.com/research/zillow-rent-index-methodology-2393/ | |
| For-Sale Listing/Inventory Metrics: Zillow provides many variables capturing current and historical for-sale listings availability, generally from 2012 to current. These variables include median list prices and inventory counts, both by various property types. Variables capturing for-sale market competitiveness including share of listings with a price cut, median price cut size, age of inventory, and the days a listing spend on Zillow before the sale is final. | |
| Home Sales Metrics: Zillow provides data on sold homes including median sale price by various housing types, sale counts (methodology here: http://www.zillow.com/research/home-sales-methodology-7733/), and a normalized view of sale volume referred to as turnover. The prevalence of foreclosures is also provided as ratio of the housing stock and the share of all sales in which the home was previously foreclosed upon. | |
| For-Rent Listing Metrics: Zillow provides median rents prices and median rent price per square foot by property type and bedroom count. | |
| Housing type definitions: | |
| All Homes: Zillow defines all homes as single-family, condominium and co-operative homes with a county record. Unless specified, all series cover this segment of the housing stock. | |
| Condo/Co-op: Condominium and co-operative homes. | |
| Multifamily 5+ units: Units in buildings with 5 or more housing units, that are not a condominiums or co-ops. | |
| Duplex/Triplex: Housing units in buildings with 2 or 3 housing units. | |
| Tiers: By metro, we determine price tier cutoffs that divide the all homes housing stock into thirds using the full distribution of estimated home values. We then estimate real estate metrics within the property sets, Bottom, Middle, and Top, defined by these cutoffs. When reported at the national level, all Bottom Tier homes defined at the metro level are pooled together to form the national bottom tier. The same holds for Middle and Top Tier homes. | |
| Regional Availability: | |
| Zillow metrics are reported for common US geographies including Nation, State, Metro (2013 Census Defined CBSAs), County, City, ZIP code, and Neighborhood. | |
| We provide a crosswalk between colloquial Zillow region names and federally defined region names and linking variables such as County FIPS codes and CBSA codes. Cities and Neighborhoods do not match standard jurisdictional boundaries. Zillow city boundaries reflect mailing address conventions and so are often visually similar to collections of ZIP codes. Zillow neighborhood boundaries can be found here. | |
| Suppression Rules: To ensure reliability of reported values the Zillow Economic Research team applies suppression rules triggered by low sample sizes and excessive volatility. These rules are customized to the metric and region type and explain most missingness found in the provided datasets. | |
| Additional Data Products | |
| The following data products and more are available for free download exclusively at Zillow.com/Data: | |
| Zillow Home Value Forecast | |
| Zillow Rent Forecast | |
| Negative Equity (the share of mortgaged properties worth less than mortgage balance) | |
| Zillow Home Price Expectations Survey | |
| Zillow Housing Aspirations Report | |
| Zillow Rising Sea Levels Research | |
| Cash Buyers Time Series | |
| Buy vs. Rent Breakeven Horizon | |
| Mortgage Affordability, Rental Affordability, Price-to-Income Ratio | |
| Conventional 30-year Fixed Mortgage Rate, Weekly Time Series | |
| Jumbo 30-year Fixed Mortgage Rates, Weekly Time Series | |
| Acknowledgements | |
| The mission of the Zillow Economic Research Team is to be the most open, authoritative source for timely and accurate housing data and unbiased insight. We aim to empower consumers, industry professionals, policy makers and researchers looking to better understand the housing market. | |
| To see more of our mission in action, we invite you to learn more about us and to check out our collection of research briefs, stories, data tools and past presentations at https://www.zillow.com/research/ | |
| Inspiration | |
| Zillow, and the Zillow Economic Research Team, firmly believe that not only do data want to be free, data are going to be free. Instead of simply publishing raw data, we believe in the power of pushing data up the ladder from raw data bits, to actionable information and finally to unique insight. We aim to answer questions of all kinds, even questions our users may not have known they had before coming to us. When done right, we firmly believe this process of turning data into insight can be transformational in people's lives. | |
| Please join us on this journey, and we're excited to see what insights you can discover hidden amongst our data!" | |
| "House Sales in King County, USA",Predict house price using regression,harlfoxem,249,"Version 1,2016-08-25","home | |
| finance",CSV,2 MB,CC0,"100,966 views","16,286 downloads",406 kernels,11 topics,https://www.kaggle.com/harlfoxem/housesalesprediction,"This dataset contains house sale prices for King County, which includes Seattle. It includes homes sold between May 2014 and May 2015. | |
| It's a great dataset for evaluating simple regression models." | |
| The Movies Dataset,"Metadata on over 45,000 movies. 26 million ratings from over 270,000 users.",Rounak Banik,246,"Version 7,2017-11-10|Version 6,2017-11-10|Version 5,2017-10-25|Version 4,2017-10-25|Version 3,2017-10-25|Version 2,2017-10-25|Version 1,2017-10-25","popular culture | |
| film",CSV,900 MB,CC0,"51,102 views","7,834 downloads",22 kernels,4 topics,https://www.kaggle.com/rounakbanik/the-movies-dataset,"Context | |
| These files contain metadata for all 45,000 movies listed in the Full MovieLens Dataset. The dataset consists of movies released on or before July 2017. Data points include cast, crew, plot keywords, budget, revenue, posters, release dates, languages, production companies, countries, TMDB vote counts and vote averages. | |
| This dataset also has files containing 26 million ratings from 270,000 users for all 45,000 movies. Ratings are on a scale of 1-5 and have been obtained from the official GroupLens website. | |
| Content | |
| This dataset consists of the following files: | |
| movies_metadata.csv: The main Movies Metadata file. Contains information on 45,000 movies featured in the Full MovieLens dataset. Features include posters, backdrops, budget, revenue, release dates, languages, production countries and companies. | |
| keywords.csv: Contains the movie plot keywords for our MovieLens movies. Available in the form of a stringified JSON Object. | |
| credits.csv: Consists of Cast and Crew Information for all our movies. Available in the form of a stringified JSON Object. | |
| links.csv: The file that contains the TMDB and IMDB IDs of all the movies featured in the Full MovieLens dataset. | |
| links_small.csv: Contains the TMDB and IMDB IDs of a small subset of 9,000 movies of the Full Dataset. | |
| ratings_small.csv: The subset of 100,000 ratings from 700 users on 9,000 movies. | |
| The Full MovieLens Dataset consisting of 26 million ratings and 750,000 tag applications from 270,000 users on all the 45,000 movies in this dataset can be accessed here | |
| Acknowledgements | |
| This dataset is an ensemble of data collected from TMDB and GroupLens. The Movie Details, Credits and Keywords have been collected from the TMDB Open API. This product uses the TMDb API but is not endorsed or certified by TMDb. Their API also provides access to data on many additional movies, actors and actresses, crew members, and TV shows. You can try it for yourself here. | |
| The Movie Links and Ratings have been obtained from the Official GroupLens website. The files are a part of the dataset available here | |
| Inspiration | |
| This dataset was assembled as part of my second Capstone Project for Springboard's Data Science Career Track. I wanted to perform an extensive EDA on Movie Data to narrate the history and the story of Cinema and use this metadata in combination with MovieLens ratings to build various types of Recommender Systems. | |
| Both my notebooks are available as kernels with this dataset: The Story of Film and Movie Recommender Systems | |
| Some of the things you can do with this dataset: Predicting movie revenue and/or movie success based on a certain metric. What movies tend to get higher vote counts and vote averages on TMDB? Building Content Based and Collaborative Filtering Based Recommendation Engines." | |
| New York Stock Exchange,S&P 500 companies historical prices with fundamental data,Dominik Gawlik,236,"Version 3,2017-02-22|Version 2,2017-02-20|Version 1,2017-02-19",finance,CSV,101 MB,CC0,"67,291 views","11,236 downloads",209 kernels,7 topics,https://www.kaggle.com/dgawlik/nyse,"Context | |
| This dataset is a playground for fundamental and technical analysis. It is said that 30% of traffic on stocks is already generated by machines, can trading be fully automated? If not, there is still a lot to learn from historical data. | |
| Content | |
| Dataset consists of following files: | |
| prices.csv: raw, as-is daily prices. Most of data spans from 2010 to the end 2016, for companies new on stock market date range is shorter. There have been approx. 140 stock splits in that time, this set doesn't account for that. | |
| prices-split-adjusted.csv: same as prices, but there have been added adjustments for splits. | |
| securities.csv: general description of each company with division on sectors | |
| fundamentals.csv: metrics extracted from annual SEC 10K fillings (2012-2016), should be enough to derive most of popular fundamental indicators. | |
| Acknowledgements | |
| Prices were fetched from Yahoo Finance, fundamentals are from Nasdaq Financials, extended by some fields from EDGAR SEC databases. | |
| Mining | |
| Here is couple of things one could try out with this data: | |
| Technical | |
| One day ahead prediction: Rolling Linear Regression, ARIMA, Neural Networks, LSTM | |
| Momentum/Mean-Reversion Strategies | |
| Security clustering, portfolio construction/hedging | |
| Fundamental | |
| Which company has biggest chance of being bankrupt? Which one is undervalued (how prices behaved afterwards), what is Return on Investment?" | |
| Cryptocurrency Historical Prices,"Prices of top cryptocurrencies including Bitcoin, Ethereum, Ripple, Bitcoin cash",SRK,235,"Version 13,2018-02-21|Version 12,2017-11-08|Version 11,2017-10-04|Version 10,2017-09-19|Version 9,2017-09-19|Version 8,2017-09-06|Version 7,2017-08-30|Version 6,2017-08-29|Version 5,2017-08-28|Version 4,2017-08-27|Version 3,2017-08-27|Version 2,2017-08-16|Version 1,2017-08-09","history | |
| finance",CSV,2 MB,CC0,"87,120 views","11,620 downloads",30 kernels,10 topics,https://www.kaggle.com/sudalairajkumar/cryptocurrencypricehistory,"Context | |
| In the last few days, I have been hearing a lot of buzz around cryptocurrencies. Things like Block chain, Bitcoin, Bitcoin cash, Ethereum, Ripple etc are constantly coming in the news articles I read. So I wanted to understand more about it and this post helped me get started. Once the basics are done, the DS guy sleeping inside me (always lazy.!) woke up and started raising questions like | |
| How many such cryptocurrencies are there and what are their prices and valuations? | |
| Why is there a sudden surge in the interest in recent days? Is it due to the increase in the price in the last few days? etc. | |
| For getting answers to all these questions (and if possible to predict the future prices ;)), I started getting the data from coinmarketcap about the cryptocurrencies. | |
| Update : Bitcoin dataset | |
| So what next.? Now that we have the price data, I wanted to dig a little more about the factors affecting the price of coins. I started of with Bitcoin and there are quite a few parameters which affect the price of Bitcoin. Thanks to Blockchain Info, I was able to get quite a few parameters on once in two day basis. | |
| This will help understand the other factors related to Bitcoin price and also help one make future predictions in a better way than just using the historical price. | |
| Update2: Ethereum Dataset: | |
| This dataset has features related to Ethereum. This is very similar to the bitcoin dataset and is available on a daily basis. Data is taken from Etherscan and the credits go to them for allowing us to use. | |
| Content | |
| This dataset has the historical price information of some of the top cryptocurrencies by market capitalization. The currencies included are | |
| Bitcoin | |
| Ethereum | |
| Ripple | |
| Bitcoin cash | |
| Bitconnect | |
| Dash | |
| Ethereum Classic | |
| Iota | |
| Litecoin | |
| Monero | |
| Nem | |
| Neo | |
| Numeraire | |
| Stratis | |
| Waves | |
| In case if you are interested in the prices of some other currencies, please post in comments section and I will try to add them in the next version. I am planning to revise it once in a week. | |
| Dataset has one csv file for each currency. Price history is available on a daily basis from April 28, 2013. The columns in the csv file are | |
| Date : date of observation | |
| Open : Opening price on the given day | |
| High : Highest price on the given day | |
| Low : Lowest price on the given day | |
| Close : Closing price on the given day | |
| Volume : Volume of transactions on the given day | |
| Market Cap : Market capitalization in USD | |
| Bitcoin Dataset (bitcoin_dataset.csv) : | |
| This dataset has the following features. | |
| Date : Date of observation | |
| btc_market_price : Average USD market price across major bitcoin exchanges. | |
| btc_total_bitcoins : The total number of bitcoins that have already been mined. | |
| btc_market_cap : The total USD value of bitcoin supply in circulation. | |
| btc_trade_volume : The total USD value of trading volume on major bitcoin exchanges. | |
| btc_blocks_size : The total size of all block headers and transactions. | |
| btc_avg_block_size : The average block size in MB. | |
| btc_n_orphaned_blocks : The total number of blocks mined but ultimately not attached to the main Bitcoin blockchain. | |
| btc_n_transactions_per_block : The average number of transactions per block. | |
| btc_median_confirmation_time : The median time for a transaction to be accepted into a mined block. | |
| btc_hash_rate : The estimated number of tera hashes per second the Bitcoin network is performing. | |
| btc_difficulty : A relative measure of how difficult it is to find a new block. | |
| btc_miners_revenue : Total value of coinbase block rewards and transaction fees paid to miners. | |
| btc_transaction_fees : The total value of all transaction fees paid to miners. | |
| btc_cost_per_transaction_percent : miners revenue as percentage of the transaction volume. | |
| btc_cost_per_transaction : miners revenue divided by the number of transactions. | |
| btc_n_unique_addresses : The total number of unique addresses used on the Bitcoin blockchain. | |
| btc_n_transactions : The number of daily confirmed Bitcoin transactions. | |
| btc_n_transactions_total : Total number of transactions. | |
| btc_n_transactions_excluding_popular : The total number of Bitcoin transactions, excluding the 100 most popular addresses. | |
| btc_n_transactions_excluding_chains_longer_than_100 : The total number of Bitcoin transactions per day excluding long transaction chains. | |
| btc_output_volume : The total value of all transaction outputs per day. | |
| btc_estimated_transaction_volume : The total estimated value of transactions on the Bitcoin blockchain. | |
| btc_estimated_transaction_volume_usd : The estimated transaction value in USD value. | |
| Ethereum Dataset (ethereum_dataset.csv): | |
| This dataset has the following features | |
| Date(UTC) : Date of transaction | |
| UnixTimeStamp : unix timestamp | |
| eth_etherprice : price of ethereum | |
| eth_tx : number of transactions per day | |
| eth_address : Cumulative address growth | |
| eth_supply : Number of ethers in supply | |
| eth_marketcap : Market cap in USD | |
| eth_hashrate : hash rate in GH/s | |
| eth_difficulty : Difficulty level in TH | |
| eth_blocks : number of blocks per day | |
| eth_uncles : number of uncles per day | |
| eth_blocksize : average block size in bytes | |
| eth_blocktime : average block time in seconds | |
| eth_gasprice : Average gas price in Wei | |
| eth_gaslimit : Gas limit per day | |
| eth_gasused : total gas used per day | |
| eth_ethersupply : new ether supply per day | |
| eth_chaindatasize : chain data size in bytes | |
| eth_ens_register : Ethereal Name Service (ENS) registrations per day | |
| Acknowledgements | |
| This data is taken from coinmarketcap and it is free to use the data. | |
| Bitcoin dataset is obtained from Blockchain Info. | |
| Ethereum dataset is obtained from Etherscan. | |
| Cover Image : Photo by Thomas Malama on Unsplash | |
| Inspiration | |
| Some of the questions which could be inferred from this dataset are: | |
| How did the historical prices / market capitalizations of various currencies change over time? | |
| Predicting the future price of the currencies | |
| Which currencies are more volatile and which ones are more stable? | |
| How does the price fluctuations of currencies correlate with each other? | |
| Seasonal trend in the price fluctuations | |
| Bitcoin / Ethereum dataset could be used to look at the following: | |
| Factors affecting the bitcoin / ether price. | |
| Directional prediction of bitcoin / ether price. (refer this paper for more inspiration) | |
| Actual bitcoin price prediction." | |
| TED Talks,"Data about TED Talks on the TED.com website until September 21st, 2017",Rounak Banik,233,"Version 3,2017-09-26|Version 2,2017-09-10|Version 1,2017-09-09","society | |
| linguistics",CSV,34 MB,CC4,"54,830 views","6,135 downloads",28 kernels,4 topics,https://www.kaggle.com/rounakbanik/ted-talks,"Context | |
| These datasets contain information about all audio-video recordings of TED Talks uploaded to the official TED.com website until September 21st, 2017. The TED main dataset contains information about all talks including number of views, number of comments, descriptions, speakers and titles. The TED transcripts dataset contains the transcripts for all talks available on TED.com. | |
| Content (for the CSV files) | |
| TED Main Dataset | |
| name: The official name of the TED Talk. Includes the title and the speaker. | |
| title: The title of the talk | |
| description: A blurb of what the talk is about. | |
| main_speaker: The first named speaker of the talk. | |
| speaker_occupation: The occupation of the main speaker. | |
| num_speaker: The number of speakers in the talk. | |
| duration: The duration of the talk in seconds. | |
| event: The TED/TEDx event where the talk took place. | |
| film_date: The Unix timestamp of the filming. | |
| published_date: The Unix timestamp for the publication of the talk on TED.com | |
| comments: The number of first level comments made on the talk. | |
| tags: The themes associated with the talk. | |
| languages: The number of languages in which the talk is available. | |
| ratings: A stringified dictionary of the various ratings given to the talk (inspiring, fascinating, jaw dropping, etc.) | |
| related_talks: A list of dictionaries of recommended talks to watch next. | |
| url: The URL of the talk. | |
| views: The number of views on the talk. | |
| TED Transcripts Dataset | |
| url: The URL of the talk | |
| transcript: The official English transcript of the talk. | |
| Acknowledgements | |
| The data has been scraped from the official TED Website and is available under the Creative Commons License. | |
| Inspiration | |
| I've always been fascinated by TED Talks and the immense diversity of content that it provides for free. I was also thoroughly inspired by a TED Talk that visually explored TED Talks stats and I was motivated to do the same thing, albeit on a much less grander scale. | |
| Some of the questions that can be answered with this dataset: 1. How is each TED Talk related to every other TED Talk? 2. Which are the most viewed and most favorited Talks of all time? Are they mostly the same? What does this tell us? 3. What kind of topics attract the maximum discussion and debate (in the form of comments)? 4. Which months are most popular among TED and TEDx chapters? 5. Which themes are most popular amongst TEDsters?" | |
| Health Insurance Marketplace,Explore health and dental plans data in the US Health Insurance Marketplace,US Department of Health and Human Services,232,"Version 2,2017-05-02|Version 1,2016-01-20","healthcare | |
| economics",CSV,11 GB,CC0,"79,817 views","10,565 downloads",203 kernels,12 topics,https://www.kaggle.com/hhs/health-insurance-marketplace,"The Health Insurance Marketplace Public Use Files contain data on health and dental plans offered to individuals and small businesses through the US Health Insurance Marketplace. | |
| Exploration Ideas | |
| To help get you started, here are some data exploration ideas: | |
| How do plan rates and benefits vary across states? | |
| How do plan benefits relate to plan rates? | |
| How do plan rates vary by age? | |
| How do plans vary across insurance network providers? | |
| See this forum thread for more ideas, and post there if you want to add your own ideas or answer some of the open questions! | |
| Data Description | |
| This data was originally prepared and released by the Centers for Medicare & Medicaid Services (CMS). Please read the CMS Disclaimer-User Agreement before using this data. | |
| Here, we've processed the data to facilitate analytics. This processed version has three components: | |
| 1. Original versions of the data | |
| The original versions of the 2014, 2015, 2016 data are available in the ""raw"" directory of the download and ""../input/raw"" on Kaggle Scripts. Search for ""dictionaries"" on this page to find the data dictionaries describing the individual raw files. | |
| 2. Combined CSV files that contain | |
| In the top level directory of the download (""../input"" on Kaggle Scripts), there are six CSV files that contain the combined at across all years: | |
| BenefitsCostSharing.csv | |
| BusinessRules.csv | |
| Network.csv | |
| PlanAttributes.csv | |
| Rate.csv | |
| ServiceArea.csv | |
| Additionally, there are two CSV files that facilitate joining data across years: | |
| Crosswalk2015.csv - joining 2014 and 2015 data | |
| Crosswalk2016.csv - joining 2015 and 2016 data | |
| 3. SQLite database | |
| The ""database.sqlite"" file contains tables corresponding to each of the processed CSV files. | |
| The code to create the processed version of this data is available on GitHub." | |
| (MBTI) Myers-Briggs Personality Type Dataset,Includes a large number of people's MBTI type and content written by them,Mitchell J,225,"Version 1,2017-09-22","personality | |
| demographics | |
| linguistics | |
| + 2 more...",CSV,60 MB,CC0,"42,504 views","4,267 downloads",19 kernels,2 topics,https://www.kaggle.com/datasnaek/mbti-type,"Context | |
| The Myers Briggs Type Indicator (or MBTI for short) is a personality type system that divides everyone into 16 distinct personality types across 4 axis: | |
| Introversion (I) – Extroversion (E) | |
| Intuition (N) – Sensing (S) | |
| Thinking (T) – Feeling (F) | |
| Judging (J) – Perceiving (P) | |
| (More can be learned about what these mean here) | |
| So for example, someone who prefers introversion, intuition, thinking and perceiving would be labelled an INTP in the MBTI system, and there are lots of personality based components that would model or describe this person’s preferences or behaviour based on the label. | |
| It is one of, if not the, the most popular personality test in the world. It is used in businesses, online, for fun, for research and lots more. A simple google search reveals all of the different ways the test has been used over time. It’s safe to say that this test is still very relevant in the world in terms of its use. | |
| From scientific or psychological perspective it is based on the work done on cognitive functions by Carl Jung i.e. Jungian Typology. This was a model of 8 distinct functions, thought processes or ways of thinking that were suggested to be present in the mind. Later this work was transformed into several different personality systems to make it more accessible, the most popular of which is of course the MBTI. | |
| Recently, its use/validity has come into question because of unreliability in experiments surrounding it, among other reasons. But it is still clung to as being a very useful tool in a lot of areas, and the purpose of this dataset is to help see if any patterns can be detected in specific types and their style of writing, which overall explores the validity of the test in analysing, predicting or categorising behaviour. | |
| Content | |
| This dataset contains over 8600 rows of data, on each row is a person’s: | |
| Type (This persons 4 letter MBTI code/type) | |
| A section of each of the last 50 things they have posted (Each entry separated by ""|||"" (3 pipe characters)) | |
| Acknowledgements | |
| This data was collected through the PersonalityCafe forum, as it provides a large selection of people and their MBTI personality type, as well as what they have written. | |
| Inspiration | |
| Some basic uses could include: | |
| Use machine learning to evaluate the MBTIs validity and ability to predict language styles and behaviour online. | |
| Production of a machine learning algorithm that can attempt to determine a person’s personality type based on some text they have written." | |
| World University Rankings,Investigate the best universities in the world,Myles O'Neill,224,"Version 3,2016-09-28|Version 2,2016-03-12|Version 1,2016-03-09",universities and colleges,CSV,11 MB,Other,"105,477 views","20,198 downloads",282 kernels,5 topics,https://www.kaggle.com/mylesoneill/world-university-rankings,"Of all the universities in the world, which are the best? | |
| Ranking universities is a difficult, political, and controversial practice. There are hundreds of different national and international university ranking systems, many of which disagree with each other. This dataset contains three global university rankings from very different places. | |
| University Ranking Data | |
| The Times Higher Education World University Ranking is widely regarded as one of the most influential and widely observed university measures. Founded in the United Kingdom in 2010, it has been criticized for its commercialization and for undermining non-English-instructing institutions. | |
| The Academic Ranking of World Universities, also known as the Shanghai Ranking, is an equally influential ranking. It was founded in China in 2003 and has been criticized for focusing on raw research power and for undermining humanities and quality of instruction. | |
| The Center for World University Rankings, is a less well know listing that comes from Saudi Arabia, it was founded in 2012. | |
| How do these rankings compare to each other? | |
| Are the various criticisms levied against these rankings fair or not? | |
| How does your alma mater fare against the world? | |
| Supplementary Data | |
| To further extend your analyses, we've also included two sets of supplementary data. | |
| The first of these is a set of data on educational attainment around the world. It comes from The World Data Bank and comprises information from the UNESCO Institute for Statistics and the Barro-Lee Dataset. How does national educational attainment relate to the quality of each nation's universities? | |
| The second supplementary dataset contains information about public and private direct expenditure on education across nations. This data comes from the National Center for Education Statistics. It represents expenditure as a percentage of gross domestic product. Does spending more on education lead to better international university rankings?" | |
| Mental Health in Tech Survey,Survey on Mental Health in the Tech Workplace in 2014,"Open Sourcing Mental Illness, LTD",224,"Version 3,2016-11-04|Version 2,2016-11-04|Version 1,2016-11-03","mental health | |
| healthcare | |
| human genetics | |
| + 2 more...",CSV,297 KB,CC4,"62,030 views","8,712 downloads",108 kernels,5 topics,https://www.kaggle.com/osmi/mental-health-in-tech-survey,"Dataset Information | |
| This dataset is from a 2014 survey that measures attitudes towards mental health and frequency of mental health disorders in the tech workplace. You are also encouraged to analyze data from the ongoing 2016 survey found here. | |
| Content | |
| This dataset contains the following data: | |
| Timestamp | |
| Age | |
| Gender | |
| Country | |
| state: If you live in the United States, which state or territory do you live in? | |
| self_employed: Are you self-employed? | |
| family_history: Do you have a family history of mental illness? | |
| treatment: Have you sought treatment for a mental health condition? | |
| work_interfere: If you have a mental health condition, do you feel that it interferes with your work? | |
| no_employees: How many employees does your company or organization have? | |
| remote_work: Do you work remotely (outside of an office) at least 50% of the time? | |
| tech_company: Is your employer primarily a tech company/organization? | |
| benefits: Does your employer provide mental health benefits? | |
| care_options: Do you know the options for mental health care your employer provides? | |
| wellness_program: Has your employer ever discussed mental health as part of an employee wellness program? | |
| seek_help: Does your employer provide resources to learn more about mental health issues and how to seek help? | |
| anonymity: Is your anonymity protected if you choose to take advantage of mental health or substance abuse treatment resources? | |
| leave: How easy is it for you to take medical leave for a mental health condition? | |
| mental_health_consequence: Do you think that discussing a mental health issue with your employer would have negative consequences? | |
| phys_health_consequence: Do you think that discussing a physical health issue with your employer would have negative consequences? | |
| coworkers: Would you be willing to discuss a mental health issue with your coworkers? | |
| supervisor: Would you be willing to discuss a mental health issue with your direct supervisor(s)? | |
| mental_health_interview: Would you bring up a mental health issue with a potential employer in an interview? | |
| phys_health_interview: Would you bring up a physical health issue with a potential employer in an interview? | |
| mental_vs_physical: Do you feel that your employer takes mental health as seriously as physical health? | |
| obs_consequence: Have you heard of or observed negative consequences for coworkers with mental health conditions in your workplace? | |
| comments: Any additional notes or comments | |
| Inspiration | |
| Some questions worth exploring: | |
| How does the frequency of mental health illness and attitudes towards mental health vary by geographic location? | |
| What are the strongest predictors of mental health illness or certain attitudes towards mental health in the workplace? | |
| Acknowledgements | |
| The original dataset is from Open Sourcing Mental Illness and can be downloaded here." | |
| Huge Stock Market Dataset,Historical daily prices and volumes of all U.S. stocks and ETFs,Boris Marjanovic,221,"Version 3,2017-11-16|Version 2,2017-11-14|Version 1,2017-11-13","business | |
| finance | |
| economics | |
| artificial intelligence",Other,245 MB,CC0,"22,421 views","3,593 downloads",10 kernels,6 topics,https://www.kaggle.com/borismarjanovic/price-volume-data-for-all-us-stocks-etfs,"Context | |
| High-quality financial data is expensive to acquire and is therefore rarely shared for free. Here I provide the full historical daily price and volume data for all U.S.-based stocks and ETFs trading on the NYSE, NASDAQ, and NYSE MKT. It's one of the best datasets of its kind you can obtain. | |
| Content | |
| The data (last updated 11/10/2017) is presented in CSV format as follows: Date, Open, High, Low, Close, Volume, OpenInt. Note that prices have been adjusted for dividends and splits. | |
| Acknowledgements | |
| This dataset belongs to me. I’m sharing it here for free. You may do with it as you wish. | |
| Inspiration | |
| Many have tried, but most have failed, to predict the stock market's ups and downs. Can you do any better?" | |
| Medical Appointment No Shows,Why do 30% of patients miss their scheduled appointments?,JoniHoppen,218,"Version 5,2017-08-21|Version 4,2017-08-19|Version 3,2017-02-14|Version 2,2017-02-14|Version 1,2017-02-08","brazil | |
| healthcare | |
| public health",CSV,10 MB,CC4,"78,477 views","9,988 downloads",219 kernels,35 topics,https://www.kaggle.com/joniarroba/noshowappointments,"Context | |
| A person makes a doctor appointment, receives all the instructions and no-show. Who to blame? If this is help, don´t forget to upvote :) Greatings! | |
| Content | |
| 300k medical appointments and its 15 variables (characteristics) of each. The most important one if the patient show-up or no-show the appointment. Variable names are self-explanatory, if you have doubts, just let me know! | |
| scholarship variable means this concept = https://en.wikipedia.org/wiki/Bolsa_Fam%C3%ADlia | |
| Data Dictionary | |
| PatientId - Identification of a patient AppointmentID - Identification of each appointment Gender = Male or Female . Female is the greater proportion, woman takes way more care of they health in comparison to man. DataMarcacaoConsulta = The day of the actuall appointment, when they have to visit the doctor. DataAgendamento = The day someone called or registered the appointment, this is before appointment of course. Age = How old is the patient. Neighbourhood = Where the appointment takes place. Scholarship = Ture of False . Observation, this is a broad topic, consider reading this article https://en.wikipedia.org/wiki/Bolsa_Fam%C3%ADlia Hipertension = True or False Diabetes = True or False Alcoholism = True or False Handcap = True or False SMS_received = 1 or more messages sent to the patient. No-show = True or False. | |
| Inspiration | |
| What if that possible to predict someone to no-show an appointment?" | |
| Twitter US Airline Sentiment,Analyze how travelers in February 2015 expressed their feelings on Twitter,Crowdflower,212,"Version 2,2016-10-06|Version 1,2016-01-07","linguistics | |
| internet | |
| aviation",SQLite,8 MB,CC4,"92,154 views","12,779 downloads",233 kernels,8 topics,https://www.kaggle.com/crowdflower/twitter-airline-sentiment,"This data originally came from Crowdflower's Data for Everyone library. | |
| As the original source says, | |
| A sentiment analysis job about the problems of each major U.S. airline. Twitter data was scraped from February of 2015 and contributors were asked to first classify positive, negative, and neutral tweets, followed by categorizing negative reasons (such as ""late flight"" or ""rude service""). | |
| The data we're providing on Kaggle is a slightly reformatted version of the original source. It includes both a CSV file and SQLite database. The code that does these transformations is available on GitHub | |
| For example, it contains whether the sentiment of the tweets in this set was positive, neutral, or negative for six US airlines:" | |
| Gender Recognition by Voice,Identify a voice as male or female,Kory Becker,210,"Version 1,2016-08-26","gender | |
| linguistics",CSV,1 MB,CC4,"83,213 views","8,458 downloads",393 kernels,6 topics,https://www.kaggle.com/primaryobjects/voicegender,"Voice Gender | |
| Gender Recognition by Voice and Speech Analysis | |
| This database was created to identify a voice as male or female, based upon acoustic properties of the voice and speech. The dataset consists of 3,168 recorded voice samples, collected from male and female speakers. The voice samples are pre-processed by acoustic analysis in R using the seewave and tuneR packages, with an analyzed frequency range of 0hz-280hz (human vocal range). | |
| The Dataset | |
| The following acoustic properties of each voice are measured and included within the CSV: | |
| meanfreq: mean frequency (in kHz) | |
| sd: standard deviation of frequency | |
| median: median frequency (in kHz) | |
| Q25: first quantile (in kHz) | |
| Q75: third quantile (in kHz) | |
| IQR: interquantile range (in kHz) | |
| skew: skewness (see note in specprop description) | |
| kurt: kurtosis (see note in specprop description) | |
| sp.ent: spectral entropy | |
| sfm: spectral flatness | |
| mode: mode frequency | |
| centroid: frequency centroid (see specprop) | |
| peakf: peak frequency (frequency with highest energy) | |
| meanfun: average of fundamental frequency measured across acoustic signal | |
| minfun: minimum fundamental frequency measured across acoustic signal | |
| maxfun: maximum fundamental frequency measured across acoustic signal | |
| meandom: average of dominant frequency measured across acoustic signal | |
| mindom: minimum of dominant frequency measured across acoustic signal | |
| maxdom: maximum of dominant frequency measured across acoustic signal | |
| dfrange: range of dominant frequency measured across acoustic signal | |
| modindx: modulation index. Calculated as the accumulated absolute difference between adjacent measurements of fundamental frequencies divided by the frequency range | |
| label: male or female | |
| Accuracy | |
| Baseline (always predict male) | |
| 50% / 50% | |
| Logistic Regression | |
| 97% / 98% | |
| CART | |
| 96% / 97% | |
| Random Forest | |
| 100% / 98% | |
| SVM | |
| 100% / 99% | |
| XGBoost | |
| 100% / 99% | |
| Research Questions | |
| An original analysis of the data-set can be found in the following article: | |
| Identifying the Gender of a Voice using Machine Learning | |
| The best model achieves 99% accuracy on the test set. According to a CART model, it appears that looking at the mean fundamental frequency might be enough to accurately classify a voice. However, some male voices use a higher frequency, even though their resonance differs from female voices, and may be incorrectly classified as female. To the human ear, there is apparently more than simple frequency, that determines a voice's gender. | |
| Questions | |
| What other features differ between male and female voices? | |
| Can we find a difference in resonance between male and female voices? | |
| Can we identify falsetto from regular voices? (separate data-set likely needed for this) | |
| Are there other interesting features in the data? | |
| CART Diagram | |
| Mean fundamental frequency appears to be an indicator of voice gender, with a threshold of 140hz separating male from female classifications. | |
| References | |
| The Harvard-Haskins Database of Regularly-Timed Speech | |
| Telecommunications & Signal Processing Laboratory (TSP) Speech Database at McGill University, Home | |
| VoxForge Speech Corpus, Home | |
| Festvox CMU_ARCTIC Speech Database at Carnegie Mellon University" | |
| Every Cryptocurrency Daily Market Price,"Daily crypto markets open, close, low, high data for every token ever",jvent,210,"Version 12,2018-02-22|Version 11,2018-02-07|Version 10,2018-01-28|Version 9,2018-01-11|Version 8,2018-01-03|Version 7,2017-12-19|Version 6,2017-11-07|Version 5,2017-10-22|Version 4,2017-09-30|Version 3,2017-09-18|Version 2,2017-09-10|Version 1,2017-08-16","business | |
| finance | |
| internet",CSV,16 MB,Other,"31,723 views","3,904 downloads",17 kernels,20 topics,https://www.kaggle.com/jessevent/all-crypto-currencies,"Cryptocurrency Market Data | |
| Historical Cryptocurrency Prices For ALL Tokens! | |
| Summary | |
| > Observations: 649,051 | |
| > Variables: 13 | |
| > Crypto Tokens: 1,382 | |
| > Start Date: 28/04/2017 | |
| > End Date: 03/01/2018 | |
| Description | |
| All historic open, high, low, close, trading volume and market cap info for all cryptocurrencies. | |
| I've had to go over the code with a fine tooth comb to get it compatible with CRAN so there have been significant enhancements to how some of the field conversions have been undertaken and the data being cleaned. This should eliminate a few issues around number formatting or unexpected handling of scientific notations. | |
| Data Structure | |
| Observations: 649,051 | |
| Variables: 13 | |
| $ slug <chr> ""bitcoin"", ""bitcoin"", ""bitcoin"", ""bitcoin""... | |
| $ symbol <chr> ""BTC"", ""BTC"", ""BTC"", ""BTC"", ""BTC"", ""BTC"", ... | |
| $ name <chr> ""Bitcoin"", ""Bitcoin"", ""Bitcoin"", ""Bitcoin""... | |
| $ date <date> 2013-04-28, 2013-04-29, 2013-04-30, 2013-... | |
| $ ranknow <dbl> 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ... | |
| $ open <dbl> 135.30, 134.44, 144.00, 139.00, 116.38, 10... | |
| $ high <dbl> 135.98, 147.49, 146.93, 139.89, 125.60, 10... | |
| $ low <dbl> 132.10, 134.00, 134.05, 107.72, 92.28, 79... | |
| $ close <dbl> 134.21, 144.54, 139.00, 116.99, 105.21, 97... | |
| $ volume <dbl> 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, ... | |
| $ market <dbl> 1500520000, 1491160000, 1597780000, 154282... | |
| $ close_ratio <dbl> 0.5438, 0.7813, 0.3843, 0.2882, 0.3881, 0... | |
| $ spread <dbl> 3.88, 13.49, 12.88, 32.17, 33.32, 29.03, 2... | |
| Built With :heart: R | |
| I've ported my original kernel to generate this data, across into a R package which is awaiting being published on CRAN. Run the below to go scrape all the historical tables of all the different cryptocurrencies listed on CoinMarketCap and turn it into a data frame. | |
| You can install it via the github link below, or: | |
| devtools::install_github(""jessevent/crypto"") | |
| library(crypto) | |
| will_i_get_rich <- getCoins() | |
| Authors | |
| Jesse Vent - Package Author - jessevent | |
| Acknowledgments | |
| Github - View my github repository for the full package. | |
| CoinSpot - Invest $AUD into Crypto today! | |
| CoinMarketCap - Providing amazing data @CoinMarketCap | |
| If this helps you become rich please consider making a donation! | |
| ERC-20: 0x375923Bf82F0b728d23A5704261a6e16341fd860 | |
| XRP: rK59semLsuJZEWftxBFhWuNE6uhznjz2bK | |
| LTC: LWpiZMd2cEyqCdrZrs9TjsouTLWbFFxwCj" | |
| Student Alcohol Consumption,"Social, gender and study data from secondary school students",UCI Machine Learning,207,"Version 2,2016-10-19|Version 1,2016-10-19","food and drink | |
| public health",CSV,108 KB,CC0,"108,583 views","18,426 downloads",259 kernels,6 topics,https://www.kaggle.com/uciml/student-alcohol-consumption,"Context: | |
| The data were obtained in a survey of students math and portuguese language courses in secondary school. It contains a lot of interesting social, gender and study information about students. You can use it for some EDA or try to predict students final grade. | |
| Content: | |
| Attributes for both student-mat.csv (Math course) and student-por.csv (Portuguese language course) datasets: | |
| school - student's school (binary: 'GP' - Gabriel Pereira or 'MS' - Mousinho da Silveira) | |
| sex - student's sex (binary: 'F' - female or 'M' - male) | |
| age - student's age (numeric: from 15 to 22) | |
| address - student's home address type (binary: 'U' - urban or 'R' - rural) | |
| famsize - family size (binary: 'LE3' - less or equal to 3 or 'GT3' - greater than 3) | |
| Pstatus - parent's cohabitation status (binary: 'T' - living together or 'A' - apart) | |
| Medu - mother's education (numeric: 0 - none, 1 - primary education (4th grade), 2 – 5th to 9th grade, 3 – secondary education or 4 – higher education) | |
| Fedu - father's education (numeric: 0 - none, 1 - primary education (4th grade), 2 – 5th to 9th grade, 3 – secondary education or 4 – higher education) | |
| Mjob - mother's job (nominal: 'teacher', 'health' care related, civil 'services' (e.g. administrative or police), 'at_home' or 'other') | |
| Fjob - father's job (nominal: 'teacher', 'health' care related, civil 'services' (e.g. administrative or police), 'at_home' or 'other') | |
| reason - reason to choose this school (nominal: close to 'home', school 'reputation', 'course' preference or 'other') | |
| guardian - student's guardian (nominal: 'mother', 'father' or 'other') | |
| traveltime - home to school travel time (numeric: 1 - <15 min., 2 - 15 to 30 min., 3 - 30 min. to 1 hour, or 4 - >1 hour) | |
| studytime - weekly study time (numeric: 1 - <2 hours, 2 - 2 to 5 hours, 3 - 5 to 10 hours, or 4 - >10 hours) | |
| failures - number of past class failures (numeric: n if 1<=n<3, else 4) | |
| schoolsup - extra educational support (binary: yes or no) | |
| famsup - family educational support (binary: yes or no) | |
| paid - extra paid classes within the course subject (Math or Portuguese) (binary: yes or no) | |
| activities - extra-curricular activities (binary: yes or no) | |
| nursery - attended nursery school (binary: yes or no) | |
| higher - wants to take higher education (binary: yes or no) | |
| internet - Internet access at home (binary: yes or no) | |
| romantic - with a romantic relationship (binary: yes or no) | |
| famrel - quality of family relationships (numeric: from 1 - very bad to 5 - excellent) | |
| freetime - free time after school (numeric: from 1 - very low to 5 - very high) | |
| goout - going out with friends (numeric: from 1 - very low to 5 - very high) | |
| Dalc - workday alcohol consumption (numeric: from 1 - very low to 5 - very high) | |
| Walc - weekend alcohol consumption (numeric: from 1 - very low to 5 - very high) | |
| health - current health status (numeric: from 1 - very bad to 5 - very good) | |
| absences - number of school absences (numeric: from 0 to 93) | |
| These grades are related with the course subject, Math or Portuguese: | |
| G1 - first period grade (numeric: from 0 to 20) | |
| G2 - second period grade (numeric: from 0 to 20) | |
| G3 - final grade (numeric: from 0 to 20, output target) | |
| Additional note: there are several (382) students that belong to both datasets . These students can be identified by searching for identical attributes that characterize each student, as shown in the annexed R file. | |
| Source Information | |
| P. Cortez and A. Silva. Using Data Mining to Predict Secondary School Student Performance. In A. Brito and J. Teixeira Eds., Proceedings of 5th FUture BUsiness TEChnology Conference (FUBUTEC 2008) pp. 5-12, Porto, Portugal, April, 2008, EUROSIS, ISBN 978-9077381-39-7. | |
| Fabio Pagnotta, Hossain Mohammad Amran. Email:fabio.pagnotta@studenti.unicam.it, mohammadamra.hossain '@' studenti.unicam.it University Of Camerino | |
| https://archive.ics.uci.edu/ml/datasets/STUDENT+ALCOHOL+CONSUMPTION" | |
| H-1B Visa Petitions 2011-2016,3 million petitions for H-1B visas,Sharan Naribole,205,"Version 2,2017-02-28|Version 1,2017-02-28","law | |
| international relations",CSV,469 MB,CC4,"71,190 views","8,575 downloads",150 kernels,12 topics,https://www.kaggle.com/nsharan/h-1b-visa,"Context | |
| H-1B visas are a category of employment-based, non-immigrant visas for temporary foreign workers in the United States. For a foreign national to apply for H1-B visa, a US employer must offer them a job and submit a petition for a H-1B visa to the US immigration department. This is also the most common visa status applied for and held by international students once they complete college or higher education and begin working in a full-time position. | |
| The following articles contain more information about the H-1B visa process: | |
| What is H1B LCA ? Why file it ? Salary, Processing times – DOL | |
| H1B Application Process: Step by Step Guide | |
| Content | |
| This dataset contains five year's worth of H-1B petition data, with approximately 3 million records overall. The columns in the dataset include case status, employer name, worksite coordinates, job title, prevailing wage, occupation code, and year filed. | |
| For more information on individual columns, refer to the column metadata. A detailed description of the underlying raw dataset is available in an official data dictionary. | |
| Acknowledgements | |
| The Office of Foreign Labor Certification (OFLC) generates program data, including data about H1-B visas. The disclosure data updated annually and is available online. | |
| The raw data available is messy and not immediately suitable analysis. A set of data transformations were performed making the data more accessible for quick exploration. To learn more, refer to this blog post and to the complimentary R Notebook. | |
| Inspiration | |
| Is the number of petitions with Data Engineer job title increasing over time? | |
| Which part of the US has the most Hardware Engineer jobs? | |
| Which industry has the most number of Data Scientist positions? | |
| Which employers file the most petitions each year?" | |
| Young People Survey,"Explore the preferences, interests, habits, opinions, and fears of young people",Miroslav Sabo,204,"Version 2,2016-12-06|Version 1,2016-08-22","social groups | |
| psychometrics | |
| demographics | |
| + 2 more...",CSV,448 KB,CC0,"67,506 views","17,873 downloads",126 kernels,3 topics,https://www.kaggle.com/miroslavsabo/young-people-survey,"Introduction | |
| In 2013, students of the Statistics class at FSEV UK were asked to invite their friends to participate in this survey. | |
| The data file (responses.csv) consists of 1010 rows and 150 columns (139 integer and 11 categorical). | |
| For convenience, the original variable names were shortened in the data file. See the columns.csv file if you want to match the data with the original names. | |
| The data contain missing values. | |
| The survey was presented to participants in both electronic and written form. | |
| The original questionnaire was in Slovak language and was later translated into English. | |
| All participants were of Slovakian nationality, aged between 15-30. | |
| The variables can be split into the following groups: | |
| Music preferences (19 items) | |
| Movie preferences (12 items) | |
| Hobbies & interests (32 items) | |
| Phobias (10 items) | |
| Health habits (3 items) | |
| Personality traits, views on life, & opinions (57 items) | |
| Spending habits (7 items) | |
| Demographics (10 items) | |
| Research questions | |
| Many different techniques can be used to answer many questions, e.g. | |
| Clustering: Given the music preferences, do people make up any clusters of similar behavior? | |
| Hypothesis testing: Do women fear certain phenomena significantly more than men? Do the left handed people have different interests than right handed? | |
| Predictive modeling: Can we predict spending habits of a person from his/her interests and movie or music preferences? | |
| Dimension reduction: Can we describe a large number of human interests by a smaller number of latent concepts? | |
| Correlation analysis: Are there any connections between music and movie preferences? | |
| Visualization: How to effectively visualize a lot of variables in order to gain some meaningful insights from the data? | |
| (Multivariate) Outlier detection: Small number of participants often cheats and randomly answers the questions. Can you identify them? Hint: Local outlier factor may help. | |
| Missing values analysis: Are there any patterns in missing responses? What is the optimal way of imputing the values in surveys? | |
| Recommendations: If some of user's interests are known, can we predict the other? Or, if we know what a person listen, can we predict which kind of movies he/she might like? | |
| Past research | |
| (in slovak) Sleziak, P. - Sabo, M.: Gender differences in the prevalence of specific phobias. Forum Statisticum Slovacum. 2014, Vol. 10, No. 6. [Differences (gender + whether people lived in village/town) in the prevalence of phobias.] | |
| Sabo, Miroslav. Multivariate Statistical Methods with Applications. Diss. Slovak University of Technology in Bratislava, 2014. [Clustering of variables (music preferences, movie preferences, phobias) + Clustering of people w.r.t. their interests.] | |
| Questionnaire | |
| MUSIC PREFERENCES | |
| I enjoy listening to music.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I prefer.: Slow paced music 1-2-3-4-5 Fast paced music (integer) | |
| Dance, Disco, Funk: Don't enjoy at all 1-2-3-4-5 Enjoy very much (integer) | |
| Folk music: Don't enjoy at all 1-2-3-4-5 Enjoy very much (integer) | |
| Country: Don't enjoy at all 1-2-3-4-5 Enjoy very much (integer) | |
| Classical: Don't enjoy at all 1-2-3-4-5 Enjoy very much (integer) | |
| Musicals: Don't enjoy at all 1-2-3-4-5 Enjoy very much (integer) | |
| Pop: Don't enjoy at all 1-2-3-4-5 Enjoy very much (integer) | |
| Rock: Don't enjoy at all 1-2-3-4-5 Enjoy very much (integer) | |
| Metal, Hard rock: Don't enjoy at all 1-2-3-4-5 Enjoy very much (integer) | |
| Punk: Don't enjoy at all 1-2-3-4-5 Enjoy very much (integer) | |
| Hip hop, Rap: Don't enjoy at all 1-2-3-4-5 Enjoy very much (integer) | |
| Reggae, Ska: Don't enjoy at all 1-2-3-4-5 Enjoy very much (integer) | |
| Swing, Jazz: Don't enjoy at all 1-2-3-4-5 Enjoy very much (integer) | |
| Rock n Roll: Don't enjoy at all 1-2-3-4-5 Enjoy very much (integer) | |
| Alternative music: Don't enjoy at all 1-2-3-4-5 Enjoy very much (integer) | |
| Latin: Don't enjoy at all 1-2-3-4-5 Enjoy very much (integer) | |
| Techno, Trance: Don't enjoy at all 1-2-3-4-5 Enjoy very much (integer) | |
| Opera: Don't enjoy at all 1-2-3-4-5 Enjoy very much (integer) | |
| MOVIE PREFERENCES | |
| I really enjoy watching movies.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| Horror movies: Don't enjoy at all 1-2-3-4-5 Enjoy very much (integer) | |
| Thriller movies: Don't enjoy at all 1-2-3-4-5 Enjoy very much (integer) | |
| Comedies: Don't enjoy at all 1-2-3-4-5 Enjoy very much (integer) | |
| Romantic movies: Don't enjoy at all 1-2-3-4-5 Enjoy very much (integer) | |
| Sci-fi movies: Don't enjoy at all 1-2-3-4-5 Enjoy very much (integer) | |
| War movies: Don't enjoy at all 1-2-3-4-5 Enjoy very much (integer) | |
| Tales: Don't enjoy at all 1-2-3-4-5 Enjoy very much (integer) | |
| Cartoons: Don't enjoy at all 1-2-3-4-5 Enjoy very much (integer) | |
| Documentaries: Don't enjoy at all 1-2-3-4-5 Enjoy very much (integer) | |
| Western movies: Don't enjoy at all 1-2-3-4-5 Enjoy very much (integer) | |
| Action movies: Don't enjoy at all 1-2-3-4-5 Enjoy very much (integer) | |
| HOBBIES & INTERESTS | |
| History: Not interested 1-2-3-4-5 Very interested (integer) | |
| Psychology: Not interested 1-2-3-4-5 Very interested (integer) | |
| Politics: Not interested 1-2-3-4-5 Very interested (integer) | |
| Mathematics: Not interested 1-2-3-4-5 Very interested (integer) | |
| Physics: Not interested 1-2-3-4-5 Very interested (integer) | |
| Internet: Not interested 1-2-3-4-5 Very interested (integer) | |
| PC Software, Hardware: Not interested 1-2-3-4-5 Very interested (integer) | |
| Economy, Management: Not interested 1-2-3-4-5 Very interested (integer) | |
| Biology: Not interested 1-2-3-4-5 Very interested (integer) | |
| Chemistry: Not interested 1-2-3-4-5 Very interested (integer) | |
| Poetry reading: Not interested 1-2-3-4-5 Very interested (integer) | |
| Geography: Not interested 1-2-3-4-5 Very interested (integer) | |
| Foreign languages: Not interested 1-2-3-4-5 Very interested (integer) | |
| Medicine: Not interested 1-2-3-4-5 Very interested (integer) | |
| Law: Not interested 1-2-3-4-5 Very interested (integer) | |
| Cars: Not interested 1-2-3-4-5 Very interested (integer) | |
| Art: Not interested 1-2-3-4-5 Very interested (integer) | |
| Religion: Not interested 1-2-3-4-5 Very interested (integer) | |
| Outdoor activities: Not interested 1-2-3-4-5 Very interested (integer) | |
| Dancing: Not interested 1-2-3-4-5 Very interested (integer) | |
| Playing musical instruments: Not interested 1-2-3-4-5 Very interested (integer) | |
| Poetry writing: Not interested 1-2-3-4-5 Very interested (integer) | |
| Sport and leisure activities: Not interested 1-2-3-4-5 Very interested (integer) | |
| Sport at competitive level: Not interested 1-2-3-4-5 Very interested (integer) | |
| Gardening: Not interested 1-2-3-4-5 Very interested (integer) | |
| Celebrity lifestyle: Not interested 1-2-3-4-5 Very interested (integer) | |
| Shopping: Not interested 1-2-3-4-5 Very interested (integer) | |
| Science and technology: Not interested 1-2-3-4-5 Very interested (integer) | |
| Theatre: Not interested 1-2-3-4-5 Very interested (integer) | |
| Socializing: Not interested 1-2-3-4-5 Very interested (integer) | |
| Adrenaline sports: Not interested 1-2-3-4-5 Very interested (integer) | |
| Pets: Not interested 1-2-3-4-5 Very interested (integer) | |
| PHOBIAS | |
| Flying: Not afraid at all 1-2-3-4-5 Very afraid of (integer) | |
| Thunder, lightning: Not afraid at all 1-2-3-4-5 Very afraid of (integer) | |
| Darkness: Not afraid at all 1-2-3-4-5 Very afraid of (integer) | |
| Heights: Not afraid at all 1-2-3-4-5 Very afraid of (integer) | |
| Spiders: Not afraid at all 1-2-3-4-5 Very afraid of (integer) | |
| Snakes: Not afraid at all 1-2-3-4-5 Very afraid of (integer) | |
| Rats, mice: Not afraid at all 1-2-3-4-5 Very afraid of (integer) | |
| Ageing: Not afraid at all 1-2-3-4-5 Very afraid of (integer) | |
| Dangerous dogs: Not afraid at all 1-2-3-4-5 Very afraid of (integer) | |
| Public speaking: Not afraid at all 1-2-3-4-5 Very afraid of (integer) | |
| HEALTH HABITS | |
| Smoking habits: Never smoked - Tried smoking - Former smoker - Current smoker (categorical) | |
| Drinking: Never - Social drinker - Drink a lot (categorical) | |
| I live a very healthy lifestyle.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| PERSONALITY TRAITS, VIEWS ON LIFE & OPINIONS | |
| I take notice of what goes on around me.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I try to do tasks as soon as possible and not leave them until last minute.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I always make a list so I don't forget anything.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I often study or work even in my spare time.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I look at things from all different angles before I go ahead.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I believe that bad people will suffer one day and good people will be rewarded.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I am reliable at work and always complete all tasks given to me.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I always keep my promises.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I can fall for someone very quickly and then completely lose interest.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I would rather have lots of friends than lots of money.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I always try to be the funniest one.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I can be two faced sometimes.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I damaged things in the past when angry.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I take my time to make decisions.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I always try to vote in elections.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I often think about and regret the decisions I make.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I can tell if people listen to me or not when I talk to them.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I am a hypochondriac.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I am emphatetic person.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I eat because I have to. I don't enjoy food and eat as fast as I can.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I try to give as much as I can to other people at Christmas.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I don't like seeing animals suffering.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I look after things I have borrowed from others.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I feel lonely in life.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I used to cheat at school.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I worry about my health.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I wish I could change the past because of the things I have done.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I believe in God.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I always have good dreams.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I always give to charity.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I have lots of friends.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| Timekeeping.: I am often early. - I am always on time. - I am often running late. (categorical) | |
| Do you lie to others?: Never. - Only to avoid hurting someone. - Sometimes. - Everytime it suits me. (categorical) | |
| I am very patient.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I can quickly adapt to a new environment.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| My moods change quickly.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I am well mannered and I look after my appearance.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I enjoy meeting new people.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I always let other people know about my achievements.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I think carefully before answering any important letters.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I enjoy childrens' company.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I am not afraid to give my opinion if I feel strongly about something.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I can get angry very easily.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I always make sure I connect with the right people.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I have to be well prepared before public speaking.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I will find a fault in myself if people don't like me.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I cry when I feel down or things don't go the right way.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I am 100% happy with my life.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I am always full of life and energy.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I prefer big dangerous dogs to smaller, calmer dogs.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I believe all my personality traits are positive.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| If I find something the doesn't belong to me I will hand it in.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I find it very difficult to get up in the morning.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I have many different hobbies and interests.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I always listen to my parents' advice.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I enjoy taking part in surveys.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| How much time do you spend online?: No time at all - Less than an hour a day - Few hours a day - Most of the day (categorical) | |
| SPENDING HABITS | |
| I save all the money I can.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I enjoy going to large shopping centres.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I prefer branded clothing to non branded.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I spend a lot of money on partying and socializing.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I spend a lot of money on my appearance.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I spend a lot of money on gadgets.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| I will hapilly pay more money for good, quality or healthy food.: Strongly disagree 1-2-3-4-5 Strongly agree (integer) | |
| DEMOGRAPHICS | |
| Age: (integer) | |
| Height: (integer) | |
| Weight: (integer) | |
| How many siblings do you have?: (integer) | |
| Gender: Female - Male (categorical) | |
| I am: Left handed - Right handed (categorical) | |
| Highest education achieved: Currently a Primary school pupil - Primary school - Secondary school - College/Bachelor degree (categorical) | |
| I am the only child: No - Yes (categorical) | |
| I spent most of my childhood in a: City - village (categorical) | |
| I lived most of my childhood in a: house/bungalow - block of flats (categorical)" | |
| Trending YouTube Video Statistics,Daily statistics for trending YouTube videos,Mitchell J,199,"Version 40,2018-02-23|Version 39,2018-02-19|Version 38,2018-02-15|Version 37,2018-02-12|Version 36,2018-02-08|Version 35,2018-02-02|Version 34,2018-01-26|Version 33,2018-01-22|Version 32,2018-01-16|Version 31,2018-01-09|Version 30,2018-01-06|Version 29,2018-01-04|Version 28,2018-01-02|Version 27,2017-12-25|Version 26,2017-12-23|Version 25,2017-12-19|Version 24,2017-12-13|Version 23,2017-12-11|Version 22,2017-12-08|Version 21,2017-12-07|Version 20,2017-12-04|Version 19,2017-12-02|Version 18,2017-12-01|Version 17,2017-11-29|Version 16,2017-11-29|Version 15,2017-11-28|Version 14,2017-11-27|Version 13,2017-11-24|Version 12,2017-11-24|Version 11,2017-11-23|Version 10,2017-11-22|Version 9,2017-11-21|Version 8,2017-11-20|Version 7,2017-11-18|Version 6,2017-11-18|Version 5,2017-11-17|Version 4,2017-11-16|Version 3,2017-11-15|Version 2,2017-11-14|Version 1,2017-11-14","languages | |
| popular culture | |
| statistics | |
| + 2 more...",CSV,54 MB,CC0,"33,195 views","5,856 downloads",14 kernels,7 topics,https://www.kaggle.com/datasnaek/youtube-new,"Context | |
| YouTube (the world-famous video sharing website) maintains a list of the top trending videos on the platform. According to Variety magazine, “To determine the year’s top-trending videos, YouTube uses a combination of factors including measuring users interactions (number of views, shares, comments and likes). Note that they’re not the most-viewed videos overall for the calendar year”. Top performers on the YouTube trending list are music videos (such as the famously virile “Gangam Style”), celebrity and/or reality TV performances, and the random dude-with-a-camera viral videos that YouTube is well-known for. | |
| This dataset is a daily record of the top trending YouTube videos. | |
| Note that this dataset is a structurally improved version of this dataset. | |
| Content | |
| This dataset includes several months (and counting) of data on daily trending YouTube videos. Data is included for the US, GB, DE, CA, and FR regions (USA, Great Britain, Germany, Canada, and France, respectively), with up to 200 listed trending videos per day. | |
| Each region’s data is in a separate file. Data includes the video title, channel title, publish time, tags, views, likes and dislikes, description, and comment count. | |
| The data also includes a category_id field, which varies between regions. To retrieve the categories for a specific video, find it in the associated JSON. One such file is included for each of the five regions in the dataset. | |
| For more information on specific columns in the dataset refer to the column metadata. | |
| Acknowledgements | |
| This dataset was collected using the YouTube API. | |
| Inspiration | |
| Possible uses for this dataset could include: | |
| Sentiment analysis in a variety of forms | |
| Categorising YouTube videos based on their comments and statistics. | |
| Training ML algorithms like RNNs to generate their own YouTube comments. | |
| Analysing what factors affect how popular a YouTube video will be. | |
| Statistical analysis over time . | |
| For further inspiration, see the kernels on this dataset!" | |
| Video Game Sales,"Analyze sales data from more than 16,500 games.",GregorySmith,198,"Version 2,2016-10-26|Version 1,2016-10-26",video games,CSV,1 MB,Other,"86,011 views","13,910 downloads",283 kernels,11 topics,https://www.kaggle.com/gregorut/videogamesales,"This dataset contains a list of video games with sales greater than 100,000 copies. It was generated by a scrape of vgchartz.com. | |
| Fields include | |
| Rank - Ranking of overall sales | |
| Name - The games name | |
| Platform - Platform of the games release (i.e. PC,PS4, etc.) | |
| Year - Year of the game's release | |
| Genre - Genre of the game | |
| Publisher - Publisher of the game | |
| NA_Sales - Sales in North America (in millions) | |
| EU_Sales - Sales in Europe (in millions) | |
| JP_Sales - Sales in Japan (in millions) | |
| Other_Sales - Sales in the rest of the world (in millions) | |
| Global_Sales - Total worldwide sales. | |
| The script to scrape the data is available at https://github.com/GregorUT/vgchartzScrape. It is based on BeautifulSoup using Python. There are 16,598 records. 2 records were dropped due to incomplete information." | |
| 20 Years of Games,18000+ rows of review data from ign.com,Eric Grinstein,186,"Version 2,2016-09-28|Version 1,2016-09-28","games and toys | |
| video games",CSV,2 MB,Other,"50,320 views","7,818 downloads",160 kernels,4 topics,https://www.kaggle.com/egrinstein/20-years-of-games,"This dataset is the result of a crawl I did on http://ign.com/games/reviews . | |
| It contains 18625 lines with the fields like the release date, it's platform and IGN's score. All the lines are fully filled. | |
| In 20 years, the gaming industry has grown and sophisticated. By exploring this dataset, one is able to find trends about the industry, compare consoles against eachother, search through the most popular genres and more. | |
| The dataset can also be a great place for beginners to start using Python modules such as Pandas and Seaborn. | |
| You can find the crawl I used for the retrieval here" | |
| Hillary Clinton's Emails,Uncover the political landscape in Hillary Clinton's emails,Kaggle,185,"Version 2,2016-10-06|Version 1,2015-09-11","politics | |
| telecommunications",CSV,51 MB,CC0,"115,287 views","11,464 downloads",347 kernels,12 topics,https://www.kaggle.com/kaggle/hillary-clinton-emails,"Throughout 2015, Hillary Clinton has been embroiled in controversy over the use of personal email accounts on non-government servers during her time as the United States Secretary of State. Some political experts and opponents maintain that Clinton's use of personal email accounts to conduct Secretary of State affairs is in violation of protocols and federal laws that ensure appropriate recordkeeping of government activity. Hillary's campaign has provided their own four sentence summary of her email use here. | |
| There have been a number of Freedom of Information lawsuits filed over the State Department's failure to fully release the emails sent and received on Clinton's private accounts. On Monday, August 31, the State Department released nearly 7,000 pages of Clinton's heavily redacted emails (its biggest release of emails to date). | |
| The documents were released by the State Department as PDFs. We've cleaned and normalized the released documents and are hosting them for public analysis. Kaggle's choice to host this dataset is not meant to express any particular political affiliation or intent. | |
| Here's the code that creates this data release." | |
| IBM HR Analytics Employee Attrition & Performance,Predict attrition of your valuable employees,pavansubhash,183,"Version 1,2017-03-31","employment | |
| business",CSV,223 KB,ODbL,"67,957 views","9,905 downloads",235 kernels,10 topics,https://www.kaggle.com/pavansubhasht/ibm-hr-analytics-attrition-dataset,"Uncover the factors that lead to employee attrition and explore important questions such as ‘show me a breakdown of distance from home by job role and attrition’ or ‘compare average monthly income by education and attrition’. This is a fictional data set created by IBM data scientists. | |
| Education 1 'Below College' 2 'College' 3 'Bachelor' 4 'Master' 5 'Doctor' | |
| EnvironmentSatisfaction 1 'Low' 2 'Medium' 3 'High' 4 'Very High' | |
| JobInvolvement | |
| 1 'Low' 2 'Medium' 3 'High' 4 'Very High' | |
| JobSatisfaction 1 'Low' 2 'Medium' 3 'High' 4 'Very High' | |
| PerformanceRating | |
| 1 'Low' 2 'Good' 3 'Excellent' 4 'Outstanding' | |
| RelationshipSatisfaction | |
| 1 'Low' 2 'Medium' 3 'High' 4 'Very High' | |
| WorkLifeBalance 1 'Bad' 2 'Good' 3 'Better' 4 'Best'" | |
| Chocolate Bar Ratings,"Expert ratings of over 1,700 chocolate bars",Rachael Tatman,180,"Version 1,2017-08-12","critical theory | |
| food and drink",CSV,125 KB,CC0,"35,945 views","7,128 downloads",89 kernels,2 topics,https://www.kaggle.com/rtatman/chocolate-bar-ratings,"Context | |
| Chocolate is one of the most popular candies in the world. Each year, residents of the United States collectively eat more than 2.8 billions pounds. However, not all chocolate bars are created equal! This dataset contains expert ratings of over 1,700 individual chocolate bars, along with information on their regional origin, percentage of cocoa, the variety of chocolate bean used and where the beans were grown. | |
| Flavors of Cacao Rating System: | |
| 5= Elite (Transcending beyond the ordinary limits) | |
| 4= Premium (Superior flavor development, character and style) | |
| 3= Satisfactory(3.0) to praiseworthy(3.75) (well made with special qualities) | |
| 2= Disappointing (Passable but contains at least one significant flaw) | |
| 1= Unpleasant (mostly unpalatable) | |
| Each chocolate is evaluated from a combination of both objective qualities and subjective interpretation. A rating here only represents an experience with one bar from one batch. Batch numbers, vintages and review dates are included in the database when known. | |
| The database is narrowly focused on plain dark chocolate with an aim of appreciating the flavors of the cacao when made into chocolate. The ratings do not reflect health benefits, social missions, or organic status. | |
| Flavor is the most important component of the Flavors of Cacao ratings. Diversity, balance, intensity and purity of flavors are all considered. It is possible for a straight forward single note chocolate to rate as high as a complex flavor profile that changes throughout. Genetics, terroir, post harvest techniques, processing and storage can all be discussed when considering the flavor component. | |
| Texture has a great impact on the overall experience and it is also possible for texture related issues to impact flavor. It is a good way to evaluate the makers vision, attention to detail and level of proficiency. | |
| Aftermelt is the experience after the chocolate has melted. Higher quality chocolate will linger and be long lasting and enjoyable. Since the aftermelt is the last impression you get from the chocolate, it receives equal importance in the overall rating. | |
| Overall Opinion is really where the ratings reflect a subjective opinion. Ideally it is my evaluation of whether or not the components above worked together and an opinion on the flavor development, character and style. It is also here where each chocolate can usually be summarized by the most prominent impressions that you would remember about each chocolate. | |
| Acknowledgements | |
| These ratings were compiled by Brady Brelinski, Founding Member of the Manhattan Chocolate Society. For up-to-date information, as well as additional content (including interviews with craft chocolate makers), please see his website: Flavors of Cacao | |
| Inspiration | |
| Where are the best cocoa beans grown? | |
| Which countries produce the highest-rated bars? | |
| What’s the relationship between cocoa solids percentage and rating?" | |
| Uber Pickups in New York City,Trip data for over 20 million Uber (and other for-hire vehicle) trips in NYC,FiveThirtyEight,180,"Version 2,2016-11-14|Version 1,2016-11-13",road transport,CSV,835 MB,CC0,"67,549 views","8,765 downloads",208 kernels,3 topics,https://www.kaggle.com/fivethirtyeight/uber-pickups-in-new-york-city,"Uber TLC FOIL Response | |
| This directory contains data on over 4.5 million Uber pickups in New York City from April to September 2014, and 14.3 million more Uber pickups from January to June 2015. Trip-level data on 10 other for-hire vehicle (FHV) companies, as well as aggregated data for 329 FHV companies, is also included. All the files are as they were received on August 3, Sept. 15 and Sept. 22, 2015. | |
| FiveThirtyEight obtained the data from the NYC Taxi & Limousine Commission (TLC) by submitting a Freedom of Information Law request on July 20, 2015. The TLC has sent us the data in batches as it continues to review trip data Uber and other HFV companies have submitted to it. The TLC's correspondence with FiveThirtyEight is included in the files TLC_letter.pdf, TLC_letter2.pdf and TLC_letter3.pdf. TLC records requests can be made here. | |
| This data was used for four FiveThirtyEight stories: Uber Is Serving New York’s Outer Boroughs More Than Taxis Are, Public Transit Should Be Uber’s New Best Friend, Uber Is Taking Millions Of Manhattan Rides Away From Taxis, and Is Uber Making NYC Rush-Hour Traffic Worse?. | |
| The Data | |
| The dataset contains, roughly, four groups of files: | |
| Uber trip data from 2014 (April - September), separated by month, with detailed location information | |
| Uber trip data from 2015 (January - June), with less fine-grained location information | |
| non-Uber FHV (For-Hire Vehicle) trips. The trip information varies by company, but can include day of trip, time of trip, pickup location, driver's for-hire license number, and vehicle's for-hire license number. | |
| aggregate ride and vehicle statistics for all FHV companies (and, occasionally, for taxi companies) | |
| Uber trip data from 2014 | |
| There are six files of raw data on Uber pickups in New York City from April to September 2014. The files are separated by month and each has the following columns: | |
| Date/Time : The date and time of the Uber pickup | |
| Lat : The latitude of the Uber pickup | |
| Lon : The longitude of the Uber pickup | |
| Base : The TLC base company code affiliated with the Uber pickup | |
| These files are named: | |
| uber-raw-data-apr14.csv | |
| uber-raw-data-aug14.csv | |
| uber-raw-data-jul14.csv | |
| uber-raw-data-jun14.csv | |
| uber-raw-data-may14.csv | |
| uber-raw-data-sep14.csv | |
| Uber trip data from 2015 | |
| Also included is the file uber-raw-data-janjune-15.csv This file has the following columns: | |
| Dispatching_base_num : The TLC base company code of the base that dispatched the Uber | |
| Pickup_date : The date and time of the Uber pickup | |
| Affiliated_base_num : The TLC base company code affiliated with the Uber pickup | |
| locationID : The pickup location ID affiliated with the Uber pickup | |
| The Base codes are for the following Uber bases: | |
| B02512 : Unter B02598 : Hinter B02617 : Weiter B02682 : Schmecken B02764 : Danach-NY B02765 : Grun B02835 : Dreist B02836 : Drinnen | |
| For coarse-grained location information from these pickups, the file taxi-zone-lookup.csv shows the taxi Zone (essentially, neighborhood) and Borough for each locationID. | |
| Non-Uber FLV trips | |
| The dataset also contains 10 files of raw data on pickups from 10 for-hire vehicle (FHV) companies. The trip information varies by company, but can include day of trip, time of trip, pickup location, driver's for-hire license number, and vehicle's for-hire license number. | |
| These files are named: | |
| American_B01362.csv | |
| Diplo_B01196.csv | |
| Highclass_B01717.csv | |
| Skyline_B00111.csv | |
| Carmel_B00256.csv | |
| Federal_02216.csv | |
| Lyft_B02510.csv | |
| Dial7_B00887.csv | |
| Firstclass_B01536.csv | |
| Prestige_B01338.csv | |
| Aggregate Statistics | |
| There is also a file other-FHV-data-jan-aug-2015.csv containing daily pickup data for 329 FHV companies from January 2015 through August 2015. | |
| The file Uber-Jan-Feb-FOIL.csv contains aggregated daily Uber trip statistics in January and February 2015." | |
| Getting Real about Fake News,Text & metadata from fake & biased news sources around the web,Megan Risdal,179,"Version 1,2016-11-26","news agencies | |
| languages | |
| politics",CSV,54 MB,CC0,"83,505 views","6,220 downloads",60 kernels,5 topics,https://www.kaggle.com/mrisdal/fake-news,"The latest hot topic in the news is fake news and many are wondering what data scientists can do to detect it and stymie its viral spread. This dataset is only a first step in understanding and tackling this problem. It contains text and metadata scraped from 244 websites tagged as ""bullshit"" by the BS Detector Chrome Extension by Daniel Sieradski. | |
| Warning: I did not modify the list of news sources from the BS Detector so as not to introduce my (useless) layer of bias; I'm not an authority on fake news. There may be sources whose inclusion you disagree with. It's up to you to decide how to work with the data and how you might contribute to ""improving it"". The labels of ""bs"" and ""junksci"", etc. do not constitute capital ""t"" Truth. If there are other sources you would like to include, start a discussion. If there are sources you believe should not be included, start a discussion or write a kernel analyzing the data. Or take the data and do something else productive with it. Kaggle's choice to host this dataset is not meant to express any particular political affiliation or intent. | |
| Contents | |
| The dataset contains text and metadata from 244 websites and represents 12,999 posts in total from the past 30 days. The data was pulled using the webhose.io API; because it's coming from their crawler, not all websites identified by the BS Detector are present in this dataset. Each website was labeled according to the BS Detector as documented here. Data sources that were missing a label were simply assigned a label of ""bs"". There are (ostensibly) no genuine, reliable, or trustworthy news sources represented in this dataset (so far), so don't trust anything you read. | |
| Fake news in the news | |
| For inspiration, I've included some (presumably non-fake) recent stories covering fake news in the news. This is a sensitive, nuanced topic and if there are other resources you'd like to see included here, please leave a suggestion. From defining fake, biased, and misleading news in the first place to deciding how to take action (a blacklist is not a good answer), there's a lot of information to consider beyond what can be neatly arranged in a CSV file. | |
| How Fake News Spreads (NYT) | |
| We Tracked Down A Fake-News Creator In The Suburbs. Here's What We Learned (NPR) | |
| Does Facebook Generate Over Half of its Revenue from Fake News? (Forbes) | |
| Fake News is Not the Only Problem (Points - Medium) | |
| Washington Post Disgracefully Promotes a McCarthyite Blacklist From a New, Hidden, and Very Shady Group (The Intercept) | |
| Improvements | |
| If you have suggestions for improvements or would like to contribute, please let me know. The most obvious extensions are to include data from ""real"" news sites and to address the bias in the current list. I'd be happy to include any contributions in future versions of the dataset. | |
| Acknowledgements | |
| Thanks to Anthony for pointing me to Daniel Sieradski's BS Detector. Thank you to Daniel Nouri for encouraging me to add a disclaimer to the dataset's page." | |
| SF Salaries,Explore San Francisco city employee salary data,Kaggle,172,"Version 2,2016-10-06|Version 1,2015-12-22",income,SQLite,33 MB,CC0,"95,375 views","14,929 downloads",266 kernels,7 topics,https://www.kaggle.com/kaggle/sf-salaries,"One way to understand how a city government works is by looking at who it employs and how its employees are compensated. This data contains the names, job title, and compensation for San Francisco city employees on an annual basis from 2011 to 2014. | |
| Exploration Ideas | |
| To help get you started, here are some data exploration ideas: | |
| How have salaries changed over time between different groups of people? | |
| How are base pay, overtime pay, and benefits allocated between different groups? | |
| Is there any evidence of pay discrimination based on gender in this dataset? | |
| How is budget allocated based on different groups and responsibilities? | |
| Have other ideas you're curious for someone else to explore? Post them in this forum thread. | |
| Data Description | |
| sf-salaries-release-*.zip (downloadable via the ""Download Data"" link in the header above) contains a CSV table and a SQLite database (with the same data as the CSV file). Here's the code that creates this data release. | |
| The original source for this data is here. We've taken the raw files here and combined/normalized them into a single CSV file as well as a SQLite database with an equivalently-defined table." | |
| UFO Sightings,Reports of unidentified flying object reports in the last century,National UFO Reporting Center (NUFORC),170,"Version 1,2016-11-17",space,CSV,28 MB,Other,"48,914 views","7,234 downloads",151 kernels,2 topics,https://www.kaggle.com/NUFORC/ufo-sightings,"Context | |
| This dataset contains over 80,000 reports of UFO sightings over the last century. | |
| Content | |
| There are two versions of this dataset: scrubbed and complete. The complete data includes entries where the location of the sighting was not found or blank (0.8146%) or have an erroneous or blank time (8.0237%). Since the reports date back to the 20th century, some older data might be obscured. Data contains city, state, time, description, and duration of each sighting. | |
| Inspiration | |
| What areas of the country are most likely to have UFO sightings? | |
| Are there any trends in UFO sightings over time? Do they tend to be clustered or seasonal? | |
| Do clusters of UFO sightings correlate with landmarks, such as airports or government research centers? | |
| What are the most common UFO descriptions? | |
| Acknowledgement | |
| This dataset was scraped, geolocated, and time standardized from NUFORC data by Sigmond Axel here." | |
| Death in the United States,Learn more about the leading causes of death from 2005-2015,Centers for Disease Control and Prevention,168,"Version 2,2017-08-04|Version 1,2016-03-10","death | |
| demographics",{}JSON,4 GB,CC0,"60,962 views","10,077 downloads",188 kernels,7 topics,https://www.kaggle.com/cdc/mortality,"Every year the CDC releases the country’s most detailed report on death in the United States under the National Vital Statistics Systems. This mortality dataset is a record of every death in the country for 2005 through 2015, including detailed information about causes of death and the demographic background of the deceased. | |
| It's been said that ""statistics are human beings with the tears wiped off."" This is especially true with this dataset. Each death record represents somebody's loved one, often connected with a lifetime of memories and sometimes tragically too short. | |
| Putting the sensitive nature of the topic aside, analyzing mortality data is essential to understanding the complex circumstances of death across the country. The US Government uses this data to determine life expectancy and understand how death in the U.S. differs from the rest of the world. Whether you’re looking for macro trends or analyzing unique circumstances, we challenge you to use this dataset to find your own answers to one of life’s great mysteries. | |
| Overview | |
| This dataset is a collection of CSV files each containing one year's worth of data and paired JSON files containing the code mappings, plus an ICD 10 code set. The CSVs were reformatted from their original fixed-width file formats using information extracted from the CDC's PDF manuals using this script. Please note that this process may have introduced errors as the text extracted from the pdf is not a perfect match. If you have any questions or find errors in the preparation process, please leave a note in the forums. We hope to publish additional years of data using this method soon. | |
| A more detailed overview of the data can be found here. You'll find that the fields are consistent within this time window, but some of data codes change every few years. For example, the 113_cause_recode entry 069 only covers ICD codes (I10,I12) in 2005, but by 2015 it covers (I10,I12,I15). When I post data from years prior to 2005, expect some of the fields themselves to change as well. | |
| All data comes from the CDC’s National Vital Statistics Systems, with the exception of the Icd10Code, which are sourced from the World Health Organization. | |
| Project ideas | |
| The CDC's mortality data was the basis of a widely publicized paper, by Anne Case and Nobel prize winner Angus Deaton, arguing that middle-aged whites are dying at elevated rates. One of the criticisms against the paper is that it failed to properly account for the exact ages within the broad bins available through the CDC's WONDER tool. What do these results look like with exact/not-binned age data? | |
| Similarly, how sensitive are the mortality trends being discussed in the news to the choice of bin-widths? | |
| As noted above, the data preparation process could have introduced errors. Can you find any discrepancies compared to the aggregate metrics on WONDER? If so, please let me know in the forums! | |
| WONDER is cited in numerous economics, sociology, and public health research papers. Can you find any papers whose conclusions would be altered if they used the exact data available here rather than binned data from Wonder? | |
| Differences from the first version of the dataset | |
| This version of the dataset was prepared in a completely different many. This has allowed us to provide a much larger volume of data and ensure that codes are available for every field. | |
| We've replaced the batch of sql files with a single JSON per year. Kaggle's platform currently offer's better support for JSON files, and this keeps the number of files manageable. | |
| A tutorial kernel providing a quick introduction to the new format is available here. | |
| Lastly, I apologize if the transition has interrupted anyone's work! If need be, you can still download v1." | |
| Indian Premier League,Data for all the IPL seasons,Manas,167,"Version 5,2017-11-23|Version 4,2016-11-06|Version 3,2016-11-06|Version 2,2016-11-06|Version 1,2016-11-06","cricket | |
| india",CSV,1 MB,CC4,"51,036 views","9,011 downloads",274 kernels,4 topics,https://www.kaggle.com/manasgarg/ipl,"This is the ball by ball data of all the IPL cricket matches till season 9. | |
| Source: http://cricsheet.org/ (data is available on this website in the YAML format. This is converted to CSV format by the contributors) | |
| The dataset contains 2 files: deliveries.csv and matches.csv. matches.csv contains details related to the match such as location, contesting teams, umpires, results, etc. deliveries.csv is the ball-by-ball data of all the IPL matches including data of the batting team, batsman, bowler, non-striker, runs scored, etc. | |
| Research scope: Predicting the winner of the next season of IPL based on past data, Visualizations, Perspectives, etc." | |
| Monty Python Flying Circus,"Remember, buy Whizzo butter and this dead crab.",Allan,163,"Version 1,2017-09-03","humor | |
| mass media",SQLite,4 MB,ODbL,"11,214 views","1,270 downloads",4 kernels,,https://www.kaggle.com/allank/monty-python-flying-circus,"Context | |
| 45 episodes across 4 seasons of Monty Python's Flying Circus - all of the scripts broken down into reusable bits. | |
| Content | |
| The data attempts to create a structure around the Flying Circus scripts by breaking actions down into Dialogue (someone is speaking) and Direction (instructions for the actors). Along with each action I have tried to allocate the episode number, episode name, recording date, air date, segment name, name of character and name of actor playing the character. | |
| Acknowledgements | |
| The scripts are hosted in HTML at http://www.ibras.dk/montypython/justthewords.htm I have loaded all of the code that I wrote to scrape and process the data (warning: very messy) at https://github.com/allank/monty-python | |
| Inspiration | |
| I scraped the data because I was looking at data sources for doing RNN to generate text based on an existing corpus. While the amount of data available in the Flying Circus scripts is probably not sufficient, there might be some interesting things to do with the data. For example, some Markov chain generated dialogue lines: | |
| Remember, buy Whizzo butter and this dead crab. Yeah, er, I, I personally think this is getting too silly. I don't like the sound of two bricks being bashed together." | |
| Heartbeat Sounds,Classifying heartbeat anomalies from stethoscope audio,Ed King,161,"Version 1,2016-11-27","healthcare | |
| human genetics | |
| sound technology",CSV,152 MB,CC0,"36,595 views","3,376 downloads",32 kernels,,https://www.kaggle.com/kinguistics/heartbeat-sounds,"Try your hand at automatically separating normal heartbeats from abnormal heartbeats and heart murmur with this machine learning challenge by Peter Bentley et al | |
| The Data | |
| Here's a brief overview of the format of this dataset as uploaded to Kaggle. For a more detailed description, look at the Description section below. | |
| The dataset is split into two sources, A and B: A was collected from the general public via an iPhone app, and B was collected from a clinical trial in hospitals using a digital stethoscope. | |
| The goal of the task is to first (1) identify the locations of heart sounds from the audio, and (2) to classify the heart sounds into one of several categories (normal v. various non-normal heartbeat sounds). | |
| The CSV files provided are: set_a.csv | |
| set_b.csv set_a_timing.csv | |
| The fields for set_a and set_b are as follows: | |
| dataset: a or b | |
| fname: the audio file | |
| label: either ""normal"", blank (for unlabelled data), or one of various categories of abnormal heartbeats | |
| sublabel: in set_b, some recordings are categorized as noisy, meaning they contain non-heart background noise; this field holds information on whether something is e.g. ""noisynormal"" or ""noisymurmur"" | |
| The file set_a_timing.csv contains gold-standard timing information for the ""normal"" recordings from Set A. This file contains the following fields: | |
| fname: the audio file | |
| cycle: anywhere from 1 to 19; the heartbeat cycle that the time observation refers to | |
| sound: either S1 or S2; see below for what these mean | |
| location: the time location of this sound, in audio samples | |
| Description | |
| The task, as described by the original authors | |
| Task Overview | |
| Data has been gathered from two sources: (A) from the general public via the iStethoscope Pro iPhone app, provided in Dataset A, and (B) from a clinic trial in hospitals using the digital stethoscope DigiScope, provided in Dataset B. | |
| CHALLENGE 1 - Heart Sound Segmentation | |
| The first challenge is to produce a method that can locate S1(lub) and S2(dub) sounds within audio data, segmenting the Normal audio files in both datasets. To enable your machine learning method to learn we provide the exact location of S1 and S2 sounds for some of the audio files. You need to use them to identify and locate the S1 and S2 sounds of all the heartbeats in the unlabelled group. The locations of sounds are measured in audio samples for better precision. Your method must use the same unit. | |
| CHALLENGE 2 - Heart Sound Classification | |
| The task is to produce a method that can classify real heart audio (also known as “beat classification”) into one of four categories for Dataset A: | |
| Normal | |
| Murmur | |
| Extra Heart Sound | |
| Artifact | |
| and three classes for Dataset B: | |
| Normal | |
| Murmur | |
| Extrasystole | |
| You may tackle either or both of these challenges. If you can solve the first challenge, the second will be considerably easier! The winner of each challenge will be the method best able to segment and/or classify two sets of unlabelled data into the correct categories after training on both datasets provided below. | |
| [Obviously no longer applicable -- ed.]: The creator of the winning method will receive a WiFi 32Gb iPad as the prize, awarded at a workshop at AISTATS 2012. | |
| The audio files are of varying lengths, between 1 second and 30 seconds (some have been clipped to reduce excessive noise and provide the salient fragment of the sound). Most information in heart sounds is contained in the low frequency components, with noise in the higher frequencies. It is common to apply a low-pass filter at 195 Hz. Fast Fourier transforms are also likely to provide useful information about volume and frequency over time. More domain-specific knowledge about the difference between the categories of sounds is provided below. | |
| Normal Category | |
| In the Normal category there are normal, healthy heart sounds. These may contain noise in the final second of the recording as the device is removed from the body. They may contain a variety of background noises (from traffic to radios). They may also contain occasional random noise corresponding to breathing, or brushing the microphone against clothing or skin. A normal heart sound has a clear “lub dub, lub dub” pattern, with the time from “lub” to “dub” shorter than the time from “dub” to the next “lub” (when the heart rate is less than 140 beats per minute). Note the temporal description of “lub” and “dub” locations over time in the following illustration: | |
| …lub……….dub……………. lub……….dub……………. lub……….dub……………. lub……….dub… | |
| In medicine we call the lub sound ""S1"" and the dub sound ""S2"". Most normal heart rates at rest will be between about 60 and 100 beats (‘lub dub’s) per minute. However, note that since the data may have been collected from children or adults in calm or excited states, the heart rates in the data may vary from 40 to 140 beats or higher per minute. Dataset B also contains noisy_normal data - normal data which includes a substantial amount of background noise or distortion. You may choose to use this or ignore it, however the test set will include some equally noisy examples. | |
| Murmur Category | |
| Heart murmurs sound as though there is a “whooshing, roaring, rumbling, or turbulent fluid” noise in one of two temporal locations: (1) between “lub” and “dub”, or (2) between “dub” and “lub”. They can be a symptom of many heart disorders, some serious. There will still be a “lub” and a “dub”. One of the things that confuses non-medically trained people is that murmurs happen between lub and dub or between dub and lub; not on lub and not on dub. Below, you can find an asterisk* at the locations a murmur may be. | |
| …lub..*...dub……………. lub..*..dub ……………. lub..*..dub ……………. lub..*..dub … or …lub……….dub…*….lub………. dub…*….lub ………. dub…**….lub ……….dub… | |
| Dataset B also contains noisy_murmur data - murmur data which includes a substantial amount of background noise or distortion. You may choose to use this or ignore it, however the test set will include some equally noisy examples | |
| Extra Heart Sound Category (Dataset A) | |
| Extra heart sounds can be identified because there is an additional sound, e.g. a “lub-lub dub” or a “lub dub-dub”. An extra heart sound may not be a sign of disease. However, in some situations it is an important sign of disease, which if detected early could help a person. The extra heart sound is important to be able to detect as it cannot be detected by ultrasound very well. Below, note the temporal description of the extra heart sounds: | |
| …lub.lub……….dub………..………. lub. lub……….dub…………….lub.lub……..…….dub……. or …lub………. dub.dub………………….lub.……….dub.dub………………….lub……..…….dub. dub…… | |
| Artifact Category (Dataset A) | |
| In the Artifact category there are a wide range of different sounds, including feedback squeals and echoes, speech, music and noise. There are usually no discernable heart sounds, and thus little or no temporal periodicity at frequencies below 195 Hz. This category is the most different from the others. It is important to be able to distinguish this category from the other three categories, so that someone gathering the data can be instructed to try again. | |
| Extrasystole Category (Dataset B) | |
| Extrasystole sounds may appear occasionally and can be identified because there is a heart sound that is out of rhythm involving extra or skipped heartbeats, e.g. a “lub-lub dub” or a “lub dub-dub”. (This is not the same as an extra heart sound as the event is not regularly occuring.) An extrasystole may not be a sign of disease. It can happen normally in an adult and can be very common in children. However, in some situations extrasystoles can be caused by heart diseases. If these diseases are detected earlier, then treatment is likely to be more effective. Below, note the temporal description of the extra heart sounds: …........lub……….dub………..………. lub. ………..……….dub…………….lub.lub……..…….dub……. or …lub………. dub......………………….lub.…………………dub.dub………………….lub……..…….dub.…… | |
| Acknowledgments | |
| Please use the following citation if the data is used: | |
| @misc{pascal-chsc-2011, author = ""Bentley, P. and Nordehn, G. and Coimbra, M. and Mannor, S."", title = ""The {PASCAL} {C}lassifying {H}eart {S}ounds {C}hallenge 2011 {(CHSC2011)} {R}esults"", howpublished = ""http://www.peterjbentley.com/heartchallenge/index.html""}" | |
| A 6-figure prize by soccer prediction (Live Feed),"Clean, yet rich dataset of 7300 soccer matches & LIVE FEED",Mohammad Ghahramani,161,"Version 10,2017-09-19|Version 9,2017-09-18|Version 8,2017-09-18|Version 7,2017-09-07|Version 6,2017-09-01|Version 5,2017-09-01|Version 4,2017-09-01|Version 3,2017-08-26|Version 2,2017-08-26|Version 1,2017-08-26","association football | |
| world | |
| sports",CSV,17 MB,CC0,"9,497 views","2,374 downloads",4 kernels,12 topics,https://www.kaggle.com/analystmasters/earn-your-6-figure-prize,"Live Feed | |
| Please comment on the Discussion above ""Live Feed"" and I will share details with you in a message. Hope we won't exceed server limitations. | |
| Context | |
| I have been recording available different types of data on soccer matches since 2012, live 24/7. The whole database contains more than 350,000 soccer matches held all around the world from over 27,000 teams of more than 180 countries. An all-in-one package including servers, algorithms and its database are now under the ""Analyst Masters"" research platform. The app is also free for everyone to get its predictions on Android Play Store . How could it become useful for a data scientist? | |
| Did you know that, | |
| more than 1000 soccer matches are played in a week? | |
| the average profit of the stock market from its beginning to now has been less than 10% a year? but you can earn at least 10% on a single match in 2 hours and get your profit in cash | |
| It is one of the very rare datasets that you do not need to prove to other companies your method is the most accurate one and get the prize :) . On the other hand you do not have to classify every data point to be rewarded. Just tune or focus to correctly classify only 1% of matches and there you go! Let me give you a simple hint how easily it can become a classification problem rather than a time series prediction: | |
| Example 1: Who wins based on the number of wins in a head 2 head history? | |
| Q) Consider two teams Midtjylland and Randers from Denmark. They have played against each other for very long time. Midtjyland has won Randers over 8 times in the past 10 matches in a 4 year time span. Forget any other complicated algorithm and simply predict who wins this match? | |
| A) That is easy! However, I am also gathering a lot more information than just their history. You can check their head-to-head history and the odds you could get for predicting this match is ""1.73"" check here. | |
| Example 2: Number of Goals based on their history? | |
| Q) Consider two teams ""San Martin S.J."" and ""Rosario Central"" from Argentina. Their odds for wining ""Team 1 (Home)"", ""Draw"" and ""Team 2 (away)"" is [3.16, 3.2, 2.25] respectively. They rank 22 and 13 in their league. They have recently won 45%,35% of their matches in their past 14 matches. Their average head to head goals in their last 7 matches were 1.3 full time (F) and 0.3 until half-time (HT). How many goals do you think they score in their match? (Note that a safe side of number of goals in soccer betting is Over 0.5 goals in HT, Under 1.5 goals in HT, Over 1.5 goals in FT and Under 3.5 goals in FT). Which one do you choose? | |
| A) For sure under 1.5 goals in HT (you get 35%) and under 3.5 goals in FT (you get 30%) . Bingo you get 65% in a single match in 2 hours | |
| Example 3: Based on the money placed for betting on teams who wins the match? | |
| Q) ""Memmingen"" and ""Munich 1860"" are well known in Germany. One of our reliable sources of data is the ratio of money placed on betting from 10 hours before the match until it starts. Assume that the ratio of bets on ""Munich 1860"" to ""Memmingen"" are recorded every hour as below, which team do you think will win? | |
| [bets in $ on Munich 1860]/[bets in $ on Memmingen] : {1.01, 1.02, 1.04, 1.1, 1.2, 1.4, 1.58, 2.3, 2.6, 2.8} | |
| A) in 10 hours the amount of money placed on wining Munich 1860 Vs Memmingen increased from 1.01 to 2.8, who is the winner? Easy again, Munich 1860 that gives you 160% as stated here. | |
| Try the dataset and inspect every strategy you may come up with, as I gave you three reliable examples above. Just perform well enough to predict 15 matches correctly in a row, start with $1000 and you are a millionaire. If you can't be that accurate use the Kelly Criterion to divide your whole money into smaller stakes. | |
| Let me do the math for you, if you can only get 90% accuracy on 1% of data points (10 out of 1000 matches a week) and your average profit on each match is only 20%. You earn (9*20% = 180%) and lose 100% for your error in 10 predictions. Your net profit would be 80% in a week or approximately 12% in a day. if you risk only 33% of your whole money on each match then the daily net profit becomes 4%. I guess you can easily calculate how fast you can progress @ 4% daily accumulative profit. | |
| For sure one needs a live data feed to predict the outcome before the match. If everything goes well and enough users are interested I will open the live feed of data for you in a shared folder of Dropbox saved in CSV. | |
| Content | |
| Here is what the dataset contains for 'n' matches: | |
| names6.csv | |
| : team names as the order of ""home-away"" separated using ""/"" ; size : (n x 1) | |
| results6.csv* | |
| : Scores recorded during the match, every 2 rows show scores for one match ; size : (2n x 14) | |
| fresults6.csv* | |
| : Final scores after full-time ; size : (n x 2) | |
| odds6.csv | |
| : odds in the order of: Home-Draw-Away ; size : (n x 3) | |
| dollars6.csv* | |
| : Ratio of the money spent on teams at 15 minutes intervals ; size : (n x 76) | |
| ranks6.csv | |
| : their ranks in the league at the day of the match Irrespectively ; size : (n x 2) | |
| winrate6.csv | |
| : their winrate In the last (maximum 14) matches in 2017 Irrespectively ; size : (n x 2) | |
| country6.csv | |
| : their country as some countries are difficult to analyze e.g. Belarussia ; size : (n x 1) | |
| wins6.csv | |
| : number of wins in their last (maximum 6) head to head matches ; size : (n x 1) | |
| FT_HT6.csv | |
| : average of total goals in their last (maximum 6) head to head matches FT and HT ; size : (n x 2) | |
| *. recorded every 15 minutes | |
| Try to predict the match as examples above using the given data in the zip file. I will upload the respective data from Mid of August to Mid September later on. | |
| Acknowledgements | |
| I developed various data scrappers and classifiers running on multiple servers worldwide and never published a paper due to their sensitivity. You may refer to this database by mentioning the ""Analyst Masters"" research package. | |
| Inspiration | |
| 10 years ago I invented the world's first home-size cooking robot in my father's basement but in the end after cooking for us for 2 years it ended up in nothing. So, you as a data scientist can earn money using this live data stream for yourself if you can perform accurately without outperforming others in the competition just get an acceptable accuracy and you are good to go :) | |
| For more information on the overall platform and its live, pre-match and in-play analysis read at www.analystmasters.com or download the app for FREE to get easy predictions at 5% profit per week. More details on how the app operates is available at https://youtu.be/fqlu0YEyqc0" | |
| Yelp Dataset,"A trove of reviews, businesses, users, tips, and check-in data!","Yelp, Inc.",160,"Version 6,2018-02-07|Version 5,2018-02-07|Version 4,2018-01-23|Version 3,2018-01-23|Version 2,2018-01-18|Version 1,2018-01-18",food and drink,CSV,3 GB,Other,"21,641 views","3,347 downloads",29 kernels,6 topics,https://www.kaggle.com/yelp-dataset/yelp-dataset,"Context | |
| This dataset is a subset of Yelp's businesses, reviews, and user data. It was originally put together for the Yelp Dataset Challenge which is a chance for students to conduct research or analysis on Yelp's data and share their discoveries. In the dataset you'll find information about businesses across 11 metropolitan areas in four countries. | |
| Content | |
| This dataset contains seven CSV files. The original JSON files can be found in yelp_academic_dataset.zip. | |
| You may find this documentation helpful: | |
| https://www.yelp.com/dataset/documentation/json | |
| In total, there are : | |
| 5,200,000 user reviews | |
| Information on 174,000 businesses | |
| The data spans 11 metropolitan areas | |
| Acknowledgements | |
| The dataset was converted from JSON to CSV format and we thank the team of the Yelp dataset challenge for creating this dataset. | |
| By downloading this dataset, you agree to the Yelp Dataset Terms of Use. | |
| Inspiration | |
| Natural Language Processing & Sentiment Analysis | |
| What's in a review? Is it positive or negative? Yelp's reviews contain a lot of metadata that can be mined and used to infer meaning, business attributes, and sentiment. | |
| Graph Mining | |
| We recently launched our Local Graph but can you take the graph further? How do user's relationships define their usage patterns? Where are the trend setters eating before it becomes popular?" | |
| Used cars database,"Over 370,000 used cars scraped from Ebay Kleinanzeigen",Orges Leka,156,"Version 3,2016-11-28|Version 2,2016-11-28|Version 1,2016-11-20",automobiles,CSV,65 MB,CC0,"82,725 views","11,526 downloads",209 kernels,7 topics,https://www.kaggle.com/orgesleka/used-cars-database,"If you want to download and experiment with the Scrapy script, you can do so from forum data science This page is a forum for data scientist I started, in hope , that you will participate and maybe even improve the scrapy script. | |
| Over 370000 used cars scraped with Scrapy from Ebay-Kleinanzeigen. The content of the data is in german, so one has to translate it first if one can not speak german. Those fields are included: autos.csv: | |
| dateCrawled : when this ad was first crawled, all field-values are taken from this date | |
| name : ""name"" of the car | |
| seller : private or dealer | |
| offerType | |
| price : the price on the ad to sell the car | |
| abtest | |
| vehicleType | |
| yearOfRegistration : at which year the car was first registered | |
| gearbox | |
| powerPS : power of the car in PS | |
| model | |
| kilometer : how many kilometers the car has driven | |
| monthOfRegistration : at which month the car was first registered | |
| fuelType | |
| brand | |
| notRepairedDamage : if the car has a damage which is not repaired yet | |
| dateCreated : the date for which the ad at ebay was created | |
| nrOfPictures : number of pictures in the ad (unfortunately this field contains everywhere a 0 and is thus useless (bug in crawler!) ) | |
| postalCode | |
| lastSeenOnline : when the crawler saw this ad last online | |
| The fields lastSeen and dateCreated could be used to estimate how long a car will be at least online before it is sold. | |
| brought to you by Orges Leka | |
| Regression on average Price per Year based on this dataset | |
| Table of value loss of an average used car per year | |
| The second file is produced in MySQL from the first one through the query: | |
| select | |
| count(*) as count, | |
| kilometer, | |
| yearOfRegistration, | |
| 20*round(powerPS/20) as powerPS, | |
| min(price) as minprice, | |
| max(price) as maxPrice, | |
| avg(price) as avgPreis, | |
| sqrt(variance(price)) as sdPreis from items where | |
| yearOfRegistration > 1990 and yearOfRegistration < 2016 | |
| and price > 100 and price < 100000 | |
| and powerPS < 600 and powerPS > 0 | |
| group by yearOfRegistration, round(powerPS/20),kilometer | |
| having count > 10 | |
| into outfile '/tmp/cnt_km_year_powerPS_minPrice_maxPrice_avgPrice_sdPrice.csv' | |
| fields terminated by ',' lines terminated by '\n'; | |
| Happy Coding!" | |
| Video Game Sales with Ratings,Video game sales from Vgchartz and corresponding ratings from Metacritic,Rush Kirubi,154,"Version 2,2016-12-30|Version 1,2016-12-22",video games,CSV,2 MB,Other,"61,430 views","10,594 downloads",184 kernels,8 topics,https://www.kaggle.com/rush4ratio/video-game-sales-with-ratings,"Context | |
| Motivated by Gregory Smith's web scrape of VGChartz Video Games Sales, this data set simply extends the number of variables with another web scrape from Metacritic. Unfortunately, there are missing observations as Metacritic only covers a subset of the platforms. Also, a game may not have all the observations of the additional variables discussed below. Complete cases are ~ 6,900 | |
| Content | |
| Alongside the fields: Name, Platform, Year_of_Release, Genre, Publisher, NA_Sales, EU_Sales, JP_Sales, Other_Sales, Global_Sales, we have:- | |
| Critic_score - Aggregate score compiled by Metacritic staff | |
| Critic_count - The number of critics used in coming up with the Critic_score | |
| User_score - Score by Metacritic's subscribers | |
| User_count - Number of users who gave the user_score | |
| Developer - Party responsible for creating the game | |
| Rating - The ESRB ratings | |
| Acknowledgements | |
| This repository, https://github.com/wtamu-cisresearch/scraper, after a few adjustments worked extremely well! | |
| Inspiration | |
| It would be interesting to see any machine learning techniques or continued data visualizations applied on this data set." | |
| US Baby Names,Explore naming trends from babies born in the US,Kaggle,153,"Version 2,2017-11-22|Version 1,2015-12-19",children,Other,173 MB,CC0,"93,894 views","10,822 downloads",417 kernels,10 topics,https://www.kaggle.com/kaggle/us-baby-names,"US Social Security applications are a great way to track trends in how babies born in the US are named. | |
| Data.gov releases two datasets that are helplful for this: one at the national level and another at the state level. Note that only names with at least 5 babies born in the same year (/ state) are included in this dataset for privacy. | |
| I've taken the raw files here and combined/normalized them into two CSV files (one for each dataset) as well as a SQLite database with two equivalently-defined tables. The code that did these transformations is available here. | |
| New to data exploration in R? Take the free, interactive DataCamp course, ""Data Exploration With Kaggle Scripts,"" to learn the basics of visualizing data with ggplot. You'll also create your first Kaggle Scripts along the way." | |
| "Homicide Reports, 1980-2014",Can you develop an algorithm to detect serial killer activity?,Murder Accountability Project,153,"Version 1,2017-02-10",crime,CSV,107 MB,CC4,"55,970 views","9,390 downloads",323 kernels,11 topics,https://www.kaggle.com/murderaccountability/homicide-reports,"Content | |
| The Murder Accountability Project is the most complete database of homicides in the United States currently available. This dataset includes murders from the FBI's Supplementary Homicide Report from 1976 to the present and Freedom of Information Act data on more than 22,000 homicides that were not reported to the Justice Department. This dataset includes the age, race, sex, ethnicity of victims and perpetrators, in addition to the relationship between the victim and perpetrator and weapon used. | |
| Acknowledgements | |
| The data was compiled and made available by the Murder Accountability Project, founded by Thomas Hargrove." | |
| Dota 2 Matches,Explore player behavior and predict match outcomes.,devin,139,"Version 3,2016-11-06|Version 2,2016-10-25|Version 1,2016-10-24",video games,CSV,1 GB,CC4,"35,491 views","3,684 downloads",132 kernels,2 topics,https://www.kaggle.com/devinanzelmo/dota-2-matches,"Overview | |
| This dataset contains 50000 ranked ladder matches from the Dota 2 data dump created by Opendota. It was inspired by the Dota 2 Matches data published here by Joe Ramir. This is an update and improved version of that dataset. I have kept the same image and a similar title. | |
| Dota 2 is a popular MOBA available as free to play, and can take up thousands of hours of your life. The number of games in this dataset are played about every hour. If you like the data there are an additional 2-3 million matches easily available for download. | |
| The aim of this dataset is to enable the exploration of player behavior, skill estimation, or anything you find interesting. The intent is to create an accessible, and easy to use resource, which can be expanded and modified if needed. As such I am open to a wide variety of suggestions as to what additions or changes to make. | |
| Help getting started | |
| If there is some aspect of this data you would like to explore but seems difficult to get figure out how to work with please feel free to request some starter code in one of the following two Kernels discussion section. I usually check kaggle every day or so. If you post a request about the current data I will try to get something working. | |
| Python https://www.kaggle.com/devinanzelmo/d/devinanzelmo/dota-2-matches/misc-howtos-dota-requests-welcome/ | |
| R https://www.kaggle.com/devinanzelmo/d/devinanzelmo/dota-2-matches/howtos-request-welcome/ | |
| Whats Currently Available | |
| See https://github.com/odota/core/wiki/JSON-Data-Dump for documentaion on data. I have found a few undocumented areas in the data, including the objectives information. player_slot can be used to combine most of the data, and it is available in most of the tables. Additionally all tables include match_id, and some have account_id to make it easier to look at an individual players matches. match_id, and account_id have been reencoded to save a little space. I can upload tables to allow conversion if needed. | |
| matches: contains top level information about each match. see https://wiki.teamfortress.com/wiki/WebAPI/GetMatchDetails#Tower_Status%22tower_status_dire%22:%202047) for interpreting tower and barracks status. Cluster can link matches to geographic region. | |
| players: Individual players are identified by account_id but there is an option to play anonymously and roughly one third of the account_id are not available. Anonymous users have the value of 0 for account_id. Contains totals for kills, deaths, denies, etc. Player action counts are available, and are indicated by variable names beginning with unit_order_. Counts for reasons for acquiring or losing gold, and gaining experience, have prefixes gold_, and xp_. | |
| player_time: Contains last hits, experience, and gold sampled at one minute interval for all players in all matches. The column names indicate the player_slot. For instance xp_t_1 indicates that this column has experience sums for the player in slot one. | |
| teamfights: Start and stop time of teamfights, as well as last death time. Teamfights appear to be all battles with three or more deaths. As such this does not include all battles for the entire match. | |
| teamfights_players : Additional information provided for each player in each teamfight. player_slot can be used to link this back to players.csv | |
| objectives: Gives information on all the objectives completed, by which player and at what time. | |
| chat: All chat for the 50k matches. There is plenty of profanity, and good natured trolling. | |
| test_labels: match_id and radiant_win(as integer 1 or 0) | |
| test_player: full player and match table with hero_id, player_slot, match_id, and account_id | |
| Nov 5th Update | |
| Added several additional tables. None of the previously uploaded data was altered. I plan to add several Kernels in the next week going over how to use the data, and performing some EDA. Many improvements to the player rating method I used are possible for those interested in MMR. | |
| player_ratings contains match counts, win counts, and TrueSkill rating, calculated on 900k matches which occurred prior to other uploaded data. trueskill ratings have two components, mu, which can be interpreted as the skill, with higher value being better, and sigma which is the uncertainty of the rating. | |
| match_outcomes data for ~900k matches used to calculate player ratings. Use this to improve on the ratings I uploaded. | |
| purchase_log item purchase times | |
| ability_upgrade ability upgrade times and levels | |
| cluster_region allows the mapping cluster found in match.csv to geographic region. | |
| patch_dates release dates for various patches, use start_time from match.csv to determine which patch a match was played in. | |
| ability_ids use with ability_upgrades.csv to get the names of upgraded abilities | |
| item_ids use with purchase_log.csv to get the names of purchased items | |
| Kernel showing how player skill was computed: Contains several resources on trueskill rating system. | |
| Past Research | |
| There seem to be some efforts to establish indicators for skillfull play based on specific parts of gameplay. Opendota has many statistics, and some analysis for specific benchmarks at different times in the game. Dotabuff has a lot of information I have not explored it deeply. This is an area to gather more information. | |
| Some possible directions of investigation | |
| Insight from domain experts would also be useful to help clarify what problems are interesting to work on. Some initial task ideas | |
| Predict match outcomes based on aggregates for individual players using only account_id as prior information | |
| Add hero id to this and see if there is a differences in performance | |
| Estimate player skill based on a sample of in game play(this might need an external mmr source or different definition skill) | |
| Create improved indicators of skillful play based game actions to help players target areas for improvement | |
| All of these areas have been worked on, but I am not aware of the most up to date research on dota2 gameplay. | |
| I plan on setting up several different predictive tasks in the upcoming weeks. A test set of an additional 50 to 100 thousand matches with just hero_id, and account_id included along with outcome of the match. | |
| The current dataset seems pretty small for modeling individual players. I would prefer to have a wide range of features instead of a larger dataset for the moment. | |
| Dataset idea for anyone interested in creating their own Dota 2 dataset. It would be useful to have a few full matches available to work on. They would need to be extracted from the .dem replay file to something easily parsed by R and Python as available in kernels. Given the size of a full match data only a few matches would be needed. There are files available from opendota' s website(check for replays). Looking at fine grained match details would potentially allow for the creation of better high level parsed data. I think it would be a lot of work just to get a handle on working with full match data so a sample would be good to have. | |
| Acknowledgements | |
| Orginal kaggle dataset on dota2 matches by Joe Ramir I also borrowed the image and some of the content for these acknowledgements from the above, thanks!. | |
| image source | |
| Data download source created by yasp | |
| Description of original dataset creation: https://github.com/yasp-dota/yasp/issues/924 | |
| yasp's license | |
| ""License: CC BY-SA 4.0"" | |
| ""Terms: We ask that you attribute yasp.co if you create or publish anything related to our data. Also, please seed for as long as possible."" | |
| Yasp is now known as opendota here are links to their website and github page | |
| https://www.opendota.com/ the data is used to for this site and its a easy way to get familier with it | |
| https://github.com/odota/core check here for info especially this wiki page which gives details on the schema." | |
| Airplane Crashes Since 1908,"Full history of airplane crashes throughout the world, from 1908-present",Sauro Grandi,139,"Version 4,2016-09-10|Version 3,2016-05-21|Version 2,2016-05-21|Version 1,2016-05-21",aviation,CSV,2 MB,ODbL,"63,487 views","15,228 downloads",415 kernels,5 topics,https://www.kaggle.com/saurograndi/airplane-crashes-since-1908,"Analysis of the public dataset: ""Airplane Crashes and Fatalities Since 1908"" (Full history of airplane crashes throughout the world, from 1908-present) hosted by Open Data by Socrata available at: | |
| https://opendata.socrata.com/Government/Airplane-Crashes-and-Fatalities-Since-1908/q2te-8cvq | |
| Questions | |
| Yearly how many planes crashed? how many people were on board? how many survived? how many died? | |
| Highest number of crashes by operator and Type of aircrafts. | |
| ‘Summary’ field has the details about the crashes. Find the reasons of the crash and categorize them in different clusters i.e Fire, shot down, weather (for the ‘Blanks’ in the data category can be UNKNOWN) you are open to make clusters of your choice but they should not exceed 7. | |
| Find the number of crashed aircrafts and number of deaths against each category from above step. | |
| Find any interesting trends/behaviors that you encounter when you analyze the dataset. | |
| My solution | |
| The following bar charts display the answers requested by point 1. of the assignment, in particular: | |
| the planes crashed per year | |
| people aboard per year during crashes | |
| people dead per year during crashes | |
| people survived per year during crashes | |
| The following answers regard point 2 of the assignment | |
| Highest number of crashes by operator: Aeroflot with 179 crashes | |
| By Type of aircraft: Douglas DC-3 with 334 crashes | |
| I have identified 7 clusters using k-means clustering technique on a matrix obtained by a text corpus created by using Text Analysis (plain text, remove punctuation, to lower, etc.) The following table summarize for each cluster the number of crashes and death. | |
| Cluster 1: 258 crashes, 6368 deaths | |
| Cluster 2: 500 crashes, 9408 deaths | |
| Cluster 3: 211 crashes, 3513 deaths | |
| Cluster 4: 1014 crashes, 14790 deaths | |
| Cluster 5: 2749 crashes, 58826 deaths | |
| Cluster 6: 195 crashes, 4439 deaths | |
| Cluster 7: 341 crashes, 8135 deaths | |
| The following picture shows clusters using the first 2 principal components: | |
| For each clusters I will summarize the most used words and I will try to identify the causes of the crash | |
| Cluster 1 (258) aircraft, crashed, plane, shortly, taking. No many information about this cluster can be deducted using Text Analysis | |
| Cluster 2 (500) aircraft, airport, altitude, crashed, crew, due, engine, failed, failure, fire, flight, landing, lost, pilot, plane, runway, takeoff, taking. Engine failure on the runway after landing or takeoff | |
| Cluster 3 (211): aircraft, crashed, fog Crash caused by fog | |
| Cluster 4 (1014): aircraft, airport, attempting, cargo, crashed, fire, land, landing, miles, pilot, plane, route, runway, struck, takeoff Struck a cargo during landing or takeoff | |
| Cluster 5 (2749): accident, aircraft, airport, altitude, approach, attempting, cargo, conditions, control, crashed, crew, due, engine, failed, failure, feet, fire, flight, flying, fog, ground, killed, land, landing, lost, low, miles, mountain, pilot. plane, poor, route, runway, short, shortly, struck, takeoff, taking, weather | |
| Struck a cargo due to engine failure or bad weather conditions mainly fog | |
| Cluster 6 (195): aircraft, crashed, engine, failure, fire, flight, left, pilot, plane, runway | |
| Engine failure on the runway | |
| Cluster 7 (341): accident, aircraft, altitude, cargo, control, crashed, crew, due, engine, failure, flight, landing, loss, lost, pilot, plane, takeoff | |
| Engine failure during landing or takeoff | |
| Better solutions are welcome. Thanks." | |
| US Mass Shootings,Last 50 Years (1966-2017),Zeeshan-ul-hassan Usmani,139,"Version 4,2017-11-06|Version 3,2017-11-05|Version 3,2017-11-05|Version 2,2017-10-11|Version 1,2017-10-03","united states | |
| crime | |
| violence | |
| terrorism",CSV,679 KB,ODbL,"34,332 views","4,893 downloads",40 kernels,18 topics,https://www.kaggle.com/zusmani/us-mass-shootings-last-50-years,"Context | |
| Mass Shootings in the United States of America (1966-2017) The US has witnessed 398 mass shootings in last 50 years that resulted in 1,996 deaths and 2,488 injured. The latest and the worst mass shooting of October 2, 2017 killed 58 and injured 515 so far. The number of people injured in this attack is more than the number of people injured in all mass shootings of 2015 and 2016 combined. The average number of mass shootings per year is 7 for the last 50 years that would claim 39 lives and 48 injured per year. | |
| Content | |
| Geography: United States of America | |
| Time period: 1966-2017 | |
| Unit of analysis: Mass Shooting Attack | |
| Dataset: The dataset contains detailed information of 398 mass shootings in the United States of America that killed 1996 and injured 2488 people. | |
| Variables: The dataset contains Serial No, Title, Location, Date, Summary, Fatalities, Injured, Total Victims, Mental Health Issue, Race, Gender, and Lat-Long information. | |
| Acknowledgements | |
| I’ve consulted several public datasets and web pages to compile this data. Some of the major data sources include Wikipedia, Mother Jones, Stanford, USA Today and other web sources. | |
| Inspiration | |
| With a broken heart, I like to call the attention of my fellow Kagglers to use Machine Learning and Data Sciences to help me explore these ideas: | |
| • How many people got killed and injured per year? | |
| • Visualize mass shootings on the U.S map | |
| • Is there any correlation between shooter and his/her race, gender | |
| • Any correlation with calendar dates? Do we have more deadly days, weeks or months on average | |
| • What cities and states are more prone to such attacks | |
| • Can you find and combine any other external datasets to enrich the analysis, for example, gun ownership by state | |
| • Any other pattern you see that can help in prediction, crowd safety or in-depth analysis of the event | |
| • How many shooters have some kind of mental health problem? Can we compare that shooter with general population with same condition | |
| Mass Shootings Dataset Ver 3 | |
| This is the new Version of Mass Shootings Dataset. I've added eight new variables: | |
| Incident Area (where the incident took place), | |
| Open/Close Location (Inside a building or open space) | |
| Target (possible target audience or company), | |
| Cause (Terrorism, Hate Crime, Fun (for no obvious reason etc.) | |
| Policeman Killed (how many on duty officers got killed) | |
| Age (age of the shooter) | |
| Employed (Y/N) | |
| Employed at (Employer Name) | |
| Age, Employed and Employed at (3 variables) contain shooter details | |
| Mass Shootings Dataset Ver 4 | |
| Quite a few missing values have been added | |
| Mass Shootings Dataset Ver 5 | |
| Three more recent mass shootings have been added including the Texas Church shooting of November 5, 2017 | |
| I hope it will help create more visualization and extract patterns. | |
| Keep Coding!" | |
| Can You Predict Product Backorders?,Based on historical data predict backorder risk for products,tiredgeek,138,"Version 4,2017-04-27|Version 3,2017-04-27|Version 2,2017-04-27|Version 1,2017-03-20","business | |
| management",CSV,134 MB,CC4,"45,608 views","6,034 downloads",109 kernels,6 topics,https://www.kaggle.com/tiredgeek/predict-bo-trial,"Context | |
| Part backorders is a common supply chain problem. Working to identify parts at risk of backorder before the event occurs so the business has time to react. | |
| Content | |
| Training data file contains the historical data for the 8 weeks prior to the week we are trying to predict. The data was taken as weekly snapshots at the start of each week. Columns are defined as follows: | |
| sku - Random ID for the product | |
| national_inv - Current inventory level for the part | |
| lead_time - Transit time for product (if available) | |
| in_transit_qty - Amount of product in transit from source | |
| forecast_3_month - Forecast sales for the next 3 months | |
| forecast_6_month - Forecast sales for the next 6 months | |
| forecast_9_month - Forecast sales for the next 9 months | |
| sales_1_month - Sales quantity for the prior 1 month time period | |
| sales_3_month - Sales quantity for the prior 3 month time period | |
| sales_6_month - Sales quantity for the prior 6 month time period | |
| sales_9_month - Sales quantity for the prior 9 month time period | |
| min_bank - Minimum recommend amount to stock | |
| potential_issue - Source issue for part identified | |
| pieces_past_due - Parts overdue from source | |
| perf_6_month_avg - Source performance for prior 6 month period | |
| perf_12_month_avg - Source performance for prior 12 month period | |
| local_bo_qty - Amount of stock orders overdue | |
| deck_risk - Part risk flag | |
| oe_constraint - Part risk flag | |
| ppap_risk - Part risk flag | |
| stop_auto_buy - Part risk flag | |
| rev_stop - Part risk flag | |
| went_on_backorder - Product actually went on backorder. This is the target value." | |
| SMS Spam Collection Dataset,Collection of SMS messages tagged as spam or legitimate,UCI Machine Learning,138,"Version 1,2016-12-03","languages | |
| linguistics | |
| human-computer interaction",CSV,492 KB,Other,"50,596 views","7,255 downloads",158 kernels,0 topics,https://www.kaggle.com/uciml/sms-spam-collection-dataset,"Context | |
| The SMS Spam Collection is a set of SMS tagged messages that have been collected for SMS Spam research. It contains one set of SMS messages in English of 5,574 messages, tagged acording being ham (legitimate) or spam. | |
| Content | |
| The files contain one message per line. Each line is composed by two columns: v1 contains the label (ham or spam) and v2 contains the raw text. | |
| Acknowledgements | |
| The original dataset can be found here. The creators would like to note that in case you find the dataset useful, please make a reference to previous paper and the web page: http://www.dt.fee.unicamp.br/~tiago/smsspamcollection/ in your papers, research, etc. | |
| Inspiration | |
| Can you use this dataset to build a prediction model that will accurately classify which texts are spam?" | |
| U.S. Pollution Data,Pollution in the U.S. since 2000,BrendaSo,137,"Version 1,2016-11-05","environment | |
| pollution",CSV,382 MB,ODbL,"56,876 views","7,252 downloads",79 kernels,5 topics,https://www.kaggle.com/sogun3/uspollution,"Context | |
| This dataset deals with pollution in the U.S. Pollution in the U.S. has been well documented by the U.S. EPA but it is a pain to download all the data and arrange them in a format that interests data scientists. Hence I gathered four major pollutants (Nitrogen Dioxide, Sulphur Dioxide, Carbon Monoxide and Ozone) for every day from 2000 - 2016 and place them neatly in a CSV file. | |
| Content | |
| There is a total of 28 fields. The four pollutants (NO2, O3, SO2 and O3) each has 5 specific columns. Observations totaled to over 1.4 million. This kernel provides a good introduction to this dataset! | |
| For observations on specific columns visit the Column Metadata on the Data tab. | |
| Acknowledgements | |
| All the data is scraped from the database of U.S. EPA : https://aqsdr1.epa.gov/aqsweb/aqstmp/airdata/download_files.html | |
| Inspiration | |
| I did a related project with some of my friends in college, and decided to open source our dataset so that data scientists don't need to re-scrape the U.S. EPA site for historical pollution data." | |
| 2015 Flight Delays and Cancellations,Which airline should you fly on to avoid significant delays?,Department of Transportation,136,"Version 1,2017-02-10",aviation,CSV,565 MB,CC0,"64,292 views","12,508 downloads",80 kernels,10 topics,https://www.kaggle.com/usdot/flight-delays,"Context | |
| The U.S. Department of Transportation's (DOT) Bureau of Transportation Statistics tracks the on-time performance of domestic flights operated by large air carriers. Summary information on the number of on-time, delayed, canceled, and diverted flights is published in DOT's monthly Air Travel Consumer Report and in this dataset of 2015 flight delays and cancellations. | |
| Acknowledgements | |
| The flight delay and cancellation data was collected and published by the DOT's Bureau of Transportation Statistics." | |
| FIFA 18 Complete Player Dataset,"17k+ players, 70+ attributes extracted from the latest edition of FIFA",Aman Shrivastava,135,"Version 5,2017-10-30|Version 4,2017-10-30|Version 3,2017-09-28|Version 2,2017-09-26|Version 1,2017-09-16","popular culture | |
| video games | |
| association football",CSV,15 MB,CC4,"45,156 views","5,277 downloads",33 kernels,13 topics,https://www.kaggle.com/thec03u5/fifa-18-demo-player-dataset,"The Dataset you can play with. | |
| Context | |
| Dataset for people who love data science and have grown up playing FIFA. | |
| Content | |
| Every player featuring in FIFA 18 | |
| 70+ attributes | |
| Player and Flag Images | |
| Playing Position Data | |
| Attributes based on actual data of the latest EA's FIFA 18 game | |
| Attributes include on all player style statistics like Dribbling, Aggression, GK Skills etc. | |
| Player personal data like Nationality, Photo, Club, Age, Wage, Salary etc. | |
| Upcoming Update will Include : | |
| Team (National and Club) Data | |
| Player Images in Zip folder | |
| Betting Odds | |
| The dataset contains all the statistics and playing attributes of all the players in the Full version of FIFA 18. | |
| Data Source | |
| The data is scraped from the website https://sofifa.com by extracting the Player personal data and Player Ids and then the playing and style statistics. | |
| Github Project | |
| Possible Explorations | |
| Make your dream team | |
| Analyse which Club or National Team has the best-rated players | |
| Assess the strength of a team at a particular position | |
| Analyse the team with the best dribbling speed | |
| Co-relate between Age and Overall rating | |
| Co-relate between Age and Nationality | |
| Co-relate between Age and Potential | |
| Could prove of immense value to Fantasy Premier League enthusiasts. | |
| These are just basic examples, sky is the limit. | |
| Acknowledgements | |
| The data has been crawled from the https://sofifa.com website. | |
| Inspiration | |
| Several insights and correlations between player value, wage, age, and performance can be derived from the dataset. Furthermore, how do the players in this dataset compare against themselves in last year's dataset? | |
| Contributing | |
| Changes and Improvement suggestions are welcome. Feel free to comment new additions that you think are useful or drop a PR on the github project." | |
| Meta Kaggle,"Kaggle's public data on competitions, users, submission scores, and kernels",Kaggle,134,"Version 7,2016-07-21|Version 6,2016-07-06|Version 5,2016-06-23|Version 4,2016-06-23|Version 3,2016-05-04|Version 2,2016-04-20|Version 1,2015-09-09","statistics | |
| telecommunications",CSV,2 GB,CC4,"45,477 views","3,441 downloads",414 kernels,23 topics,https://www.kaggle.com/kaggle/meta-kaggle,"We aren't saying this dataset is the Rosetta Stone of machine learning competitions, but we do think there is a lot to learn from (and a lot of fun to be had by) releasing some of our most interesting tables on Kaggle community and competition activity. | |
| Strategizing to become a Master? Wondering who, where, and what goes in to a winning team? Deciding between evaluation metrics for your next data science project? We hope the scripts published here will enrich and entertain Kagglers, spark some lively conversations, and act as a resource for the larger machine learning community. | |
| This data (available through Kaggle Scripts as CSV files and a SQLite database) contains the tables listed below. | |
| Note that this data is not a complete dump: rows, columns, and tables have been filtered out, and it is a small subset of the data that we can release publicly. Over time, we'll add more of the tables that we can release publicly to it." | |
| Complete FIFA 2017 Player dataset (Global),"15k+ players, 50+ Attributes per player from the latest EA Sports Fifa 17",Soumitra Agarwal,133,"Version 5,2017-04-13|Version 4,2017-04-12|Version 3,2017-04-12|Version 2,2017-04-02|Version 1,2017-04-01","video games | |
| association football",CSV,9 MB,ODbL,"39,473 views","5,801 downloads",91 kernels,5 topics,https://www.kaggle.com/artimous/complete-fifa-2017-player-dataset-global,"The dataset for people who double on Fifa and Data Science | |
| Content | |
| 17,000+ players | |
| 50+ attributes per player ranging from ball skills aggression etc. | |
| Player's attributes sourced from EA Sports' FIFA video game series, including the weekly updates | |
| Players from all around the globe | |
| URLs to their homepage | |
| Club logos | |
| Player images male and female | |
| National and club team data | |
| Weekly Updates would include : | |
| Real life data (Match events etc.) | |
| The fifa generated player dataset | |
| Betting odds | |
| Growth | |
| Data Source | |
| Data was scraped from https://www.fifaindex.com/ first by getting player profile url set (as stored in PlayerNames.csv) and then scraping the individual pages for their attributes | |
| Improvements | |
| You may have noticed that for a lot of players, their national details are absent (Team and kit number) even though the nationality is listed. This may be attributed to the missing data on fifa sites. | |
| GITHUB PROJECT | |
| There is much more than just 50 attributes by which fifa decides what happens to players over time, how they perform under pressure, how they grow etc. This data obviously would be well hidden by the organisation and thus would be tough to find | |
| Important note for people interested in using the scraping: The site is not uniform and thus the scraping script requires considering a lot of corner cases (i.e. interchanged position of different attributes). Also the script contains proxy preferences which may be removed if not required. | |
| Exploring the data | |
| For starters you can become a scout: | |
| Create attribute dependent or overall best teams | |
| Create the fastest/slowest teams | |
| See which areas of the world provide which attributes (like Africa : Stamina, Pace) | |
| See which players are the best at each position | |
| See which outfield players can play a better role at some other position | |
| See which youngsters have attributes which can be developed | |
| And that is just the beginning. This is the playground.. literally! | |
| Data description | |
| The file FullData.csv contains attributes describing the in game play style and also some of the real statistics such as Nationality etc. | |
| The file PlayerNames.csv contains URLs for different players from their profiles on fifaindex.com. Append the URLs after the base url fifaindex.com. | |
| The compressed file Pictures.zip contains pictures for top 1000 players in Fifa 17. | |
| The compressed file Pictures_f.zip contains pictures for top 139 female players in Fifa 17. | |
| The compressed file ClubPictures.zip contains pictures for emblems of some major clubs in Fifa 17. | |
| Inspiration | |
| I am a huge FIFA fanatic. While playing career mode I realised that I picked great young players early on every single time and since a lot of digital learning relies on how our brain works, I thought scouting great qualities in players would be something that can be worked on. Since then I started working on scraping the website and here is the data. I hope we can build something on it. | |
| With access to players attributes you can become the best scout in the world. Go for it!" | |
| Netflix Prize data,Dataset from Netflix's competition to improve their reccommendation algorithm,Netflix,131,"Version 1,2017-07-20","film | |
| artificial intelligence",Other,2 GB,Other,"40,645 views","3,967 downloads",4 kernels,2 topics,https://www.kaggle.com/netflix-inc/netflix-prize-data,"Context | |
| Netflix held the Netflix Prize open competition for the best algorithm to predict user ratings for films. The grand prize was $1,000,000 and was won by BellKor's Pragmatic Chaos team. This is the dataset that was used in that competition. | |
| Content | |
| This comes directly from the README: | |
| TRAINING DATASET FILE DESCRIPTION | |
| The file ""training_set.tar"" is a tar of a directory containing 17770 files, one per movie. The first line of each file contains the movie id followed by a colon. Each subsequent line in the file corresponds to a rating from a customer and its date in the following format: | |
| CustomerID,Rating,Date | |
| MovieIDs range from 1 to 17770 sequentially. | |
| CustomerIDs range from 1 to 2649429, with gaps. There are 480189 users. | |
| Ratings are on a five star (integral) scale from 1 to 5. | |
| Dates have the format YYYY-MM-DD. | |
| MOVIES FILE DESCRIPTION | |
| Movie information in ""movie_titles.txt"" is in the following format: | |
| MovieID,YearOfRelease,Title | |
| MovieID do not correspond to actual Netflix movie ids or IMDB movie ids. | |
| YearOfRelease can range from 1890 to 2005 and may correspond to the release of corresponding DVD, not necessarily its theaterical release. | |
| Title is the Netflix movie title and may not correspond to titles used on other sites. Titles are in English. | |
| QUALIFYING AND PREDICTION DATASET FILE DESCRIPTION | |
| The qualifying dataset for the Netflix Prize is contained in the text file ""qualifying.txt"". It consists of lines indicating a movie id, followed by a colon, and then customer ids and rating dates, one per line for that movie id. The movie and customer ids are contained in the training set. Of course the ratings are withheld. There are no empty lines in the file. | |
| MovieID1: | |
| CustomerID11,Date11 | |
| CustomerID12,Date12 | |
| ... | |
| MovieID2: | |
| CustomerID21,Date21 | |
| CustomerID22,Date22 | |
| For the Netflix Prize, your program must predict the all ratings the customers gave the movies in the qualifying dataset based on the information in the training dataset. | |
| The format of your submitted prediction file follows the movie and customer id, date order of the qualifying dataset. However, your predicted rating takes the place of the corresponding customer id (and date), one per line. | |
| For example, if the qualifying dataset looked like: | |
| 111: | |
| 3245,2005-12-19 | |
| 5666,2005-12-23 | |
| 6789,2005-03-14 | |
| 225: | |
| 1234,2005-05-26 | |
| 3456,2005-11-07 | |
| then a prediction file should look something like: | |
| 111: | |
| 3.0 | |
| 3.4 | |
| 4.0 | |
| 225: | |
| 1.0 | |
| 2.0 | |
| which predicts that customer 3245 would have rated movie 111 3.0 stars on the 19th of Decemeber, 2005, that customer 5666 would have rated it slightly higher at 3.4 stars on the 23rd of Decemeber, 2005, etc. | |
| You must make predictions for all customers for all movies in the qualifying dataset. | |
| THE PROBE DATASET FILE DESCRIPTION | |
| To allow you to test your system before you submit a prediction set based on the qualifying dataset, we have provided a probe dataset in the file ""probe.txt"". This text file contains lines indicating a movie id, followed by a colon, and then customer ids, one per line for that movie id. | |
| MovieID1: | |
| CustomerID11 | |
| CustomerID12 | |
| ... | |
| MovieID2: | |
| CustomerID21 | |
| CustomerID22 | |
| Like the qualifying dataset, the movie and customer id pairs are contained in the training set. However, unlike the qualifying dataset, the ratings (and dates) for each pair are contained in the training dataset. | |
| If you wish, you may calculate the RMSE of your predictions against those ratings and compare your RMSE against the Cinematch RMSE on the same data. See http://www.netflixprize.com/faq#probe for that value. | |
| Acknowledgements | |
| The training data came in 17,000+ files. In the interest of keeping files together and file sizes as low as possible, I combined them into four text files: combined_data_(1,2,3,4).txt | |
| The contest was originally hosted at http://netflixprize.com/index.html | |
| The dataset was downloaded from https://archive.org/download/nf_prize_dataset.tar | |
| Inspiration | |
| This is a fun dataset to work with. You can read about the winning algorithm by BellKor's Pragmatic Chaos here" | |
| The Enron Email Dataset,"500,000+ emails from 150 employees of the Enron Corporation",William Cukierski,128,"Version 2,2016-06-17|Version 1,2016-06-16","crime | |
| linguistics",CSV,1 GB,Other,"62,396 views","6,596 downloads",144 kernels,2 topics,https://www.kaggle.com/wcukierski/enron-email-dataset,"The Enron email dataset contains approximately 500,000 emails generated by employees of the Enron Corporation. It was obtained by the Federal Energy Regulatory Commission during its investigation of Enron's collapse. | |
| This is the May 7, 2015 Version of dataset, as published at https://www.cs.cmu.edu/~./enron/" | |
| NBA shot logs,"Moneyball data, for basketball.",DanB,127,"Version 1,2016-08-18","basketball | |
| sports",CSV,16 MB,Other,"45,505 views","5,458 downloads",148 kernels,18 topics,https://www.kaggle.com/dansbecker/nba-shot-logs,"Data on shots taken during the 2014-2015 season, who took the shot, where on the floor was the shot taken from, who was the nearest defender, how far away was the nearest defender, time on the shot clock, and much more. The column titles are generally self-explanatory. | |
| Useful for evaluating who the best shooter is, who the best defender is, the hot-hand hypothesis, etc. | |
| Scraped from NBA's REST API." | |
| Synthetic Financial Datasets For Fraud Detection,Synthetic datasets generated by the PaySim mobile money simulator,TESTIMON @ NTNU,127,"Version 2,2017-04-03|Version 1,2017-03-31","crime | |
| finance",CSV,471 MB,CC4,"45,657 views","5,222 downloads",67 kernels,10 topics,https://www.kaggle.com/ntnu-testimon/paysim1,"Context | |
| There is a lack of public available datasets on financial services and specially in the emerging mobile money transactions domain. Financial datasets are important to many researchers and in particular to us performing research in the domain of fraud detection. Part of the problem is the intrinsically private nature of financial transactions, that leads to no publicly available datasets. | |
| We present a synthetic dataset generated using the simulator called PaySim as an approach to such a problem. PaySim uses aggregated data from the private dataset to generate a synthetic dataset that resembles the normal operation of transactions and injects malicious behaviour to later evaluate the performance of fraud detection methods. | |
| Content | |
| PaySim simulates mobile money transactions based on a sample of real transactions extracted from one month of financial logs from a mobile money service implemented in an African country. The original logs were provided by a multinational company, who is the provider of the mobile financial service which is currently running in more than 14 countries all around the world. | |
| This synthetic dataset is scaled down 1/4 of the original dataset and it is created just for Kaggle. | |
| Headers | |
| This is a sample of 1 row with headers explanation: | |
| 1,PAYMENT,1060.31,C429214117,1089.0,28.69,M1591654462,0.0,0.0,0,0 | |
| step - maps a unit of time in the real world. In this case 1 step is 1 hour of time. Total steps 744 (30 days simulation). | |
| type - CASH-IN, CASH-OUT, DEBIT, PAYMENT and TRANSFER. | |
| amount - amount of the transaction in local currency. | |
| nameOrig - customer who started the transaction | |
| oldbalanceOrg - initial balance before the transaction | |
| newbalanceOrig - new balance after the transaction | |
| nameDest - customer who is the recipient of the transaction | |
| oldbalanceDest - initial balance recipient before the transaction. Note that there is not information for customers that start with M (Merchants). | |
| newbalanceDest - new balance recipient after the transaction. Note that there is not information for customers that start with M (Merchants). | |
| isFraud - This is the transactions made by the fraudulent agents inside the simulation. In this specific dataset the fraudulent behavior of the agents aims to profit by taking control or customers accounts and try to empty the funds by transferring to another account and then cashing out of the system. | |
| isFlaggedFraud - The business model aims to control massive transfers from one account to another and flags illegal attempts. An illegal attempt in this dataset is an attempt to transfer more than 200.000 in a single transaction. | |
| Past Research | |
| There are 5 similar files that contain the run of 5 different scenarios. These files are better explained at my PhD thesis chapter 7 (PhD Thesis Available here http://urn.kb.se/resolve?urn=urn:nbn:se:bth-12932). | |
| We ran PaySim several times using random seeds for 744 steps, representing each hour of one month of real time, which matches the original logs. Each run took around 45 minutes on an i7 intel processor with 16GB of RAM. The final result of a run contains approximately 24 million of financial records divided into the 5 types of categories: CASH-IN, CASH-OUT, DEBIT, PAYMENT and TRANSFER. | |
| Acknowledgements | |
| This work is part of the research project ”Scalable resource-efficient systems for big data analytics” funded by the Knowledge Foundation (grant: 20140032) in Sweden. | |
| Please refer to this dataset using the following citations: | |
| PaySim first paper of the simulator: | |
| E. A. Lopez-Rojas , A. Elmir, and S. Axelsson. ""PaySim: A financial mobile money simulator for fraud detection"". In: The 28th European Modeling and Simulation Symposium-EMSS, Larnaca, Cyprus. 2016" | |
| YouTube Faces With Facial Keypoints,Videos of Celebrity Faces with Facial Keypoints for each Image Frame,Selfish Gene,126,"Version 22,2017-11-14|Version 21,2017-11-13|Version 20,2017-11-13|Version 19,2017-11-12|Version 18,2017-11-12|Version 17,2017-11-11|Version 16,2017-11-10|Version 15,2017-11-10|Version 14,2017-10-27|Version 13,2017-10-27|Version 12,2017-10-27|Version 11,2017-10-27|Version 10,2017-10-23|Version 9,2017-10-23|Version 8,2017-10-23|Version 7,2017-10-15|Version 6,2017-10-15|Version 5,2017-10-15|Version 4,2017-10-14|Version 3,2017-10-13|Version 2,2017-10-13|Version 1,2017-10-13","popular culture | |
| celebrity | |
| humans | |
| internet",Other,10 GB,CC0,"16,902 views","1,133 downloads",3 kernels,,https://www.kaggle.com/selfishgene/youtube-faces-with-facial-keypoints,"YouTube Faces Dataset with Facial Keypoints | |
| This dataset is a processed version of the YouTube Faces Dataset, that basically contained short videos of celebrities that are publicly available and were downloaded from YouTube. There are multiple videos of each celebrity (up to 6 videos per celebrity). I've cropped the original videos around the faces, plus kept only consecutive frames of up to 240 frames for each original video. This is done also for reasons of disk space, but mainly to make the dataset easier to use. | |
| Additionally, for this kaggle version of the dataset I've extracted facial keypoints for each frame of each video using this amazing 2D and 3D Face alignment library that was recently published. please check out this video demonstrating the library. It's performance is really amazing, and I feel I'm quite qualified to say that after manually curating many thousands of individual frames and their corresponding keypoints. I removed all videos with extremely bad keypoints labeling. The end result of my curation process is approximately 2800 videos. Right now only 1293 of those videos are uploaded due to dataset size limitations (10GB), but since overall this totals into 155,560 single image frames, I think this is more than enough to do a lot of interesting kernels as well as potentially very interesting research. | |
| Context | |
| Kaggle datasets platform and its integration with kernels is really amazing, but it's yet to have a videos dataset (at least that I'm aware of). Videos are special in the fact that they contain rich spatial patterns (in this case images of human faces) and rich temporal patterns (in this case how the faces move in time). | |
| I was also inspired by the Face Images with Marked Landmark Points dataset uploaded by DrGuillermo and decided to create and share a dataset that would be similar but would also add something extra. | |
| Acknowledgements | |
| If you use The YouTube Faces Dataset, or refer to its results, please cite the following paper: | |
| Lior Wolf, Tal Hassner and Itay Maoz | |
| Face Recognition in Unconstrained Videos with Matched Background Similarity. | |
| IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2011. (pdf) | |
| if you use the 2D or 3D keypoints, or refer to its results, please cite the following paper: | |
| Adrian Bulat and Georgios Tzimiropoulos. | |
| How far are we from solving the 2D & 3D Face Alignment problem? | |
| (and a dataset of 230,000 3D facial landmarks), arxiv, 2017. (pdf) | |
| Also, I would like to thank Gil Levi for pointing out YouTube Faces to me a few years back. | |
| Inspiration | |
| The YouTube Faces Dataset was originally intended to be used for face recognition across videos, i.e. given two videos, are those videos of the same person or not? | |
| I think it can be used to serve many additional goals, especially when combined with the keypoints information. For example, can we build a face movement model and predict what facial expression will come next? | |
| This dataset can also be used to test transfer learning between other face datasets (like Face Images with Marked Landmark Points that I mentioned earlier), or even other types of faces like cat or dog faces (like here or here). Also, using the pre-trained Keras models might be useful (example kernel). | |
| Have Fun!" | |
| Default of Credit Card Clients Dataset,Default Payments of Credit Card Clients in Taiwan from 2005,UCI Machine Learning,124,"Version 1,2016-11-03",finance,CSV,3 MB,CC0,"74,601 views","7,420 downloads",135 kernels,5 topics,https://www.kaggle.com/uciml/default-of-credit-card-clients-dataset,"Dataset Information | |
| This dataset contains information on default payments, demographic factors, credit data, history of payment, and bill statements of credit card clients in Taiwan from April 2005 to September 2005. | |
| Content | |
| There are 25 variables: | |
| ID: ID of each client | |
| LIMIT_BAL: Amount of given credit in NT dollars (includes individual and family/supplementary credit | |
| SEX: Gender (1=male, 2=female) | |
| EDUCATION: (1=graduate school, 2=university, 3=high school, 4=others, 5=unknown, 6=unknown) | |
| MARRIAGE: Marital status (1=married, 2=single, 3=others) | |
| AGE: Age in years | |
| PAY_0: Repayment status in September, 2005 (-1=pay duly, 1=payment delay for one month, 2=payment delay for two months, ... 8=payment delay for eight months, 9=payment delay for nine months and above) | |
| PAY_2: Repayment status in August, 2005 (scale same as above) | |
| PAY_3: Repayment status in July, 2005 (scale same as above) | |
| PAY_4: Repayment status in June, 2005 (scale same as above) | |
| PAY_5: Repayment status in May, 2005 (scale same as above) | |
| PAY_6: Repayment status in April, 2005 (scale same as above) | |
| BILL_AMT1: Amount of bill statement in September, 2005 (NT dollar) | |
| BILL_AMT2: Amount of bill statement in August, 2005 (NT dollar) | |
| BILL_AMT3: Amount of bill statement in July, 2005 (NT dollar) | |
| BILL_AMT4: Amount of bill statement in June, 2005 (NT dollar) | |
| BILL_AMT5: Amount of bill statement in May, 2005 (NT dollar) | |
| BILL_AMT6: Amount of bill statement in April, 2005 (NT dollar) | |
| PAY_AMT1: Amount of previous payment in September, 2005 (NT dollar) | |
| PAY_AMT2: Amount of previous payment in August, 2005 (NT dollar) | |
| PAY_AMT3: Amount of previous payment in July, 2005 (NT dollar) | |
| PAY_AMT4: Amount of previous payment in June, 2005 (NT dollar) | |
| PAY_AMT5: Amount of previous payment in May, 2005 (NT dollar) | |
| PAY_AMT6: Amount of previous payment in April, 2005 (NT dollar) | |
| default.payment.next.month: Default payment (1=yes, 0=no) | |
| Inspiration | |
| Some ideas for exploration: | |
| How does the probability of default payment vary by categories of different demographic variables? | |
| Which variables are the strongest predictors of default payment? | |
| Acknowledgements | |
| Any publications based on this dataset should acknowledge the following: | |
| Lichman, M. (2013). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science. | |
| The original dataset can be found here at the UCI Machine Learning Repository." | |
| Emergency - 911 Calls,"Montgomery County, PA",Mike Chirico,121,"Version 48,2017-12-30|Version 47,2017-12-30|Version 46,2017-11-04|Version 45,2017-09-21|Version 44,2017-08-19|Version 43,2017-05-29|Version 42,2017-04-14|Version 41,2017-04-10|Version 40,2017-03-16|Version 39,2017-03-09|Version 38,2017-03-09|Version 37,2017-02-25|Version 36,2017-02-18|Version 35,2017-02-10|Version 34,2017-02-09|Version 33,2017-02-09|Version 32,2017-01-28|Version 31,2017-01-18|Version 30,2017-01-09|Version 29,2017-01-01|Version 28,2016-12-30|Version 27,2016-12-24|Version 26,2016-12-18|Version 25,2016-12-18|Version 24,2016-12-18|Version 23,2016-12-14|Version 22,2016-11-30|Version 21,2016-11-20|Version 20,2016-11-13|Version 19,2016-11-05|Version 18,2016-11-03|Version 17,2016-10-30|Version 16,2016-10-29|Version 15,2016-10-26|Version 14,2016-10-17|Version 13,2016-10-13|Version 12,2016-10-08|Version 11,2016-10-07|Version 10,2016-09-30|Version 9,2016-09-25|Version 8,2016-09-21|Version 7,2016-09-17|Version 6,2016-09-09|Version 5,2016-09-07|Version 4,2016-08-31|Version 3,2016-08-24|Version 2,2016-08-18|Version 1,2016-08-18","crime | |
| law",CSV,11 MB,ODbL,"58,170 views","6,687 downloads",309 kernels,3 topics,https://www.kaggle.com/mchirico/montcoalert,"Emergency (911) Calls: Fire, Traffic, EMS for Montgomery County, PA | |
| You can get a quick introduction to this Dataset with this kernel: Dataset Walk-through | |
| Acknowledgements: Data provided by montcoalert.org" | |
| Pima Indians Diabetes Database,Predict the onset of diabetes based on diagnostic measures,UCI Machine Learning,120,"Version 1,2016-10-07",healthcare,CSV,23 KB,CC0,"65,473 views","9,433 downloads",368 kernels,7 topics,https://www.kaggle.com/uciml/pima-indians-diabetes-database,"This dataset is originally from the National Institute of Diabetes and Digestive and Kidney Diseases. The objective is to predict based on diagnostic measurements whether a patient has diabetes. | |
| Dataset information | |
| Several constraints were placed on the selection of these instances from a larger database. In particular, all patients here are females at least 21 years old of Pima Indian heritage. | |
| Relevant papers | |
| Smith, J.W., Everhart, J.E., Dickson, W.C., Knowler, W.C., & Johannes, R.S. (1988). Using the ADAP learning algorithm to forecast the onset of diabetes mellitus. In Proceedings of the Symposium on Computer Applications and Medical Care (pp. 261--265). IEEE Computer Society Press." | |
| Exoplanet Hunting in Deep Space,Kepler labelled time series data,WΔ,116,"Version 3,2017-04-12|Version 2,2017-04-04|Version 1,2017-04-03","astronomy | |
| space",CSV,278 MB,CC0,"25,599 views","2,231 downloads",30 kernels,6 topics,https://www.kaggle.com/keplersmachines/kepler-labelled-time-series-data,"The Search for New Earths | |
| GitHub | |
| The data describe the change in flux (light intensity) of several thousand stars. Each star has a binary label of 2 or 1. 2 indicated that that the star is confirmed to have at least one exoplanet in orbit; some observations are in fact multi-planet systems. | |
| As you can imagine, planets themselves do not emit light, but the stars that they orbit do. If said star is watched over several months or years, there may be a regular 'dimming' of the flux (the light intensity). This is evidence that there may be an orbiting body around the star; such a star could be considered to be a 'candidate' system. Further study of our candidate system, for example by a satellite that captures light at a different wavelength, could solidify the belief that the candidate can in fact be 'confirmed'. | |
| In the above diagram, a star is orbited by a blue planet. At t = 1, the starlight intensity drops because it is partially obscured by the planet, given our position. The starlight rises back to its original value at t = 2. The graph in each box shows the measured flux (light intensity) at each time interval. | |
| Description | |
| Trainset: | |
| 5087 rows or observations. | |
| 3198 columns or features. | |
| Column 1 is the label vector. Columns 2 - 3198 are the flux values over time. | |
| 37 confirmed exoplanet-stars and 5050 non-exoplanet-stars. | |
| Testset: | |
| 570 rows or observations. | |
| 3198 columns or features. | |
| Column 1 is the label vector. Columns 2 - 3198 are the flux values over time. | |
| 5 confirmed exoplanet-stars and 565 non-exoplanet-stars. | |
| Acknowledgements | |
| The data presented here are cleaned and are derived from observations made by the NASA Kepler space telescope. The Mission is ongoing - for instance data from Campaign 12 was released on 8th March 2017. Over 99% of this dataset originates from Campaign 3. To boost the number of exoplanet-stars in the dataset, confirmed exoplanets from other campaigns were also included. | |
| To be clear, all observations from Campaign 3 are included. And in addition to this, confirmed exoplanet-stars from other campaigns are also included. | |
| The datasets were prepared late-summer 2016. | |
| Campaign 3 was used because 'it was felt' that this Campaign is unlikely to contain any undiscovered (i.e. wrongly labelled) exoplanets. | |
| NASA open-sources the original Kepler Mission data and it is hosted at the Mikulski Archive. After being beamed down to Earth, NASA applies de-noising algorithms to remove artefacts generated by the telescope. The data - in the .fits format - is stored online. And with the help of a seasoned astrophysicist, anyone with an internet connection can embark on a search to find and retrieve the datafiles from the Archive. | |
| The cover image is copyright © 2011 by Dan Lessmann" | |
| May 2015 Reddit Comments,Get personal with a dataset of comments from May 2015,Reddit,115,"Version 1,2015-08-05","linguistics | |
| internet",Other,0 B,Other,"46,111 views","4,626 downloads",608 kernels,11 topics,https://www.kaggle.com/reddit/reddit-comments-may-2015,"Recently Reddit released an enormous dataset containing all ~1.7 billion of their publicly available comments. The full dataset is an unwieldy 1+ terabyte uncompressed, so we've decided to host a small portion of the comments here for Kagglers to explore. (You don't even need to leave your browser!) | |
| You can find all the comments from May 2015 on scripts for your natural language processing pleasure. What had redditors laughing, bickering, and NSFW-ing this spring? | |
| Who knows? Top visualizations may just end up on Reddit. | |
| Data Description | |
| The database has one table, May2015, with the following fields: | |
| created_utc | |
| ups | |
| subreddit_id | |
| link_id | |
| name | |
| score_hidden | |
| author_flair_css_class | |
| author_flair_text | |
| subreddit | |
| id | |
| removal_reason | |
| gilded | |
| downs | |
| archived | |
| author | |
| score | |
| retrieved_on | |
| body | |
| distinguished | |
| edited | |
| controversiality | |
| parent_id" | |
| Bitcoin Blockchain,Complete live historical Bitcoin blockchain data,Google BigQuery,115,"Version 1,2018-02-01","finance | |
| money | |
| internet | |
| bigquery",BigQuery,821 GB,CC0,"21,005 views",0 downloads,405 kernels,4 topics,https://www.kaggle.com/bigquery/bitcoin-blockchain,"Context | |
| Blockchain technology, first implemented by Satoshi Nakamoto in 2009 as a core component of Bitcoin, is a distributed, public ledger recording transactions. Its usage allows secure peer-to-peer communication by linking blocks containing hash pointers to a previous block, a timestamp, and transaction data. Bitcoin is a decentralized digital currency (cryptocurrency) which leverages the Blockchain to store transactions in a distributed manner in order to mitigate against flaws in the financial industry. | |
| Nearly ten years after its inception, Bitcoin and other cryptocurrencies experienced an explosion in popular awareness. The value of Bitcoin, on the other hand, has experienced more volatility. Meanwhile, as use cases of Bitcoin and Blockchain grow, mature, and expand, hype and controversy have swirled. | |
| Content | |
| In this dataset, you will have access to information about blockchain blocks and transactions. All historical data are in the bigquery-public-data:bitcoin_blockchain dataset. It’s updated it every 10 minutes. The data can be joined with historical prices in kernels. See available similar datasets here: https://www.kaggle.com/datasets?search=bitcoin. | |
| Querying BigQuery tables | |
| You can use the BigQuery Python client library to query tables in this dataset in Kernels. Note that methods available in Kernels are limited to querying data. Tables are at bigquery-public-data.bitcoin_blockchain.[TABLENAME]. Fork this kernel to get started. | |
| Method & Acknowledgements | |
| Allen Day (Twitter | Medium), Google Cloud Developer Advocate & Colin Bookman, Google Cloud Customer Engineer retrieve data from the Bitcoin network using a custom client available on GitHub that they built with the bitcoinj Java library. Historical data from the origin block to 2018-01-31 were loaded in bulk to two BigQuery tables, blocks_raw and transactions. These tables contain fresh data, as they are now appended when new blocks are broadcast to the Bitcoin network. For additional information visit the Google Cloud Big Data and Machine Learning Blog post ""Bitcoin in BigQuery: Blockchain analytics on public data"". | |
| Photo by Andre Francois on Unsplash. | |
| Inspiration | |
| How many bitcoins are sent each day? | |
| How many addresses receive bitcoin each day? | |
| Compare transaction volume to historical prices by joining with other available data sources" | |
| "Stack Overflow Developer Survey, 2017","A look into the lives of over 64,000 Stack Overflow developers",Stack Overflow,113,"Version 2,2017-06-15|Version 1,2017-06-15","information technology | |
| internet",CSV,89 MB,ODbL,"23,607 views","2,603 downloads",121 kernels,,https://www.kaggle.com/stackoverflow/so-survey-2017,"Every year, Stack Overflow conducts a massive survey of people on the site, covering all sorts of information like programming languages, salary, code style and various other information. This year, they amassed more than 64,000 responses fielded from 213 countries. | |
| Data | |
| The data is made up of two files: | |
| 1. survey_results_public.csv - CSV file with main survey results, one respondent per row and one column per answer | |
| 2. survey_results_schema.csv - CSV file with survey schema, i.e., the questions that correspond to each column name m | |
| Acknowledgements | |
| Data is directly taken from StackOverflow and licensed under the ODbL license." | |
| US Dept of Education: College Scorecard,Raise the curtain on the true cost of higher education,Kaggle,112,"Version 3,2017-11-10|Version 2,2017-05-02|Version 1,2015-09-25","education | |
| finance",CSV,4 GB,CC0,"56,642 views","9,242 downloads",191 kernels,8 topics,https://www.kaggle.com/kaggle/college-scorecard,"It's no secret that US university students often graduate with debt repayment obligations that far outstrip their employment and income prospects. While it's understood that students from elite colleges tend to earn more than graduates from less prestigious universities, the finer relationships between future income and university attendance are quite murky. In an effort to make educational investments less speculative, the US Department of Education has matched information from the student financial aid system with federal tax returns to create the College Scorecard dataset. | |
| Kaggle is hosting the College Scorecard dataset in order to facilitate shared learning and collaboration. Insights from this dataset can help make the returns on higher education more transparent and, in turn, more fair. | |
| Data Description | |
| Here's a script showing an exploratory overview of some of the data. | |
| college-scorecard-release-*.zip contains a compressed version of the same data available through Kaggle Scripts. | |
| It consists of three components: | |
| All the raw data files released in version 1.40 of the college scorecard data | |
| Scorecard.csv, a single CSV file with all the years data combined. In it, we've converted categorical variables represented by integer keys in the original data to their labels and added a Year column | |
| database.sqlite, a SQLite database containing a single Scorecard table that contains the same information as Scorecard.csv | |
| New to data exploration in R? Take the free, interactive DataCamp course, ""Data Exploration With Kaggle Scripts,"" to learn the basics of visualizing data with ggplot. You'll also create your first Kaggle Scripts along the way." | |
| Anime Recommendations Database,"Recommendation data from 76,000 users at myanimelist.net",CooperUnion,111,"Version 1,2016-12-21","popular culture | |
| film",CSV,107 MB,CC0,"28,899 views","3,440 downloads",96 kernels,12 topics,https://www.kaggle.com/CooperUnion/anime-recommendations-database,"Context | |
| This data set contains information on user preference data from 73,516 users on 12,294 anime. Each user is able to add anime to their completed list and give it a rating and this data set is a compilation of those ratings. | |
| Content | |
| Anime.csv | |
| anime_id - myanimelist.net's unique id identifying an anime. | |
| name - full name of anime. | |
| genre - comma separated list of genres for this anime. | |
| type - movie, TV, OVA, etc. | |
| episodes - how many episodes in this show. (1 if movie). | |
| rating - average rating out of 10 for this anime. | |
| members - number of community members that are in this anime's ""group"". | |
| Rating.csv | |
| user_id - non identifiable randomly generated user id. | |
| anime_id - the anime that this user has rated. | |
| rating - rating out of 10 this user has assigned (-1 if the user watched it but didn't assign a rating). | |
| Acknowledgements | |
| Thanks to myanimelist.net API for providing anime data and user ratings. | |
| Inspiration | |
| Building a better anime recommendation system based only on user viewing history." | |
| Board Games Dataset,What makes a game a good game?,Gabriele Baldassarre,110,"Version 5,2017-06-25|Version 4,2017-06-25|Version 3,2017-06-25|Version 2,2017-05-04|Version 1,2017-05-04",board games,SQLite,140 MB,Other,"19,357 views","2,575 downloads",20 kernels,,https://www.kaggle.com/gabrio/board-games-dataset,"About | |
| A dataset containing the attributes and the ratings for around 94,000 among board games and expansions as get from BoardGameGeek. | |
| Resources | |
| The same data and the scripts used to crawl it from BoardGameGeek are available in the form of R package on Github." | |
| EEG brain wave for confusion,For variable selection and causal inference. Challenging for classification,Haohan Wang,110,"Version 3,2016-12-01|Version 2,2016-08-28|Version 1,2016-08-19","healthcare | |
| neuroscience",CSV,115 MB,CC0,"29,814 views","2,454 downloads",81 kernels,6 topics,https://www.kaggle.com/wanghaohan/eeg-brain-wave-for-confusion,"Description | |
| We collected EEG signal data from 10 college students while they watched MOOC video clips. We extracted online education videos that are assumed not to be confusing for college students, such as videos of the introduction of basic algebra or geometry. We also prepare videos that are expected to confuse a typical college student if a student is not familiar with the video topics like Quantum Mechanics, and Stem Cell Research. We prepared 20 videos, 10 in each category. Each video was about 2 minutes long. We chopped the two-minute clip in the middle of a topic to make the videos more confusing. The students wore a single-channel wireless MindSet that measured activity over the frontal lobe. The MindSet measures the voltage between an electrode resting on the forehead and two electrodes (one ground and one reference) each in contact with an ear. After each session, the student rated his/her confusion level on a scale of 1-7, where one corresponded to the least confusing and seven corresponded to the most confusing. These labels if further normalized into labels of whether the students are confused or not. This label is offered as self-labelled confusion in addition to our predefined label of confusion. | |
| Data information: | |
| -----data.csv | |
| Column 1: Subject ID | |
| Column 2: Video ID | |
| Column 3: Attention (Proprietary measure of mental focus) | |
| Column 4: Mediation (Proprietary measure of calmness) | |
| Column 5: Raw (Raw EEG signal) | |
| Column 6: Delta (1-3 Hz of power spectrum) | |
| Column 7: Theta (4-7 Hz of power spectrum) | |
| Column 8: Alpha 1 (Lower 8-11 Hz of power spectrum) | |
| Column 9: Alpha 2 (Higher 8-11 Hz of power spectrum) | |
| Column 10: Beta 1 (Lower 12-29 Hz of power spectrum) | |
| Column 11: Beta 2 (Higher 12-29 Hz of power spectrum) | |
| Column 12: Gamma 1 (Lower 30-100 Hz of power spectrum) | |
| Column 13: Gamma 2 (Higher 30-100 Hz of power spectrum) | |
| Column 14: predefined label (whether the subject is expected to be confused) | |
| Column 15: user-defined label (whether the subject is actually confused) | |
| -----subject demographic | |
| Column 1: Subject ID | |
| Column 2: Age | |
| Column 3: Ethnicity (Categorized according to https://en.wikipedia.org/wiki/List_of_contemporary_ethnic_groups) | |
| Column 4: Gender | |
| -----video data | |
| Each video lasts roughly two-minute long, we remove the first 30 seconds and last 30 seconds, only collect the EEG data during the middle 1 minute. | |
| Format | |
| These data are collected from ten students, each watching ten videos. | |
| Therefore, it can be seen as only 100 data points for these 12000+ rows. If you look at this way, then each data point consists of 120+ rows, which is sampled every 0.5 seconds (so each data point is a one minute video). Signals with higher frequency are reported as the mean value during each 0.5 second. | |
| Reference: | |
| Wang, H., Li, Y., Hu, X., Yang, Y., Meng, Z., & Chang, K. M. (2013, June). Using EEG to Improve Massive Open Online Courses Feedback Interaction. In AIED Workshops. [PDF] | |
| Data Collection | |
| The data is collected from a software that we implemented ourselves. Check HaohanWang/Bioimaging for the source code. | |
| Inspiration | |
| This dataset is an extremely challenging data set to perform binary classification. 65% of prediction accuracy is quite decent according to our experience. | |
| It is an interesting data set to carry out the variable selection (causal inference) task that may help further research. Past research has indicated that Theta signal is correlated with confusion level. | |
| It is also an interesting data set for confounding factors correction model because we offer two labels (subject id and video id) that could profoundly confound the results. | |
| Warning | |
| The data for subject 3 might be corrupted. | |
| Other Resources | |
| Promotion Video | |
| Source Code of Data Collection Software | |
| Contact | |
| Haohan Wang" | |
| The History of Baseball,A complete history of major league baseball stats from 1871 to 2015,SeanLahman,109,"Version 2,2016-09-08|Version 1,2016-03-09","baseball | |
| history",CSV,66 MB,CC3,"27,811 views","4,751 downloads",129 kernels,2 topics,https://www.kaggle.com/seanlahman/the-history-of-baseball,"Baffled why your team traded for that 34-year-old pitcher? Convinced you can create a new and improved version of WAR? Wondering what made the 1907 Cubs great and if can they do it again? | |
| The History of Baseball is a reformatted version of the famous Lahman’s Baseball Database. It contains Major League Baseball’s complete batting and pitching statistics from 1871 to 2015, plus fielding statistics, standings, team stats, park stats, player demographics, managerial records, awards, post-season data, and more. | |
| Scripts, Kaggle’s free, in-browser analytics tool, makes it easy to share detailed sabermetrics, predict the next hall of fame inductee, illustrate how speed scores runs, or publish a definitive analysis on why the Los Angeles Dodgers will never win another World Series. | |
| We have more ideas for analysis than games in a season, but here are a few we’d really love to see: | |
| Is there a most error-prone position? | |
| When do players at different positions peak? | |
| Are the best performers selected for all-star game? | |
| How many walks does it take for a starting pitcher to get pulled? | |
| Do players with a high ground into double play (GIDP) have a lower batting average? | |
| Which players are the most likely to choke during the post-season? | |
| Why should or shouldn’t the National League adopt the designated hitter rule? | |
| See the full SQLite schema." | |
| 2016 Parties in New York,225k noise complaints to the police about ongoing parties in the city,Evgenii Vasilev,108,"Version 4,2017-08-23|Version 3,2017-08-23|Version 2,2017-07-29|Version 1,2017-07-28","parties | |
| cities",CSV,53 MB,CC0,"12,035 views","1,834 downloads",19 kernels,0 topics,https://www.kaggle.com/somesnm/partynyc,"Context | |
| This dataset contains all noise complaints calls that were received by the city police with complaint type ""Loud music/Party"" in 2016. The data contains the time of the call, time of the police response, coordinates and part of the city. | |
| This data should help match taxi rides from ""New York City Taxi Trip Duration"" competition to the night rides of partygoers. | |
| Content | |
| The New York city hotline receives non-urgent community concerns, which are made public by the city through NYC Open Data portal. The full dataset contains a variety of complaints ranging from illegal parking to customer complaints. This dataset focuses on Noise complaints that were collected in 2016 and indicate ongoing party in a given neighborhood. | |
| parties_in_nyc.csv: | |
| Columns: | |
| Created Date - time of the call | |
| Closed Date - time when ticket was closed by police | |
| Location Type - type of the location | |
| Incident Zip - zip code of the location | |
| City - name of the city (almost the same as the Borough field) | |
| Borough - administrative division of the city | |
| Latitude - latitude of the location | |
| Longitude - longitude of the location | |
| test_parties and train_parties: | |
| Columns: | |
| id - id of the ride | |
| num_complaints - number of noise complaints about ongoing parties within ~500 meters and within 2 hours of pickup place and time | |
| Acknowledgements | |
| https://opendata.cityofnewyork.us/ - NYC Open Data portal contains many other interesting datasets Photo by Yvette de Wit on Unsplash | |
| Inspiration | |
| After a fun night out in the city majority of people are too exhausted to travel by public transport, so they catch a cab to their home. I hope this data will help the community to find the patterns in the data that will lead to better solutions." | |
| Breast Cancer Proteomes,Dividing breast cancer patients into separate sub-classes,kajot,107,"Version 3,2016-07-03|Version 2,2016-07-03|Version 1,2016-07-03",healthcare,CSV,12 MB,Other,"38,640 views","4,430 downloads",76 kernels,7 topics,https://www.kaggle.com/piotrgrabo/breastcancerproteomes,"Context: This data set contains published iTRAQ proteome profiling of 77 breast cancer samples generated by the Clinical Proteomic Tumor Analysis Consortium (NCI/NIH). It contains expression values for ~12.000 proteins for each sample, with missing values present when a given protein could not be quantified in a given sample. | |
| Content: | |
| File: 77_cancer_proteomes_CPTAC_itraq.csv | |
| RefSeq_accession_number: RefSeq protein ID (each protein has a unique ID in a RefSeq database) | |
| gene_symbol: a symbol unique to each gene (every protein is encoded by some gene) | |
| gene_name: a full name of that gene Remaining columns: log2 iTRAQ ratios for each sample (protein expression data, most important), three last columns are from healthy individuals | |
| File: clinical_data_breast_cancer.csv | |
| First column ""Complete TCGA ID"" is used to match the sample IDs in the main cancer proteomes file (see example script). All other columns have self-explanatory names, contain data about the cancer classification of a given sample using different methods. 'PAM50 mRNA' classification is being used in the example script. | |
| File: PAM50_proteins.csv | |
| Contains the list of genes and proteins used by the PAM50 classification system. The column RefSeqProteinID contains the protein IDs that can be matched with the IDs in the main protein expression data set. | |
| Past Research: The original study: http://www.nature.com/nature/journal/v534/n7605/full/nature18003.html (paywall warning) | |
| In brief: the data were used to assess how the mutations in the DNA are affecting the protein expression landscape in breast cancer. Genes in our DNA are first transcribed into RNA molecules which then are translated into proteins. Changing the information content of DNA has impact on the behavior of the proteome, which is the main functional unit of cells, taking care of cell division, DNA repair, enzymatic reactions and signaling etc. They performed K-means clustering on the protein data to divide the breast cancer patients into sub-types, each having unique protein expression signature. They found that the best clustering was achieved using 3 clusters (original PAM50 gene set yields four different subtypes using RNA data). | |
| Inspiration: | |
| This is an interesting study and I myself wanted to use this breast cancer proteome data set for other types of analyses using machine learning that I am performing as a part of my PhD. However, I though that the Kaggle community (or at least that part with biomedical interests) would enjoy playing with it. I added a simple K-means clustering example for that data with some comments, the same approach as used in the original paper. One thing is that there is a panel of genes, the PAM50 which is used to classify breast cancers into subtypes. This panel was originally based on the RNA expression data which is (in my opinion) not as robust as the measurement of mRNA's final product, the protein. Perhaps using this data set, someone could find a different set of proteins (they all have unique NP_/XP_ identifiers) that would divide the data set even more robustly? Perhaps into a higher numbers of clusters with very distinct protein expression signatures? | |
| Example K-means analysis script: http://pastebin.com/A0Wj41DP" | |
| 55000+ Song Lyrics,Lyrics for 55000+ songs in English from LyricsFreak,Sergey Kuznetsov,107,"Version 1,2017-01-05","languages | |
| music | |
| linguistics",CSV,69 MB,CC0,"22,801 views","2,901 downloads",55 kernels,0 topics,https://www.kaggle.com/mousehead/songlyrics,"Context | |
| These are the lyrics for 57650 songs. They can be used for Natural Language Processing purposes, such as clustering of the words with similar meanings or predicting artist by the song. The dataset can be expanded with some more features for more advanced research like sentiment analysis. The data is not modified, only slightly cleaned, which gives a lot of freedom to devise your own applications. | |
| Mining | |
| I have mined this dataset as a corpus for my NLP studies. However, before performing any transformation to bag-of-words or bag-of-N-grams, I decided to share the data. The data has been acquired from LyricsFreak through scraping. Then I did some very basic work on removing inconvenient data: non-English lyrics, extremely short and extremely long lyrics, lyrics with non-ASCII symbols. However, there's still work to be done in terms of data preparation. | |
| Content | |
| The dataset contains 4 columns: | |
| Artist | |
| Song Name | |
| Link to a webpage with the song (for reference). This is to be concatenated with http://www.lyricsfreak.com to form a real URL. | |
| Lyrics of the song, unmodified. | |
| Acknowledgements | |
| I would like to acknowledge LyricsFreak, which is the direct source of the data." | |
| Kickstarter projects,"More than 300,000 kickstarter projects",Kemical,107,"Version 7,2018-02-08|Version 6,2018-01-26|Version 5,2018-01-03|Version 4,2018-01-02|Version 3,2018-01-02|Version 2,2017-11-27|Version 1,2017-11-06","finance | |
| crowdfunding",CSV,37 MB,CC4,"13,492 views","2,525 downloads",10 kernels,5 topics,https://www.kaggle.com/kemical/kickstarter-projects,"Context | |
| I'm a crowdfunding enthusiast and i'm watching kickstarter since its early days. Right now I just collect data and the only app i've made is this twitter bot which tweet any project reaching some milestone: @bloomwatcher . I have a lot of other ideas, but sadly not enough time to develop them... But I hope you can! | |
| Content | |
| You'll find most useful data for project analysis. Columns are self explanatory except: | |
| usd_pledged: conversion in US dollars of the pledged column (conversion done by kickstarter). | |
| usd pledge real: conversion in US dollars of the pledged column (conversion from Fixer.io API). | |
| usd goal real: conversion in US dollars of the goal column (conversion from Fixer.io API). | |
| Acknowledgements | |
| Data are collected from Kickstarter Platform | |
| usd conversion (usd_pledged_real and usd_goal_real columns) were generated from convert ks pledges to usd script done by tonyplaysguitar | |
| Inspiration | |
| I hope to see great projects, and why not a model to predict if a project will be successful before it is released? :)" | |
| Students' Academic Performance Dataset,xAPI-Educational Mining Dataset,IbrahimAljarah,107,"Version 6,2016-11-27|Version 5,2016-11-26|Version 4,2016-11-26|Version 3,2016-11-26|Version 2,2016-11-24|Version 1,2016-11-24",education,CSV,37 KB,CC4,"74,553 views","8,845 downloads",159 kernels,5 topics,https://www.kaggle.com/aljarah/xAPI-Edu-Data,"Students' Academic Performance Dataset (xAPI-Edu-Data) | |
| Data Set Characteristics: Multivariate | |
| Number of Instances: 480 | |
| Area: E-learning, Education, Predictive models, Educational Data Mining | |
| Attribute Characteristics: Integer/Categorical | |
| Number of Attributes: 16 | |
| Date: 2016-11-8 | |
| Associated Tasks: Classification | |
| Missing Values? No | |
| File formats: xAPI-Edu-Data.csv | |
| Source: | |
| Elaf Abu Amrieh, Thair Hamtini, and Ibrahim Aljarah, The University of Jordan, Amman, Jordan, http://www.Ibrahimaljarah.com www.ju.edu.jo | |
| Dataset Information: | |
| This is an educational data set which is collected from learning management system (LMS) called Kalboard 360. Kalboard 360 is a multi-agent LMS, which has been designed to facilitate learning through the use of leading-edge technology. Such system provides users with a synchronous access to educational resources from any device with Internet connection. | |
| The data is collected using a learner activity tracker tool, which called experience API (xAPI). The xAPI is a component of the training and learning architecture (TLA) that enables to monitor learning progress and learner’s actions like reading an article or watching a training video. The experience API helps the learning activity providers to determine the learner, activity and objects that describe a learning experience. The dataset consists of 480 student records and 16 features. The features are classified into three major categories: (1) Demographic features such as gender and nationality. (2) Academic background features such as educational stage, grade Level and section. (3) Behavioral features such as raised hand on class, opening resources, answering survey by parents, and school satisfaction. | |
| The dataset consists of 305 males and 175 females. The students come from different origins such as 179 students are from Kuwait, 172 students are from Jordan, 28 students from Palestine, 22 students are from Iraq, 17 students from Lebanon, 12 students from Tunis, 11 students from Saudi Arabia, 9 students from Egypt, 7 students from Syria, 6 students from USA, Iran and Libya, 4 students from Morocco and one student from Venezuela. | |
| The dataset is collected through two educational semesters: 245 student records are collected during the first semester and 235 student records are collected during the second semester. | |
| The data set includes also the school attendance feature such as the students are classified into two categories based on their absence days: 191 students exceed 7 absence days and 289 students their absence days under 7. | |
| This dataset includes also a new category of features; this feature is parent parturition in the educational process. Parent participation feature have two sub features: Parent Answering Survey and Parent School Satisfaction. There are 270 of the parents answered survey and 210 are not, 292 of the parents are satisfied from the school and 188 are not. | |
| (See the related papers for more details). | |
| Attributes | |
| 1 Gender - student's gender (nominal: 'Male' or 'Female’) | |
| 2 Nationality- student's nationality (nominal:’ Kuwait’,’ Lebanon’,’ Egypt’,’ SaudiArabia’,’ USA’,’ Jordan’,’ Venezuela’,’ Iran’,’ Tunis’,’ Morocco’,’ Syria’,’ Palestine’,’ Iraq’,’ Lybia’) | |
| 3 Place of birth- student's Place of birth (nominal:’ Kuwait’,’ Lebanon’,’ Egypt’,’ SaudiArabia’,’ USA’,’ Jordan’,’ Venezuela’,’ Iran’,’ Tunis’,’ Morocco’,’ Syria’,’ Palestine’,’ Iraq’,’ Lybia’) | |
| 4 Educational Stages- educational level student belongs (nominal: ‘lowerlevel’,’MiddleSchool’,’HighSchool’) | |
| 5 Grade Levels- grade student belongs (nominal: ‘G-01’, ‘G-02’, ‘G-03’, ‘G-04’, ‘G-05’, ‘G-06’, ‘G-07’, ‘G-08’, ‘G-09’, ‘G-10’, ‘G-11’, ‘G-12 ‘) | |
| 6 Section ID- classroom student belongs (nominal:’A’,’B’,’C’) | |
| 7 Topic- course topic (nominal:’ English’,’ Spanish’, ‘French’,’ Arabic’,’ IT’,’ Math’,’ Chemistry’, ‘Biology’, ‘Science’,’ History’,’ Quran’,’ Geology’) | |
| 8 Semester- school year semester (nominal:’ First’,’ Second’) | |
| 9 Parent responsible for student (nominal:’mom’,’father’) | |
| 10 Raised hand- how many times the student raises his/her hand on classroom (numeric:0-100) | |
| 11- Visited resources- how many times the student visits a course content(numeric:0-100) | |
| 12 Viewing announcements-how many times the student checks the new announcements(numeric:0-100) | |
| 13 Discussion groups- how many times the student participate on discussion groups (numeric:0-100) | |
| 14 Parent Answering Survey- parent answered the surveys which are provided from school or not (nominal:’Yes’,’No’) | |
| 15 Parent School Satisfaction- the Degree of parent satisfaction from school(nominal:’Yes’,’No’) | |
| 16 Student Absence Days-the number of absence days for each student (nominal: above-7, under-7) | |
| The students are classified into three numerical intervals based on their total grade/mark: | |
| Low-Level: interval includes values from 0 to 69, | |
| Middle-Level: interval includes values from 70 to 89, | |
| High-Level: interval includes values from 90-100. | |
| Relevant Papers: | |
| Amrieh, E. A., Hamtini, T., & Aljarah, I. (2016). Mining Educational Data to Predict Student’s academic Performance using Ensemble Methods. International Journal of Database Theory and Application, 9(8), 119-136. | |
| Amrieh, E. A., Hamtini, T., & Aljarah, I. (2015, November). Preprocessing and analyzing educational data set using X-API for improving student's performance. In Applied Electrical Engineering and Computing Technologies (AEECT), 2015 IEEE Jordan Conference on (pp. 1-5). IEEE. | |
| Citation Request: | |
| Please include these citations if you plan to use this dataset: | |
| Amrieh, E. A., Hamtini, T., & Aljarah, I. (2016). Mining Educational Data to Predict Student’s academic Performance using Ensemble Methods. International Journal of Database Theory and Application, 9(8), 119-136. | |
| Amrieh, E. A., Hamtini, T., & Aljarah, I. (2015, November). Preprocessing and analyzing educational data set using X-API for improving student's performance. In Applied Electrical Engineering and Computing Technologies (AEECT), 2015 IEEE Jordan Conference on (pp. 1-5). IEEE." | |
| S&P 500 stock data,Historical stock data for all current S&P 500 companies,Cam Nugent,105,"Version 4,2018-02-10|Version 3,2018-02-10|Version 2,2017-08-13|Version 1,2017-08-11",finance,Other,19 MB,CC0,"28,224 views","3,663 downloads",13 kernels,4 topics,https://www.kaggle.com/camnugent/sandp500,"Context | |
| Stock market data can be interesting to analyze and as a further incentive, strong predictive models can have large financial payoff. The amount of financial data on the web is seemingly endless. A large and well structured dataset on a wide array of companies can be hard to come by. Here I provide a dataset with historical stock prices (last 5 years) for all companies currently found on the S&P 500 index. | |
| The script I used to acquire all of these .csv files can be found in this GitHub repository In the future if you wish for a more up to date dataset, this can be used to acquire new versions of the .csv files. | |
| Feb 2018 note: I have just updated the dataset to include data up to Feb 2018. I have also accounted for changes in the stocks on the S&P 500 index (RIP whole foods etc. etc.). | |
| Content | |
| The data is presented in a couple of formats to suit different individual's needs or computational limitations. I have included files containing 5 years of stock data (in the all_stocks_5yr.csv and corresponding folder). | |
| The folder individual_stocks_5yr contains files of data for individual stocks, labelled by their stock ticker name. The all_stocks_5yr.csv contains the same data, presented in a merged .csv file. Depending on the intended use (graphing, modelling etc.) the user may prefer one of these given formats. | |
| All the files have the following columns: Date - in format: yy-mm-dd | |
| Open - price of the stock at market open (this is NYSE data so all in USD) | |
| High - Highest price reached in the day | |
| Low Close - Lowest price reached in the day | |
| Volume - Number of shares traded | |
| Name - the stock's ticker name | |
| Acknowledgements | |
| Due to volatility in google finance, for the newest version I have switched over to acquiring the data from The Investor's Exchange api, the simple script I use to do this is found here. Special thanks to Kaggle, Github, pandas_datareader and The Market. | |
| Inspiration | |
| This dataset lends itself to a some very interesting visualizations. One can look at simple things like how prices change over time, graph an compare multiple stocks at once, or generate and graph new metrics from the data provided. From these data informative stock stats such as volatility and moving averages can be easily calculated. The million dollar question is: can you develop a model that can beat the market and allow you to make statistically informed trades!" | |
| Cervical Cancer Risk Classification,prediction of cancer indicators; Please download; run kernel & upvote,Gokagglers,103,"Version 6,2017-08-31|Version 5,2017-08-30|Version 4,2017-08-30|Version 3,2017-08-24|Version 2,2017-08-24|Version 1,2017-08-18","healthcare | |
| human genetics",CSV,100 KB,Other,"21,513 views","4,466 downloads",23 kernels,18 topics,https://www.kaggle.com/loveall/cervical-cancer-risk-classification,"Cervical Cancer Risk Factors for Biopsy: This Dataset is Obtained from UCI Repository and kindly acknowledged! | |
| This file contains a List of Risk Factors for Cervical Cancer leading to a Biopsy Examination! | |
| About 11,000 new cases of invasive cervical cancer are diagnosed each year in the U.S. However, the number of new cervical cancer cases has been declining steadily over the past decades. Although it is the most preventable type of cancer, each year cervical cancer kills about 4,000 women in the U.S. and about 300,000 women worldwide. In the United States, cervical cancer mortality rates plunged by 74% from 1955 - 1992 thanks to increased screening and early detection with the Pap test. AGE Fifty percent of cervical cancer diagnoses occur in women ages 35 - 54, and about 20% occur in women over 65 years of age. The median age of diagnosis is 48 years. About 15% of women develop cervical cancer between the ages of 20 - 30. Cervical cancer is extremely rare in women younger than age 20. However, many young women become infected with multiple types of human papilloma virus, which then can increase their risk of getting cervical cancer in the future. Young women with early abnormal changes who do not have regular examinations are at high risk for localized cancer by the time they are age 40, and for invasive cancer by age 50. SOCIOECONOMIC AND ETHNIC FACTORS Although the rate of cervical cancer has declined among both Caucasian and African-American women over the past decades, it remains much more prevalent in African-Americans -- whose death rates are twice as high as Caucasian women. Hispanic American women have more than twice the risk of invasive cervical cancer as Caucasian women, also due to a lower rate of screening. These differences, however, are almost certainly due to social and economic differences. Numerous studies report that high poverty levels are linked with low screening rates. In addition, lack of health insurance, limited transportation, and language difficulties hinder a poor woman’s access to screening services. HIGH SEXUAL ACTIVITY Human papilloma virus (HPV) is the main risk factor for cervical cancer. In adults, the most important risk factor for HPV is sexual activity with an infected person. Women most at risk for cervical cancer are those with a history of multiple sexual partners, sexual intercourse at age 17 years or younger, or both. A woman who has never been sexually active has a very low risk for developing cervical cancer. Sexual activity with multiple partners increases the likelihood of many other sexually transmitted infections (chlamydia, gonorrhea, syphilis).Studies have found an association between chlamydia and cervical cancer risk, including the possibility that chlamydia may prolong HPV infection. FAMILY HISTORY Women have a higher risk of cervical cancer if they have a first-degree relative (mother, sister) who has had cervical cancer. USE OF ORAL CONTRACEPTIVES Studies have reported a strong association between cervical cancer and long-term use of oral contraception (OC). Women who take birth control pills for more than 5 - 10 years appear to have a much higher risk HPV infection (up to four times higher) than those who do not use OCs. (Women taking OCs for fewer than 5 years do not have a significantly higher risk.) The reasons for this risk from OC use are not entirely clear. Women who use OCs may be less likely to use a diaphragm, condoms, or other methods that offer some protection against sexual transmitted diseases, including HPV. Some research also suggests that the hormones in OCs might help the virus enter the genetic material of cervical cells. HAVING MANY CHILDREN Studies indicate that having many children increases the risk for developing cervical cancer, particularly in women infected with HPV. SMOKING Smoking is associated with a higher risk for precancerous changes (dysplasia) in the cervix and for progression to invasive cervical cancer, especially for women infected with HPV. IMMUNOSUPPRESSION Women with weak immune systems, (such as those with HIV / AIDS), are more susceptible to acquiring HPV. Immunocompromised patients are also at higher risk for having cervical precancer develop rapidly into invasive cancer. DIETHYLSTILBESTROL (DES) From 1938 - 1971, diethylstilbestrol (DES), an estrogen-related drug, was widely prescribed to pregnant women to help prevent miscarriages. The daughters of these women face a higher risk for cervical cancer. DES is no longer prsecribed." | |
| US Stocks Fundamentals (XBRL),"Fundamental data for 12,129 companies based on XBRL",usfundamentals,102,"Version 3,2016-09-07|Version 2,2016-09-04|Version 1,2016-08-27",finance,CSV,145 MB,CC0,"23,982 views","3,659 downloads",74 kernels,3 topics,https://www.kaggle.com/usfundamentals/us-stocks-fundamentals,"This dataset contains US stocks fundamental data, such as income statement, balance sheet and cash flows. | |
| 12,129 companies | |
| 8,526 unique indicators | |
| ~20 indicators comparable across most companies | |
| Five years of data, yearly | |
| The data is provided by http://usfundamentals.com." | |
| Nutrition Facts for McDonald's Menu,"Calories, fat, and sugar for every cheeseburger, fries, and milkshake on menu",McDonald's,102,"Version 1,2017-03-03","food and drink | |
| health",CSV,29 KB,Other,"53,015 views","8,441 downloads",188 kernels,2 topics,https://www.kaggle.com/mcdonalds/nutrition-facts,"Context | |
| Ray Kroc wanted to build a restaurant system that would be famous for providing food of consistently high quality and uniform methods of preparation. He wanted to serve burgers, buns, fries and beverages that tasted just the same in Alaska as they did in Alabama. To achieve this, he chose a unique path: persuading both franchisees and suppliers to buy into his vision, working not for McDonald’s but for themselves, together with McDonald’s. Many of McDonald’s most famous menu items – like the Big Mac, Filet-O-Fish, and Egg McMuffin – were created by franchisees. | |
| Content | |
| This dataset provides a nutrition analysis of every menu item on the US McDonald's menu, including breakfast, beef burgers, chicken and fish sandwiches, fries, salads, soda, coffee and tea, milkshakes, and desserts. | |
| Acknowledgements | |
| The menu items and nutrition facts were scraped from the McDonald's website. | |
| Inspiration | |
| How many calories does the average McDonald's value meal contain? How much do beverages, like soda or coffee, contribute to the overall caloric intake? Does ordered grilled chicken instead of crispy increase a sandwich's nutritional value? What about ordering egg whites instead of whole eggs? What is the least number of items could you order from the menu to meet one day's nutritional requirements? | |
| Start a new kernel" | |
| Philadelphia Crime Data,"Ten Years of Crime Data, by OpenDataPhilly",Mike Chirico,101,"Version 19,2017-03-24|Version 18,2017-03-15|Version 17,2017-02-26|Version 16,2017-02-09|Version 15,2017-01-28|Version 14,2017-01-03|Version 13,2016-12-24|Version 12,2016-12-12|Version 11,2016-12-01|Version 10,2016-11-29|Version 9,2016-11-20|Version 8,2016-11-13|Version 7,2016-11-07|Version 6,2016-10-26|Version 5,2016-10-13|Version 4,2016-09-26|Version 3,2016-09-15|Version 2,2016-09-10|Version 1,2016-08-19",crime,CSV,296 MB,Other,"42,461 views","5,158 downloads",201 kernels,9 topics,https://www.kaggle.com/mchirico/philadelphiacrimedata,"Crime Data for Philadelphia | |
| To get started quickly, take a look at Philly Data Crime Walk-through. | |
| Data was provided by OpenDataPhilly" | |
| Human Activity Recognition with Smartphones,Recordings of 30 study participants performing activities of daily living,UCI Machine Learning,101,"Version 1,2016-10-07","sociology | |
| human-computer interaction | |
| telecommunications",CSV,64 MB,CC0,"46,442 views","4,398 downloads",109 kernels,6 topics,https://www.kaggle.com/uciml/human-activity-recognition-with-smartphones,"The Human Activity Recognition database was built from the recordings of 30 study participants performing activities of daily living (ADL) while carrying a waist-mounted smartphone with embedded inertial sensors. The objective is to classify activities into one of the six activities performed. | |
| Description of experiment | |
| The experiments have been carried out with a group of 30 volunteers within an age bracket of 19-48 years. Each person performed six activities (WALKING, WALKING_UPSTAIRS, WALKING_DOWNSTAIRS, SITTING, STANDING, LAYING) wearing a smartphone (Samsung Galaxy S II) on the waist. Using its embedded accelerometer and gyroscope, we captured 3-axial linear acceleration and 3-axial angular velocity at a constant rate of 50Hz. The experiments have been video-recorded to label the data manually. The obtained dataset has been randomly partitioned into two sets, where 70% of the volunteers was selected for generating the training data and 30% the test data. | |
| The sensor signals (accelerometer and gyroscope) were pre-processed by applying noise filters and then sampled in fixed-width sliding windows of 2.56 sec and 50% overlap (128 readings/window). The sensor acceleration signal, which has gravitational and body motion components, was separated using a Butterworth low-pass filter into body acceleration and gravity. The gravitational force is assumed to have only low frequency components, therefore a filter with 0.3 Hz cutoff frequency was used. From each window, a vector of features was obtained by calculating variables from the time and frequency domain. | |
| Attribute information | |
| For each record in the dataset the following is provided: | |
| Triaxial acceleration from the accelerometer (total acceleration) and the estimated body acceleration. | |
| Triaxial Angular velocity from the gyroscope. | |
| A 561-feature vector with time and frequency domain variables. | |
| Its activity label. | |
| An identifier of the subject who carried out the experiment. | |
| Relevant papers | |
| Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra and Jorge L. Reyes-Ortiz. Human Activity Recognition on Smartphones using a Multiclass Hardware-Friendly Support Vector Machine. International Workshop of Ambient Assisted Living (IWAAL 2012). Vitoria-Gasteiz, Spain. Dec 2012 | |
| Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra, Jorge L. Reyes-Ortiz. Energy Efficient Smartphone-Based Activity Recognition using Fixed-Point Arithmetic. Journal of Universal Computer Science. Special Issue in Ambient Assisted Living: Home Care. Volume 19, Issue 9. May 2013 | |
| Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra and Jorge L. Reyes-Ortiz. Human Activity Recognition on Smartphones using a Multiclass Hardware-Friendly Support Vector Machine. 4th International Workshop of Ambient Assited Living, IWAAL 2012, Vitoria-Gasteiz, Spain, December 3-5, 2012. Proceedings. Lecture Notes in Computer Science 2012, pp 216-223. | |
| Jorge Luis Reyes-Ortiz, Alessandro Ghio, Xavier Parra-Llanas, Davide Anguita, Joan Cabestany, Andreu Català. Human Activity and Motion Disorder Recognition: Towards Smarter Interactive Cognitive Environments. 21st European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, ESANN 2013. Bruges, Belgium 24-26 April 2013. | |
| Citation | |
| Davide Anguita, Alessandro Ghio, Luca Oneto, Xavier Parra and Jorge L. Reyes-Ortiz. A Public Domain Dataset for Human Activity Recognition Using Smartphones. 21st European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, ESANN 2013. Bruges, Belgium 24-26 April 2013." | |
| Ethereum Historical Data,All Ethereum data from the start to Dec 2017,LiamLarsen,100,"Version 7,2018-02-02|Version 6,2017-12-16|Version 5,2017-12-13|Version 4,2017-06-14|Version 3,2017-05-06|Version 2,2017-04-16|Version 1,2017-04-14","history | |
| finance",CSV,437 KB,Other,"22,270 views","2,188 downloads",12 kernels,0 topics,https://www.kaggle.com/kingburrito666/ethereum-historical-data,"Context | |
| The Ethereum blockchain gives a revolutionary way of decentralized applications and provides its own cryptocurrency. Ethereum is a decentralized platform that runs smart contracts: applications that run exactly as programmed without any possibility of downtime, censorship, fraud or third party interference. These apps run on a custom built blockchain, an enormously powerful shared global infrastructure that can move value around and represent the ownership of property. This enables developers to create markets, store registries of debts or promises, move funds in accordance with instructions given long in the past (like a will or a futures contract) and many other things that have not been invented yet, all without a middle man or counterparty risk. | |
| Content | |
| What you may see in the CSVs are just numbers, but there is more to this. Numbers make machine learning easy. I've labeled each column, the first in all of them is the day; it may look weird but it makes sense if you look closely. | |
| Note: | |
| TIMESTAMP FORMAT | |
| How to convert timestamp in python: | |
| import datetime as dt | |
| # The (would-be) timestamp value is below | |
| timestamp = 1339521878.04 | |
| # Technechly you would iterate through and change them all if you were graphing | |
| timeValue = dt.datetime.fromtimestamp(timestamp) | |
| #Year, month, day, hour, minute, second | |
| print(timeValue.strftime('%Y-%m-%d %H:%M:%S')) | |
| Acknowledgements | |
| MR. Vitalik Buterin. co-founder of Ethereum and as a co-founder of Bitcoin Magazine. | |
| Hit a brother up | |
| 0x767e8b211f70c5b8b4caa38c2efe05bf8eac0da7 | |
| Will be updating every month with new Ethereum history!" | |
| E-Commerce Data,Actual transactions from UK retailer,Carrie,99,"Version 1,2017-08-17",,CSV,43 MB,Other,"25,892 views","4,091 downloads",20 kernels,0 topics,https://www.kaggle.com/carrie1/ecommerce-data,"Context | |
| Typically e-commerce datasets are proprietary and consequently hard to find among publicly available data. However, The UCI Machine Learning Repository has made this dataset containing actual transactions from 2010 and 2011. The dataset is maintained on their site, where it can be found by the title ""Online Retail"". | |
| Content | |
| ""This is a transnational data set which contains all the transactions occurring between 01/12/2010 and 09/12/2011 for a UK-based and registered non-store online retail.The company mainly sells unique all-occasion gifts. Many customers of the company are wholesalers."" | |
| Acknowledgements | |
| Per the UCI Machine Learning Repository, this data was made available by Dr Daqing Chen, Director: Public Analytics group. chend '@' lsbu.ac.uk, School of Engineering, London South Bank University, London SE1 0AA, UK. | |
| Image from stocksnap.io. | |
| Inspiration | |
| Analyses for this dataset could include time series, clustering, classification and more." | |
| FiveThirtyEight,80 Datasets published by FiveThirtyEight,FiveThirtyEight,96,"Version 7,2017-01-30|Version 6,2017-01-30|Version 5,2017-01-30|Version 4,2017-01-30|Version 3,2017-01-30|Version 2,2017-01-30|Version 1,2017-01-30",news agencies,Other,14 MB,Other,"28,075 views","3,097 downloads",85 kernels,,https://www.kaggle.com/fivethirtyeight/fivethirtyeight,"Introduction | |
| Data and code behind the stories and interactives at FiveThirtyEight. There are 80 Datasets here, included in one place. This allows you to make comparisons and utilize multiple Datasets. | |
| airline-safety | |
| alcohol-consumption | |
| avengers | |
| bad-drivers | |
| bechdel | |
| biopics | |
| births | |
| bob-ross | |
| buster-posey-mvp | |
| classic-rock | |
| college-majors | |
| comic-characters | |
| comma-survey-data | |
| congress-age | |
| cousin-marriage | |
| ... For the complete list, see the zip files listed here | |
| Python | |
| If you're planning to use Python, you might want to take a look at How to read datasets, which helps navigate the zipped Datasets. Again, there are multiple, 80 Datasets, in this Dataset. | |
| R | |
| If you're planning to use R, it's included in the kernels. No need to unzip files. For example, if you wanted to load ""bechdel"", you could use the following commands. | |
| library(fivethirtyeight) | |
| data(package = ""fivethirtyeight"") | |
| head(bechdel) | |
| Reference R Quick Start, for an example. | |
| References | |
| The following is FiveThirtyEight's public repository on Github. | |
| https://github.com/fivethirtyeight/data | |
| The CRAN package is maintained at the following link: | |
| https://github.com/rudeboybert/fivethirtyeight | |
| License | |
| Copyright (c) 2014 ESPN Internet Ventures | |
| Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the ""Software""), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: | |
| The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. | |
| THE SOFTWARE IS PROVIDED ""AS IS"", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE." | |
| GitHub Repos,Code and comments from 2.8 million repos,Github,93,"Version 1,2017-12-06","programming languages | |
| programming | |
| software engineering | |
| bigquery",BigQuery,3 TB,Other,"12,297 views",0 downloads,348 kernels,3 topics,https://www.kaggle.com/github/github-repos,"GitHub is how people build software and is home to the largest community of open source developers in the world, with over 12 million people contributing to 31 million projects on GitHub since 2008. | |
| This 3TB+ dataset comprises the largest released source of GitHub activity to date. It contains a full snapshot of the content of more than 2.8 million open source GitHub repositories including more than 145 million unique commits, over 2 billion different file paths, and the contents of the latest revision for 163 million files, all of which are searchable with regular expressions. | |
| Querying BigQuery tables | |
| You can use the BigQuery Python client library to query tables in this dataset in Kernels. Note that methods available in Kernels are limited to querying data. Tables are at bigquery-public-data.github_repos.[TABLENAME]. Fork this kernel to get started to learn how to safely manage analyzing large BigQuery datasets. | |
| Acknowledgements | |
| This dataset was made available per GitHub's terms of service. | |
| Inspiration | |
| This is the perfect dataset for fighting language wars. | |
| Can you identify any signals that predict which packages or languages will become popular, in advance of their mass adoption?" | |
| The Simpsons by the Data,"27 seasons of ""Simpsons did it.""",William Cukierski,91,"Version 1,2016-09-29",popular culture,CSV,34 MB,Other,"38,229 views","4,172 downloads",69 kernels,9 topics,https://www.kaggle.com/wcukierski/the-simpsons-by-the-data,"This dataset contains the characters, locations, episode details, and script lines for approximately 600 Simpsons episodes, dating back to 1989. | |
| Inspiration and credit for gathering the data goes to Todd Schneider: | |
| http://toddwschneider.com/posts/the-simpsons-by-the-data/ | |
| https://github.com/toddwschneider/flim-springfield" | |
| 2016 US Presidential Debates,Full transcripts of the face-off between Clinton & Trump,Megan Risdal,88,"Version 4,2016-10-24|Version 3,2016-10-10|Version 2,2016-10-05|Version 1,2016-09-28","politics | |
| political science",CSV,366 KB,CC0,"29,298 views","3,042 downloads",169 kernels,4 topics,https://www.kaggle.com/mrisdal/2016-us-presidential-debates,"Context | |
| In November, the United States will elect a new president. Before then, three presidential debates will take place between Hillary Clinton and Donald Trump as well as one vice presidential debate between Tim Kaine and Mike Pence. While you can watch the debates live, why not also read deeply into the candidates' responses with text analytics? | |
| You can now answer any questions you have about the platforms of our presidential hopefuls or their speaking skills. | |
| Which candidate is the most given to loquaciousness? | |
| How many times does Clinton get interrupted? | |
| Who gets the most audience applause? | |
| When is positive sentiment at its highest during the candidates' word play? | |
| Content & Acknowledgements | |
| For consistency, full transcripts of the debates were all taken from The Washington Post who made annotated transcripts available following each debate: | |
| First debate taking place September 26th, 2016 was obtained from The Washington Post. | |
| The vice presidential debate from October 4th, 2016 was similarly obtained here. | |
| The ""town hall"" presidential debate on October 9th is found here. | |
| The final presidential debate taking place on October 19th is found here. | |
| Please make any dataset suggestions or requests on the forum. Word cloud from Debate Visualization by Jane Yu." | |
| Lower Back Pain Symptoms Dataset,Collection of physical spine data,sammy123,87,"Version 1,2016-08-19",healthcare,CSV,42 KB,Other,"26,864 views","3,212 downloads",87 kernels,7 topics,https://www.kaggle.com/sammy123/lower-back-pain-symptoms-dataset,"310 Observations, 13 Attributes (12 Numeric Predictors, 1 Binary Class Attribute - No Demographics) | |
| Lower back pain can be caused by a variety of problems with any parts of the complex, interconnected network of spinal muscles, nerves, bones, discs or tendons in the lumbar spine. Typical sources of low back pain include: | |
| The large nerve roots in the low back that go to the legs may be irritated | |
| The smaller nerves that supply the low back may be irritated | |
| The large paired lower back muscles (erector spinae) may be strained | |
| The bones, ligaments or joints may be damaged | |
| An intervertebral disc may be degenerating | |
| An irritation or problem with any of these structures can cause lower back pain and/or pain that radiates or is referred to other parts of the body. Many lower back problems also cause back muscle spasms, which don't sound like much but can cause severe pain and disability. | |
| While lower back pain is extremely common, the symptoms and severity of lower back pain vary greatly. A simple lower back muscle strain might be excruciating enough to necessitate an emergency room visit, while a degenerating disc might cause only mild, intermittent discomfort. | |
| This data set is about to identify a person is abnormal or normal using collected physical spine details/data." | |
| NBA Players stats since 1950,"3000+ Players over 60+ Seasons, and 50+ features per player",DrGuillermo,86,"Version 1,2017-06-06",basketball,CSV,5 MB,Other,"20,492 views","3,474 downloads",33 kernels,4 topics,https://www.kaggle.com/drgilermo/nba-players-stats,"Content | |
| The data-set contains aggregate individual statistics for 67 NBA seasons. from basic box-score attributes such as points, assists, rebounds etc., to more advanced money-ball like features such as Value Over Replacement. | |
| Acknowledgements | |
| The data was scraped from Basketball-reference Take a look in their glossary for a detailed column description Glossary" | |
| Pokemon- Weedle's Cave,Welcome to Weedle's cave,T7 - Pokemon Challenge,86,"Version 1,2017-09-21","popular culture | |
| video games",CSV,682 KB,CC0,"15,187 views","6,606 downloads",38 kernels,4 topics,https://www.kaggle.com/terminus7/pokemon-challenge,"Welcome to Weedle's cave. Will you be able to predict the outcome of future matches? | |
| To do it you will have the pokemon characteristics and the results of previous combats. | |
| Three files are available. The first one contains the pokemon characteristics (the first column being the id of the pokemon). The second one contains information about previous combats. The first two columns contain the ids of the combatants and the third one the id of the winner. Important: The pokemon in the first columns attacks first. | |
| The goal is to develop a Machine Learning model able to predict the result of future pokemon combats. | |
| If you have any questions, please email: t7pokemonchallenge@intelygenz.com | |
| DISCLAIMER | |
| In Intelygenz we are against animal abuse. No animal real or imaginary should be forced to fight against other. Freedom for the pokemons !" | |
| Craft Beers Dataset,2K+ craft canned beers from the US and 500+ breweries in the United States.,Jean-NicholasHould,85,"Version 1,2017-01-18",food and drink,CSV,178 KB,ODbL,"34,226 views","4,236 downloads",90 kernels,6 topics,https://www.kaggle.com/nickhould/craft-cans,"This dataset contains a list of 2410 US craft beers and 510 US breweries. The beers and breweries are linked together with an ""id"". This data was collected in January 2017 on CraftCans.com. The dataset is an a tidy format and values have been cleaned up for your enjoyment. | |
| If you are interested in learning more about how this dataset was acquired, I wrote an extensive blogpost about it (http://www.jeannicholashould.com/python-web-scraping-tutorial-for-craft-beers.html). | |
| Enjoy!" | |
| Starbucks Locations Worldwide,"Name, ownership type, and location of every Starbucks store in operation",Starbucks,85,"Version 1,2017-02-14",food and drink,CSV,4 MB,Other,"35,348 views","5,208 downloads",103 kernels,7 topics,https://www.kaggle.com/starbucks/store-locations,"Context | |
| Starbucks started as a roaster and retailer of whole bean and ground coffee, tea and spices with a single store in Seattle’s Pike Place Market in 1971. The company now operates more than 24,000 retail stores in 70 countries. | |
| Content | |
| This dataset includes a record for every Starbucks or subsidiary store location currently in operation as of February 2017. | |
| Acknowledgements | |
| This data was scraped from the Starbucks store locator webpage by Github user chrismeller. | |
| Inspiration | |
| What city or country has the highest number of Starbucks stores per capita? What two Starbucks locations are the closest in proximity to one another? What location on Earth is farthest from a Starbucks? How has Starbucks expanded overseas?" | |
| The Holy Quran,Understanding God - Sacred Meanings,Zeeshan-ul-hassan Usmani,83,"Version 3,2017-11-20|Version 2,2017-11-10|Version 1,2017-11-08","languages | |
| faith and traditions | |
| islam",CSV,16 MB,CC0,"10,214 views",498 downloads,8 kernels,7 topics,https://www.kaggle.com/zusmani/the-holy-quran,"Context | |
| The Holy Quran is the central text for 1.5 billion Muslims around the world. It literally means ""The Recitation."" It is undoubtedly the finest work in Arabic literature and revealed by Allah (God) to His Messenger Prophet Muhammed (Peace Be Upon Him) through angel Gabriel. It was revealed verbally from December 22, 609 (AD) to 632 AD (when Prophet Muhammed (Peace Be Upon Him) died) | |
| The book is divided into 30 parts, 114 Chapters and 6,000+ verses. | |
| There has been a lot of questions and comments on the text of this holy book given the contemporary Geo-political situation of the world, wars in the Middle East and Afghanistan and the ongoing terrorism. | |
| I have put this dataset together to call my fellow data scientists to run their NLP algorithms and Kernels to find and explore the sacred text by them selves. | |
| Content | |
| The data contains complete Holy Quran in following 21 languages (so data scientists from different parts of the world can work with it). The original text was revealed in Arabic. Other 20 files are the translations of the original text. | |
| Arabic (Original Book by God) | |
| English (Transalation by Yusuf Ali) | |
| Persian (Makarim Sheerazi) | |
| Urdu (Jalandhari) | |
| Turkish (Y. N. Ozturk) | |
| Portuguese (El. Hayek) | |
| Dutch (Keyzer) | |
| Norwegian (Einar Berg) | |
| Italian (Piccardo) | |
| French (Hamidullah) | |
| German (Zaidan) | |
| Swedish (Rashad Kalifa) | |
| Indonesia (Bhasha Indoenisan) | |
| Bangla | |
| Chinese/Madarin | |
| Japanese | |
| Malay | |
| Malayalam | |
| Russian | |
| Tamil | |
| Uzbek | |
| Inspiration | |
| Here are some ideas to explore: | |
| Can we make a word cloud for each chapter | |
| Can we make a word cloud of the whole book and find out the frequency of each word | |
| Can we describe or annotate subjects in each chapter and verse | |
| Can we find how many times The Quran has mentioned Humans, Women, Humility, Heaven or Hell | |
| Can we compare the text with other famous books and see the correlation | |
| Can we compare the text with laws in multiple countries to see the resemblance | |
| Any other ideas you can think of | |
| I am looking forward to see your work and ideas and will keep adding more ideas to explore | |
| Welcome on board to learn the finest text on earth with Data Sciences and Machine Learning! | |
| Updates | |
| Complete Verse-By-Verse Dataset has been shared. A good contribution by Zohaib Ali - https://www.kaggle.com/zohaib1111 (Nov 20, 2017)" | |
| Deep-NLP,natural language processing,samdeeplearning,83,"Version 1,2017-03-01","languages | |
| linguistics",CSV,663 KB,Other,"25,369 views","1,667 downloads",49 kernels,2 topics,https://www.kaggle.com/samdeeplearning/deepnlp,"What's In The Deep-NLP Dataset? | |
| Sheet_1.csv contains 80 user responses, in the response_text column, to a therapy chatbot. Bot said: 'Describe a time when you have acted as a resource for someone else'. User responded. If a response is 'not flagged', the user can continue talking to the bot. If it is 'flagged', the user is referred to help. | |
| Sheet_2.csv contains 125 resumes, in the resume_text column. Resumes were queried from Indeed.com with keyword 'data scientist', location 'Vermont'. If a resume is 'not flagged', the applicant can submit a modified resume version at a later date. If it is 'flagged', the applicant is invited to interview. | |
| What Do I Do With This? | |
| Classify new resumes/responses as flagged or not flagged. | |
| There are two sets of data here - resumes and responses. Split the data into a train set and a test set to test the accuracy of your classifier. Bonus points for using the same classifier for both problems. | |
| Good luck. | |
| Acknowledgements | |
| Thank you to Parsa Ghaffari (Aylien), without whom these visuals (cover photo is in Parsa Ghaffari's excellent LinkedIn article on English, Spanish and German postive v. negative sentiment analysis) would not exist. | |
| There Is A 'deep natural language processing' Kernel. I will update it. I Hope You Find It Useful. | |
| You can use any of the code in that kernel anywhere, on or off Kaggle. Ping me at @_samputnam for questions." | |
| Japan Trade Statistics,Japan's international trade by country and type of good,TadashiNagao,81,"Version 41,2018-02-04|Version 40,2017-12-30|Version 39,2017-12-02|Version 38,2017-10-31|Version 37,2017-09-29|Version 36,2017-09-24|Version 35,2017-08-19|Version 34,2017-08-19|Version 33,2017-08-11|Version 32,2017-07-29|Version 31,2017-07-22|Version 30,2017-07-15|Version 29,2017-07-05|Version 28,2017-06-11|Version 27,2017-06-01|Version 26,2017-04-30|Version 25,2017-04-20|Version 24,2017-03-26|Version 23,2017-03-26|Version 22,2017-02-26|Version 21,2017-02-05|Version 20,2017-01-05|Version 19,2016-10-30|Version 18,2016-10-09|Version 17,2016-10-09|Version 16,2016-10-02|Version 15,2016-09-10|Version 14,2016-09-07|Version 13,2016-09-04|Version 12,2016-09-04|Version 11,2016-09-02|Version 10,2016-09-02|Version 9,2016-09-02|Version 8,2016-08-31|Version 7,2016-08-31|Version 6,2016-08-31|Version 5,2016-08-02|Version 4,2016-08-01|Version 3,2016-07-31|Version 2,2016-06-29|Version 1,2016-06-24","business | |
| finance",CSV,241 MB,CC4,"46,192 views","4,354 downloads",166 kernels,16 topics,https://www.kaggle.com/zanjibar/japan-trade-statistics,"Because of memory limitations,data format change csv -> db (sqlite format) | |
| This dataset includes yearly and monthly versions of Japan's international trading data (segmented by country , the type of good and local custom ). | |
| Japan trade statistics is searchable here" | |
| Hacker News,All posts from Y Combinator's social news website from 2006 to late 2017,Hacker News,81,"Version 1,2017-12-05","journalism | |
| information technology | |
| internet | |
| bigquery",BigQuery,14 GB,CC0,"14,579 views",0 downloads,723 kernels,,https://www.kaggle.com/hacker-news/hacker-news,"Context | |
| This dataset contains all stories and comments from Hacker News from its launch in 2006. Each story contains a story id, the author that made the post, when it was written, and the number of points the story received. Hacker News is a social news website focusing on computer science and entrepreneurship. It is run by Paul Graham's investment fund and startup incubator, Y Combinator. In general, content that can be submitted is defined as ""anything that gratifies one's intellectual curiosity"". | |
| Content | |
| Each story contains a story ID, the author that made the post, when it was written, and the number of points the story received. | |
| Please note that the text field includes profanity. All texts are the author’s own, do not necessarily reflect the positions of Kaggle or Hacker News, and are presented without endorsement. | |
| Querying BigQuery tables | |
| You can use the BigQuery Python client library to query tables in this dataset in Kernels. Note that methods available in Kernels are limited to querying data. Tables are at bigquery-public-data.hacker_news.[TABLENAME]. Fork this kernel to get started. | |
| Acknowledgements | |
| This dataset was kindly made publicly available by Hacker News under the MIT license. | |
| Inspiration | |
| Recent studies have found that many forums tend to be dominated by a very small fraction of users. Is this true of Hacker News? | |
| Hacker News has received complaints that the site is biased towards Y Combinator startups. Do the data support this? | |
| Is the amount of coverage by Hacker News predictive of a startup’s success?" | |
| Zika Virus Epidemic,Analyze the ongoing spread of this infectious disease,Centers for Disease Control and Prevention,79,"Version 1,2016-07-16","brazil | |
| healthcare | |
| epidemiology",CSV,11 MB,CC0,"36,494 views","4,487 downloads",113 kernels,3 topics,https://www.kaggle.com/cdc/zika-virus-epidemic,"An outbreak of the Zika virus, an infection transmitted mostly by the Aedes species mosquito (Ae. aegypti and Ae. albopictus), has been sweeping across the Americas and the Pacific since mid-2015. Although first isolated in 1947 in Uganda, a lack of previous research has challenged the scientific community to quickly understand its devastating effects as the epidemic continues to spread. | |
| All Countries & Territories with Active Zika Virus Transmission | |
| The data | |
| This dataset shares publicly available data related to the ongoing Zika epidemic. It is being provided as a resource to the scientific community engaged in the public health response. The data provided here is not official and should be considered provisional and non-exhaustive. The data in reports may change over time, reflecting delays in reporting or changes in classifications. And while accurate representation of the reported data is the objective in the machine readable files shared here, that accuracy is not guaranteed. Before using any of these data, it is advisable to review the original reports and sources, which are provided whenever possible along with further information on the CDC Zika epidemic GitHub repo. | |
| The dataset includes the following fields: | |
| report_date - The report date is the date that the report was published. The date should be specified in standard ISO format (YYYY-MM-DD). | |
| location - A location is specified for each observation following the specific names specified in the country place name database. This may be any place with a 'location_type' as listed below, e.g. city, state, country, etc. It should be specified at up to three hierarchical levels in the following format: [country]-[state/province]-[county/municipality/city], always beginning with the country name. If the data is for a particular city, e.g. Salvador, it should be specified: Brazil-Bahia-Salvador. | |
| location_type - A location code is included indicating: city, district, municipality, county, state, province, or country. If there is need for an additional 'location_type', open an Issue to create a new 'location_type'. | |
| data_field - The data field is a short description of what data is represented in the row and is related to a specific definition defined by the report from which it comes. | |
| data_field_code - This code is defined in the country data guide. It includes a two letter country code (ISO-3166 alpha-2, list), followed by a 4-digit number corresponding to a specific report type and data type. | |
| time_period - Optional. If the data pertains to a specific period of time, for example an epidemiological week, that number should be indicated here and the type of time period in the 'time_period_type', otherwise it should be NA. | |
| time_period_type - Required only if 'time_period' is specified. Types will also be specified in the country data guide. Otherwise should be NA. | |
| value - The observation indicated for the specific 'report_date', 'location', 'data_field' and when appropriate, 'time_period'. | |
| unit - The unit of measurement for the 'data_field'. This should conform to the 'data_field' unit options as described in the country-specific data guide. | |
| If you find the data useful, please support data sharing by referencing this dataset and the original data source. If you're interested in contributing to the Zika project from GitHub, you can read more here. The source for the Zika virus structure is available here." | |
| Melbourne Housing Market,Melbourne housing clearance data from Jan 2016,Tony Pino,78,"Version 19,2018-02-13|Version 18,2018-01-11|Version 17,2017-11-27|Version 16,2017-11-23|Version 15,2017-09-26|Version 14,2017-08-18|Version 13,2017-08-18|Version 12,2017-08-16|Version 11,2017-07-05|Version 10,2017-07-04|Version 9,2017-05-26|Version 8,2017-05-22|Version 7,2017-04-10|Version 6,2017-04-10|Version 5,2017-03-22|Version 4,2017-03-07|Version 3,2017-02-15|Version 2,2016-12-13|Version 1,2016-11-23","housing | |
| demographics",CSV,929 KB,CC4,"25,357 views","3,534 downloads",77 kernels,8 topics,https://www.kaggle.com/anthonypino/melbourne-housing-market,"Update 28/11/2017 - Last few weeks clearance levels starting to decrease (I may just be seeing a pattern I want to see.. maybe I'm just evil). Anyway, can any of you magicians make any sense of it? | |
| Melbourne is currently experiencing a housing bubble (some experts say it may burst soon). Maybe someone can find a trend or give a prediction? Which suburbs are the best to buy in? Which ones are value for money? Where's the expensive side of town? And more importantly where should I buy a 2 bedroom unit? | |
| Content & Acknowledgements | |
| This data was scraped from publicly available results posted every week from Domain.com.au, I've cleaned it as best I can, now it's up to you to make data analysis magic. The dataset includes Address, Type of Real estate, Suburb, Method of Selling, Rooms, Price, Real Estate Agent, Date of Sale and distance from C.B.D. | |
| ....Now with extra data including including property size, land size and council area, you may need to change your code! | |
| Some Key Details | |
| Suburb: Suburb | |
| Address: Address | |
| Rooms: Number of rooms | |
| Price: Price in dollars | |
| Method: S - property sold; SP - property sold prior; PI - property passed in; PN - sold prior not disclosed; SN - sold not disclosed; NB - no bid; VB - vendor bid; W - withdrawn prior to auction; SA - sold after auction; SS - sold after auction price not disclosed. N/A - price or highest bid not available. | |
| Type: br - bedroom(s); h - house,cottage,villa, semi,terrace; u - unit, duplex; t - townhouse; dev site - development site; o res - other residential. | |
| SellerG: Real Estate Agent | |
| Date: Date sold | |
| Distance: Distance from CBD | |
| Regionname: General Region (West, North West, North, North east ...etc) | |
| Propertycount: Number of properties that exist in the suburb. | |
| Bedroom2 : Scraped # of Bedrooms (from different source) | |
| Bathroom: Number of Bathrooms | |
| Car: Number of carspots | |
| Landsize: Land Size | |
| BuildingArea: Building Size | |
| YearBuilt: Year the house was built | |
| CouncilArea: Governing council for the area | |
| Lattitude: Self explanitory | |
| Longtitude: Self explanitory" | |
| Crimes in Chicago,"An extensive dataset of crimes in Chicago (2001-2017), by City of Chicago",Currie32,78,"Version 1,2017-01-28",crime,CSV,2 GB,Other,"42,154 views","6,867 downloads",95 kernels,7 topics,https://www.kaggle.com/currie32/crimes-in-chicago,"Context | |
| This dataset reflects reported incidents of crime (with the exception of murders where data exists for each victim) that occurred in the City of Chicago from 2001 to present, minus the most recent seven days. Data is extracted from the Chicago Police Department's CLEAR (Citizen Law Enforcement Analysis and Reporting) system. In order to protect the privacy of crime victims, addresses are shown at the block level only and specific locations are not identified. Should you have questions about this dataset, you may contact the Research & Development Division of the Chicago Police Department at 312.745.6071 or RDAnalysis@chicagopolice.org. Disclaimer: These crimes may be based upon preliminary information supplied to the Police Department by the reporting parties that have not been verified. The preliminary crime classifications may be changed at a later date based upon additional investigation and there is always the possibility of mechanical or human error. Therefore, the Chicago Police Department does not guarantee (either expressed or implied) the accuracy, completeness, timeliness, or correct sequencing of the information and the information should not be used for comparison purposes over time. The Chicago Police Department will not be responsible for any error or omission, or for the use of, or the results obtained from the use of this information. All data visualizations on maps should be considered approximate and attempts to derive specific addresses are strictly prohibited. The Chicago Police Department is not responsible for the content of any off-site pages that are referenced by or that reference this web page other than an official City of Chicago or Chicago Police Department web page. The user specifically acknowledges that the Chicago Police Department is not responsible for any defamatory, offensive, misleading, or illegal conduct of other users, links, or third parties and that the risk of injury from the foregoing rests entirely with the user. The unauthorized use of the words ""Chicago Police Department,"" ""Chicago Police,"" or any colorable imitation of these words or the unauthorized use of the Chicago Police Department logo is unlawful. This web page does not, in any way, authorize such use. Data are updated daily. The dataset contains more than 6,000,000 records/rows of data and cannot be viewed in full in Microsoft Excel. To access a list of Chicago Police Department - Illinois Uniform Crime Reporting (IUCR) codes, go to http://data.cityofchicago.org/Public-Safety/Chicago-Police-Department-Illinois-Uniform-Crime-R/c7ck-438e | |
| Content | |
| ID - Unique identifier for the record. | |
| Case Number - The Chicago Police Department RD Number (Records Division Number), which is unique to the incident. | |
| Date - Date when the incident occurred. this is sometimes a best estimate. | |
| Block - The partially redacted address where the incident occurred, placing it on the same block as the actual address. | |
| IUCR - The Illinois Unifrom Crime Reporting code. This is directly linked to the Primary Type and Description. See the list of IUCR codes at https://data.cityofchicago.org/d/c7ck-438e. | |
| Primary Type - The primary description of the IUCR code. | |
| Description - The secondary description of the IUCR code, a subcategory of the primary description. | |
| Location Description - Description of the location where the incident occurred. | |
| Arrest - Indicates whether an arrest was made. | |
| Domestic - Indicates whether the incident was domestic-related as defined by the Illinois Domestic Violence Act. | |
| Beat - Indicates the beat where the incident occurred. A beat is the smallest police geographic area – each beat has a dedicated police beat car. Three to five beats make up a police sector, and three sectors make up a police district. The Chicago Police Department has 22 police districts. See the beats at https://data.cityofchicago.org/d/aerh-rz74. | |
| District - Indicates the police district where the incident occurred. See the districts at https://data.cityofchicago.org/d/fthy-xz3r. | |
| Ward - The ward (City Council district) where the incident occurred. See the wards at https://data.cityofchicago.org/d/sp34-6z76. | |
| Community Area - Indicates the community area where the incident occurred. Chicago has 77 community areas. See the community areas at https://data.cityofchicago.org/d/cauq-8yn6. | |
| FBI Code - Indicates the crime classification as outlined in the FBI's National Incident-Based Reporting System (NIBRS). See the Chicago Police Department listing of these classifications at http://gis.chicagopolice.org/clearmap_crime_sums/crime_types.html. | |
| X Coordinate - The x coordinate of the location where the incident occurred in State Plane Illinois East NAD 1983 projection. This location is shifted from the actual location for partial redaction but falls on the same block. | |
| Y Coordinate - The y coordinate of the location where the incident occurred in State Plane Illinois East NAD 1983 projection. This location is shifted from the actual location for partial redaction but falls on the same block. | |
| Year - Year the incident occurred. | |
| Updated On - Date and time the record was last updated. | |
| Latitude - The latitude of the location where the incident occurred. This location is shifted from the actual location for partial redaction but falls on the same block. | |
| Longitude - The longitude of the location where the incident occurred. This location is shifted from the actual location for partial redaction but falls on the same block. | |
| Location - The location where the incident occurred in a format that allows for creation of maps and other geographic operations on this data portal. This location is shifted from the actual location for partial redaction but falls on the same block. | |
| Acknowledgements | |
| I really want to say thank you to the City of Chicago and the Chicago Police Department for making this comprehensive data set available to everyone! | |
| Inspiration | |
| How has crime changed over the years? Is it possible to predict where or when a crime will be committed? Which areas of the city have evolved over this time span?" | |
| Predict'em All,Predict where Pokemon appear in PokemonGo based on historical data,SemionKorchevskiy,78,"Version 3,2016-10-12|Version 2,2016-10-04|Version 1,2016-10-01","games and toys | |
| video games",Other,763 MB,Other,"32,888 views","3,876 downloads",99 kernels,6 topics,https://www.kaggle.com/semioniy/predictemall,"Overview | |
| PokemonGo is a mobile augmented reality game developed by Niantic inc. for iOS, Android, and Apple Watch devices. It was initially released in selected countries in July 2016. In the game, players use a mobile device's GPS capability to locate, capture, battle, and train virtual creatures, called Pokémon, who appear on the screen as if they were in the same real-world location as the player. | |
| Dataset | |
| Dataset consists of roughly 293,000 pokemon sightings (historical appearances of Pokemon), having coordinates, time, weather, population density, distance to pokestops/ gyms etc. as features. The target is to train a machine learning algorithm so that it can predict where pokemon appear in future. So, can you predict'em all?) | |
| Feature description | |
| pokemonId - the identifier of a pokemon, should be deleted to not affect predictions. (numeric; ranges between 1 and 151) | |
| latitude, longitude - coordinates of a sighting (numeric) | |
| appearedLocalTime - exact time of a sighting in format yyyy-mm-dd'T'hh-mm-ss.ms'Z' (nominal) | |
| cellId 90-5850m - geographic position projected on a S2 Cell, with cell sizes ranging from 90 to 5850m (numeric) | |
| appearedTimeOfDay - time of the day of a sighting (night, evening, afternoon, morning) | |
| appearedHour/appearedMinute - local hour/minute of a sighting (numeric) | |
| appearedDayOfWeek - week day of a sighting (Monday, Tuesday, Wednesday, Thursday, Friday, Saturday, Sunday) | |
| appearedDay/appearedMonth/appearedYear - day/month/year of a sighting (numeric) | |
| terrainType - terrain where pokemon appeared described with help of GLCF Modis Land Cover (numeric) | |
| closeToWater - did pokemon appear close (100m or less) to water (Boolean, same source as above) | |
| city - the city of a sighting (nominal) | |
| continent (not always parsed right) - the continent of a sighting (nominal) | |
| weather - weather type during a sighting (Foggy Clear, PartlyCloudy, MostlyCloudy, Overcast, Rain, BreezyandOvercast, LightRain, Drizzle, BreezyandPartlyCloudy, HeavyRain, BreezyandMostlyCloudy, Breezy, Windy, WindyandFoggy, Humid, Dry, WindyandPartlyCloudy, DryandMostlyCloudy, DryandPartlyCloudy, DrizzleandBreezy, LightRainandBreezy, HumidandPartlyCloudy, HumidandOvercast, RainandWindy) // Source for all weather features | |
| temperature - temperature in celsius at the location of a sighting (numeric) | |
| windSpeed - speed of the wind in km/h at the location of a sighting (numeric) | |
| windBearing - wind direction (numeric) | |
| pressure - atmospheric pressure in bar at the location of a sighting (numeric) | |
| weatherIcon - a compact representation of the weather at the location of a sighting (fog, clear-night, partly-cloudy-night, partly-cloudy-day, cloudy, clear-day, rain, wind) | |
| sunriseMinutesMidnight-sunsetMinutesBefore - time of appearance relatively to sunrise/sunset Source | |
| population density - what is the population density per square km of a sighting (numeric, Source) | |
| urban-rural - how urban is location where pokemon appeared (Boolean, built on Population density, <200 for rural, >=200 and <400 for midUrban, >=400 and <800 for subUrban, >800 for urban) | |
| gymDistanceKm, pokestopDistanceKm - how far is the nearest gym/pokestop in km from a sighting? (numeric, extracted from this dataset) | |
| gymIn100m-pokestopIn5000m - is there a gym/pokestop in 100/200/etc meters? (Boolean) | |
| cooc 1-cooc 151 - co-occurrence with any other pokemon (pokemon ids range between 1 and 151) within 100m distance and within the last 24 hours (Boolean) | |
| class - says which pokemonId it is, to be predicted. | |
| Data dump | |
| All pokemon sightings (in JSON file, without features) can be found in Discussion ""Datadump""" | |
| Ocean Ship Logbooks (1750-1850),Explore changing climatology with data from early shipping logs,CWILOC,78,"Version 2,2017-11-16|Version 1,2015-08-19","environment | |
| climate | |
| shipping",CSV,19 MB,CC0,"23,833 views","2,814 downloads",91 kernels,11 topics,https://www.kaggle.com/cwiloc/climate-data-from-ocean-ships,"In the mid-eighteenth to nineteenth centuries, navigating the open ocean was an imprecise and often dangerous feat. In order to calculate their daily progress and avoid running/sailing into the unknown, a ship's crew kept a detailed logbook with data on winds, waves, and any remarkable weather. | |
| Handwritten in archived logbooks, these rich datasets were nearly impossible to study until the European Union funded their digitization in 2001. You can visit the EU project website for detailed information on the countries and ships included. | |
| We're hosting the full 1750-1850 dataset on Kaggle to promote the exploration of this unique and invaluable climatology resource. | |
| Data Description | |
| This data comes from the Climatological Database for the World's Oceans 1750-1850 (CLIWOC), version 1.5 data release. | |
| The primary data file is CLIWOC15.csv. The columns in this table are described on this page (scroll down to the table that starts with ""Field abbreviation""). It includes 280,280 observational records of ship locations weather data, and other associated information. | |
| The ancillary data files are described on the above site." | |
| 1.88 Million US Wildfires,24 years of geo-referenced wildfire records,Rachael Tatman,77,"Version 1,2017-09-14","climate | |
| firefighting",SQLite,759 MB,CC0,"13,794 views","1,590 downloads",9 kernels,,https://www.kaggle.com/rtatman/188-million-us-wildfires,"Context: | |
| This data publication contains a spatial database of wildfires that occurred in the United States from 1992 to 2015. It is the third update of a publication originally generated to support the national Fire Program Analysis (FPA) system. The wildfire records were acquired from the reporting systems of federal, state, and local fire organizations. The following core data elements were required for records to be included in this data publication: discovery date, final fire size, and a point location at least as precise as Public Land Survey System (PLSS) section (1-square mile grid). The data were transformed to conform, when possible, to the data standards of the National Wildfire Coordinating Group (NWCG). Basic error-checking was performed and redundant records were identified and removed, to the degree possible. The resulting product, referred to as the Fire Program Analysis fire-occurrence database (FPA FOD), includes 1.88 million geo-referenced wildfire records, representing a total of 140 million acres burned during the 24-year period. | |
| Content: | |
| This dataset is an SQLite database that contains the following information: | |
| Fires: Table including wildfire data for the period of 1992-2015 compiled from US federal, state, and local reporting systems. | |
| FOD_ID = Global unique identifier. | |
| FPA_ID = Unique identifier that contains information necessary to track back to the original record in the source dataset. | |
| SOURCE_SYSTEM_TYPE = Type of source database or system that the record was drawn from (federal, nonfederal, or interagency). | |
| SOURCE_SYSTEM = Name of or other identifier for source database or system that the record was drawn from. See Table 1 in Short (2014), or \Supplements\FPA_FOD_source_list.pdf, for a list of sources and their identifier. | |
| NWCG_REPORTING_AGENCY = Active National Wildlife Coordinating Group (NWCG) Unit Identifier for the agency preparing the fire report (BIA = Bureau of Indian Affairs, BLM = Bureau of Land Management, BOR = Bureau of Reclamation, DOD = Department of Defense, DOE = Department of Energy, FS = Forest Service, FWS = Fish and Wildlife Service, IA = Interagency Organization, NPS = National Park Service, ST/C&L = State, County, or Local Organization, and TRIBE = Tribal Organization). | |
| NWCG_REPORTING_UNIT_ID = Active NWCG Unit Identifier for the unit preparing the fire report. | |
| NWCG_REPORTING_UNIT_NAME = Active NWCG Unit Name for the unit preparing the fire report. | |
| SOURCE_REPORTING_UNIT = Code for the agency unit preparing the fire report, based on code/name in the source dataset. | |
| SOURCE_REPORTING_UNIT_NAME = Name of reporting agency unit preparing the fire report, based on code/name in the source dataset. | |
| LOCAL_FIRE_REPORT_ID = Number or code that uniquely identifies an incident report for a particular reporting unit and a particular calendar year. | |
| LOCAL_INCIDENT_ID = Number or code that uniquely identifies an incident for a particular local fire management organization within a particular calendar year. | |
| FIRE_CODE = Code used within the interagency wildland fire community to track and compile cost information for emergency fire suppression (https://www.firecode.gov/). | |
| FIRE_NAME = Name of the incident, from the fire report (primary) or ICS-209 report (secondary). | |
| ICS_209_INCIDENT_NUMBER = Incident (event) identifier, from the ICS-209 report. | |
| ICS_209_NAME = Name of the incident, from the ICS-209 report. | |
| MTBS_ID = Incident identifier, from the MTBS perimeter dataset. | |
| MTBS_FIRE_NAME = Name of the incident, from the MTBS perimeter dataset. | |
| COMPLEX_NAME = Name of the complex under which the fire was ultimately managed, when discernible. | |
| FIRE_YEAR = Calendar year in which the fire was discovered or confirmed to exist. | |
| DISCOVERY_DATE = Date on which the fire was discovered or confirmed to exist. | |
| DISCOVERY_DOY = Day of year on which the fire was discovered or confirmed to exist. | |
| DISCOVERY_TIME = Time of day that the fire was discovered or confirmed to exist. | |
| STAT_CAUSE_CODE = Code for the (statistical) cause of the fire. | |
| STAT_CAUSE_DESCR = Description of the (statistical) cause of the fire. | |
| CONT_DATE = Date on which the fire was declared contained or otherwise controlled (mm/dd/yyyy where mm=month, dd=day, and yyyy=year). | |
| CONT_DOY = Day of year on which the fire was declared contained or otherwise controlled. | |
| CONT_TIME = Time of day that the fire was declared contained or otherwise controlled (hhmm where hh=hour, mm=minutes). | |
| FIRE_SIZE = Estimate of acres within the final perimeter of the fire. | |
| FIRE_SIZE_CLASS = Code for fire size based on the number of acres within the final fire perimeter expenditures (A=greater than 0 but less than or equal to 0.25 acres, B=0.26-9.9 acres, C=10.0-99.9 acres, D=100-299 acres, E=300 to 999 acres, F=1000 to 4999 acres, and G=5000+ acres). | |
| LATITUDE = Latitude (NAD83) for point location of the fire (decimal degrees). | |
| LONGITUDE = Longitude (NAD83) for point location of the fire (decimal degrees). | |
| OWNER_CODE = Code for primary owner or entity responsible for managing the land at the point of origin of the fire at the time of the incident. | |
| OWNER_DESCR = Name of primary owner or entity responsible for managing the land at the point of origin of the fire at the time of the incident. | |
| STATE = Two-letter alphabetic code for the state in which the fire burned (or originated), based on the nominal designation in the fire report. | |
| COUNTY = County, or equivalent, in which the fire burned (or originated), based on nominal designation in the fire report. | |
| FIPS_CODE = Three-digit code from the Federal Information Process Standards (FIPS) publication 6-4 for representation of counties and equivalent entities. | |
| FIPS_NAME = County name from the FIPS publication 6-4 for representation of counties and equivalent entities. | |
| NWCG_UnitIDActive_20170109: Look-up table containing all NWCG identifiers for agency units that were active (i.e., valid) as of 9 January 2017, when the list was downloaded from https://www.nifc.blm.gov/unit_id/Publish.html and used as the source of values available to populate the following fields in the Fires table: NWCG_REPORTING_AGENCY, NWCG_REPORTING_UNIT_ID, and NWCG_REPORTING_UNIT_NAME. | |
| UnitId = NWCG Unit ID. | |
| GeographicArea = Two-letter code for the geographic area in which the unit is located (NA=National, IN=International, AK=Alaska, CA=California, EA=Eastern Area, GB=Great Basin, NR=Northern Rockies, NW=Northwest, RM=Rocky Mountain, SA=Southern Area, and SW=Southwest). | |
| Gacc = Seven or eight-letter code for the Geographic Area Coordination Center in which the unit is located or primarily affiliated with (CAMBCIFC=Canadian Interagency Forest Fire Centre, USAKCC=Alaska Interagency Coordination Center, USCAONCC=Northern California Area Coordination Center, USCAOSCC=Southern California Coordination Center, USCORMCC=Rocky Mountain Area Coordination Center, USGASAC=Southern Area Coordination Center, USIDNIC=National Interagency Coordination Center, USMTNRC=Northern Rockies Coordination Center, USNMSWC=Southwest Area Coordination Center, USORNWC=Northwest Area Coordination Center, USUTGBC=Western Great Basin Coordination Center, USWIEACC=Eastern Area Coordination Center). | |
| WildlandRole = Role of the unit within the wildland fire community. | |
| UnitType = Type of unit (e.g., federal, state, local). | |
| Department = Department (or state/territory) to which the unit belongs (AK=Alaska, AL=Alabama, AR=Arkansas, AZ=Arizona, CA=California, CO=Colorado, CT=Connecticut, DE=Delaware, DHS=Department of Homeland Security, DOC= Department of Commerce, DOD=Department of Defense, DOE=Department of Energy, DOI= Department of Interior, DOL=Department of Labor, FL=Florida, GA=Georgia, IA=Iowa, IA/GC=Non-Departmental Agencies, ID=Idaho, IL=Illinois, IN=Indiana, KS=Kansas, KY=Kentucky, LA=Louisiana, MA=Massachusetts, MD=Maryland, ME=Maine, MI=Michigan, MN=Minnesota, MO=Missouri, MS=Mississippi, MT=Montana, NC=North Carolina, NE=Nebraska, NG=Non-Government, NH=New Hampshire, NJ=New Jersey, NM=New Mexico, NV=Nevada, NY=New York, OH=Ohio, OK=Oklahoma, OR=Oregon, PA=Pennsylvania, PR=Puerto Rico, RI=Rhode Island, SC=South Carolina, SD=South Dakota, ST/L=State or Local Government, TN=Tennessee, Tribe=Tribe, TX=Texas, USDA=Department of Agriculture, UT=Utah, VA=Virginia, VI=U. S. Virgin Islands, VT=Vermont, WA=Washington, WI=Wisconsin, WV=West Virginia, WY=Wyoming). | |
| Agency = Agency or bureau to which the unit belongs (AG=Air Guard, ANC=Alaska Native Corporation, BIA=Bureau of Indian Affairs, BLM=Bureau of Land Management, BOEM=Bureau of Ocean Energy Management, BOR=Bureau of Reclamation, BSEE=Bureau of Safety and Environmental Enforcement, C&L=County & Local, CDF=California Department of Forestry & Fire Protection, DC=Department of Corrections, DFE=Division of Forest Environment, DFF=Division of Forestry Fire & State Lands, DFL=Division of Forests and Land, DFR=Division of Forest Resources, DL=Department of Lands, DNR=Department of Natural Resources, DNRC=Department of Natural Resources and Conservation, DNRF=Department of Natural Resources Forest Service, DOA=Department of Agriculture, DOC=Department of Conservation, DOE=Department of Energy, DOF=Department of Forestry, DVF=Division of Forestry, DWF=Division of Wildland Fire, EPA=Environmental Protection Agency, FC=Forestry Commission, FEMA=Federal Emergency Management Agency, FFC=Bureau of Forest Fire Control, FFP=Forest Fire Protection, FFS=Forest Fire Service, FR=Forest Rangers, FS=Forest Service, FWS=Fish & Wildlife Service, HQ=Headquarters, JC=Job Corps, NBC=National Business Center, NG=National Guard, NNSA=National Nuclear Security Administration, NPS=National Park Service, NWS=National Weather Service, OES=Office of Emergency Services, PRI=Private, SF=State Forestry, SFS=State Forest Service, SP=State Parks, TNC=The Nature Conservancy, USA=United States Army, USACE=United States Army Corps of Engineers, USAF=United States Air Force, USGS=United States Geological Survey, USN=United States Navy). | |
| Parent = Agency subgroup to which the unit belongs (A concatenation of State and Unit from this report - https://www.nifc.blm.gov/unit_id/publish/UnitIdReport.rtf). | |
| Country = Country in which the unit is located (e.g. US = United States). | |
| State = Two-letter code for the state in which the unit is located (or primarily affiliated). | |
| Code = Unit code (follows state code to create UnitId). | |
| Name = Unit name. | |
| Acknowledgements: | |
| These data were collected using funding from the U.S. Government and can be used without additional permissions or fees. If you use these data in a publication, presentation, or other research product please use the following citation: | |
| Short, Karen C. 2017. Spatial wildfire occurrence data for the United States, 1992-2015 [FPA_FOD_20170508]. 4th Edition. Fort Collins, CO: Forest Service Research Data Archive. https://doi.org/10.2737/RDS-2013-0009.4 | |
| Inspiration: | |
| Have wildfires become more or less frequent over time? | |
| What counties are the most and least fire-prone? | |
| Given the size, location and date, can you predict the cause of a fire wildfire?" | |
| Trending YouTube Video Statistics and Comments,"Daily statistics (views, likes, category, comments+) for trending YouTube videos",Mitchell J,77,"Version 24,2017-10-26|Version 23,2017-10-22|Version 22,2017-10-20|Version 21,2017-10-17|Version 20,2017-10-16|Version 19,2017-10-12|Version 18,2017-10-10|Version 17,2017-10-07|Version 16,2017-10-05|Version 15,2017-10-03|Version 14,2017-10-02|Version 13,2017-09-28|Version 12,2017-09-27|Version 11,2017-09-25|Version 10,2017-09-24|Version 9,2017-09-23|Version 8,2017-09-22|Version 7,2017-09-21|Version 6,2017-09-21|Version 5,2017-09-20|Version 4,2017-09-18|Version 3,2017-09-16|Version 2,2017-09-16|Version 1,2017-09-16","languages | |
| popular culture | |
| statistics | |
| + 2 more...",CSV,149 MB,CC0,"26,908 views","3,111 downloads",10 kernels,,https://www.kaggle.com/datasnaek/youtube,"UPDATED: https://www.kaggle.com/datasnaek/youtube-new | |
| Plan | |
| Data collected from the (up to) 200 listed trending YouTube videos every day in the US and the UK. | |
| Description | |
| The dataset includes data gathered from videos on YouTube that are contained within the trending category each day. | |
| There are two kinds of data files, one includes comments and one includes video statistics. They are linked by the unique video_id field. | |
| The headers in the video file are: | |
| video_id (Common id field to both comment and video csv files) | |
| title | |
| channel_title | |
| category_id (Can be looked up using the included JSON files, but varies per region so use the appropriate JSON file for the CSV file's country) | |
| tags (Separated by | character, [none] is displayed if there are no tags) | |
| views | |
| likes | |
| dislikes | |
| thumbnail_link | |
| date (Formatted like so: [day].[month]) | |
| The headers in the comments file are: | |
| video_id (Common id field to both comment and video csv files) | |
| comment_text | |
| likes | |
| replies | |
| Extra info: The YouTube API is not effective at formatting comments by relevance, although it claims to do so. As a result, the most relevant comments do not align with the top comments at all, they aren't even sorted by likes or replies. | |
| Inspiration | |
| Possible uses for this dataset could include: | |
| Sentiment analysis in a variety of forms | |
| Categorising YouTube videos based on their comments and statistics. | |
| Training ML algorithms to generate their own YouTube comments. | |
| Analysing what factors affect how popular a YouTube video will be. | |
| Although there are likely many more possibilities, including analysis of changes over time etc." | |
| 2013 American Community Survey,Find insights in the 2013 American Community Survey,US Census Bureau,77,"Version 2,2017-05-02|Version 1,2015-07-18","social groups | |
| demographics | |
| sociology",CSV,4 GB,CC0,"39,307 views","3,187 downloads",662 kernels,23 topics,https://www.kaggle.com/census/2013-american-community-survey,"The American Community Survey is an ongoing survey from the US Census Bureau. In this survey, approximately 3.5 million households per year are asked detailed questions about who they are and how they live. Many topics are covered, including ancestry, education, work, transportation, internet use, and residency. | |
| The responses reveal a fascinating, granular snapshot into the lives of many Americans. | |
| We''re publishing this data on scripts to make it easy for you to explore this rich dataset, share your work, and collaborate with other data scientists. No data download or local environment needed! We''ve also added shapefiles to simplify publishing maps. | |
| What surprising insights can you find in this data? We look forward to seeing and sharing what you discover on scripts! | |
| Data Description | |
| Here''s a data dictionary. | |
| There are two types of survey data provided, housing and population. | |
| For the housing data, each row is a housing unit, and the characteristics are properties like rented vs. owned, age of home, etc. | |
| For the population data, each row is a person and the characteristics are properties like age, gender, whether they work, method/length of commute, etc. | |
| Each data set is divided in two pieces, ""a"" and ""b"" (where ""a"" contains states 1 to 25 and ""b"" contains states 26 to 50). | |
| Both data sets have weights associated with them. Weights are included to account for the fact that individuals are not sampled with equal probably (people who have a greater chance of being sampled have a lower weight to reflect this). | |
| Weight variable for the housing data: WGTP | |
| Weight variable for the population data: PWGTP | |
| In Kaggle Scripts, these files can be accessed at: | |
| ../input/pums/ss13husa.csv (housing, a) | |
| ../input/pums/ss13husb.csv (housing, b) | |
| ../input/pums/ss13pusa.csv (population, a) | |
| ../input/pums/ss13pusb.csv (population, b) | |
| You can download the data from the census website: | |
| housing | |
| population | |
| In scripts, they are accessed at: | |
| ../input/shapefiles/pums/tl_2013_[state]_puma10.[extension]. | |
| The shapefiles can also be downloaded here. | |
| DataCamp and Kaggle have teamed up to bring you the basics of Data Exploration With Kaggle Scripts. Take the free, interactive course here and start building your data science portfolio." | |
| How ISIS Uses Twitter,Analyze how ISIS fanboys have been using Twitter since 2015 Paris Attacks,Fifth Tribe,77,"Version 4,2016-05-18|Version 3,2016-05-16|Version 2,2016-05-14|Version 1,2016-05-14","crime | |
| internet",CSV,6 MB,CC0,"52,509 views","4,804 downloads",99 kernels,12 topics,https://www.kaggle.com/fifthtribe/how-isis-uses-twitter,"We scraped over 17,000 tweets from 100+ pro-ISIS fanboys from all over the world since the November 2015 Paris Attacks. We are working with content producers and influencers to develop effective counter-messaging measures against violent extremists at home and abroad. In order to maximize our impact, we need assistance in quickly analyzing message frames. | |
| The dataset includes the following: | |
| Name | |
| Username | |
| Description | |
| Location | |
| Number of followers at the time the tweet was downloaded | |
| Number of statuses by the user when the tweet was downloaded | |
| Date and timestamp of the tweet | |
| The tweet itself | |
| Based on this data, here are some useful ways of deriving insights and analysis: | |
| Social Network Cluster Analysis: Who are the major players in the pro-ISIS twitter network? Ideally, we would like this visualized via a cluster network with the biggest influencers scaled larger than smaller influencers. | |
| Keyword Analysis: Which keywords derived from the name, username, description, location, and tweets were the most commonly used by ISIS fanboys? Examples include: ""baqiyah"", ""dabiq"", ""wilayat"", ""amaq"" | |
| Data Categorization of Links: Which websites are pro-ISIS fanboys linking to? Categories include: Mainstream Media, Altermedia, Jihadist Websites, Image Upload, Video Upload, | |
| Sentiment Analysis: Which clergy do pro-ISIS fanboys quote the most and which ones do they hate the most? Search the tweets for names of prominent clergy and classify the tweet as positive, negative, or neutral and if negative, include the reasons why. Examples of clergy they like the most: ""Anwar Awlaki"", ""Ahmad Jibril"", ""Ibn Taymiyyah"", ""Abdul Wahhab"". Examples of clergy that they hate the most: ""Hamza Yusuf"", ""Suhaib Webb"", ""Yaser Qadhi"", ""Nouman Ali Khan"", ""Yaqoubi"". | |
| Timeline View: Visualize all the tweets over a timeline and identify peak moments | |
| Further Reading: ""ISIS Has a Twitter Strategy and It is Terrifying [Infographic]"" | |
| About Fifth Tribe | |
| Fifth Tribe is a digital agency based out of DC that serves businesses, non-profits, and government agencies. We provide our clients with product development, branding, web/mobile development, and digital marketing services. Our client list includes Oxfam, Ernst and Young, Kaiser Permanente, Aetna Innovation Health, the U.S. Air Force, and the U.S. Peace Corps. Along with Goldman Sachs International and IBM, we serve on the Private Sector Committee of the Board of the Global Community Engagement and Resilience Fund (GCERF), the first global effort to support local, community-level initiatives aimed at strengthening resilience against violent extremism. In December 2014, we won the anti-ISIS ""Hedaya Hack"" organized by Affinis Labs and hosted at the ""Global Countering Violent Extremism (CVE) Expo "" in Abu Dhabi. Since then, we've been actively involved in working with the open-source community and community content producers in developing counter-messaging campaigns and tools." | |
| Celebrity Deaths,All wikipedia-listed celebrity deaths from 2006,HugoDarwood,76,"Version 4,2017-01-14|Version 3,2016-12-31|Version 2,2016-12-29|Version 1,2016-12-29","biography | |
| celebrity | |
| death",CSV,2 MB,CC4,"24,667 views","2,487 downloads",136 kernels,8 topics,https://www.kaggle.com/hugodarwood/celebrity-deaths,"Context | |
| I created this dataset to investigate the claim that 2016 had an unnaturally large number of celebrity deaths. | |
| Content | |
| Points listed by Name, Age, Cause of death and Reason for fame | |
| Acknowledgements | |
| Lifted from: https://en.wikipedia.org/wiki/Deaths_in_2016 for all years" | |
| Hospital Charges for Inpatients,How inpatient hospital charges can differ among different providers in the US,Pranay Aryal,75,"Version 2,2016-09-19|Version 1,2016-09-19","healthcare | |
| finance",CSV,26 MB,Other,"22,926 views","2,887 downloads",71 kernels,6 topics,https://www.kaggle.com/speedoheck/inpatient-hospital-charges,"Variation of hospital charges in the various hospitals in the US for the top 100 diagnoses. | |
| The dataset is owned by the US government. It is freely available on data.gov The dataset keeps getting updated periodically here | |
| This dataset will show you how price for the same diagnosis and the same treatment and in the same city can vary differently across different providers. It might help you or your loved one find a better hospital for your treatment. You can also analyze to detect fraud among providers." | |
| Kickstarter Project Statistics,4000 live projects plus 4000 most backed projects,Cathie So,75,"Version 1,2016-11-01",finance,CSV,3 MB,CC4,"30,850 views","3,817 downloads",104 kernels,2 topics,https://www.kaggle.com/socathie/kickstarter-project-statistics,"Crowdfunding has become one of the main sources of initial capital for small businesses and start-up companies that are looking to launch their first products. Websites like Kickstarter and Indiegogo provide a platform for millions of creators to present their innovative ideas to the public. This is a win-win situation where creators could accumulate initial fund while the public get access to cutting-edge prototypical products that are not available in the market yet. | |
| At any given point, Indiegogo has around 10,000 live campaigns while Kickstarter has 6,000. It has become increasingly difficult for projects to stand out of the crowd. Of course, advertisements via various channels are by far the most important factor to a successful campaign. However, for creators with a smaller budget, this leaves them wonder, | |
| ""How do we increase the probability of success of our campaign starting from the very moment we create our project on these websites?"" | |
| Data Sources | |
| All of my raw data are scraped from Kickstarter.com. | |
| First 4000 live projects that are currently campaigning on Kickstarter (live.csv) | |
| Last updated: 2016-10-29 5pm PDT | |
| amt.pledged: amount pledged (float) | |
| blurb: project blurb (string) | |
| by: project creator (string) | |
| country: abbreviated country code (string of length 2) | |
| currency: currency type of amt.pledged (string of length 3) | |
| end.time: campaign end time (string ""YYYY-MM-DDThh:mm:ss-TZD"") | |
| location: mostly city (string) | |
| pecentage.funded: unit % (int) | |
| state: mostly US states (string of length 2) and others (string) | |
| title: project title (string) | |
| type: type of location (string: County/Island/LocalAdmin/Suburb/Town/Zip) | |
| url: project url after domain (string) | |
| Top 4000 most backed projects ever on Kickstarter (most_backed.csv) | |
| Last updated: 2016-10-30 10pm PDT | |
| amt.pledged | |
| blurb | |
| by | |
| category: project category (string) | |
| currency | |
| goal: original pledge goal (float) | |
| location | |
| num.backers: total number of backers (int) | |
| num.backers.tier: number of backers corresponds to the pledge amount in pledge.tier (int[len(pledge.tier)]) | |
| pledge.tier: pledge tiers in USD (float[]) | |
| title | |
| url | |
| See more at http://datapolymath.paperplane.io/" | |
| Airlines Delay,Airline on-time statistics and delay causes,Giovanni Gonzalez,75,"Version 2,2016-11-12|Version 1,2016-11-12",aviation,Other,239 MB,Other,"41,946 views","6,056 downloads",53 kernels,4 topics,https://www.kaggle.com/giovamata/airlinedelaycauses,"The U.S. Department of Transportation's (DOT) Bureau of Transportation Statistics (BTS) tracks the on-time performance of domestic flights operated by large air carriers. Summary information on the number of on-time, delayed, canceled and diverted flights appears in DOT's monthly Air Travel Consumer Report, published about 30 days after the month's end, as well as in summary tables posted on this website. BTS began collecting details on the causes of flight delays in June 2003. Summary statistics and raw data are made available to the public at the time the Air Travel Consumer Report is released. | |
| This version of the dataset was compiled from the Statistical Computing Statistical Graphics 2009 Data Expo and is also available here." | |
| League of Legends,"Competitive matches, 2015 to 2018",Chuck Ephron,75,"Version 7,2018-01-30|Version 6,2018-01-09|Version 5,2017-05-25|Version 4,2017-04-27|Version 3,2017-04-06|Version 2,2017-04-06|Version 1,2017-04-04",video games,CSV,30 MB,CC0,"21,150 views","2,875 downloads",46 kernels,6 topics,https://www.kaggle.com/chuckephron/leagueoflegends,"League of Legends competitive matches between 2015-2017. The matches include the NALCS, EULCS, LCK, LMS, and CBLoL leagues as well as the World Championship and Mid-Season Invitational tournaments." | |
| A Million News Headlines,News headlines published over a period of 14 years.,Rohk,75,"Version 6,2018-01-02|Version 5,2017-10-11|Version 4,2017-08-31|Version 3,2017-08-18|Version 2,2017-07-30|Version 1,2017-07-23","news agencies | |
| historiography | |
| linguistics | |
| sociology",CSV,19 MB,CC4,"14,723 views","1,281 downloads",23 kernels,4 topics,https://www.kaggle.com/therohk/million-headlines,"Context | |
| This contains data of news headlines published over a period of 15 years. From the reputable Australian news source ABC (Australian Broadcasting Corp.) | |
| Site: http://www.abc.net.au/ | |
| Prepared by Rohit Kulkarni | |
| Content | |
| Format: CSV Rows: 1,103,665 | |
| Column 1: publish_date (yyyyMMdd format) | |
| Column 2: headline_text (ascii, lowercase) | |
| Start Date: 2003-02-19 End Date: 2017-12-31 | |
| Acknowledgements | |
| Special thanks to the java jsoup library. | |
| This dataset is free to use with citation: | |
| Rohit Kulkarni (2017), A Million News Headlines [CSV Data file], doi:10.7910/DVN/SYBGZL, Retrieved from: [this url] | |
| Inspiration | |
| I look at this news dataset as a summarised historical record of noteworthy events in the globe from early-2003 to end-2017 with a more granular focus on Australia. | |
| This includes the entire corpus of articles published by the ABC website in the given time range. With a volume of 200 articles per day and a good focus on international news, we can be fairly certain that every event of significance has been captured here. | |
| Digging into the keywords, one can see all the important episodes shaping the last decade and how they evolved over time. Ex: financial crisis, iraq war, multiple US elections, ecological disasters, terrorism, famous people, Australian crimes etc. | |
| Similar Work | |
| Your kernals can be reused with minimal changes across all these datasets | |
| 3M Clickbait Headlines for 6 years: Examine the Examiner | |
| 1.3M Global Headlines from 20K sources over 1 week: Global News Week | |
| 2.6M News Headlines from India from 2001-2017: Headlines of India" | |
| 1 million Sudoku games,1 million numpy array pairs of Sudoku games and solutions,Kyubyong Park,74,"Version 3,2016-12-29|Version 2,2016-12-28|Version 1,2016-12-27",games and toys,CSV,156 MB,CC0,"28,707 views","1,584 downloads",20 kernels,3 topics,https://www.kaggle.com/bryanpark/sudoku,"Context | |
| Sudoku is a popular number puzzle that requires you to fill blanks in a 9X9 grid with digits so that each column, each row, and each of the nine 3×3 subgrids contains all of the digits from 1 to 9. Sudoku-solving has gained much attention from various fields. As a deep learning researcher, I was inclined to investigate the possibilities of neural networks solving Sudoku. This dataset was prepared for that. | |
| Content | |
| There are dozens of source codes to generate Sudoku games available. I picked one of them, and ran the code. It took approximately 6 hours to generate 1 million games ( + solutions). | |
| A Sudoku puzzle is represented as a 9x9 Python numpy array. The blanks were replaced with 0's. You can easily load and explore the data by running this. | |
| import numpy as np | |
| quizzes = np.load('sudoku_quizzes.npy') # shape = (1000000, 9, 9) | |
| solutions = np.load('sudoku_solutions.npy') # shape = (1000000, 9, 9) | |
| for quiz, solution in zip(quizzes[:10], solutions[:10]): | |
| print(quiz) | |
| print(solution) | |
| ** Updates for Version 3. ** | |
| I converted NumPy arrays to csv so they are easily accessible, irrespective of language. In each line, a Sudoku quiz and its corresponding solution are separated by a comma. You can restore the csv file content to Numpy arrays if needed as follows: | |
| import numpy as np | |
| quizzes = np.zeros((1000000, 81), np.int32) | |
| solutions = np.zeros((1000000, 81), np.int32) | |
| for i, line in enumerate(open('sudoku.csv', 'r').read().splitlines()[1:]): | |
| quiz, solution = line.split("","") | |
| for j, q_s in enumerate(zip(quiz, solution)): | |
| q, s = q_s | |
| quizzes[i, j] = q | |
| solutions[i, j] = s | |
| quizzes = quizzes.reshape((-1, 9, 9)) | |
| solutions = solutions.reshape((-1, 9, 9)) | |
| Acknowledgements | |
| I'm grateful to Arel Cordero, who wrote and shared this great Sudoku generation code. https://www.ocf.berkeley.edu/~arel/sudoku/main.html. | |
| Inspiration | |
| Check https://github.com/Kyubyong/sudoku to see if CNNs can crack Sudoku puzzles. | |
| Also, reinforcement learning can be a promising alternative to this task. | |
| Feel free to challenge Sudoku puzzles." | |
| U.S. Opiate Prescriptions/Overdoses,Can you save lives through predictive modeling?,"Alan ""AJ"" Pryor",74,"Version 2,2016-10-24|Version 1,2016-10-23",healthcare,CSV,14 MB,CC0,"22,101 views","3,366 downloads",41 kernels,4 topics,https://www.kaggle.com/apryor6/us-opiate-prescriptions,"U.S. Opiate Prescriptions | |
| Accidental death by fatal drug overdose is a rising trend in the United States. What can you do to help? | |
| This dataset contains summaries of prescription records for 250 common opioid and non-opioid drugs written by 25,000 unique licensed medical professionals in 2014 in the United States for citizens covered under Class D Medicare as well as some metadata about the doctors themselves. This is a small subset of data that was sourced from cms.gov. The full dataset contains almost 24 million prescription instances in long format. I have cleaned and compiled this data here in a format with 1 row per prescriber and limited the approximately 1 million total unique prescribers down to 25,000 to keep it manageable. If you are interested in more data, you can get the script I used to assemble the dataset here and run it yourself. The main data is in prescriber-info.csv. There is also opioids.csv that contains the names of all opioid drugs included in the data and overdoses.csv that contains information on opioid related drug overdose fatalities. | |
| The increase in overdose fatalities is a well-known problem, and the search for possible solutions is an ongoing effort. My primary interest in this dataset is detecting sources of significant quantities of opiate prescriptions. However, there is plenty of other studies to perform, and I am interested to see what other Kagglers will come up with, or if they can improve the model I have already built. | |
| The data consists of the following characteristics for each prescriber | |
| NPI – unique National Provider Identifier number | |
| Gender - (M/F) | |
| State - U.S. State by abbreviation | |
| Credentials - set of initials indicative of medical degree | |
| Specialty - description of type of medicinal practice | |
| A long list of drugs with numeric values indicating the total number of prescriptions written for the year by that individual | |
| Opioid.Prescriber - a boolean label indicating whether or not that individual prescribed opiate drugs more than 10 times in the year" | |
| US Consumer Finance Complaints,US consumer complaints on financial products and company responses,Consumer Financial Protection Bureau,73,"Version 1,2016-04-27",finance,CSV,361 MB,Other,"28,190 views","3,448 downloads",76 kernels,,https://www.kaggle.com/cfpb/us-consumer-finance-complaints,"Each week the CFPB sends thousands of consumers’ complaints about financial products and services to companies for response. Those complaints are published here after the company responds or after 15 days, whichever comes first. By adding their voice, consumers help improve the financial marketplace." | |
| Demonetization in India Twitter Data,Data extracted from Twitter regarding the recent currency demonetization,Amandeep Rathee,72,"Version 3,2017-04-22|Version 2,2016-11-24|Version 1,2016-11-24","finance | |
| human-computer interaction | |
| internet",CSV,5 MB,Other,"30,471 views","3,258 downloads",168 kernels,4 topics,https://www.kaggle.com/arathee2/demonetization-in-india-twitter-data,"Context | |
| The demonetization of ₹500 and ₹1000 banknotes was a step taken by the Government of India on 8 November 2016, ceasing the usage of all ₹500 and ₹1000 banknotes of the Mahatma Gandhi Series as a form of legal tender in India from 9 November 2016. | |
| The announcement was made by the Prime Minister of India Narendra Modi in an unscheduled live televised address to the nation at 20:15 Indian Standard Time (IST) the same day. In the announcement, Modi declared circulation of all ₹500 and ₹1000 banknotes of the Mahatma Gandhi Series as invalid and announced the issuance of new ₹500 and ₹2000 banknotes of the Mahatma Gandhi New Series in exchange for the old banknotes. | |
| Content | |
| The data contains 6000 most recent tweets on #demonetization. There are 6000 rows(one for each tweet) and 14 columns. | |
| Metadata: | |
| Text (Tweets) | |
| favorited | |
| favoriteCount | |
| replyToSN | |
| created | |
| truncated | |
| replyToSID | |
| id | |
| replyToUID | |
| statusSource | |
| screenName | |
| retweetCount | |
| isRetweet | |
| retweeted | |
| Acknowledgement | |
| The data was collected using the ""twitteR"" package in R using the twitter API. | |
| Past Research | |
| I have performed my own analysis on the data. I only did a sentiment analysis and formed a word cloud. | |
| Click here to see the analysis on GitHub | |
| Inspiration | |
| What percentage of tweets are negative, positive or neutral ? | |
| What are the most famous/re-tweeted tweets ?" | |
| Open Exoplanet Catalogue,Characteristics of all discovered extrasolar planets,Megan Risdal,71,"Version 2,2017-06-09|Version 1,2016-09-11","astronomy | |
| space",CSV,455 KB,Other,"15,670 views","1,541 downloads",90 kernels,,https://www.kaggle.com/mrisdal/open-exoplanet-catalogue,"Our first glimpse at planets outside of the solar system we call home came in 1992 when several terrestrial-mass planets were detected orbiting the pulsar PSR B1257+12. In this dataset, you can become a space explorer too by analyzing the characteristics of all discovered exoplanets (plus some familiar faces like Mars, Saturn, and even Earth). Data fields include planet and host star attributes, discovery methods, and (of course) date of discovery. | |
| Data was originally collected and continues to be updated by Hanno Rein at the Open Exoplanet Catalogue Github repository. If you discover any new exoplanets, please submit a pull request there. | |
| Constants | |
| Jupiter mass: 1.8991766e+27 kg | |
| Solar mass: 1.9891e+30 kg | |
| Jupiter radius: 69911000 m | |
| Solar radius: 6.96e+08 m | |
| License | |
| The database is licensed under an MIT license. If you use it for a scientific publication, please include a reference to the Open Exoplanet Catalogue on GitHub or to this arXiv paper." | |
| Diamonds,"Analyze diamonds by their cut, color, clarity, price, and other attributes",shivamagrawal,71,"Version 1,2017-05-25","clothing | |
| finance",CSV,3 MB,Other,"19,685 views","3,349 downloads",53 kernels,3 topics,https://www.kaggle.com/shivam2503/diamonds,"Context | |
| This classic dataset contains the prices and other attributes of almost 54,000 diamonds. It's a great dataset for beginners learning to work with data analysis and visualization. | |
| Content | |
| price price in US dollars (\$326--\$18,823) | |
| carat weight of the diamond (0.2--5.01) | |
| cut quality of the cut (Fair, Good, Very Good, Premium, Ideal) | |
| color diamond colour, from J (worst) to D (best) | |
| clarity a measurement of how clear the diamond is (I1 (worst), SI2, SI1, VS2, VS1, VVS2, VVS1, IF (best)) | |
| x length in mm (0--10.74) | |
| y width in mm (0--58.9) | |
| z depth in mm (0--31.8) | |
| depth total depth percentage = z / mean(x, y) = 2 * z / (x + y) (43--79) | |
| table width of top of diamond relative to widest point (43--95)" | |
| Red Wine Quality,Simple and clean practice dataset for regression or classification modelling,UCI Machine Learning,71,"Version 2,2017-11-28|Version 1,2017-11-12","food and drink | |
| beginner | |
| regression analysis",CSV,99 KB,ODbL,"15,724 views","3,359 downloads",32 kernels,4 topics,https://www.kaggle.com/uciml/red-wine-quality-cortez-et-al-2009,"Context | |
| The two datasets are related to red and white variants of the Portuguese ""Vinho Verde"" wine. For more details, consult the reference [Cortez et al., 2009]. Due to privacy and logistic issues, only physicochemical (inputs) and sensory (the output) variables are available (e.g. there is no data about grape types, wine brand, wine selling price, etc.). | |
| These datasets can be viewed as classification or regression tasks. The classes are ordered and not balanced (e.g. there are much more normal wines than excellent or poor ones). | |
| This dataset is also available from the UCI machine learning repository, https://archive.ics.uci.edu/ml/datasets/wine+quality , I just shared it to kaggle for convenience. (If I am mistaken and the public license type disallowed me from doing so, I will take this down if requested.) | |
| Content | |
| For more information, read [Cortez et al., 2009]. | |
| Input variables (based on physicochemical tests): | |
| 1 - fixed acidity | |
| 2 - volatile acidity | |
| 3 - citric acid | |
| 4 - residual sugar | |
| 5 - chlorides | |
| 6 - free sulfur dioxide | |
| 7 - total sulfur dioxide | |
| 8 - density | |
| 9 - pH | |
| 10 - sulphates | |
| 11 - alcohol | |
| Output variable (based on sensory data): | |
| 12 - quality (score between 0 and 10) | |
| Tips | |
| What might be an interesting thing to do, is aside from using regression modelling, is to set an arbitrary cutoff for your dependent variable (wine quality) at e.g. 7 or higher getting classified as 'good/1' and the remainder as 'not good/0'. This allows you to practice with hyper parameter tuning on e.g. decision tree algorithms looking at the ROC curve and the AUC value. Without doing any kind of feature engineering or overfitting you should be able to get an AUC of .88 (without even using random forest algorithm) | |
| KNIME is a great tool (GUI) that can be used for this. | |
| 1 - File Reader (for csv) to linear correlation node and to interactive histogram for basic EDA. | |
| 2- File Reader to 'Rule Engine Node' to turn the 10 point scale to dichtome variable (good wine and rest), the code to put in the rule engine is something like this: | |
| - $quality$ > 6.5 => ""good"" | |
| - TRUE => ""bad"" | |
| 3- Rule Engine Node output to input of Column Filter node to filter out your original 10point feature (this prevent leaking) | |
| 4- Column Filter Node output to input of Partitioning Node (your standard train/tes split, e.g. 75%/25%, choose 'random' or 'stratified') | |
| 5- Partitioning Node train data split output to input of Train data split to input Decision Tree Learner node and | |
| 6- Partitioning Node test data split output to input Decision Tree predictor Node | |
| 7- Decision Tree learner Node output to input Decision Tree Node input | |
| 8- Decision Tree output to input ROC Node.. (here you can evaluate your model base on AUC value) | |
| Inspiration | |
| Use machine learning to determine which physiochemical properties make a wine 'good'! | |
| Acknowledgements | |
| This dataset is also available from the UCI machine learning repository, https://archive.ics.uci.edu/ml/datasets/wine+quality , I just shared it to kaggle for convenience. (I am mistaken and the public license type disallowed me from doing so, I will take this down at first request. I am not the owner of this dataset. | |
| Please include this citation if you plan to use this database: P. Cortez, A. Cerdeira, F. Almeida, T. Matos and J. Reis. Modeling wine preferences by data mining from physicochemical properties. In Decision Support Systems, Elsevier, 47(4):547-553, 2009. | |
| Relevant publication | |
| P. Cortez, A. Cerdeira, F. Almeida, T. Matos and J. Reis. Modeling wine preferences by data mining from physicochemical properties. In Decision Support Systems, Elsevier, 47(4):547-553, 2009." | |
| Keras Pretrained models,This dataset helps to use pretrained keras models in Kernels.,beluga,70,"Version 11,2017-11-17|Version 10,2017-11-17|Version 9,2017-10-13|Version 8,2017-10-13|Version 7,2017-10-05|Version 6,2017-10-04|Version 5,2017-10-04|Version 4,2017-10-04|Version 3,2017-10-04|Version 2,2017-10-04|Version 1,2017-10-03","artificial intelligence | |
| pre-trained model",Other,943 MB,CC4,"7,870 views","1,401 downloads",27 kernels,,https://www.kaggle.com/gaborfodor/keras-pretrained-models,"Context | |
| Kaggle has more and more computer vision challenges. Although Kernel resources were increased recently we still can not train useful CNNs without GPU. The other main problem is that Kernels can't use network connection to download pretrained keras model weights. This dataset helps you to apply your favorite pretrained model in the Kaggle Kernel environment. | |
| Happy data exploration and transfer learning! | |
| Content | |
| Model (Top-1 Accuracy | Top -5 Accuracy) | |
| Xception (0.790 | 0.945) | |
| VGG16 (0.715 | 0.901) | |
| VGG19 (0.727 | 0.910) | |
| ResNet50 (0.759 | 0.929) | |
| InceptionV3 (0.788 | 0.944) | |
| InceptionResNetV2 (0.804 | 0.953) (could not upload due to 500 MB limit) | |
| For more information see https://keras.io/applications/ | |
| Acknowledgements | |
| Thanks to François Chollet for collecting these models and for the awesome keras." | |
| "Significant Earthquakes, 1965-2016","Date, time, and location of all earthquakes with magnitude of 5.5 or higher",US Geological Survey,70,"Version 1,2017-01-27",earth sciences,CSV,2 MB,CC0,"24,603 views","4,104 downloads",121 kernels,,https://www.kaggle.com/usgs/earthquake-database,"Context | |
| The National Earthquake Information Center (NEIC) determines the location and size of all significant earthquakes that occur worldwide and disseminates this information immediately to national and international agencies, scientists, critical facilities, and the general public. The NEIC compiles and provides to scientists and to the public an extensive seismic database that serves as a foundation for scientific research through the operation of modern digital national and global seismograph networks and cooperative international agreements. The NEIC is the national data center and archive for earthquake information. | |
| Content | |
| This dataset includes a record of the date, time, location, depth, magnitude, and source of every earthquake with a reported magnitude 5.5 or higher since 1965. | |
| Start a new kernel" | |
| 2016 New Coder Survey,"A survey of 15,000+ people who are new to software development",Free Code Camp,70,"Version 1,2016-06-03","employment | |
| computing and society",CSV,10 MB,ODbL,"31,112 views","3,282 downloads",159 kernels,6 topics,https://www.kaggle.com/freecodecamp/2016-new-coder-survey-,"Context | |
| Free Code Camp is an open source community where you learn to code and build projects for nonprofits. CodeNewbie.org is the most supportive community of people learning to code. Together, we surveyed more than 15,000 people who are actively learning to code. We reached them through the twitter accounts and email lists of various organizations that help people learn to code. Our goal was to understand these people's motivations in learning to code, how they're learning to code, their demographics, and their socioeconomic background. We've written in depth about this dataset. | |
| In May 2017 we just released an even bigger open dataset with our 2017 survey results." | |
| Twitter User Gender Classification,Predict user gender based on Twitter profile information,Crowdflower,70,"Version 1,2016-11-21","gender | |
| internet",CSV,8 MB,CC0,"32,808 views","3,851 downloads",61 kernels,5 topics,https://www.kaggle.com/crowdflower/twitter-user-gender-classification,"This data set was used to train a CrowdFlower AI gender predictor. You can read all about the project here. Contributors were asked to simply view a Twitter profile and judge whether the user was a male, a female, or a brand (non-individual). The dataset contains 20,000 rows, each with a user name, a random tweet, account profile and image, location, and even link and sidebar color. | |
| Inspiration | |
| Here are a few questions you might try to answer with this dataset: | |
| how well do words in tweets and profiles predict user gender? | |
| what are the words that strongly predict male or female gender? | |
| how well do stylistic factors (like link color and sidebar color) predict user gender? | |
| Acknowledgments | |
| Data was provided by the Data For Everyone Library on Crowdflower. | |
| Our Data for Everyone library is a collection of our favorite open data jobs that have come through our platform. They're available free of charge for the community, forever. | |
| The Data | |
| The dataset contains the following fields: | |
| _unit_id: a unique id for user | |
| _golden: whether the user was included in the gold standard for the model; TRUE or FALSE | |
| _unit_state: state of the observation; one of finalized (for contributor-judged) or golden (for gold standard observations) | |
| _trusted_judgments: number of trusted judgments (int); always 3 for non-golden, and what may be a unique id for gold standard observations | |
| _last_judgment_at: date and time of last contributor judgment; blank for gold standard observations | |
| gender: one of male, female, or brand (for non-human profiles) | |
| gender:confidence: a float representing confidence in the provided gender | |
| profile_yn: ""no"" here seems to mean that the profile was meant to be part of the dataset but was not available when contributors went to judge it | |
| profile_yn:confidence: confidence in the existence/non-existence of the profile | |
| created: date and time when the profile was created | |
| description: the user's profile description | |
| fav_number: number of tweets the user has favorited | |
| gender_gold: if the profile is golden, what is the gender? | |
| link_color: the link color on the profile, as a hex value | |
| name: the user's name | |
| profile_yn_gold: whether the profile y/n value is golden | |
| profileimage: a link to the profile image | |
| retweet_count: number of times the user has retweeted (or possibly, been retweeted) | |
| sidebar_color: color of the profile sidebar, as a hex value | |
| text: text of a random one of the user's tweets | |
| tweet_coord: if the user has location turned on, the coordinates as a string with the format ""[latitude, longitude]"" | |
| tweet_count: number of tweets that the user has posted | |
| tweet_created: when the random tweet (in the text column) was created | |
| tweet_id: the tweet id of the random tweet | |
| tweet_location: location of the tweet; seems to not be particularly normalized | |
| user_timezone: the timezone of the user" | |
| CT Medical Image Analysis Tutorial,CT images from cancer imaging archive with contrast and patient age,Kevin Mader,69,"Version 6,2017-05-23|Version 5,2017-05-23|Version 4,2017-04-20|Version 3,2017-03-20|Version 2,2017-03-20|Version 1,2017-03-20","healthcare | |
| tutorial | |
| image data",Other,437 MB,Other,"20,239 views","1,732 downloads",24 kernels,4 topics,https://www.kaggle.com/kmader/siim-medical-image-analysis-tutorial,"Overview | |
| The dataset is designed to allow for different methods to be tested for examining the trends in CT image data associated with using contrast and patient age. The basic idea is to identify image textures, statistical patterns and features correlating strongly with these traits and possibly build simple tools for automatically classifying these images when they have been misclassified (or finding outliers which could be suspicious cases, bad measurements, or poorly calibrated machines) | |
| Data | |
| The data are a tiny subset of images from the cancer imaging archive. They consist of the middle slice of all CT images taken where valid age, modality, and contrast tags could be found. This results in 475 series from 69 different patients. | |
| TCIA Archive Link - https://wiki.cancerimagingarchive.net/display/Public/TCGA-LUAD | |
| License | |
| http://creativecommons.org/licenses/by/3.0/ | |
| After the publication embargo period ends these collections are freely available to browse, download, and use for commercial, scientific and educational purposes as outlined in the Creative Commons Attribution 3.0 Unported License. Questions may be directed to help@cancerimagingarchive.net. Please be sure to acknowledge both this data set and TCIA in publications by including the following citations in your work: | |
| Data Citation | |
| Albertina, B., Watson, M., Holback, C., Jarosz, R., Kirk, S., Lee, Y., … Lemmerman, J. (2016). Radiology Data from The Cancer Genome Atlas Lung Adenocarcinoma [TCGA-LUAD] collection. The Cancer Imaging Archive. http://doi.org/10.7937/K9/TCIA.2016.JGNIHEP5 | |
| TCIA Citation | |
| Clark K, Vendt B, Smith K, Freymann J, Kirby J, Koppel P, Moore S, Phillips S, Maffitt D, Pringle M, Tarbox L, Prior F. The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository, Journal of Digital Imaging, Volume 26, Number 6, December, 2013, pp 1045-1057. (paper)" | |
| International football results from 1872 to 2018,"An up-to-date dataset of nearly 40,000 international football results",Mart Jürisoo,69,"Version 14,2018-01-27|Version 13,2017-11-18|Version 12,2017-11-18|Version 11,2017-11-16|Version 10,2017-11-16|Version 9,2017-11-16|Version 8,2017-11-16|Version 7,2017-11-11|Version 6,2017-11-11|Version 5,2017-11-11|Version 4,2017-11-10|Version 3,2017-11-10|Version 2,2017-11-10|Version 1,2017-11-10","association football | |
| countries | |
| sports | |
| + 2 more...",CSV,475 KB,CC0,"9,293 views","1,394 downloads",8 kernels,4 topics,https://www.kaggle.com/martj42/international-football-results-from-1872-to-2017,"Context | |
| Well, basically what happened was I was looking for a semi-definite easy to read list of international football matches and couldn't find anything decent. So I took it upon myself to collect it for my own use. I might as well share it. | |
| Content | |
| This dataset includes 38,759 results of international football matches starting from the very first official match in 1972 up to 2018. The matches range from World Cup to Baltic Cup to regular friendly matches. The matches are strictly men's full internationals and the data does not include Olympic Games or matches where at least one of the teams was the nation's B-team, U-23 or a league select team. | |
| results.csv includes the following columns: | |
| date | |
| home_team | |
| away_team | |
| home_score | |
| away_score | |
| tournament | |
| city | |
| country | |
| Acknowledgements | |
| The data is gathered from several sources including but not limited to wikipedia, fifa.com, rsssf.com and individual football associations' websites. | |
| Inspiration | |
| Some directions to take when exploring the data: | |
| Which teams dominated different eras of football | |
| Who is the best team of all time | |
| What trends have there been in international football throughout the ages - home advantage, total goals scored, distribution of teams' strength etc | |
| Can we say anything about geopolitics from football fixtures - how has the number of countries changed, which teams like to play each other | |
| Which countries host the most matches where they themselves are not participating in | |
| How much, if at all, does hosting a major tournament help a country's chances in said tournament | |
| Which teams are the most active in playing friendlies and friendly tournaments - does it help or harm them | |
| Do you dare to make any predictions for 2018 World Cup based on this data? | |
| and so on... | |
| The world's your oyster, my friend." | |
| Spotify's Worldwide Daily Song Ranking,The 200 daily most streamed songs in 53 countries,Eduardo,69,"Version 3,2018-01-13|Version 2,2018-01-12|Version 1,2017-08-21",music,CSV,43 MB,Other,"11,139 views","1,522 downloads",5 kernels,0 topics,https://www.kaggle.com/edumucelli/spotifys-worldwide-daily-song-ranking,"Context | |
| Music streaming is ubiquitous. Currently, Spotify plays an important part on that. This dataset enable us to explore how artists and songs' popularity varies in time. | |
| Content | |
| This dataset contains the daily ranking of the 200 most listened songs in 53 countries from 2017 and 2018 by Spotify users. It contains more than 2 million rows, which comprises 6629 artists, 18598 songs for a total count of one hundred five billion streams count. | |
| The data spans from 1st January 2017 to 9th January 2018 and will be kept up-to-date on following versions. It has been collected from Spotify's regional chart data. | |
| Inspiration | |
| Can you predict what is the rank position or the number of streams a song will have in the future? | |
| How long does songs ""resist"" on the top 3, 5, 10, 20 ranking? | |
| What are the signs of a song that gets into the top rank to stay? | |
| Do continents share same top ranking artists or songs? | |
| Are people listening to the very same top ranking songs on countries far away from each other? | |
| How long time does a top ranking song takes to get into the ranking of neighbor countries? | |
| Example | |
| To start out, you can take a look into a simple Kernel I have made in order to read the data, filter data from a song, plot is temporal tendency per country than make a simple forecast of the its streams count here. | |
| Crawler | |
| The crawler used to collect this data can be found here." | |
| Smart meters in London,Smart meter data from London area,Jean-Michel D.,69,"Version 10,2017-12-13|Version 9,2017-12-12|Version 8,2017-11-29|Version 7,2017-11-22|Version 6,2017-11-14|Version 5,2017-11-10|Version 4,2017-11-09|Version 3,2017-11-07|Version 2,2017-11-07|Version 1,2017-11-05","weather | |
| home | |
| demographics | |
| energy",CSV,1 GB,ODbL,"11,025 views","1,060 downloads",2 kernels,6 topics,https://www.kaggle.com/jeanmidev/smart-meters-in-london,"Context | |
| To better follow the energy consumption, the government wants energy suppliers to install smart meters in every home in England, Wales and Scotland. There are more than 26 million homes for the energy suppliers to get to, with the goal of every home having a smart meter by 2020. | |
| This roll out of meter is lead by the European Union who asked all member governments to look at smart meters as part of measures to upgrade our energy supply and tackle climate change. After an initial study, the British government decided to adopt smart meters as part of their plan to update our ageing energy system. | |
| In this dataset, you will find a refactorised version of the data from the London data store, that contains the energy consumption readings for a sample of 5,567 London Households that took part in the UK Power Networks led Low Carbon London project between November 2011 and February 2014. The data from the smart meters seems associated only to the electrical consumption. | |
| There is infomations on the ACORN classification details that you can find in this report or the website of CACI. | |
| I added weather data for London area, I used the darksky api to collect this data. | |
| Content | |
| There is 19 files in this dataset : | |
| informations_households.csv : this file that contains all the information on the households in the panel (their acorn group, their tariff) and in which block.csv.gz file their data are stored | |
| halfhourly_dataset.zip: Zip file that contains the block files with the half-hourly smart meter measurement | |
| daily_dataset.zip: Zip file that contains the block files with the daily information like the number of measures, minimum, maximum, mean, median, sum and std. | |
| acorn_details.csv : Details on the acorn groups and their profile of the people in the group, it's come from this xlsx spreadsheet.The first three columns are the attributes studied, the ACORN-X is the index of the attribute. At a national scale, the index is 100 if for one column the value is 150 it means that there are 1.5 times more people with this attribute in the ACORN group than at the national scale. You can find an explanation on the CACI website | |
| weather_daily_darksky.csv : that contains the daily data from darksky api. You can find more details about the parameters in the documentation of the api | |
| weather_hourly_darksky.csv : that contains the hourly data from darksky api. You can find more details about the parameters in the documentation of the api | |
| Acknowledgements | |
| All the big work of data collection has been done by the UK power networks for the smart meter data. | |
| The details related at the acorn group are provided by the CACI. | |
| The weather data are from darksky. | |
| Inspiration | |
| For me some ideas to analyze the data: | |
| Segmentation of the consumption daily pattern | |
| Disaggregation of the electricity load curve | |
| Cross the consumption result and the acorn information | |
| Forecast the electricity consumption of a household, I wrote an article on this subject | |
| What if I add electrical heating system ? an EV battery system ? | |
| Forecast at a global scale (London consumption)" | |
| LEGO Database,The LEGO Parts/Sets/Colors and Inventories of every official LEGO set,Rachael Tatman,69,"Version 1,2017-07-15",games and toys,CSV,12 MB,CC0,"21,060 views","3,013 downloads",34 kernels,,https://www.kaggle.com/rtatman/lego-database,"Context: | |
| LEGO is a popular brand of toy building bricks. They are often sold in sets with in order to build a specific object. Each set contains a number of parts in different shapes, sizes and colors. This database contains information on which parts are included in different LEGO sets. It was originally compiled to help people who owned some LEGO sets already figure out what other sets they could build with the pieces they had. | |
| Content: | |
| This dataset contains the LEGO Parts/Sets/Colors and Inventories of every official LEGO set in the Rebrickable database. These files are current as of July 2017. If you need it to be more recent data, you can use Rebrickable’s API which provides up to date data, and additional features. | |
| Acknowledgements: | |
| This dataset was compiled by Rebrickable, which is a website to help identify what LEGO sets can be built given bricks and pieces from other LEGO sets. You can use these files for any purpose. | |
| Inspiration: | |
| This is a very rich dataset that offers lots of rooms for exploration, especially since the “sets” file includes the year in which a set was first released. | |
| How have the size of sets changed over time? | |
| What colors are associated with witch themes? Could you predict which theme a set is from just by the bricks it contains? | |
| What sets have the most-used pieces in them? What sets have the rarest pieces in them? | |
| Have the colors of LEGOs included in sets changed over time?" | |
| 1.6 million UK traffic accidents,Visualise and analyse traffic demographics,Dave Fisher-Hickey,69,"Version 10,2017-09-17|Version 9,2017-09-17|Version 8,2017-09-16|Version 7,2017-09-16|Version 6,2017-09-16|Version 5,2017-09-15|Version 4,2017-09-14|Version 3,2017-09-14|Version 2,2017-09-13|Version 1,2017-09-11","climate | |
| automobiles | |
| road transport | |
| taxi services",CSV,621 MB,ODbL,"23,349 views","2,474 downloads",10 kernels,2 topics,https://www.kaggle.com/daveianhickey/2000-16-traffic-flow-england-scotland-wales,"Context | |
| The UK government amassed traffic data from 2000 and 2016, recording over 1.6 million accidents in the process and making this one of the most comprehensive traffic data sets out there. It's a huge picture of a country undergoing change. | |
| Note that all the contained accident data comes from police reports, so this data does not include minor incidents. | |
| Content | |
| ukTrafficAADF.csv tracks how much traffic there was on all major roads in the given time period (2000 through 2016). AADT, the core statistic included in this file, stands for ""Average Annual Daily Flow"", and is a measure of how activity a road segment based on how many vehicle trips traverse it. The AADT page on Wikipedia is a good reference on the subject. | |
| Accidents data is split across three CSV files: accidents_2005_to_2007.csv, accidents_2009_to_2011.csv, and accidents_2012_to_2014.csv. These three files together constitute 1.6 million traffic accidents. The total time period is 2005 through 2014, but 2008 is missing. | |
| A data dictionary for the raw dataset at large is available from the UK Department of Transport website here. For descriptions of individual columns, see the column metadata. | |
| Acknowledgements | |
| The license for this dataset is the Open Givernment Licence used by all data on data.gov.uk (here). The raw datasets are available from the UK Department of Transport website here. | |
| Inspiration | |
| How has changing traffic flow impacted accidents? | |
| Can we predict accident rates over time? What might improve accident rates? | |
| Plot interactive maps of changing trends, e.g. How has London has changed for cyclists? Busiest roads in the nation? | |
| Which areas never change and why? Identify infrastructure needs, failings and successes. | |
| How have Rural and Urban areas differed (see RoadCategory)? How about the differences between England, Scotland, and Wales? | |
| The UK government also like to look at miles driven. You can do this by multiplying the AADF by the corresponding length of road (link length) and by the number of days in the years. What does this tell you about UK roads?" | |
| Adult Census Income,Predict whether income exceeds $50K/yr based on census data,UCI Machine Learning,69,"Version 3,2016-10-08|Version 2,2016-10-08|Version 1,2016-10-07","employment | |
| demographics",CSV,4 MB,CC0,"42,472 views","5,017 downloads",119 kernels,6 topics,https://www.kaggle.com/uciml/adult-census-income,"This data was extracted from the 1994 Census bureau database by Ronny Kohavi and Barry Becker (Data Mining and Visualization, Silicon Graphics). A set of reasonably clean records was extracted using the following conditions: ((AAGE>16) && (AGI>100) && (AFNLWGT>1) && (HRSWK>0)). The prediction task is to determine whether a person makes over $50K a year. | |
| Description of fnlwgt (final weight) | |
| The weights on the Current Population Survey (CPS) files are controlled to independent estimates of the civilian noninstitutional population of the US. These are prepared monthly for us by Population Division here at the Census Bureau. We use 3 sets of controls. These are: | |
| A single cell estimate of the population 16+ for each state. | |
| Controls for Hispanic Origin by age and sex. | |
| Controls by Race, age and sex. | |
| We use all three sets of controls in our weighting program and ""rake"" through them 6 times so that by the end we come back to all the controls we used. The term estimate refers to population totals derived from CPS by creating ""weighted tallies"" of any specified socio-economic characteristics of the population. People with similar demographic characteristics should have similar weights. There is one important caveat to remember about this statement. That is that since the CPS sample is actually a collection of 51 state samples, each with its own probability of selection, the statement only applies within state. | |
| Relevant papers | |
| Ron Kohavi, ""Scaling Up the Accuracy of Naive-Bayes Classifiers: a Decision-Tree Hybrid"", Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, 1996. (PDF)" | |
| Horses For Courses,"Daily horse racing (thoroughbred) data, machine learning for fun and profit",lukebyrne,68,"Version 18,2017-01-11|Version 17,2016-12-12|Version 16,2016-11-21|Version 15,2016-11-07|Version 14,2016-10-17|Version 13,2016-10-17|Version 12,2016-10-05|Version 11,2016-09-27|Version 10,2016-09-21|Version 9,2016-09-12|Version 8,2016-09-05|Version 7,2016-09-05|Version 6,2016-08-29|Version 5,2016-08-29|Version 4,2016-08-29|Version 3,2016-08-25|Version 2,2016-08-22|Version 1,2016-08-20",horse racing,CSV,26 MB,CC4,"36,251 views","3,270 downloads",122 kernels,17 topics,https://www.kaggle.com/lukebyrne/horses-for-courses,"Context: Daily horse racing (thoroughbred) information that has(is) being actively collected and aggregated from a variety of sources. Years covered are just 2016, country is irrelevant to the dataset. | |
| Acknowledgements: This data has(is) being actively collected and aggregated from a variety of sources, all in the public domain. | |
| Past Research: None of merit, data is used currently to influence some betting decisions but no solid machine learning model(s) have been developed. | |
| Have thrown various versions of the data into: | |
| Google Prediction | |
| Amazon Machine Learning | |
| Azure Machine Learning | |
| Watson Analytics | |
| as a way to learn how these systems work. | |
| Inspiration: Probably one of the hardest things to do is pick stocks and horses. I have been involved in the stocks and horses industry for many years and through publishing previous libraries and software I have met many interesting people and also one of my long term clients/friends. | |
| I am currently trying enhance my software development skills by learning data science / machine learning. | |
| I have a done a few tutorials and I am hoping that by publishing this data I can learn and collaborate with members of the Kaggle Community. | |
| Content: | |
| markets.csv | |
| id | |
| start_time | |
| what time did the race start, datetime in UTC | |
| venue_id | |
| race_number | |
| distance(m) | |
| condition_id | |
| track condition, see conditions.csv | |
| weather_id | |
| weather on day, see weathers.csv | |
| total_pool_win_one | |
| rough $ amount wagered across all runners for win market | |
| total_pool_place_one | |
| rough $ amount wagered across all runners for place market | |
| total_pool_win_two | |
| total_pool_place_two | |
| total_pool_win_three | |
| total_pool_place_three | |
| runners.csv | |
| id | |
| collected | |
| what time was this row created/data collected, datetime in UTC | |
| market_id | |
| position | |
| THIS IS THE FIELD WE WANT TO PREDICT!!!! | |
| Will either be 1,2,3,4,5,6 etc or 0/null if the horse was scratched or failed to finish | |
| If all positions for a market_id are null it means we were unable to match up the positional data for this market | |
| place_paid | |
| Will either be 1/0 or null | |
| If you see a race that only has 2 booleans of 1 it means that the race only paid out places on the first two positions | |
| margin | |
| If the runner didnt win, how many lengths behind the 1st place was it | |
| horse_id | |
| see horses.csv | |
| trainer_id | |
| rider_id | |
| see riders.csv | |
| handicap_weight | |
| number | |
| barrier | |
| blinkers | |
| emergency | |
| did it come into the race at the last minute | |
| form_rating_one | |
| form_rating_two | |
| form_rating_three | |
| last_five_starts | |
| favourite_odds_win | |
| from one of the odds sources, will it win - true/false | |
| favourite_odds_place | |
| from one of the odds sources, will it win - true/false | |
| favourite_pool_win | |
| favourite_pool_place | |
| tip_one_win | |
| from a tipster, will it win - true/false | |
| tip_one_place | |
| from a tipster, will it place - true/false | |
| tip_two_win | |
| tip_two_place | |
| tip_three_win | |
| tip_three_place | |
| tip_four_win | |
| tip_four_place | |
| tip_five_win | |
| tip_five_place | |
| tip_six_win | |
| tip_six_place | |
| tip_seven_win | |
| tip_seven_place | |
| tip_eight_win | |
| tip_eight_place | |
| tip_nine_win | |
| tip_nine_place | |
| odds.csv (collected for every runner 10 minutes out from race start until race starts) | |
| runner_id | |
| collected | |
| what time was this row created/data collected, datetime in UTC | |
| odds_one_win | |
| from odds source, win odds | |
| odds_one_win_wagered | |
| from odds source, rough $ amount wagered on win | |
| odds_one_place | |
| from odds source, place odds | |
| odds_one_place_wagered | |
| from odds source, rough $ amount wagered on place | |
| odds_two_win | |
| odds_two_win_wagered | |
| odds_two_place | |
| odds_two_place_wagered | |
| odds_three_win | |
| odds_three_win_wagered | |
| odds_three_place | |
| odds_three_place_wagered | |
| odds_four_win | |
| odds_four_win_wagered | |
| odds_four_place | |
| odds_four_place_wagered | |
| forms.csv | |
| collected | |
| what time was this row created/data collected, datetime in UTC | |
| market_id | |
| horse_id | |
| runner_number | |
| last_twenty_starts | |
| e.g. f9x726x753x92222x35 | |
| f = failed to finish, 7 = finished 7th, 6 = finished 6th, 7 = finished 7th, x = runner was scratched | |
| class_level_id | |
| 1 = eq (in same class as other horses) | |
| 2 = up (up in class) | |
| 3 = dn (down in class) | |
| field_strength | |
| days_since_last_run | |
| runs_since_spell | |
| overall_starts | |
| overall_wins | |
| overall_places | |
| track_starts | |
| track_wins | |
| track_places | |
| firm_starts | |
| firm_wins | |
| firm_places | |
| good_starts | |
| good_wins | |
| good_places | |
| dead_starts | |
| dead_wins | |
| dead_places | |
| slow_starts | |
| slow_wins | |
| slow_places | |
| soft_starts | |
| soft_wins | |
| soft_places | |
| heavy_starts | |
| heavy_wins | |
| heavy_places | |
| distance_starts | |
| distance_wins | |
| distance_places | |
| class_same_starts | |
| class_same_wins | |
| class_same_places | |
| class_stronger_starts | |
| class_stronger_wins | |
| class_stronger_places | |
| first_up_starts | |
| first_up_wins | |
| first_up_places | |
| second_up_starts | |
| second_up_wins | |
| second_up_places | |
| track_distance_starts | |
| track_distance_wins | |
| track_distance_places | |
| conditions.csv | |
| id | |
| name | |
| weathers.csv | |
| id | |
| name | |
| riders.csv (jockeys) | |
| id | |
| sex | |
| horses.csv | |
| id | |
| age | |
| sex_id | |
| see horse_sexes.csv | |
| sire_id | |
| not related to horses.id, there is another table called horse_sires that is not present here | |
| dam_id | |
| not related to horses.id, there is another table called horse_dams that is not present here | |
| prize_money | |
| total aggregate prize money | |
| horse_sexes.csv | |
| id | |
| name" | |
| Cancer Inhibitors,Predict small molecules' activity targeting protein kinase,Kelvin Xiao,68,"Version 4,2016-10-28|Version 3,2016-10-26|Version 2,2016-09-03|Version 1,2016-09-02",healthcare,Other,2 GB,CC4,"18,371 views","1,912 downloads",34 kernels,6 topics,https://www.kaggle.com/xiaotawkaggle/inhibitors,"Outline | |
| It was reported that an estimated 4292,000 new cancer cases and 2814,000 cancer deaths would occur in China in 2015. Chen, W., etc. (2016), Cancer statistics in China, 2015. | |
| Small molecules play an non-trivial role in cancer chemotherapy. Here I focus on inhibitors of 8 protein kinases(name: abbr): | |
| Cyclin-dependent kinase 2: cdk2 | |
| Epidermal growth factor receptor erbB1: egfr_erbB1 | |
| Glycogen synthase kinase-3 beta: gsk3b | |
| Hepatocyte growth factor receptor: hgfr | |
| MAP kinase p38 alpha: map_k_p38a | |
| Tyrosine-protein kinase LCK: tpk_lck | |
| Tyrosine-protein kinase SRC: tpk_src | |
| Vascular endothelial growth factor receptor 2: vegfr2 | |
| For each protein kinase, several thousand inhibitors are collected from chembl database, in which molecules with IC50 lower than 10 uM are usually considered as inhibitors, otherwise non-inhibitors. | |
| Challenge | |
| Based on those labeled molecules, build your model, and try to make the right prediction. | |
| Additionally, more than 70,000 small molecules are generated from pubchem database. And you can screen these molecules to find out potential inhibitors. P.S. the majority of these molecules are non-inhibitors. | |
| DataSets(hdf5 version) | |
| There are 8 protein kinase files and 1 pubchem negative samples file. Taking ""cdk2.h5"" as an example: | |
| import h5py | |
| from scipy import sparse | |
| hf = h5py.File(""../input/cdk2.h5"", ""r"") | |
| ids = hf[""chembl_id""].value # the name of each molecules | |
| ap = sparse.csr_matrix((hf[""ap""][""data""], hf[""ap""][""indices""], hf[""ap""][""indptr""]), shape=[len(hf[""ap""][""indptr""]) - 1, 2039]) | |
| mg = sparse.csr_matrix((hf[""mg""][""data""], hf[""mg""][""indices""], hf[""mg""][""indptr""]), shape=[len(hf[""mg""][""indptr""]) - 1, 2039]) | |
| tt = sparse.csr_matrix((hf[""tt""][""data""], hf[""tt""][""indices""], hf[""tt""][""indptr""]), shape=[len(hf[""tt""][""indptr""]) - 1, 2039]) | |
| features = sparse.hstack([ap, mg, tt]).toarray() # the samples' features, each row is a sample, and each sample has 3*2039 features | |
| labels = hf[""label""].value # the label of each molecule" | |
| Food choices,College students' food and cooking preferences,BoraPajo,67,"Version 5,2017-04-23|Version 4,2017-04-23|Version 3,2017-04-18|Version 2,2017-04-18|Version 1,2017-04-18","food and drink | |
| health",Other,5 MB,CC0,"23,149 views","5,125 downloads",41 kernels,0 topics,https://www.kaggle.com/borapajo/food-choices,"Food choices and preferences of college students | |
| This dataset includes information on food choices, nutrition, preferences, childhood favorites, and other information from college students. There are 126 responses from students. Data is raw and uncleaned. Cleaning is in the process and as soon as that is done, additional versions of the data will be posted. Acknowledgements | |
| Thank you to all the students of Mercyhurst University who agreed to participate in this survey. | |
| Inspiration | |
| How important is nutrition information for today's college kids? Is their taste in food defined by their food preferences when they were children? Are kids of parents who cook more likely to make better food choices than others? Are these kids likely to have a different taste compared to others? There a number of open ended questions included in this dataset such as: What is your favorite comfort food? What is your favorite cuisine? that could work well for natural language processing" | |
| Fifa 18 More Complete Player Dataset,FIFA 18 Player Data++.,KevinH,66,"Version 5,2017-12-26|Version 4,2017-12-23|Version 3,2017-11-05|Version 2,2017-11-05|Version 2,2017-11-05|Version 1,2017-11-03","popular culture | |
| video games | |
| association football | |
| sports",CSV,5 MB,CC0,"11,400 views","2,009 downloads",9 kernels,5 topics,https://www.kaggle.com/kevinmh/fifa-18-more-complete-player-dataset,"Context | |
| This dataset is an extension of that found here. It contains several extra fields and is pre-cleaned to a much greater extent. After talking with the creator of the original dataset, he and I agreed that merging our work would require making breaking changes to the original, and that this should be published as a new dataset. | |
| Content | |
| 185 fields for every player in FIFA 18. | |
| Player info such as age, club, league, nationality, salary and physical attributes | |
| All playing attributes, such as finishing and dribbling | |
| Special attributes like skill moves and international reputation | |
| Traits and specialities | |
| Overall, potential, and ratings for each position | |
| Differences | |
| Here are the columns in this dataset that aren't in the original: | |
| birth_date | |
| eur_release_clause | |
| height_cm | |
| weight_kg | |
| body_type | |
| real_face | |
| league | |
| Headline attributes: pac, sho, pas, dri, def, and phy. These are what appear on Ultimate Team cards | |
| international_reputation | |
| skill_moves | |
| weak_foot | |
| work_rate_att | |
| work_rate_def | |
| preferred_foot | |
| all traits and specialities as dummy variables | |
| all position preferences as dummy variables | |
| Acknowledgements | |
| Credit goes to Aman Shrivastava for building the original dataset. And thanks of course to https://sofifa.com for not banning my IP when I scraped over 18000 pages to get this data. | |
| Inspiration | |
| What insights can this data give us, not only into FIFA 18 but into real-world football? The kernels on last year's dataset are a good place to find ideas. | |
| Contributing | |
| Contributions to the GitHub project are more than welcome. Do let me know if you think of ways to improve either the code or the dataset!" | |
| First GOP Debate Twitter Sentiment,Analyze tweets on the first 2016 GOP Presidential Debate,Crowdflower,65,"Version 2,2016-10-06|Version 1,2015-12-28","politics | |
| internet",SQLite,8 MB,CC4,"29,918 views","3,990 downloads",96 kernels,3 topics,https://www.kaggle.com/crowdflower/first-gop-debate-twitter-sentiment,"This data originally came from Crowdflower's Data for Everyone library. | |
| As the original source says, | |
| We looked through tens of thousands of tweets about the early August GOP debate in Ohio and asked contributors to do both sentiment analysis and data categorization. Contributors were asked if the tweet was relevant, which candidate was mentioned, what subject was mentioned, and then what the sentiment was for a given tweet. We've removed the non-relevant messages from the uploaded dataset. | |
| The data we're providing on Kaggle is a slightly reformatted version of the original source. It includes both a CSV file and SQLite database. The code that does these transformations is available on GitHub" | |
| NIPS 2015 Papers,Explore and analyze this year's NIPS papers,Ben Hamner,64,"Version 3,2017-05-02|Version 2,2016-02-29|Version 1,2015-12-09","writing | |
| linguistics | |
| artificial intelligence",CSV,28 MB,ODbL,"45,351 views","2,323 downloads",76 kernels,5 topics,https://www.kaggle.com/benhamner/nips-2015-papers,"Neural Information Processing Systems (NIPS) is one of the top machine learning conferences in the world. It covers topics ranging from deep learning and computer vision to cognitive science and reinforcement learning. | |
| This year, Kaggle is hosting the NIPS 2015 paper dataset to facilitate and showcase exploratory analytics on the NIPS data. We've extracted the paper text from the raw PDF files and are releasing that both in CSV files and as a SQLite database. Here's a quick script that gives an overview of what's included in the data. | |
| We encourage you to explore this data and share what you find through Kaggle Scripts! | |
| Data Description | |
| Overview of the data in Kaggle Scripts. | |
| nips-2015-papers-release-*.zip (downloadable from the link above) contains the below files/folders. All this data's available through Kaggle Scripts as well, and you can create a new script to immediately start exploring the data in R, Python, Julia, or SQLite. | |
| This dataset is available in two formats: three CSV files and a single SQLite database (consisting of three tables with content identical to the CSV files). | |
| You can see the code used to create this dataset on Github. | |
| Papers.csv | |
| This file contains one row for each of the 403 NIPS papers from this year's conference. It includes the following fields | |
| Id - unique identifier for the paper (equivalent to the one in NIPS's system) | |
| Title - title of the paper | |
| EventType - whether it's a poster, oral, or spotlight presentation | |
| PdfName - filename for the PDF document | |
| Abstract - text for the abstract (scraped from the NIPS website) | |
| PaperText - raw text from the PDF document (created using the tool pdftotext) | |
| Authors.csv | |
| This file contains id's and names for each of the authors on this year's NIPS papers. | |
| Id - unique identifier for the author (equivalent to the one in NIPS's system) | |
| Name - author's name | |
| PaperAuthors.csv | |
| This file links papers to their corresponding authors. | |
| Id - unique identifier | |
| PaperId - id for the paper | |
| AuthorId - id for the author | |
| database.sqlite | |
| This SQLite database contains the tables with equivalent data and formatting as the Papers.csv, Authors.csv, and PaperAuthors.csv files. | |
| pdfs | |
| This folder contains the raw pdf files for each of the papers." | |
| Hearthstone Cards,Explore the entire collection of Hearthstone cards,Jerad Rose,64,"Version 3,2017-01-05|Version 2,2016-11-01|Version 1,2016-10-05","games and toys | |
| video games",CSV,2 MB,CC0,"14,964 views","1,245 downloads",31 kernels,0 topics,https://www.kaggle.com/jeradrose/hearthstone-cards,"This dataset contains data for the entire collection of cards for Hearthstone, the popular online card game by Blizzard. Launching to the public on March 11, 2011 after being under development for almost 5 years, Hearthstone has gained popularity as a freemium game, launching into eSports across the globe, and the source of many Twitch channels. | |
| The data in this dataset was extracted from hearthstonejson.com, and the documentation for all the data can be found on the cards.json documentation page. | |
| The original data was extracted from the actual card data files used in the game, so all of the data should be here, enabling explorations like: | |
| Card strengths and weaknesses | |
| Card strengths relative to cost and rarity | |
| Comparisons across player classes, bosses, and sets | |
| Whether a set of optimal cards can be determined per class | |
| The cards can be explored in one of four ways: | |
| cards.json: The raw JSON pulled from hearthstonejson.com | |
| cards_flat.csv: A flat CSV containing a row for each card, and any n:m data stored as arrays in single fields | |
| database.sqlite: A SQLite database containing relational data of the cards | |
| cards.csv, mechanics.csv, dust_costs.csv, play_requirements.csv, and entourages.csv: the normalized data in CSV format. | |
| This dataset will be updated as new releases and expansions are made to Hearthstone. | |
| Currently, any localized string values are in en-us, but I may look into adding other languages if the demand seems to be there." | |
| Detailed NFL Play-by-Play Data 2015,An NFL dataset generated by the nflscrapR R-package & primed for analysis,Max Horowitz,64,"Version 2,2016-10-03|Version 1,2016-05-21",american football,CSV,15 MB,Other,"33,699 views","3,321 downloads",86 kernels,8 topics,https://www.kaggle.com/maxhorowitz/nflplaybyplay2015,"Introduction | |
| The lack of publicly available National Football League (NFL) data sources has been a major obstacle in the creation of modern, reproducible research in football analytics. While clean play-by-play data is available via open-source software packages in other sports (e.g. nhlscrapr for hockey; PitchF/x data in baseball; the NBA API for basketball), the equivalent datasets are not freely available for researchers interested in the statistical analysis of the NFL. To solve this issue, a group of Carnegie Mellon University statistical researchers led by recent graduate, Maksim Horowitz, built and released nflscrapR an R package which uses an API maintained by the NFL to scrape, clean, parse, and output clean datasets at the individual play, player, game, and season levels. These datasets allow for the advancement of NFL research in the public domain by allowing analysts to develop from a common source in order to create reproducible NFL research, similar to what is being done currently in other professional sports. | |
| 2015 NFL Play-by-Play Dataset | |
| The dataset made available on Kaggle contains all the regular season plays from the 2015-2016 NFL season. The dataset contain 46,129 rows and 63 columns. Each play is broken down into great detail containing information on; game situation, players involved and results. Detailed information about the dataset can be found in the nflscrapR documentation. | |
| Downloading and Installing nflscrapR: | |
| Use the following code in your R console: | |
| # Must install the devtools package using the below code | |
| install.packages('devtools') | |
| library(devtools) | |
| # For now you must install nflscrapR from github | |
| if (!is.element(""nflscrapR"", installed.packages())) { | |
| # Print Installing nflscrapR | |
| devtools::install_github(repo = ""maksimhorowitz/nflscrapR"") | |
| } | |
| library(nflscrapR)" | |
| Pokémon for Data Mining and Machine Learning,(Almost) all Pokémon stats until generation 6: 21 variables per Pokémon,alopez247,64,"Version 2,2017-03-05|Version 1,2017-01-08",video games,CSV,800 KB,CC4,"69,501 views","2,256 downloads",30 kernels,0 topics,https://www.kaggle.com/alopez247/pokemon,"Context | |
| With the rise of the popularity of machine learning, this is a good opportunity to share a wide database of the even more popular video-game Pokémon by Nintendo, Game freak, and Creatures, originally released in 1996. | |
| Pokémon started as a Role Playing Game (RPG), but due to its increasing popularity, its owners ended up producing many TV series, manga comics, and so on, as well as other types of video-games (like the famous Pokémon Go!). | |
| This dataset is focused on the stats and features of the Pokémon in the RPGs. Until now (08/01/2017) seven generations of Pokémon have been published. All in all, this dataset does not include the data corresponding to the last generation, since 1) I created the databased when the seventh generation was not released yet, and 2) this database is a modification+extension of the database ""721 Pokemon with stats"" by Alberto Barradas (https://www.kaggle.com/abcsds/pokemon), which does not include (of course) the latest generation either. | |
| Content | |
| This database includes 21 variables per each of the 721 Pokémon of the first six generations, plus the Pokémon ID and its name. These variables are briefly described next: | |
| Number. Pokémon ID in the Pokédex. | |
| Name. Name of the Pokémon. | |
| Type_1. Primary type. | |
| Type_2. Second type, in case the Pokémon has it. | |
| Total. Sum of all the base stats (Health Points, Attack, Defense, Special Attack, Special Defense, and Speed). | |
| HP. Base Health Points. | |
| Attack. Base Attack. | |
| Defense. Base Defense. | |
| Sp_Atk. Base Special Attack. | |
| Sp_Def. Base Special Defense. | |
| Speed. Base Speed. | |
| Generation. Number of the generation when the Pokémon was introduced. | |
| isLegendary. Boolean that indicates whether the Pokémon is Legendary or not. | |
| Color. Color of the Pokémon according to the Pokédex. | |
| hasGender. Boolean that indicates if the Pokémon can be classified as female or male. | |
| Pr_male. In case the Pokémon has Gender, the probability of its being male. The probability of being female is, of course, 1 minus this value. | |
| Egg_Group_1. Egg Group of the Pokémon. | |
| Egg_Group_2. Second Egg Group of the Pokémon, in case it has two. | |
| hasMegaEvolution. Boolean that indicates whether the Pokémon is able to Mega-evolve or not. | |
| Height_m. Height of the Pokémon, in meters. | |
| Weight_kg. Weight of the Pokémon, in kilograms. | |
| Catch_Rate. Catch Rate. | |
| Body_Style. Body Style of the Pokémon according to the Pokédex. | |
| Notes | |
| Please note that many Pokémon are multi-form, and also some of them can Mega-evolve. I wanted to keep the structure of the dataset as simple and general as possible, as well as the Number variable (the ID of the Pokémon) unique. Hence, in the cases of the multi-form Pokémon, or the ones capable of Mega-evolve, I just chose one of the forms, the one I (and my brother) considered the standard and/or the most common. The specific choice for each of this Pokémon are shown below: | |
| Mega-Evolutions are not considered as Pokémon. | |
| Kyogre, Groudon. Primal forms not considered. | |
| Deoxis. Only normal form considered. | |
| Wormadam. Only plant form considered. | |
| Rotom. Only normal form considered, the one with types Electric and Ghost. | |
| Giratina. Origin form considered. | |
| Shaymin. Land form considered. | |
| Darmanitan. Standard mode considered. | |
| Tornadus, Thundurus, Landorus. Incarnate form considered. | |
| Kyurem. Normal form considered, not white or black forms. | |
| Meloetta. Aria form considered. | |
| Mewstic. Both female and male forms are equal in the considered variables. | |
| Aegislash. Shield form considered. | |
| Pumpkaboo, Gourgeist. Average size considered. | |
| Zygarde. 50% form considered. | |
| Hoopa. Confined form considered. | |
| Acknowledgements | |
| As said at the beginning, this database was based on the Kaggle database ""721 Pokemon with stats"" by Alberto Barradas (https://www.kaggle.com/abcsds/pokemon). The other resources I mainly used are listed below: | |
| WikiDex (http://es.pokemon.wikia.com/wiki/WikiDex). | |
| Bulbapedia, the community driven Pokémon encyclopedia (http://bulbapedia.bulbagarden.net/wiki/Main_Page). | |
| Smogon University (http://www.smogon.com/). | |
| Possible future work | |
| This dataset can be used with different objectives, such as, Pokémon clustering, trying to find relations or dependencies between the variables, and also for supervised classification purposes, where the class could be the Primary Type, but also many of the other variables. | |
| Author | |
| Asier López Zorrilla" | |
| Ships in Satellite Imagery,Classify ships in San Franciso Bay using Planet satellite imagery,rhammell,64,"Version 7,2017-11-14|Version 6,2017-10-16|Version 5,2017-10-11|Version 4,2017-10-11|Version 3,2017-10-10|Version 2,2017-10-09|Version 1,2017-10-09",business,Other,148 MB,CC4,"8,227 views",954 downloads,9 kernels,3 topics,https://www.kaggle.com/rhammell/ships-in-satellite-imagery,"Context | |
| Satellite imagery provides unique insights into various markets, including agriculture, defense and intelligence, energy, and finance. New commercial imagery providers, such as Planet and BlackSky, are using constellations of small satellites to exponentially increase the amount of images of the earth captured every day. | |
| This flood of new imagery is outgrowing the ability for organizations to manually look at each image that gets captured, and there is a need for machine learning and computer vision algorithms to help automate the analysis process. | |
| The aim of this dataset is to help address the difficult task of detecting the location of large ships in satellite images. Automating this process can be applied to many issues including monitoring port activity levels and supply chain analysis. | |
| Continusouly updates will be made to this dataset as new Planet imagery released. Current images were collected as late as September 2017. | |
| Content | |
| The dataset consists of image chips extracted from Planet satellite imagery collected over the San Franciso Bay area. It includes 2800 80x80 RGB images labeled with either a ""ship"" or ""no-ship"" classification. Image chips were derived from PlanetScope full-frame visual scene products, which are orthorectified to a 3 meter pixel size. | |
| Provided is a zipped directory shipsnet.7z that contains the entire dataset as .png image chips. Each individual image filename follows a specific format: {label} __ {scene id} __ {longitude} _ {latitude}.png | |
| label: Valued 1 or 0, representing the ""ship"" class and ""no-ship"" class, respectively. | |
| scene id: The unique identifier of the PlanetScope visual scene the image chip was extracted from. The scene id can be used with the Planet API to discover and download the entire scene. | |
| longitude_latitude: The longitude and latitude coordinates of the image center point, with values separated by a single underscore. | |
| The dataset is also distributed as a JSON formatted text file shipsnet.json. The loaded object contains data, label, scene_ids, and location lists. | |
| The pixel value data for each 80x80 RGB image is stored as a list of 19200 integers within the data list. The first 6400 entries contain the red channel values, the next 6400 the green, and the final 6400 the blue. The image is stored in row-major order, so that the first 80 entries of the array are the red channel values of the first row of the image. | |
| The list values at index i in labels, scene_ids, and locations each correspond to the i-th image in the data list. | |
| Class Labels | |
| The ""ship"" class includes 700 images. Images in this class are near-centered on the body of a single ship. Ships of different ship sizes, orientations, and atmospheric collection conditions are included. Example images from this class are shown below. | |
| The ""no-ship"" class includes 2100 images. A third of these are a random sampling of different landcover features - water, vegetion, bare earth, buildings, etc. - that do not include any portion of an ship. The next third are ""partial ships"" that contain only a portion of an ship, but not enough to meet the full definition of the ""ship"" class. The last third are images that have previously been mislabeled by machine learning models, typically caused by bright pixels or strong linear features. Example images from this class are shown below. | |
| Acknowledgements | |
| Satellite imagery used to build this dataset is made available through Planet's Open California dataset, which is openly licensed. As such, this dataset is also available under the same CC-BY-SA license. Users can sign up for a free Planet account to search, view, and download thier imagery and gain access to their API." | |
| Historical Air Quality,Air Quality Data Collected at Outdoor Monitors Across the US,US Environmental Protection Agency,64,"Version 1,2017-12-01","pollution | |
| bigquery",BigQuery,323 GB,CC0,"13,461 views",0 downloads,14 kernels,,https://www.kaggle.com/epa/epa-historical-air-quality,"The AQS Data Mart is a database containing all of the information from AQS. It has every measured value the EPA has collected via the national ambient air monitoring program. It also includes the associated aggregate values calculated by EPA (8-hour, daily, annual, etc.). The AQS Data Mart is a copy of AQS made once per week and made accessible to the public through web-based applications. The intended users of the Data Mart are air quality data analysts in the regulatory, academic, and health research communities. It is intended for those who need to download large volumes of detailed technical data stored at EPA and does not provide any interactive analytical tools. It serves as the back-end database for several Agency interactive tools that could not fully function without it: AirData, AirCompare, The Remote Sensing Information Gateway, the Map Monitoring Sites KML page, etc. | |
| AQS must maintain constant readiness to accept data and meet high data integrity requirements, thus is limited in the number of users and queries to which it can respond. The Data Mart, as a read only copy, can allow wider access. | |
| The most commonly requested aggregation levels of data (and key metrics in each) are: | |
| Sample Values (2.4 billion values back as far as 1957, national consistency begins in 1980, data for 500 substances routinely collected) The sample value converted to standard units of measure (generally 1-hour averages as reported to EPA, sometimes 24-hour averages) Local Standard Time (LST) and GMT timestamps Measurement method Measurement uncertainty, where known Any exceptional events affecting the data NAAQS Averages NAAQS average values (8-hour averages for ozone and CO, 24-hour averages for PM2.5) Daily Summary Values (each monitor has the following calculated each day) Observation count Observation per cent (of expected observations) Arithmetic mean of observations Max observation and time of max AQI (air quality index) where applicable Number of observations > Standard where applicable Annual Summary Values (each monitor has the following calculated each year) Observation count and per cent Valid days Required observation count Null observation count Exceptional values count Arithmetic Mean and Standard Deviation 1st - 4th maximum (highest) observations Percentiles (99, 98, 95, 90, 75, 50) Number of observations > Standard Site and Monitor Information FIPS State Code (the first 5 items on this list make up the AQS Monitor Identifier) FIPS County Code Site Number (unique within the county) Parameter Code (what is measured) POC (Parameter Occurrence Code) to distinguish from different samplers at the same site Latitude Longitude Measurement method information Owner / operator / data-submitter information Monitoring Network to which the monitor belongs Exemptions from regulatory requirements Operational dates City and CBSA where the monitor is located Quality Assurance Information Various data fields related to the 19 different QA assessments possible | |
| Querying BigQuery tables | |
| You can use the BigQuery Python client library to query tables in this dataset in Kernels. Note that methods available in Kernels are limited to querying data. Tables are at bigquery-public-data.epa_historical_air_quality.[TABLENAME]. Fork this kernel to get started. | |
| Acknowledgements | |
| Data provided by the US Environmental Protection Agency Air Quality System Data Mart." | |
| Question Pairs Dataset,Can you identify duplicate questions?,Quora,64,"Version 2,2017-02-02|Version 1,2017-01-31","languages | |
| linguistics | |
| artificial intelligence",CSV,58 MB,Other,"18,252 views","1,473 downloads",80 kernels,4 topics,https://www.kaggle.com/quora/question-pairs-dataset,"Context | |
| Quora's first public dataset is related to the problem of identifying duplicate questions. At Quora, an important product principle is that there should be a single question page for each logically distinct question. For example, the queries “What is the most populous state in the USA?” and “Which state in the United States has the most people?” should not exist separately on Quora because the intent behind both is identical. Having a canonical page for each logically distinct query makes knowledge-sharing more efficient in many ways: for example, knowledge seekers can access all the answers to a question in a single location, and writers can reach a larger readership than if that audience was divided amongst several pages. | |
| The dataset is based on actual data from Quora and will give anyone the opportunity to train and test models of semantic equivalence. | |
| Content | |
| There are over 400,000 lines of potential question duplicate pairs. Each line contains IDs for each question in the pair, the full text for each question, and a binary value that indicates whether the line truly contains a duplicate pair. | |
| Acknowledgements | |
| For more information on this dataset, check out Quora's first dataset release page. | |
| License | |
| This data is subject to Quora's Terms of Service, allowing for non-commercial use." | |
| Weather Data for Recruit Restaurant Competition,Data From Weather Stations in Japan and the Recruit Restaurant Competition Data,Hunter McGushion,64,"Version 5,2018-01-05|Version 4,2018-01-04|Version 3,2018-01-03|Version 2,2017-12-21|Version 1,2017-12-20","food and drink | |
| time series | |
| geography",CSV,11 MB,CC4,"5,944 views","1,900 downloads",5 kernels,9 topics,https://www.kaggle.com/huntermcgushion/rrv-weather-data,"Version 5 Description | |
| TLDR | |
| The directory 1-1-16_5-31-17_Weather contains 1663 files (one for each of the 1663 stations in Japan) | |
| As its name implies, the data is from the same date window as the competition's date_info file | |
| If a station file's name ends with four underscores and a date, the date indicates when the station was terminated | |
| Please, read on... | |
| Context | |
| This dataset contains the dataset from the Recruit Restaurant Visitor Forecasting competition (active from 11-28-17 to 2-6-18). | |
| The focus is on using data about reservations made at various restaurants throughout Japan, along with restaurant location and genre information to predict the actual number of visitors a restaurant will have on a given day. | |
| This dataset augments the above with the addition of information about the weather at various locations in Japan over time to produce an exciting, multi-faceted dataset that deals with time, geography, weather, and delicious food. | |
| Thank you for your interest in this dataset! Please let me know if you have any suggestions, questions or problems! | |
| Content | |
| The core of the dataset comprises the files in the following directory: | |
| 1-1-16_5-31-17_Weather (1663 .csv files): | |
| This directory contains translated weather data for the time period denoted by the directory’s name (from 1-1-16 through 5-31-17). | |
| Each .csv file in this directory is of shape (517, 15), and is named according to the id values in the below weather_stations file. | |
| There are a few reasons why there may seem to be a lot of null values: | |
| The primary reason is that different types of stations/sensors are used, and some just don't capture as much data as others | |
| Questionable data is sometimes removed by the Agency | |
| If the station was terminated, its values are null | |
| These are the features for all translated weather files (I won’t hazard a description for the features that aren’t already self-explanatory because I’m no meteorologist, and I’d hate for you to get that impression): | |
| calendar_date - the observation date, formatted thusly ""yyyy-mm-dd"" | |
| avg_temperature | |
| high_temperature | |
| low_temperature | |
| precipitation | |
| hours_sunlight | |
| solar_radiation | |
| deepest_snowfall | |
| total_snowfall | |
| avg_wind_speed | |
| avg_vapor_pressure | |
| avg_local_pressure | |
| avg_humidity | |
| avg_sea_pressure | |
| cloud_cover | |
| This dataset adds the following .csv files regarding weather stations, and their relations to the competition data: | |
| weather_stations.csv (1663, 8): | |
| This file contains the location and termination dates for 1,663 weather stations in Japan. | |
| id - the join of a station’s prefecture, first_name, and second_name, with ""__"" (double underscores) | |
| Note: If date_terminated is not null, id will end with four underscores and the date_terminated | |
| prefecture - the prefecture in which this station is located (see note 1) | |
| first_name - the first name given to specify a location (see note 2) | |
| second_name - the second name given to specify a location (see note 2) | |
| latitude - latitude of the station, converted from degrees, minutes, seconds to decimal degrees for consistency | |
| longitude - longitude of the station, converted from degrees, minutes, seconds to decimal degrees for consistency | |
| altitude - altitude of the station | |
| date_terminated - If the station was terminated, the date of its termination (formatted thusly ""yyyy-mm-dd"") else null | |
| nearby_active_stations.csv (62, 8): | |
| This file is a subset of weather_stations.csv (above) selected via the following criteria: | |
| 1) the station was not terminated, and | |
| 2) the station was the closest station to at least one store in air_store_info or hpg_store_info | |
| As you can see, there is a lot of overlap here, because while the weather stations seem to generally be scattered throughout Japan, the store locations tend to be clustered around several areas. | |
| Column names and descriptions are identical to those of weather_stations. | |
| feature_manifest.csv (1663, 15): | |
| This file contains information about each station's ""coverage"" of each weather feature. | |
| Values of 0.0 for any of the below features except id mean that station collected no data on that feature. | |
| Values of 1.0 for any of the below features except id mean that station collected data on that feature for every day. | |
| id - the id of this weather station | |
| avg_temperature - ratio of non-null values for this feature at this station | |
| high_temperature - ratio of non-null values for this feature at this station | |
| low_temperature - ratio of non-null values for this feature at this station | |
| precipitation - ratio of non-null values for this feature at this station | |
| hours_sunlight - ratio of non-null values for this feature at this station | |
| solar_radiation - ratio of non-null values for this feature at this station | |
| deepest_snowfall - ratio of non-null values for this feature at this station | |
| total_snowfall - ratio of non-null values for this feature at this station | |
| avg_wind_speed - ratio of non-null values for this feature at this station | |
| avg_vapor_pressure - ratio of non-null values for this feature at this station | |
| avg_local_pressure - ratio of non-null values for this feature at this station | |
| avg_humidity - ratio of non-null values for this feature at this station | |
| avg_sea_pressure - ratio of non-null values for this feature at this station | |
| cloud_cover - ratio of non-null values for this feature at this station | |
| air_station_distances.csv (1663, 111): | |
| This file contains the Vincenty distance from every weather station to every unique latitude/longitude pair in the air system. | |
| station_id - the id of this weather station | |
| station_latitude - station latitude (in decimal degrees) | |
| station_longitude - station longitude (in decimal degrees) | |
| <<UNIQUE AIR COORDINATE PAIRS>> - The remaining 108 columns are stringified versions of all unique air coordinate pairs | |
| They are formatted thusly: (##.###, ##.###), where the first float is the store latitude and the second is the store longitude (see note 3) | |
| EX) The first of these columns is: (34.6951242, 135.1978525) | |
| hpg_station_distances.csv (1663, 132): | |
| This file contains the Vincenty distance from every weather station to every unique latitude/longitude pair in the hpg system. | |
| station_id - the id of this weather station | |
| station_latitude - station latitude (in decimal degrees) | |
| station_longitude - station longitude (in decimal degrees) | |
| <<UNIQUE HPG COORDINATE PAIRS>> - The remaining 129 columns are stringified versions of all unique hpg coordinate pairs | |
| They are formatted thusly: (##.###, ##.###), where the first float is the store latitude and the second is the store longitude (see note 3) | |
| EX) The first of these columns is: (35.6436746642265, 139.668220854814) | |
| air_store_info_with_nearest_active_station.csv (829, 12): | |
| hpg_store_info_with_nearest_active_station.csv (4690, 12): | |
| These two files are supplemented versions of air_store_info and hpg_store_info, and they contain the original competition data for the store_info file specified in the file’s name, plus the following features: | |
| latitude_str - a stringified version of this store's latitude for lookup in air_station_distances and hpg_station_distances | |
| longitude_str - a stringified version of this store's longitude for lookup in air_station_distances and hpg_station_distances | |
| station_id - the id of the weather station nearest to this store | |
| station_latitude - the latitude (in decimal degrees) of the weather station nearest to this store | |
| station_longitude - the longitude (in decimal degrees) of the weather station nearest to this store | |
| station_vincenty - the Vincenty distance between this store and the station to which it is closest | |
| station_great_circle - the Great Circle distance between this store and the station to which it is closest | |
| Other Available Data | |
| Date ranges outside of the one used for this competition | |
| Records seem to go fairly far back, but of course the data we can actually get will be subject to the activation and termination dates of each individual station | |
| Observation periods other than daily values | |
| Hourly, every “x” days, monthly, seasonal | |
| Option to compare data with average from the past 30 years | |
| Other features: | |
| Day’s maximum instantaneous wind speed | |
| Day’s maximum wind direction | |
| Day’s minimum relative humidity | |
| Day’s minimum sea pressure | |
| Option to show the time at which the observation period’s maximum and minimum values occurred | |
| Acknowledgements | |
| All weather data contained herein came from the Japan Meteorological Agency | |
| All data pertaining to restaurants, reservations, and visitors came from the Recruit Restaurant Visitor Forecasting competition | |
| Disclaimers | |
| The site has one station (that I know of) that I did not include in the list of all stations. That station is located in the Antarctic, and I didn’t include it because the Antarctic is not located in Japan | |
| I have not tested the feasibility of actually gathering and processing the “other available data” listed above | |
| Below is a list of my qualifications for processing weather data for Japan, from a site written in Japanese: | |
| 1 | |
| 2 | |
| 3 | |
| Look upon the list and see that it is empty. I’m just a guy that knows how to get stuff done and enjoys a challenge | |
| Again, read all translations with several grains of salt | |
| Plans/Ideas to Expand This Dataset (Tell Me If These Interest You) | |
| Getting data from previous years to get a better idea of how the weather should be on a given day (next version) | |
| Averaging data from multiple stations to maybe create some representation of the weather for that prefecture/area | |
| For stores that are particularly far from their nearest station, doing a similar sort of averaging as above to attempt to minimize any risk posed by using weather data too far from the target store | |
| Notes | |
| 1) The site separates the prefecture ""Hokkaido"" into its 14 subprefectures: | |
| Therefore, when you might expect prefecture to be ""hokkaido"", you will actually see ""hokkaido_<subprefecture>"" | |
| i.e., the concatenation of the three strings ""hokkaido"", ""_"" (single underscore), and the subprefecture name | |
| 2) Formatting of first_name and last_name: | |
| Spaces were replaced by ""-"" (single dash) | |
| If a value was not provided or could not be translated, the value is ""NONE"" | |
| I would suggest you don’t rely too heavily on these features, as they are the result of several different translation APIs, which were rarely in agreement on the translation. These features are provided more as a convenience, and in an effort to give you as much data as possible | |
| 3) Column formatting in air_station_distances.csv and hpg_station_distances.csv: | |
| These files show the coordinates with the precision shown in the original air_store_info.csv and hpg_store_info.csv files; however, Pandas, doesn't read these values in with the same precision | |
| If you plan on using these files, you must use the latitude_str and longitude_str features in the appropriate ..._store_info_with_nearest_active_station file to look up coordinates | |
| For more information, see this discussion post" | |
| "380,000+ lyrics from MetroLyrics","Lyrics, Artist , Genre, Year",GyanendraMishra,63,"Version 2,2017-01-11|Version 1,2017-01-09","writing | |
| music | |
| linguistics",CSV,310 MB,CC4,"11,838 views","1,624 downloads",18 kernels,5 topics,https://www.kaggle.com/gyani95/380000-lyrics-from-metrolyrics,"Context | |
| I tried to gather as many lyrics as I could. I ran my code on a a free ec2 instance and ran out of storage space. I have attached the code below so if any one wants to try out it and get all lyrics, please do. | |
| Content | |
| There are around 380,000+ lyrics in the data set from a lot of different artists from a lot of different genres arranged by year. Structure is artist/year/song. Every artist folder has a genre.txt that tells what is the genre of the musician. Find the crawler here. | |
| Acknowledgements | |
| I would like to thank Shruti Jasoria, SJasoria on GitHub for writing the multi-threaded version. | |
| Inspiration | |
| I wanted to find out what genre and what artist abuses what substance. Do rapstars like cocaine or liquor? If liquor then what Liquor? Does Eminem prefer Hennesy over Jack Daniels? Do Rockstars love pot?" | |
| Loan Data,This dataset includes customers who have paid off their loans or not,Zhijin,63,"Version 1,2017-04-11",finance,CSV,43 KB,CC0,"21,936 views","3,470 downloads",47 kernels,2 topics,https://www.kaggle.com/zhijinzhai/loandata,"Context | |
| This data set includes customers who have paid off their loans, who have been past due and put into collection without paying back their loan and interests, and who have paid off only after they were put in collection. The financial product is a bullet loan that customers should pay off all of their loan debt in just one time by the end of the term, instead of an installment schedule. Of course, they could pay off earlier than their pay schedule. | |
| Content | |
| Loan_id A unique loan number assigned to each loan customers | |
| Loan_status Whether a loan is paid off, in collection, new customer yet to payoff, or paid off after the collection efforts | |
| Principal Basic principal loan amount at the origination | |
| terms Can be weekly (7 days), biweekly, and monthly payoff schedule | |
| Effective_date When the loan got originated and took effects | |
| Due_date Since it’s one-time payoff schedule, each loan has one single due date | |
| Paidoff_time The actual time a customer pays off the loan | |
| Pastdue_days How many days a loan has been past due | |
| Age, education, gender A customer’s basic demographic information" | |
| The Complete Pokemon Dataset,Data on more than 800 Pokemon from all 7 Generations.,Rounak Banik,63,"Version 1,2017-09-30","popular culture | |
| video games",CSV,157 KB,CC0,"15,580 views","1,544 downloads",16 kernels,0 topics,https://www.kaggle.com/rounakbanik/pokemon,"Context | |
| This dataset contains information on all 802 Pokemon from all Seven Generations of Pokemon. The information contained in this dataset include Base Stats, Performance against Other Types, Height, Weight, Classification, Egg Steps, Experience Points, Abilities, etc. The information was scraped from http://serebii.net/ | |
| Content | |
| name: The English name of the Pokemon | |
| japanese_name: The Original Japanese name of the Pokemon | |
| pokedex_number: The entry number of the Pokemon in the National Pokedex | |
| percentage_male: The percentage of the species that are male. Blank if the Pokemon is genderless. | |
| type1: The Primary Type of the Pokemon | |
| type2: The Secondary Type of the Pokemon | |
| classification: The Classification of the Pokemon as described by the Sun and Moon Pokedex | |
| height_m: Height of the Pokemon in metres | |
| weight_kg: The Weight of the Pokemon in kilograms | |
| capture_rate: Capture Rate of the Pokemon | |
| base_egg_steps: The number of steps required to hatch an egg of the Pokemon | |
| abilities: A stringified list of abilities that the Pokemon is capable of having | |
| experience_growth: The Experience Growth of the Pokemon | |
| base_happiness: Base Happiness of the Pokemon | |
| against_?: Eighteen features that denote the amount of damage taken against an attack of a particular type | |
| hp: The Base HP of the Pokemon | |
| attack: The Base Attack of the Pokemon | |
| defense: The Base Defense of the Pokemon | |
| sp_attack: The Base Special Attack of the Pokemon | |
| sp_defense: The Base Special Defense of the Pokemon | |
| speed: The Base Speed of the Pokemon | |
| generation: The numbered generation which the Pokemon was first introduced | |
| is_legendary: Denotes if the Pokemon is legendary. | |
| Acknowledgements | |
| The data was scraped from http://serebii.net/. | |
| Inspiration | |
| Pokemon holds a very special place in my heart as it is probably the only video game I have judiciously followed for more than 10 years. With this dataset, I wanted to be able to answer the following questions: | |
| Is it possible to build a classifier to identify legendary Pokemon? | |
| How does height and weight of a Pokemon correlate with its various base stats? | |
| What factors influence the Experience Growth and Egg Steps? Are these quantities correlated? | |
| Which type is the strongest overall? Which is the weakest? | |
| Which type is the most likely to be a legendary Pokemon? | |
| Can you build a Pokemon dream team? A team of 6 Pokemon that inflicts the most damage while remaining relatively impervious to any other team of 6 Pokemon." | |
| Amazon Reviews for Sentiment Analysis,A few million Amazon reviews in fastText format,Adam Mathias Bittlingmayer,62,"Version 2,2017-05-24|Version 1,2017-05-23","business | |
| linguistics | |
| internet",Other,493 MB,Other,"27,962 views","3,335 downloads",9 kernels,,https://www.kaggle.com/bittlingmayer/amazonreviews,"This dataset consists of a few million Amazon customer reviews (input text) and star ratings (output labels) for learning how to train fastText for sentiment analysis. | |
| The idea here is a dataset is more than a toy - real business data on a reasonable scale - but can be trained in minutes on a modest laptop. | |
| Content | |
| The fastText supervised learning tutorial requires data in the following format: | |
| __label__<X> __label__<Y> ... <Text> | |
| where X and Y are the class names. No quotes, all on one line. | |
| In this case, the classes are __label__1 and __label__2, and there is only one class per row. | |
| __label__1 corresponds to 1- and 2-star reviews, and __label__2 corresponds to 4- and 5-star reviews. | |
| (3-star reviews i.e. reviews with neutral sentiment were not included in the original), | |
| The review titles, followed by ':' and a space, are prepended to the text. | |
| Most of the reviews are in English, but there are a few in other languages, like Spanish. | |
| Source | |
| The data was lifted from Xiang Zhang's Google Drive dir, but it was in .csv format, not suitable for fastText. | |
| Training and Testing | |
| Follow the basic instructions at fastText supervised learning tutorial to set up the directory. | |
| To train: | |
| ./fasttext supervised -input train.ft.txt -output model_amzn | |
| This should take a few minutes. | |
| To test: | |
| ./fasttext test model_amzn.bin test.ft.txt | |
| Expect precision and recall of 0.916 if all is in order. | |
| You can also train and test in Python, see Kernel." | |
| "Govt. of India Census, 2001 District-Wise","One billion hearts, a single CSV",PreetSinghKhalsa,62,"Version 1,2017-01-18","demographics | |
| sociology",CSV,355 KB,ODbL,"14,182 views","2,104 downloads",71 kernels,,https://www.kaggle.com/bazuka/census2001,"Context | |
| Census of India is a rich database which can tell stories of over a billion Indians. It is important not only for research point of view, but commercially as well for the organizations that want to understand India's complex yet strongly knitted heterogeneity. However, nowhere on the web, there exists a single database that combines the district- wise information of all the variables (most include no more than 4-5 out of over 50 variables!). Extracting and using data from Census of India 2001 is quite a laborious task since all data is made available in scattered PDFs district wise. Individual PDFs can be extracted from http://www.censusindia.gov.in/(S(ogvuk1y2e5sueoyc5eyc0g55))/Tables_Published/Basic_Data_Sheet.aspx. | |
| Content | |
| This database has been extracted from Census of 2001 and includes data of 590 districts, having around 80 variables each. | |
| In case of confusion regarding the context of the variable, refer to the following PDF and you will be able to make sense out of it: http://censusindia.gov.in/Dist_File/datasheet-2923.pdf | |
| All the extraction work can be found @ https://github.com/preetskhalsa97/census2001auto The final CSV can be found at finalCSV/all.csv | |
| The subtle hack that was used to automate extraction to a great extent was the the URLs of all the PDFs were same except the four digits (that were respective state and district codes). | |
| A few abbreviations used for states: | |
| AN- Andaman and Nicobar CG- Chhattisgarh D_D- Daman and Diu D_N_H- Dadra and Nagar Haveli JK- Jammu and Kashmir MP- Madhya Pradesh TN- Tamil Nadu UP- Uttar Pradesh WB- West Bengal | |
| A few variables for clarification: Growth..1991...2001- population growth from 1991 to 2001 X0..4 years- People in age group 0 to 4 years SC1- Scheduled Class with highest population | |
| Acknowledgements | |
| Inspiration | |
| This is a massive dataset which can be used to explain the interplay between education, caste, development, gender and much more. It really can explain a lot about India and propel data driven research. Happy Number Crunching!" | |
| Indian Startup Funding,Funding details of the startups in India,SRK,61,"Version 2,2017-08-11|Version 1,2017-08-11","india | |
| finance | |
| lending",CSV,305 KB,CC0,"10,948 views","1,873 downloads",22 kernels,3 topics,https://www.kaggle.com/sudalairajkumar/indian-startup-funding,"Context | |
| Interested in the Indian startup ecosystem just like me? Wanted to know what type of startups are getting funded in the last few years? Wanted to know who are the important investors? Wanted to know the hot fields that get a lot of funding these days? This dataset is a chance to explore the Indian start up scene. Deep dive into funding data and derive insights into the future! | |
| Content | |
| This dataset has funding information of the Indian startups from January 2015 to August 2017. It includes columns with the date funded, the city the startup is based out of, the names of the funders, and the amount invested (in USD). | |
| For more information on the values of individual fields, check out the Column Metadata. | |
| Acknowledgements | |
| Thanks to trak.in who are generous enough to share the data publicly for free. | |
| Inspiration | |
| Possible questions which could be answered are: | |
| How does the funding ecosystem change with time? | |
| Do cities play a major role in funding? | |
| Which industries are favored by investors for funding? | |
| Who are the important investors in the Indian Ecosystem? | |
| How much funds does startups generally get in India?" | |
| Retailrocket recommender system dataset,"Ecommerce data: web events, item properties (with texts), category tree",Retailrocket,61,"Version 4,2017-03-24|Version 3,2017-03-24|Version 2,2017-03-24|Version 1,2017-03-24","business | |
| internet",CSV,942 MB,CC4,"23,263 views","2,357 downloads",16 kernels,6 topics,https://www.kaggle.com/retailrocket/ecommerce-dataset,"Context | |
| The dataset consists of three files: a file with behaviour data (events.csv), a file with item properties (item_properties.сsv) and a file, which describes category tree (category_tree.сsv). The data has been collected from a real-world ecommerce website. It is raw data, i.e. without any content transformations, however, all values are hashed due to confidential issues. The purpose of publishing is to motivate researches in the field of recommender systems with implicit feedback. | |
| Content | |
| The behaviour data, i.e. events like clicks, add to carts, transactions, represent interactions that were collected over a period of 4.5 months. A visitor can make three types of events, namely “view”, “addtocart” or “transaction”. In total there are 2 756 101 events including 2 664 312 views, 69 332 add to carts and 22 457 transactions produced by 1 407 580 unique visitors. For about 90% of events corresponding properties can be found in the “item_properties.csv” file. | |
| For example: | |
| “1439694000000,1,view,100,” means visitorId = 1, clicked the item with id = 100 at 1439694000000 (Unix timestamp) | |
| “1439694000000,2,transaction,1000,234” means visitorId = 2 purchased the item with id = 1000 in transaction with id = 234 at 1439694000000 (Unix timestamp) | |
| The file with item properties (item_properties.csv) includes 20 275 902 rows, i.e. different properties, describing 417 053 unique items. File is divided into 2 files due to file size limitations. Since the property of an item can vary in time (e.g., price changes over time), every row in the file has corresponding timestamp. In other words, the file consists of concatenated snapshots for every week in the file with the behaviour data. However, if a property of an item is constant over the observed period, only a single snapshot value will be present in the file. For example, we have three properties for single item and 4 weekly snapshots, like below: | |
| timestamp,itemid,property,value | |
| 1439694000000,1,100,1000 | |
| 1439695000000,1,100,1000 | |
| 1439696000000,1,100,1000 | |
| 1439697000000,1,100,1000 | |
| 1439694000000,1,200,1000 | |
| 1439695000000,1,200,1100 | |
| 1439696000000,1,200,1200 | |
| 1439697000000,1,200,1300 | |
| 1439694000000,1,300,1000 | |
| 1439695000000,1,300,1000 | |
| 1439696000000,1,300,1100 | |
| 1439697000000,1,300,1100 | |
| After snapshot merge it would looks like: | |
| 1439694000000,1,100,1000 | |
| 1439694000000,1,200,1000 | |
| 1439695000000,1,200,1100 | |
| 1439696000000,1,200,1200 | |
| 1439697000000,1,200,1300 | |
| 1439694000000,1,300,1000 | |
| 1439696000000,1,300,1100 | |
| Because property=100 is constant over time, property=200 has different values for all snapshots, property=300 has been changed once. | |
| Item properties file contain timestamp column because all of them are time dependent, since properties may change over time, e.g. price, category, etc. Initially, this file consisted of snapshots for every week in the events file and contained over 200 millions rows. We have merged consecutive constant property values, so it's changed from snapshot form to change log form. Thus, constant values would appear only once in the file. This action has significantly reduced the number of rows in 10 times. | |
| All values in the “item_properties.csv” file excluding ""categoryid"" and ""available"" properties were hashed. Value of the ""categoryid"" property contains item category identifier. Value of the ""available"" property contains availability of the item, i.e. 1 means the item was available, otherwise 0. All numerical values were marked with ""n"" char at the beginning, and have 3 digits precision after decimal point, e.g., ""5"" will become ""n5.000"", ""-3.67584"" will become ""n-3.675"". All words in text values were normalized (stemming procedure: https://en.wikipedia.org/wiki/Stemming) and hashed, numbers were processed as above, e.g. text ""Hello world 2017!"" will become ""24214 44214 n2017.000"" | |
| The category tree file has 1669 rows. Every row in the file specifies a child categoryId and the corresponding parent. For example: | |
| Line “100,200” means that categoryid=1 has parent with categoryid=200 | |
| Line “300,” means that categoryid hasn’t parent in the tree | |
| Acknowledgements | |
| Retail Rocket (retailrocket.io) helps web shoppers make better shopping decisions by providing personalized real-time recommendations through multiple channels with over 100MM unique monthly users and 1000+ retail partners over the world. | |
| Inspiration | |
| How to use item properties and category tree data to improve collaborative filtering model? | |
| Recurrent Neural Networks with Top-k Gains for Session-based Recommendations https://github.com/hidasib/GRU4Rec and paper https://arxiv.org/abs/1706.03847 | |
| https://www.researchgate.net/publication/280538158_Application_of_Kullback-Leibler_divergence_for_short-term_user_interest_detection | |
| https://pdfs.semanticscholar.org/66dc/1724c4ed1e74fe6b22e636b52031a33c8ebe.pdf https://www.slideshare.net/LukasLerche/adaptation-and-evaluation-of-recommendationsfor-shortterm-shopping-goals Adaptation and Evaluation of Recommendations for Short-term Shopping Goals | |
| Tasks | |
| Task 1 | |
| When a customer comes to an e-commerce site, he looks for a product with particular properties: price range, vendor, product type and etc. These properties are implicit, so it's hard to determine them through clicks log. | |
| Try to create an algorithm which predicts properties of items in ""addtocart"" event by using data from ""view"" events for any visitor in the published log. | |
| Task 2 | |
| Description: | |
| Process of analyzing ecommerce data include very important part of data cleaning. Researchers noticed that in some cases browsing data include up to 40% of abnormal traffic. | |
| Firstly, abnormal users add a lot of noise into data and make recommendation system less effective. In order to increase efficiency of recommendation system, abnormal users should be removed from the raw data. | |
| Secondly, abnormal users add bias to results of split tests, so this type of users should be removed also from split test data. | |
| Goals: | |
| The main goal is to find abnormal users of e-shop. | |
| Subgoals: | |
| Generate features | |
| Build a model | |
| Create a metric that helps to evaluate quality of the model" | |
| Social Power NBA,"NBA on the court performance with Social Influence, Popularity and Power",Noah Gift,61,"Version 1,2017-08-01","basketball | |
| social groups",CSV,8 MB,CC4,"10,136 views","1,914 downloads",37 kernels,,https://www.kaggle.com/noahgift/social-power-nba,"Context | |
| This data set contains combined on-court performance data for NBA players in the 2016-2017 season, alongside salary, Twitter engagement, and Wikipedia traffic data. | |
| Further information can be found in a series of articles for IBM Developerworks: ""Explore valuation and attendance using data science and machine learning"" and ""Exploring the individual NBA players"". | |
| Acknowledgement | |
| Data sources include ESPN, Basketball-Reference, Twitter, Five-ThirtyEight, and Wikipedia. The source code for this dataset (in Python and R) can be found on GitHub. Links to more writing can be found at noahgift.com. | |
| Inspiration | |
| Do NBA fans know more about who the best players are, or do owners? | |
| What is the true worth of the social media presence of athletes in the NBA?" | |
| Fruits 360 dataset,A dataset with 60 fruits and 38409 images,Mihai Oltean,61,"Version 13,2018-02-08|Version 12,2018-02-07|Version 11,2018-02-05|Version 10,2018-01-27|Version 9,2018-01-26|Version 8,2018-01-19|Version 7,2018-01-14|Version 6,2018-01-11|Version 5,2018-01-11|Version 4,2018-01-02|Version 3,2017-12-30|Version 2,2017-12-27|Version 1,2017-12-02","food and drink | |
| image data | |
| multiclass classification",Other,180 MB,CC4,"7,198 views","1,075 downloads",3 kernels,0 topics,https://www.kaggle.com/moltean/fruits,"Context | |
| A high-quality, dataset of images containing fruits. The following fruits are included: Apples - (different varieties: Golden, Golden-Red, Granny Smith, Red, Red Delicious), Apricot, Avocado, Avocado ripe, Banana (Yellow, Red), Cactus fruit, Carambula, Cherry, Clementine, Cocos, Dates, Granadilla, Grape (Pink, White, White2), Grapefruit (Pink, White), Guava, Huckleberry, Kiwi, Kaki, Kumsquats, Lemon (normal, Meyer), Lime, Litchi, Mandarine, Mango, Maracuja, Nectarine, Orange, Papaya, Passion fruit, Peach, Pepino, Pear (different varieties, Abate, Monster, Williams), Pineapple, Pitahaya Red, Plum, Pomegranate, Quince, Raspberry, Salak, Strawberry, Tamarillo, Tangelo. | |
| Dataset properties | |
| Training set size: 28736 images. | |
| Validation set size: 9673 images. | |
| Number of classes: 60 (fruits). | |
| Image size: 100x100 pixels. | |
| Filename format: image_index_100.jpg (e.g. 32_100.jpg) or r_image_index_100.jpg (e.g. r_32_100.jpg). ""r"" stands for rotated fruit. ""100"" comes from image size (100x100 pixels). | |
| Different varieties of the same fruit (apple for instance) are shown having different labels. | |
| Content | |
| Fruits were planted in the shaft of a low speed motor (3 rpm) and a short movie of 20 seconds was recorded. | |
| A Logitech C920 camera was used for filming the fruits. This is one of the best webcams available. | |
| Behind the fruits we placed a white sheet of paper as background. | |
| However due to the variations in the lighting conditions, the background was not uniform and we wrote a dedicated algorithm which extract the fruit from the background. This algorithm is of flood fill type: we start from each edge of the image and we mark all pixels there, then we mark all pixels found in the neighborhood of the already marked pixels for which the distance between colors is less than a prescribed value. We repeat the previous step until no more pixels can be marked. | |
| All marked pixels are considered as being background (which is then filled with white) and the rest of pixels are considered as belonging to the object. | |
| The maximum value for the distance between 2 neighbor pixels is a parameter of the algorithm and is set (by trial and error) for each movie. | |
| How to cite | |
| Horea Muresan, Mihai Oltean, Fruit recognition from images using deep learning, Technical Report, Babes-Bolyai University, 2017 | |
| Alternate download | |
| This dataset is also available for download from GitHub: Fruits-360 dataset | |
| History | |
| Fruits were filmed at the dates given below: | |
| 2017.02.25 - Apple (golden). | |
| 2017.02.28 - Apple (red-yellow, red, golden2), Kiwi, Pear, Grapefruit, Lemon, Orange, Strawberry, Banana. | |
| 2017.03.05 - Apple (golden3, Braeburn, Granny Smith, red2). | |
| 2017.03.07 - Apple (red3). | |
| 2017.05.10 - Plum, Peach, Peach flat, Apricot, Nectarine, Pomegranate. | |
| 2017.05.27 - Avocado, Papaya, Grape, Cherrie. | |
| 2017.12.25 - Carambula, Cactus fruit, Granadilla, Kaki, Kumsquats, Passion fruit, Avocado ripe, Quince. | |
| 2017.12.28 - Clementine, Cocos, Mango, Lime, Litchi. | |
| 2017.12.31 - Apple Red Delicious, Pear Monster, Grape White. | |
| 2018.01.14 - Ananas, Grapefruit Pink, Mandarine, Pineapple, Tangelo. | |
| 2018.01.19 - Huckleberry, Raspberry. | |
| 2018.01.26 - Dates, Maracuja, Salak, Tamarillo. | |
| 2018.02.05 - Guava, Grape White 2, Lemon Meyer | |
| 2018.02.07 - Banana Red, Pepino, Pitahaya Red. | |
| 2018.02.08 - Pear Abate, Pear Williams." | |
| Online Courses from Harvard and MIT,What subjects or courses are the most popular on edX?,edX,61,"Version 1,2017-01-27",education,CSV,65 KB,Other,"15,138 views","2,046 downloads",59 kernels,0 topics,https://www.kaggle.com/edx/course-study,"Context | |
| In 2012, the Massachusetts Institute of Technology (MIT) and Harvard University launched open online courses on edX, a non-profit learning platform co-founded by the two institutions. Four years later, what have we learned about these online “classrooms” and the global community of learners who take them? | |
| Content | |
| This report provides data on 290 Harvard and MIT online courses, 250 thousand certifications, 4.5 million participants, and 28 million participant hours on the edX platform since 2012. | |
| Acknowledgements | |
| Isaac Chuang, a professor at MIT, and Andrew Ho, a professor at Harvard University, published this data as an appendix to their paper ""HarvardX and MITx: Four Years of Open Online Courses""." | |
| goodbooks-10k,"Ten thousand books, one million ratings. Also books marked to read, and tags.",Foxtrot,60,"Version 5,2017-09-02|Version 4,2017-08-29|Version 3,2017-08-24|Version 2,2017-08-22|Version 1,2017-08-14",books,CSV,41 MB,CC4,"16,076 views","1,803 downloads",19 kernels,6 topics,https://www.kaggle.com/zygmunt/goodbooks-10k,"This version of the dataset is obsolete. It contains duplicate ratings (same user_id,book_id), as reported by Philipp Spachtholz in his illustrious notebook. | |
| The current version has duplicates removed, and more ratings (six million), sorted by time. Book and user IDs are the same. | |
| It is available at https://github.com/zygmuntz/goodbooks-10k. | |
| There have been good datasets for movies (Netflix, Movielens) and music (Million Songs) recommendation, but not for books. That is, until now. | |
| This dataset contains ratings for ten thousand popular books. As to the source, let's say that these ratings were found on the internet. Generally, there are 100 reviews for each book, although some have less - fewer - ratings. Ratings go from one to five. | |
| Both book IDs and user IDs are contiguous. For books, they are 1-10000, for users, 1-53424. All users have made at least two ratings. Median number of ratings per user is 8. | |
| There are also books marked to read by the users, book metadata (author, year, etc.) and tags. | |
| Contents | |
| ratings.csv contains ratings and looks like that: | |
| book_id,user_id,rating | |
| 1,314,5 | |
| 1,439,3 | |
| 1,588,5 | |
| 1,1169,4 | |
| 1,1185,4 | |
| to_read.csv provides IDs of the books marked ""to read"" by each user, as user_id,book_id pairs. | |
| books.csv has metadata for each book (goodreads IDs, authors, title, average rating, etc.). | |
| The metadata have been extracted from goodreads XML files, available in the third version of this dataset as books_xml.tar.gz. The archive contains 10000 XML files. One of them is available as sample_book.xml. To make the download smaller, these files are absent from the current version. Download version 3 if you want them. | |
| book_tags.csv contains tags/shelves/genres assigned by users to books. Tags in this file are represented by their IDs. | |
| tags.csv translates tag IDs to names. | |
| See the notebook for some basic stats of the dataset. | |
| goodreads IDs | |
| Each book may have many editions. goodreads_book_id and best_book_id generally point to the most popular edition of a given book, while goodreads work_id refers to the book in the abstract sense. | |
| You can use the goodreads book and work IDs to create URLs as follows: | |
| https://www.goodreads.com/book/show/2767052 | |
| https://www.goodreads.com/work/editions/2792775" | |
| Top 500 Indian Cities,What story do the top 500 cities of India tell to the world?,Arijit Mukherjee,60,"Version 4,2016-12-21|Version 3,2016-12-21|Version 2,2016-12-17|Version 1,2016-12-17","cities | |
| sociology",CSV,73 KB,CC0,"23,972 views","2,823 downloads",171 kernels,9 topics,https://www.kaggle.com/zed9941/top-500-indian-cities,"Context | |
| I created this data set merging the census 2011 of Indian Cities with Population more than 1 Lac and City wise number of Graduates from the Census 2011, to create a visualization of where the future cities of India stands today, I will try to add more columns [ fertility rate, religion distribution, health standards, number of schools, Mortality rate ] in the future, hope people will contribute. | |
| Content | |
| Data of 500 Cities with population more than 1 Lac by Census 2011 | |
| 'name_of_city' : Name of the City | |
| 'state_code' : State Code of the City | |
| 'state_name' : State Name of the City | |
| 'dist_code' : District Code where the city belongs ( 99 means multiple district ) | |
| 'population_total' : Total Population | |
| 'population_male' : Male Population | |
| 'population_female' : Female Population | |
| '0-6_population_total' : 0-6 Age Total Population | |
| '0-6_population_male' : 0-6 Age Male Population | |
| '0-6_population_female' : 0-6 Age Female Population | |
| 'literates_total' : Total Literates | |
| 'literates_male' : Male Literates | |
| 'literates_female' : Female Literates | |
| 'sex_ratio' : Sex Ratio | |
| 'child_sex_ratio' : Sex ratio in 0-6 | |
| 'effective_literacy_rate_total' : Literacy rate over Age 7 | |
| 'effective_literacy_rate_male' : Male Literacy rate over Age 7 | |
| 'effective_literacy_rate_female': Female Literacy rate over Age 7 | |
| 'location' : Lat,Lng | |
| 'total_graduates' : Total Number of Graduates | |
| 'male_graduates' : Male Graduates | |
| 'female_graduates' : Female Graduates | |
| Acknowledgements | |
| Census 2011 | |
| http://censusindia.gov.in/2011-prov-results/paper2/data_files/India2/Table_2_PR_Cities_1Lakh_and_Above.xls | |
| Google Geocoder for Location Fetching. | |
| Graduation Data Census 2011 | |
| http://www.censusindia.gov.in/2011census/C-series/DDWCT-0000C-08.xlsx | |
| Inspiration | |
| What story do the top 500 cities of India tell to the world? I wrote a post in my blog about the dataset ." | |
| Detailed NFL Play-by-Play Data 2009-2016,nflscrapR generated NFL dataset wiith expected points and win probability,Max Horowitz,59,"Version 3,2018-01-10|Version 2,2017-07-28|Version 1,2017-07-26","american football | |
| sports",CSV,67 MB,Other,"11,493 views","1,149 downloads",2 kernels,7 topics,https://www.kaggle.com/maxhorowitz/nflplaybyplay2009to2016,"Introduction | |
| The lack of publicly available National Football League (NFL) data sources has been a major obstacle in the creation of modern, reproducible research in football analytics. While clean play-by-play data is available via open-source software packages in other sports (e.g. nhlscrapr for hockey; PitchF/x data in baseball; the Basketball Reference for basketball), the equivalent datasets are not freely available for researchers interested in the statistical analysis of the NFL. To solve this issue, a group of Carnegie Mellon University statistical researchers including Maksim Horowitz, Ron Yurko, and Sam Ventura, built and released nflscrapR an R package which uses an API maintained by the NFL to scrape, clean, parse, and output clean datasets at the individual play, player, game, and season levels. Using the data outputted by the package, the trio went on to develop reproducible methods for building expected point and win probability models for the NFL. The outputs of these models are included in this dataset and can be accessed using the nflscrapR package. | |
| Content | |
| The dataset made available on Kaggle contains all the regular season plays from the 2009-2016 NFL seasons. The dataset has 356,768 rows and 100 columns. Each play is broken down into great detail containing information on: game situation, players involved, results, and advanced metrics such as expected point and win probability values. Detailed information about the dataset can be found at the following web page, along with more NFL data: https://github.com/ryurko/nflscrapR-data. | |
| Acknowledgements | |
| This dataset was compiled by Ron Yurko, Sam Ventura, and myself. Special shout-out to Ron for improving our current expected points and win probability models and compiling this dataset. All three of us are proud founders of the Carnegie Mellon Sports Analytics Club. | |
| Inspiration | |
| This dataset is meant to both grow and bring together the community of sports analytics by providing clean and easily accessible NFL data that has never been availabe on this scale for free." | |
| The Simpsons Characters Data,Image dataset of 20 characters from The Simpsons,alexattia,58,"Version 3,2017-07-01|Version 2,2017-06-20|Version 1,2017-06-15","popular culture | |
| image data | |
| object detection",Other,588 MB,CC4,"16,589 views","2,701 downloads",5 kernels,5 topics,https://www.kaggle.com/alexattia/the-simpsons-characters-dataset,"Feel free to check and recommend my Medium post Part 1 on a classification model and Part 2 on a detection model (Faster R-CNN) about this dataset and what I am doing with it. | |
| You can also find the related GitHub repo here . | |
| Context | |
| As a big Simpsons fan, I have watched a lot (and still watching) of The Simpson episodes -multiple times each- over the years. I wanted to build a neural network which can recognize characters | |
| Content | |
| I am still building this dataset (labeling pictures), I will upload new versions of this dataset. Please check the files there are descriptions and explanations. | |
| File simpson-set.tar.gz : This is an image dataset: 20 folders (one for each character) with 400-2000 pictures in each folder. | |
| File simpson-test-set.zip. : Preview of the image dataset | |
| File weights.best.h5 : Weights computed, in order to predict in Kernels. | |
| File annotation.txt : Annotation file for bounding boxes for each character | |
| Help me to build this dataset | |
| If someone wants to contribute and make this dataset bigger and more relevant, any help will be appreciated. | |
| Acknowledgements | |
| Data is directly taken and labeled from TV show episodes." | |
| Glass Classification,Can you correctly identify glass type?,UCI Machine Learning,58,"Version 1,2017-01-28","chemistry | |
| artificial intelligence",CSV,10 KB,ODbL,"28,823 views","3,249 downloads",222 kernels,3 topics,https://www.kaggle.com/uciml/glass,"Context | |
| This is a Glass Identification Data Set from UCI. It contains 10 attributes including id. The response is glass type(discrete 7 values) | |
| Content | |
| Attribute Information: | |
| Id number: 1 to 214 (removed from CSV file) | |
| RI: refractive index | |
| Na: Sodium (unit measurement: weight percent in corresponding oxide, as are attributes 4-10) | |
| Mg: Magnesium | |
| Al: Aluminum | |
| Si: Silicon | |
| K: Potassium | |
| Ca: Calcium | |
| Ba: Barium | |
| Fe: Iron | |
| Type of glass: (class attribute) -- 1 building_windows_float_processed -- 2 building_windows_non_float_processed -- 3 vehicle_windows_float_processed -- 4 vehicle_windows_non_float_processed (none in this database) -- 5 containers -- 6 tableware -- 7 headlamps | |
| Acknowledgements | |
| https://archive.ics.uci.edu/ml/datasets/Glass+Identification Source: | |
| Creator: B. German Central Research Establishment Home Office Forensic Science Service Aldermaston, Reading, Berkshire RG7 4PN | |
| Donor: Vina Spiehler, Ph.D., DABFT Diagnostic Products Corporation (213) 776-0180 (ext 3014) | |
| Inspiration | |
| Data exploration of this dataset reveals two important characteristics : 1) The variables are highly corelated with each other including the response variables: So which kind of ML algorithm is most suitable for this dataset Random Forest , KNN or other? Also since dataset is too small is there any chance of applying PCA or it should be completely avoided? | |
| 2) Highly Skewed Data: Is scaling sufficient or are there any other techniques which should be applied to normalize data? Like BOX-COX Power transformation?" | |
| Python Questions from Stack Overflow,Full text of Stack Overflow Q&A about the Python programming language,Stack Overflow,58,"Version 1,2016-10-21","internet | |
| programming languages",CSV,2 GB,Other,"16,001 views","1,534 downloads",48 kernels,,https://www.kaggle.com/stackoverflow/pythonquestions,"Context | |
| Full text of all questions and answers from Stack Overflow that are tagged with the python tag. Useful for natural language processing and community analysis. See also the dataset of R questions. | |
| Content | |
| This dataset is organized as three tables: | |
| Questions contains the title, body, creation date, score, and owner ID for each Python question. | |
| Answers contains the body, creation date, score, and owner ID for each of the answers to these questions. The ParentId column links back to the Questions table. | |
| Tags contains the tags on each question besides the Python tag. | |
| Questions may be deleted by the user who posted them. They can also be closed by community vote, if the question is deemed off-topic for instance. Such questions are not included in this dataset. | |
| The dataset contains questions all questions asked between August 2, 2008 and Ocotober 19, 2016. | |
| License | |
| All Stack Overflow user contributions are licensed under CC-BY-SA 3.0 with attribution required." | |
| Biodiversity in National Parks,Plant and animal species found in the American national park system,National Park Service,58,"Version 3,2017-01-20|Version 2,2017-01-20|Version 1,2017-01-17","ecology | |
| animals | |
| plants",CSV,17 MB,CC0,"9,304 views","1,631 downloads",16 kernels,3 topics,https://www.kaggle.com/nationalparkservice/park-biodiversity,"Context | |
| The National Park Service publishes a database of animal and plant species identified in individual national parks and verified by evidence — observations, vouchers, or reports that document the presence of a species in a park. All park species records are available to the public on the National Park Species portal; exceptions are made for sensitive, threatened, or endangered species when widespread distribution of information could pose a risk to the species in the park. | |
| Content | |
| National Park species lists provide information on the presence and status of species in our national parks. These species lists are works in progress and the absence of a species from a list does not necessarily mean the species is absent from a park. The time and effort spent on species inventories varies from park to park, which may result in data gaps. Species taxonomy changes over time and reflects regional variations or preferences; therefore, records may be listed under a different species name. | |
| Each park species record includes a species ID, park name, taxonomic information, scientific name, one or more common names, record status, occurrence (verification of species presence in park), nativeness (species native or foreign to park), abundance (presence and visibility of species in park), seasonality (season and nature of presence in park), and conservation status (species classification according to US Fish & Wildlife Service). Taxonomic classes have been translated from Latin to English for species categorization; order, family, and scientific name (genus, species, subspecies) are in Latin. | |
| Acknowledgements | |
| The National Park Service species list database is managed and updated by staff at individual national parks and the systemwide Inventory and Monitoring department." | |
| Formula 1 Race Data,Race data from 1950 to 2017,Chris G,57,"Version 1,2017-11-29","auto racing | |
| sports",CSV,6 MB,CC4,"7,684 views","1,572 downloads",,2 topics,https://www.kaggle.com/cjgdev/formula-1-race-data-19502017,"Context | |
| Formula One (also Formula 1 or F1 and officially the FIA Formula One World Championship) is the highest class of single-seat auto racing that is sanctioned by the Fédération Internationale de l'Automobile (FIA). The FIA Formula One World Championship has been one of the premier forms of racing around the world since its inaugural season in 1950. | |
| Content | |
| This dataset contains data from 1950 all the way through the 2017 season, and consists of tables describing constructors, race drivers, lap times, pit stops and more. | |
| Acknowledgements | |
| The data was downloaded from http://ergast.com/mrd/ at the conclusion of the 2017 season. The data was originally gathered and published to the public domain by Chris Newell. | |
| Inspiration | |
| I think this dataset offers an exciting insight into a $ billion industry, enjoyed by hundreds of millions of viewers all over the world. So please, explore and enjoy!" | |
| Prescription-based prediction,Predicting doctor attributes from prescription behavior,Roam Analytics,56,"Version 1,2016-10-17","healthcare | |
| artificial intelligence",{}JSON,156 MB,CC4,"16,920 views","1,518 downloads",38 kernels,,https://www.kaggle.com/roamresearch/prescriptionbasedprediction,"This is the dataset used in the Roam blog post Prescription-based prediction. It is derived from a variety of US open health datasets, but the bulk of the data points come from the Medicare Part D dataset and the National Provider Identifier dataset. | |
| The prescription vector for each doctor tells a rich story about that doctor's attributes, including specialty, gender, age, and region. There are 239,930 doctors in the dataset. | |
| The file is in JSONL format (one JSON record per line): | |
| { | |
| 'provider_variables': | |
| { | |
| 'brand_name_rx_count': int, | |
| 'gender': 'M' or 'F', | |
| 'generic_rx_count': int, | |
| 'region': 'South' or 'MidWest' or 'Northeast' or 'West', | |
| 'settlement_type': 'non-urban' or 'urban' | |
| 'specialty': str | |
| 'years_practicing': int | |
| }, | |
| 'npi': str, | |
| 'cms_prescription_counts': | |
| { | |
| `drug_name`: int, | |
| `drug_name`: int, | |
| ... | |
| } | |
| } | |
| The brand/generic classifications behind brand_name_rx_count and generic_rx_count are defined heuristically. For more details, see the blog post or go directly to the associated code." | |
| Meteorite Landings,Data on over 45k meteorites that have struck Earth,NASA,56,"Version 2,2016-11-06|Version 1,2016-11-05","astronomy | |
| space",CSV,4 MB,CC0,"18,161 views","2,212 downloads",60 kernels,2 topics,https://www.kaggle.com/nasa/meteorite-landings,"The Meteoritical Society collects data on meteorites that have fallen to Earth from outer space. This dataset includes the location, mass, composition, and fall year for over 45,000 meteorites that have struck our planet. | |
| Notes on missing or incorrect data points: | |
| a few entries here contain date information that was incorrectly parsed into the NASA database. As a spot check: any date that is before 860 CE or after 2016 are incorrect; these should actually be BCE years. There may be other errors and we are looking for a way to identify them. | |
| a few entries have latitude and longitude of 0N/0E (off the western coast of Africa, where it would be quite difficult to recover meteorites). Many of these were actually discovered in Antarctica, but exact coordinates were not given. 0N/0E locations should probably be treated as NA. | |
| The starter kernel for this dataset has a quick way to filter out these observations using dplyr in R, provided here for convenience: | |
| meteorites.geo <- meteorites.all %>% | |
| filter(year>=860 & year<=2016) %>% # filter out weird years | |
| filter(reclong<=180 & reclong>=-180 & (reclat!=0 | reclong!=0)) # filter out weird locations | |
| The Data | |
| Note that a few column names start with ""rec"" (e.g., recclass, reclat, reclon). These are the recommended values of these variables, according to The Meteoritical Society. In some cases, there were historical reclassification of a meteorite, or small changes in the data on where it was recovered; this dataset gives the currently recommended values. | |
| The dataset contains the following variables: | |
| name: the name of the meteorite (typically a location, often modified with a number, year, composition, etc) | |
| id: a unique identifier for the meteorite | |
| nametype: one of: | |
| -- valid: a typical meteorite | |
| -- relict: a meteorite that has been highly degraded by weather on Earth | |
| recclass: the class of the meteorite; one of a large number of classes based on physical, chemical, and other characteristics (see the Wikipedia article on meteorite classification for a primer) | |
| mass: the mass of the meteorite, in grams | |
| fall: whether the meteorite was seen falling, or was discovered after its impact; one of: | |
| -- Fell: the meteorite's fall was observed | |
| -- Found: the meteorite's fall was not observed | |
| year: the year the meteorite fell, or the year it was found (depending on the value of fell) | |
| reclat: the latitude of the meteorite's landing | |
| reclong: the longitude of the meteorite's landing | |
| GeoLocation: a parentheses-enclose, comma-separated tuple that combines reclat and reclong | |
| What can we do with this data? | |
| Here are a couple of thoughts on questions to ask and ways to look at this data: | |
| how does the geographical distribution of observed falls differ from that of found meteorites? -- this would be great overlaid on a cartogram or alongside a high-resolution population density map | |
| are there any geographical differences or differences over time in the class of meteorites that have fallen to Earth? | |
| Acknowledgements | |
| This dataset was downloaded from NASA's Data Portal, and is based on The Meteoritical Society's Meteoritical Bulletin Database (this latter database provides additional information such as meteorite images, links to primary sources, etc.)." | |
| NIPS Papers,"Titles, authors, abstracts, and extracted text for all NIPS papers (1987-2017)",Ben Hamner,55,"Version 2,2017-12-06|Version 1,2016-12-04","linguistics | |
| artificial intelligence",CSV,142 MB,ODbL,"20,790 views","1,519 downloads",68 kernels,0 topics,https://www.kaggle.com/benhamner/nips-papers,"Neural Information Processing Systems (NIPS) is one of the top machine learning conferences in the world. It covers topics ranging from deep learning and computer vision to cognitive science and reinforcement learning. | |
| This dataset includes the title, authors, abstracts, and extracted text for all NIPS papers to date (ranging from the first 1987 conference to the current 2016 conference). I've extracted the paper text from the raw PDF files and are releasing that both in CSV files and as a SQLite database. The code to scrape and create this dataset is on GitHub. Here's a quick RMarkdown exploratory overview of what's in the data. We encourage you to explore this data and share what you find through Kaggle Kernels!" | |
| New York City Taxi with OSRM,Helpful dataset for New York Taxi Playground,oscarleo,55,"Version 5,2017-08-06|Version 4,2017-07-30|Version 3,2017-07-29|Version 2,2017-07-27|Version 1,2017-07-26",taxi services,CSV,2 GB,ODbL,"12,073 views","3,702 downloads",45 kernels,10 topics,https://www.kaggle.com/oscarleo/new-york-city-taxi-with-osrm,"Context | |
| I created this data set to help with the New York City Taxi Trip Duration playground. I used OSRM to extract information about the fastest routes for each data point. | |
| I think it will be very useful for anyone doing work in that competition. Please try it out and tell me what you want me to add. I intend to improve and add features. | |
| Content | |
| starting_street | |
| The street where the taxi-trip starts. In version 1 this field contained a lot of empty values. That has been dealt with. | |
| end_street | |
| The street where the taxi-trip ends. In version 1 this field contained a lot of empty values. That has been dealt with. | |
| total_distance | |
| The total distance is measured between the pickup coordinates and the drop-off coordinates in train.csv and test.csv. The unit is meters. | |
| total_travel_time | |
| The total travel time for that data point in seconds | |
| number_of_steps | |
| The number of steps on that trip. One step consists of some driving and an action the taxi needs to perform. It can be something like a turn or going on to a highway. See step_maneuvers for more information. | |
| street_for_each_step | |
| A list of streets where each step occurs. Multiple steps can be performed on the same street. Therefore there might the same street might occur multiple times. | |
| (The values are stored as a string separated by '|') | |
| distance_per_step | |
| The distance for each step. | |
| (The values are stored as a string separated by '|') | |
| travel_time_per_step | |
| The travel time for each step | |
| (The values are stored as a string separated by '|') | |
| step_maneuvers | |
| The action (or maneuver) performed in each step. The possible maneuvers are: | |
| turn: a basic turn | |
| new name: no turn is taken/possible, but the road name changes. | |
| depart: The trip starts | |
| arrive: The trip ends | |
| merge: Merge onto a street (e.g. getting on the highway from a ramp) | |
| on ramp: Entering a highway (direction given my modifier ) | |
| off ramp: Exiting a highway | |
| fork: The road forks | |
| end of road: The road ends in a T intersection | |
| continue: Turn to stay on the same road | |
| roundabout: A roundabout | |
| rotary A traffic circle (a bigger roundabout) | |
| roundabout turn: A small roundabout that can be treated as a regular turn | |
| (The values are stored as a string separated by '|') | |
| step_direction | |
| The direction for each action (or maneuver) | |
| step_location_list | |
| The coordinates for each action (or maneuver)" | |
| NIH Chest X-rays,"Over 112,000 Chest X-ray images from more than 30,000 unique patients",National Institutes of Health Chest X-Ray Dataset,55,"Version 3,2018-02-22|Version 2,2017-12-06|Version 1,2017-12-02","medicine | |
| machine learning",Other,42 GB,CC0,"8,668 views","2,051 downloads",8 kernels,0 topics,https://www.kaggle.com/nih-chest-xrays/data,"NIH Chest X-ray Dataset | |
| National Institutes of Health Chest X-Ray Dataset | |
| Chest X-ray exams are one of the most frequent and cost-effective medical imaging examinations available. However, clinical diagnosis of a chest X-ray can be challenging and sometimes more difficult than diagnosis via chest CT imaging. The lack of large publicly available datasets with annotations means it is still very difficult, if not impossible, to achieve clinically relevant computer-aided detection and diagnosis (CAD) in real world medical sites with chest X-rays. One major hurdle in creating large X-ray image datasets is the lack resources for labeling so many images. Prior to the release of this dataset, Openi was the largest publicly available source of chest X-ray images with 4,143 images available. | |
| This NIH Chest X-ray Dataset is comprised of 112,120 X-ray images with disease labels from 30,805 unique patients. To create these labels, the authors used Natural Language Processing to text-mine disease classifications from the associated radiological reports. The labels are expected to be >90% accurate and suitable for weakly-supervised learning. The original radiology reports are not publicly available but you can find more details on the labeling process in this Open Access paper: ""ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases."" (Wang et al.) | |
| Link to paper | |
| Data limitations: | |
| The image labels are NLP extracted so there could be some erroneous labels but the NLP labeling accuracy is estimated to be >90%. | |
| Very limited numbers of disease region bounding boxes (See BBox_list_2017.csv) | |
| Chest x-ray radiology reports are not anticipated to be publicly shared. Parties who use this public dataset are encouraged to share their “updated” image labels and/or new bounding boxes in their own studied later, maybe through manual annotation | |
| File contents | |
| Image format: 112,120 total images with size 1024 x 1024 | |
| images_001.zip: Contains 4999 images | |
| images_002.zip: Contains 10,000 images | |
| images_003.zip: Contains 10,000 images | |
| images_004.zip: Contains 10,000 images | |
| images_005.zip: Contains 10,000 images | |
| images_006.zip: Contains 10,000 images | |
| images_007.zip: Contains 10,000 images | |
| images_008.zip: Contains 10,000 images | |
| images_009.zip: Contains 10,000 images | |
| images_010.zip: Contains 10,000 images | |
| images_011.zip: Contains 10,000 images | |
| images_012.zip: Contains 7,121 images | |
| README_ChestXray.pdf: Original README file | |
| BBox_list_2017.csv: Bounding box coordinates. Note: Start at x,y, extend horizontally w pixels, and vertically h pixels | |
| Image Index: File name | |
| Finding Label: Disease type (Class label) | |
| Bbox x | |
| Bbox y | |
| Bbox w | |
| Bbox h | |
| Data_entry_2017.csv: Class labels and patient data for the entire dataset | |
| Image Index: File name | |
| Finding Labels: Disease type (Class label) | |
| Follow-up # | |
| Patient ID | |
| Patient Age | |
| Patient Gender | |
| View Position: X-ray orientation | |
| OriginalImageWidth | |
| OriginalImageHeight | |
| OriginalImagePixelSpacing_x | |
| OriginalImagePixelSpacing_y | |
| Class descriptions | |
| There are 15 classes (14 diseases, and one for ""No findings""). Images can be classified as ""No findings"" or one or more disease classes: | |
| Atelectasis | |
| Consolidation | |
| Infiltration | |
| Pneumothorax | |
| Edema | |
| Emphysema | |
| Fibrosis | |
| Effusion | |
| Pneumonia | |
| Pleural_thickening | |
| Cardiomegaly | |
| Nodule Mass | |
| Hernia | |
| Full Dataset Content | |
| There are 12 zip files in total and range from ~2 gb to 4 gb in size. Additionally, we randomly sampled 5% of these images and created a smaller dataset for use in Kernels. The random sample contains 5606 X-ray images and class labels. | |
| Sample: sample.zip | |
| Modifications to original data | |
| Original TAR archives were converted to ZIP archives to be compatible with the Kaggle platform | |
| CSV headers slightly modified to be more explicit in comma separation and also to allow fields to be self-explanatory | |
| Citations | |
| Wang X, Peng Y, Lu L, Lu Z, Bagheri M, Summers RM. ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases. IEEE CVPR 2017, ChestX-ray8_Hospital-Scale_Chest_CVPR_2017_paper.pdf | |
| NIH News release: NIH Clinical Center provides one of the largest publicly available chest x-ray datasets to scientific community | |
| Original source files and documents: https://nihcc.app.box.com/v/ChestXray-NIHCC/folder/36938765345 | |
| Acknowledgements | |
| This work was supported by the Intramural Research Program of the NClinical Center (clinicalcenter.nih.gov) and National Library of Medicine (www.nlm.nih.gov)." | |
| US Permanent Visa Applications,Detailed information on 374k visa decisions,Jacob Boysen,54,"Version 1,2017-08-25","government agencies | |
| immigration",CSV,285 MB,CC0,"8,287 views","1,278 downloads",7 kernels,0 topics,https://www.kaggle.com/jboysen/us-perm-visas,"Context: | |
| A permanent labor certification issued by the Department of Labor (DOL) allows an employer to hire a foreign worker to work permanently in the United States. In most instances, before the U.S. employer can submit an immigration petition to the Department of Homeland Security's U.S. Citizenship and Immigration Services (USCIS), the employer must obtain a certified labor certification application from the DOL's Employment and Training Administration (ETA). The DOL must certify to the USCIS that there are not sufficient U.S. workers able, willing, qualified and available to accept the job opportunity in the area of intended employment and that employment of the foreign worker will not adversely affect the wages and working conditions of similarly employed U.S. workers. | |
| Content: | |
| Data covers 2012-2017 and includes information on employer, position, wage offered, job posting history, employee education and past visa history, associated lawyers, and final decision. | |
| Acknowledgements: | |
| This data was collected and distributed by the US Department of Labor. | |
| Inspiration: | |
| Can you predict visa decisions based on employee/employer/wage? | |
| How does this data compare to H1B decisions in this dataset?" | |
| League of Legends Ranked Matches,180000 ranked games of League of Legends starting from 2014,Paolo Campanelli,54,"Version 9,2017-10-26|Version 8,2017-10-20|Version 7,2017-10-20|Version 6,2017-10-20|Version 5,2017-10-18|Version 4,2017-10-18|Version 3,2017-10-18|Version 2,2017-10-18|Version 1,2017-10-18",video games,CSV,696 MB,Other,"8,653 views","1,104 downloads",6 kernels,7 topics,https://www.kaggle.com/paololol/league-of-legends-ranked-matches,"League of Legends Ranked Matches | |
| Data about 184070 League of Legends ranked solo games, spanning across several years | |
| Content | |
| Matches | |
| Player and team stats | |
| Bans | |
| Acknowledgements | |
| I found this data on a SQL database and exported it to CSV. All data belongs ultimately to Riot Games and their data policies applies. These files are presented only as a simpler way to obtain a large dataset without stressing the Riot API and are in no way associated with Riot Games. The data is provided as-is without any warranty on its correctness. If your algorithm catches fire, don't blame me or Riot. If you are Rito and are opposed to sharing this data here, contact me and it will be removed immediately. | |
| Possible questions | |
| Can we predict the winner given the teams? | |
| Can ranked matchmaking be assumed to be unbiased (or adjusted for red-side advantage)? | |
| Does the region affect significantly win rates? | |
| Can we compare the data in relation to competitive data (also available on Kaggle)? | |
| Can we assess information on the different metas?" | |
| Steam Video Games,Recommend video games from 200k steam user interactions.,Tamber,54,"Version 3,2017-03-09|Version 2,2017-03-04|Version 1,2017-03-04",video games,CSV,9 MB,ODbL,"16,123 views","1,832 downloads",35 kernels,2 topics,https://www.kaggle.com/tamber/steam-video-games,"Context | |
| Steam is the world's most popular PC Gaming hub, with over 6,000 games and a community of millions of gamers. With a massive collection that includes everything from AAA blockbusters to small indie titles, great discovery tools are a highly valuable asset for Steam. How can we make them better? | |
| Content | |
| This dataset is a list of user behaviors, with columns: user-id, game-title, behavior-name, value. The behaviors included are 'purchase' and 'play'. The value indicates the degree to which the behavior was performed - in the case of 'purchase' the value is always 1, and in the case of 'play' the value represents the number of hours the user has played the game. | |
| Acknowledgements | |
| This dataset is generated entirely from public Steam data, so we want to thank Steam for building such an awesome platform and community! | |
| Inspiration | |
| The dataset is formatted to be compatible with Tamber. Build a Tamber engine and take it for a spin! | |
| Combine our collaborative filter's results with your favorite Machine Learning techniques with Ensemble Learning, or make Tamber do battle with something else you've built. | |
| Have fun, The Tamber Team" | |
| SF Bay Area Bike Share,Anonymized bike trip data from August 2013 to August 2015,Ben Hamner,53,"Version 1,2016-06-14","cycling | |
| road transport",CSV,4 GB,Other,"30,915 views","5,130 downloads",155 kernels,2 topics,https://www.kaggle.com/benhamner/sf-bay-area-bike-share,"The Bay Area Bike Share enables quick, easy, and affordable bike trips around the San Francisco Bay Area. They make regular open data releases (this dataset is a transformed version of the data from this link), plus maintain a real-time API. | |
| Exploration Ideas | |
| How does weather impact bike trips? | |
| How do bike trip patterns vary by time of day and the day of the week?" | |
| eCommerce Item Data,500 SKUs and their descriptions. Great for content engine training!,cclark,53,"Version 1,2016-08-18",business,CSV,553 KB,Other,"38,272 views","4,575 downloads",34 kernels,4 topics,https://www.kaggle.com/cclark/product-item-data,500 actual SKUs from an outdoor apparel brand's product catalog. It's somewhat rare to get real item level data in a real-world format. Very useful for testing things like recommendation engines. In fact...maybe I'll publish some code along with this :) | |
| Star Cluster Simulations,Direct N-body simulation of a star cluster: Position and velocities of stars,Mario Pasquato,53,"Version 1,2017-01-07","astronomy | |
| space",CSV,109 MB,CC0,"12,745 views","1,327 downloads",21 kernels,2 topics,https://www.kaggle.com/mariopasquato/star-cluster-simulations,"Context | |
| Stars mostly form in clusters and associations rather than in isolation. Milky Way star clusters are easily observable with small telescopes, and in some cases even with the naked eye. Depending on a variety of conditions, star clusters may dissolve quickly or be very long lived. The dynamical evolution of star clusters is a topic of very active research in astrophysics. Some popular models of star clusters are the so-called direct N-body simulations [1, 2], where every star is represented by a point particle that interacts gravitationally with every other particle. This kind of simulation is computationally expensive, as it scales as O(N^2) where N is the number of particles in the simulated cluster. In the following, the words ""particle"" and ""star"" are used interchangeably. | |
| Content | |
| This dataset contains the positions and velocities of simulated stars (particles) in a direct N-body simulation of a star cluster. In the cluster there are initially 64000 stars distributed in position-velocity space according to a King model [3]. Each .csv file named c_xxxx.csv corresponds to a snapshot of the simulation at time t = xxxx. For example, c_0000.csv contains the initial conditions (positions and velocities of stars at time t=0). Times are measured in standard N-body units [4]. This is a system of units where G = M = −4E = 1 (G is the gravitational constant, M the total mass of the cluster, and E its total energy). | |
| x, y, z Columns 1, 2, and 3 of each file are the x, y, z positions of the stars. They are also expressed in standard N-body units [4]. You can switch to units of the median radius of the cluster by finding the cluster center and calculating the median distance of stars from it, and then dividing x, y, and z by this number. In general, the median radius changes in time. The initial conditions are approximately spherically symmetric (you can check) so there is no particular physical meaning attached to the choice of x, y, and z. | |
| vx, vy, vz Columns 4, 5, and 6 contain the x, y, and z velocity, also in N-body units. A scale velocity for the stars can be obtained by taking the standard deviation of velocity along one direction (e.g. z). You may check that the ratio between the typical radius (see above) and the typical velocity is of order unity. | |
| m Column 7 is the mass of each star. For this simulation this is identically 1.5625e-05, i.e. 1/64000. The total mass of the cluster is initially 1. More realistic simulations (coming soon) have a spectrum of different masses and live stelar evolution, that results in changes in the mass of stars. This simulation is a pure N-body problem instead. | |
| Star id number The id numbers of each particle are listed in the last column (8) of the files under the header ""id"". The ids are unique and can be used to trace the position and velocity of a star across all files. There are initially 64000 particles. At end of the simulation there are 63970. This is because some particles escape the cluster. | |
| Acknowledgements | |
| This simulation was run on a Center for Galaxy Evolution Research (CGER) workstation at Yonsei University (Seoul, Korea), using the NBODY6 software (https://www.ast.cam.ac.uk/~sverre/web/pages/nbody.htm). | |
| Inspiration | |
| Some stars hover around the center of the cluster, while some other get kicked out to the cluster outskirts or even leave the cluster altogether. Can we predict where a star will be at any given time based on its initial position and velocity? Can we predict its velocity? | |
| How correlated are the motions of stars? Can we predict the velocity of a given star based on the velocity of its neighbours? | |
| The size of the cluster can be measured by defining a center (see below) and finding the median distance of stars from it. This is called the three-dimensional effective radius. Can we predict how it evolves over time? What are its properties as a time series? What can we say about other quantiles of the radius? | |
| How to define the cluster center? Just as the mode of a KDE of the distribution of stars? How does it move over time and how to quantify the properties of its fluctuations? Is the cluster symmetric around this center? | |
| Some stars leave the cluster: over time they exchange energy in close encounters with other stars and reach the escape velocity. This can be seen by comparing later snapshots with the initial one: some IDs are missing and there is overall a lower number of stars. Can we predict which stars are more likely to escape? When will a given star escape? | |
| References | |
| [1] Heggie, D., Hut, P. 2003, The Gravitational Million-Body Problem: A Multidisciplinary Approach to Star Cluster Dynamics ~ Cambridge University Press, 2003 | |
| [2] Aarseth, S.~J. 2003, Gravitational N-Body Simulations - Cambridge University Press, 2003 | |
| [3] King, I. 1966, AJ, 71, 64 | |
| [4] Heggie, D. C., Mathieu, R. D. 1986, Lecture Notes in Physics, Vol. 267, The Use of Supercomputers in Stellar Dynamics, Berlin, Springer" | |
| Global Shark Attacks,Data compiled by the global shark attack file,toby jolly,53,"Version 6,2018-01-02|Version 5,2017-06-21|Version 4,2017-06-21|Version 3,2016-09-30|Version 2,2016-09-30|Version 1,2016-09-29",oceans,CSV,543 KB,Other,"21,471 views","3,450 downloads",80 kernels,4 topics,https://www.kaggle.com/teajay/global-shark-attacks,"Context | |
| This is a table of shark attack incidents compiled by the Global Shark Attack File: Please see their website for more details on where this data comes from. | |
| Acknowledgements | |
| This data was downloaded with permission from the Global Shark Attack File's website" | |
| Pakistan Drone Attacks,"Most authentic count of drone strikes in Pakistan, 2004-2016",Zeeshan-ul-hassan Usmani,52,"Version 8,2017-12-01|Version 7,2017-10-26|Version 6,2017-10-26|Version 5,2017-10-10|Version 4,2017-05-02|Version 3,2017-03-23|Version 2,2017-01-29|Version 1,2017-01-25",military,CSV,475 KB,CC0,"11,241 views",975 downloads,44 kernels,3 topics,https://www.kaggle.com/zusmani/pakistandroneattacks,"Context | |
| Pakistan Drone Attacks (2004-2016) | |
| The United States has targeted militants in the Federally Administered Tribal Areas [FATA] and the province of Khyber Pakhtunkhwa [KPK] in Pakistan via its Predator and Reaper drone strikes since year 2004. Pakistan Body Count (www.PakistanBodyCount.org) is the oldest and most accurate running tally of drone strikes in Pakistan. The given database (PakistanDroneAttacks.CSV) has been populated by using majority of the data from Pakistan Body Count, and building up on it by canvassing open source newspapers, media reports, think tank analyses, and personal contacts in media and law enforcement agencies. We provide a count of the people killed and injured in drone strikes, including the ones who died later in hospitals or homes due to injuries caused or aggravated by drone strikes, making it the most authentic source for drone related data in this region. | |
| We will keep releasing the updates every quarter at this page. | |
| Content | |
| Geography: Pakistan | |
| Time period: 2004-2016 | |
| Unit of analysis: Attack | |
| Dataset: The dataset contains detailed information of 397 drone attacks in Pakistan that killed an estimated 3,558 and injured 1,333 people including 2,539 civilians. | |
| Variables: The dataset contains Serial No, Incident Day & Date, Approximate Time of the attack, Specific Location, City, Province, Number of people killed who claimed to be from Al-Qaeeda, Number of people killed who claimed to be from Taliban, minimum and maximum count of foreigners killed, minimum and maximum count of civilians killed, minimum and maximum count of civilians injured, special mention (more details) and comments about the attack, longitude and latitude of the location. Sources: Unclassified media articles, hospital reports, think tank analysis and reports, and government official press releases. | |
| Acknowledgements & References | |
| Pakistan Body Count has been leveraged extensively in scholarly publications, reports, media articles and books. The website and the dataset has been collected and curated by the founder Zeeshan-ul-hassan Usmani. Users are allowed to use, copy, distribute and cite the dataset as follows: “Zeeshan-ul-hassan Usmani, Pakistan Body Count, Drone Attacks Dataset, Kaggle Dataset Repository, Jan 25, 2017.” | |
| Past Research | |
| Zeeshan-ul-hassan Usmani and Hira Bashir, “The Impact of Drone Strikes in Pakistan”, Cost of War Project, Brown University, December 16, 2014 | |
| Inspiration | |
| Some ideas worth exploring: | |
| • How many people got killed and injured per year in last 12 years? | |
| • How many attacks involved killing of actual terrorists from Al-Qaeeda and Taliban? | |
| • How many attacks involved women and children? | |
| • Visualize drone attacks on timeline | |
| • Find out any correlation with number of drone attacks with specific date and time, for example, do we have more drone attacks in September? | |
| • Find out any correlation with drone attacks and major global events (US funding to Pakistan and/or Afghanistan, Friendly talks with terrorist outfits by local or foreign government?) | |
| • The number of drone attacks in Bush Vs Obama tenure? | |
| • The number of drone attacks versus the global increase/decrease in terrorism? | |
| • Correlation between number of drone strikes and suicide bombings in Pakistan | |
| Questions? | |
| For detailed visit www.PakistanBodyCount.org | |
| Or contact Pakistan Body Count staff at info@pakistanbodycount.org" | |
| South Park Dialogue,"More than 70,000 lines of dialogue by season, episode, and character",Ksenia Sukhova,52,"Version 1,2017-03-15","popular culture | |
| linguistics",CSV,5 MB,Other,"9,836 views","1,061 downloads",37 kernels,4 topics,https://www.kaggle.com/tovarischsukhov/southparklines,"South Park cartoon lines | |
| +70k lines, annotated with season, episode and speaker | |
| It is interesting to practice NLP with ML techniques in order to guess who is speaking. Later there will be file with pre-proccesed data to train" | |
| 2015 Traffic Fatalities,National Highway Traffic Safety Administration,NHTSA,52,"Version 3,2016-11-26|Version 2,2016-11-26|Version 1,2016-11-21",road transport,CSV,88 MB,ODbL,"23,938 views","3,337 downloads",188 kernels,5 topics,https://www.kaggle.com/nhtsa/2015-traffic-fatalities,"Quick Start | |
| For a quick introduction to this Dataset, take a look at the Kernel Traffic Fatalities Getting Started. | |
| See the Fatality Analysis Reporting System FARS User’s Manual for understanding the column abbreviations and possible values. | |
| Also, see the following reference | |
| Original source of this data containing all files can be obtained here | |
| Below are the files released by the (NHTSA) National Highway Traffic Safety Administration, in their original format. Additional files can be found in the extra folder. Reference Traffic Fatalities Getting Started. for how to access this extra folder with contents. | |
| Data Compared to 2014 | |
| A few interesting notes about this data compared to 2014 | |
| Pedalcyclist fatalities increased by 89 (12.2 percent) | |
| Motorcyclist fatalities increased by 382 (8.3-percent increase) | |
| Alcohol-impaired driving fatalities increased by 3.2 percent, from 9,943 in 2014 to 10,265 in 2015 | |
| Vehicle miles traveled (VMT) increased by 3.5 percent from 2014 to 2015, the largest increase since 1992, nearly 25 years ago. | |
| See TRAFFIC SAFETY FACTS for more detail on the above findings." | |
| World of Warcraft Avatar History,Track the players of this popular online game,Myles O'Neill,51,"Version 3,2016-06-14|Version 2,2016-05-21|Version 1,2016-05-20","games and toys | |
| video games",CSV,614 MB,CC0,"22,550 views","1,634 downloads",35 kernels,5 topics,https://www.kaggle.com/mylesoneill/warcraft-avatar-history,"Overview | |
| The World of Warcraft Avatar History Dataset is a collection of records that detail information about player characters in the game over time. It includes information about their character level, race, class, location, and social guild. The Kaggle version of this dataset includes only the information from 2008 (and the dataset in general only includes information from the 'Horde' faction of players in the game from a single game server). | |
| Full Dataset Source and Information: http://mmnet.iis.sinica.edu.tw/dl/wowah/ | |
| Code used to clean the data: https://github.com/myles-oneill/WoWAH-parser | |
| Ideas for Using the Dataset | |
| From the perspective of game system designers, players' behavior is one of the most important factors they must consider when designing game systems. To gain a fundamental understanding of the game play behavior of online gamers, exploring users' game play time provides a good starting point. This is because the concept of game play time is applicable to all genres of games and it enables us to model the system workload as well as the impact of system and network QoS on users' behavior. It can even help us predict players' loyalty to specific games. | |
| Open Questions | |
| Understand user gameplay behavior (game sessions, movement, leveling) | |
| Understand user interactions (guilds) | |
| Predict players unsubscribing from the game based on activity | |
| What are the most popular zones in WoW, what level players tend to inhabit each? | |
| Wrath of the Lich King | |
| An expansion to World of Warcraft, ""Wrath of the Lich King"" (Wotlk) was released on November 13, 2008. It introduced new zones for players to go to, a new character class (the death knight), and a new level cap of 80 (up from 70 previously). This event intersects nicely with the dataset and is probably interesting to investigate. | |
| Map | |
| This dataset doesn't include a shapefile (if you know of one that exists, let me know!) to show where the zones the dataset talks about are. Here is a list of zones an information from this version of the game, including their recommended levels: http://wowwiki.wikia.com/wiki/Zones_by_level_(original) . | |
| Update (Version 3): dmi3kno has generously put together some supplementary zone information files which have now been included in this dataset. Some notes about the files: | |
| Note that some zone names contain Chinese characters. Unicode names are preserved as a key to the original dataset. What this addition will allow is to understand properties of the zones a bit better - their relative location to each other, competititive properties, type of gameplay and, hopefully, their contribution to character leveling. Location coordinates contain some redundant (and possibly duplicate) records as they are collected from different sources. Working with uncleaned location coordinate data will allow users to demonstrate their data wrangling skills (both working with strings and spatial data)." | |
| "18,393 Pitchfork Reviews","Pitchfork reviews from Jan 5, 1999 to Jan 8, 2017",Nolan Conaway,51,"Version 1,2017-01-13","critical theory | |
| music",SQLite,80 MB,Other,"8,565 views",928 downloads,13 kernels,,https://www.kaggle.com/nolanbconaway/pitchfork-data,"Context | |
| Pitchfork is a music-centric online magazine. It was started in 1995 and grew out of independent music reviewing into a general publication format, but is still famed for its variety music reviews. I scraped over 18,000 Pitchfork reviews (going back to January 1999). Initially, this was done to satisfy a few of my own curiosities, but I bet Kagglers can come up with some really interesting analyses! | |
| Content | |
| This dataset is provided as a sqlite database with the following tables: artists, content, genres, labels, reviews, years. For column-level information on specific tables, refer to the Metadata tab. | |
| Inspiration | |
| Do review scores for individual artists generally improve over time, or go down? | |
| How has Pitchfork's review genre selection changed over time? | |
| Who are the most highly rated artists? The least highly rated artists? | |
| Acknowledgements | |
| Gotta love Beautiful Soup!" | |
| Retail Data Analytics,Historical sales data from 45 stores,Manjeet Singh,51,"Version 2,2017-09-01|Version 1,2017-09-01",,CSV,13 MB,CC0,"18,458 views","2,866 downloads",4 kernels,,https://www.kaggle.com/manjeetsingh/retaildataset,"Context | |
| The Challenge - One challenge of modeling retail data is the need to make decisions based on limited history. Holidays and select major events come once a year, and so does the chance to see how strategic decisions impacted the bottom line. In addition, markdowns are known to affect sales – the challenge is to predict which departments will be affected and to what extent. | |
| Content | |
| You are provided with historical sales data for 45 stores located in different regions - each store contains a number of departments. The company also runs several promotional markdown events throughout the year. These markdowns precede prominent holidays, the four largest of which are the Super Bowl, Labor Day, Thanksgiving, and Christmas. The weeks including these holidays are weighted five times higher in the evaluation than non-holiday weeks. | |
| Within the Excel Sheet, there are 3 Tabs – Stores, Features and Sales | |
| Stores | |
| Anonymized information about the 45 stores, indicating the type and size of store | |
| Features | |
| Contains additional data related to the store, department, and regional activity for the given dates. | |
| Store - the store number | |
| Date - the week | |
| Temperature - average temperature in the region | |
| Fuel_Price - cost of fuel in the region | |
| MarkDown1-5 - anonymized data related to promotional markdowns. MarkDown data is only available after Nov 2011, and is not available for all stores all the time. Any missing value is marked with an NA | |
| CPI - the consumer price index | |
| Unemployment - the unemployment rate | |
| IsHoliday - whether the week is a special holiday week | |
| Sales | |
| Historical sales data, which covers to 2010-02-05 to 2012-11-01. Within this tab you will find the following fields: | |
| Store - the store number | |
| Dept - the department number | |
| Date - the week | |
| Weekly_Sales - sales for the given department in the given store | |
| IsHoliday - whether the week is a special holiday week | |
| The Task | |
| Predict the department-wide sales for each store for the following year | |
| Model the effects of markdowns on holiday weeks | |
| Provide recommended actions based on the insights drawn, with prioritization placed on largest business impact" | |
| Health Nutrition and Population Statistics,State of human health across the world,World Bank,51,"Version 1,2016-11-18","nutrition | |
| health | |
| demographics",CSV,43 MB,Other,"19,575 views","2,935 downloads",22 kernels,,https://www.kaggle.com/theworldbank/health-nutrition-and-population-statistics,"Context | |
| HealthStats provides key health, nutrition and population statistics gathered from a variety of international sources. Themes include population dynamics, nutrition, reproductive health, health financing, medical resources and usage, immunization, infectious diseases, HIV/AIDS, DALY, population projections and lending. HealthStats also includes health, nutrition and population statistics by wealth quintiles. | |
| Content | |
| This dataset includes 345 indicators, such as immunization rates, malnutrition prevalence, and vitamin A supplementation rates across 263 countries around the world. Data was collected on a yearly basis from 1960-2016. | |
| Inspiration | |
| In your opinion, what are some of the more surprising indicators? Are there any you would consider adding? | |
| Is there a relationship between condom use and rates of children born with HIV? How do these rates compare over time? | |
| Which countries have the highest consumption of iodized salt? Has this indicator changed over time, and if so, in which countries? Are there any other indicators that seem to correlate with this one? | |
| Acknowledgements | |
| Data was acquired from the World Bank, and can be accessed in multiple formats here." | |
| World Cities Database,A database of coordinates for countries and cities,Max Mind,51,"Version 3,2017-08-24|Version 2,2017-08-24|Version 1,2017-08-18","world | |
| cities | |
| countries",Other,157 MB,Other,"8,575 views","1,045 downloads",5 kernels,0 topics,https://www.kaggle.com/max-mind/world-cities-database,"Context | |
| This dataset is meant to be used with other datasets that have features like country and city but no latitude/longitude. It is simply a list of cities in the world. Being able to put cities on a map will help people tell their stories more effectively. Another way to think about it is that you can use this make more pretty graphs! | |
| Content | |
| Fields: | |
| city | |
| region | |
| country | |
| population | |
| latitude | |
| longitude | |
| Acknowledgements | |
| These data come from Maxmind.com and have not been altered. The original source can be found by clicking here | |
| Additionally, the Maxmind sharing license has been included. | |
| Inspiration | |
| I wanted to analyze a dataset and make a map, but I was only given a city name without any latitude or longotude coordinates. I found this dataset very helpful and I hoe you do too!" | |
| Customer Support on Twitter,Over 3 million tweets and replies from the biggest brands on Twitter,Thought Vector,51,"Version 10,2017-12-04|Version 9,2017-11-30|Version 8,2017-11-28|Version 7,2017-11-22|Version 6,2017-11-14|Version 5,2017-11-10|Version 4,2017-11-09|Version 3,2017-11-08|Version 2,2017-11-07|Version 1,2017-11-07","business | |
| communication | |
| linguistics | |
| twitter",Other,167 MB,CC4,"10,715 views",708 downloads,7 kernels,,https://www.kaggle.com/thoughtvector/customer-support-on-twitter,"The Customer Support on Twitter dataset is a large, modern corpus of tweets and replies to aid innovation in natural language understanding and conversational models, and for study of modern customer support practices and impact. | |
| Context | |
| Natural language remains the densest encoding of human experience we have, and innovation in NLP has accelerated to power understanding of that data, but the datasets driving this innovation don't match the real language in use today. The Customer Support on Twitter dataset offers a large corpus of modern English (mostly) conversations between consumers and customer support agents on Twitter, and has three important advantages over other conversational text datasets: | |
| Focused - Consumers contact customer support to have a specific problem solved, and the manifold of problems to be discussed is relatively small, especially compared to unconstrained conversational datasets like the reddit Corpus. | |
| Natural - Consumers in this dataset come from a much broader segment than those in the Ubuntu Dialogue Corpus and have much more natural and recent use of typed text than the Cornell Movie Dialogs Corpus. | |
| Succinct - Twitter's brevity causes more natural responses from support agents (rather than scripted), and to-the-point descriptions of problems and solutions. Also, its convenient in allowing for a relatively low message limit size for recurrent nets. | |
| Inspiration | |
| The size and breadth of this dataset inspires many interesting questions: | |
| Can we predict company responses? Given the bounded set of subjects handled by each company, the answer seems like yes! | |
| Do requests get stale? How quickly do the best companies respond, compared to the worst? | |
| Can we learn high quality dense embeddings or representations of similarity for topical clustering? | |
| How does tone affect the customer support conversation? Does saying sorry help? | |
| Can we help companies identify new problems, or ones most affecting their customers? | |
| Content | |
| The dataset is a CSV, where each row is a tweet. The different columns are described below. Every conversation included has at least one request from a consumer and at least one response from a company. Which user IDs are company user IDs can be calculated using the inbound field. | |
| tweet_id | |
| A unique, anonymized ID for the Tweet. Referenced by response_tweet_id and in_response_to_tweet_id. | |
| author_id | |
| A unique, anonymized user ID. @s in the dataset have been replaced with their associated anonymized user ID. | |
| inbound | |
| Whether the tweet is ""inbound"" to a company doing customer support on Twitter. This feature is useful when re-organizing data for training conversational models. | |
| created_at | |
| Date and time when the tweet was sent. | |
| text | |
| Tweet content. Sensitive information like phone numbers and email addresses are replaced with mask values like __email__. | |
| response_tweet_id | |
| IDs of tweets that are responses to this tweet, comma-separated. | |
| in_response_to_tweet_id | |
| ID of the tweet this tweet is in response to, if any. | |
| Contributing | |
| Know of other brands the dataset should include? Found something that needs to be fixed? Start a discussion, or email me directly at $FIRSTNAME@$LASTNAME.com! | |
| Acknowledgements | |
| A huge thank you to my friends who helped bootstrap the list of companies that do customer support on Twitter! There are many rocks that would have been left un-turned were it not for your suggestions! | |
| Relevant Resources | |
| NLTK - casual_tokenize for social media text tokenizing, vader sentiment analysis for social media text | |
| SciKit Learn - BoW Count Vectorizer, Multinomial Naive Bayes Classifier | |
| Topic Modeling via Phrase detection with gensim | |
| facebook research - fastText text classifier" | |
| Complete Historical Cryptocurrency Financial Data,Top 200 Cryptocurrencies by Marketcap,pmohun,51,"Version 2,2018-02-12|Version 1,2018-01-05","business | |
| finance | |
| money | |
| internet",CSV,2 MB,CC0,"5,077 views",843 downloads,,,https://www.kaggle.com/philmohun/cryptocurrency-financial-data,"Context | |
| Recent growing interest in cryptocurrencies, specifically as a speculative investment vehicle, has sparked global conversation over the past 12 months. Although this data is available across various sites, there is a lack of understanding as to what is driving the exponential rise of many individual currencies. This data set is intended to be a starting point for a detailed analysis into what is driving price action, and what can be done to predict future movement. | |
| Content | |
| Consolidated financial information for the top 200 cryptocurrencies by marketcap. Pulled from CoinMarketCap.com. Attributes include: | |
| Currency name (e.g. bitcoin) | |
| Date | |
| Open | |
| High | |
| Low | |
| Close | |
| Volume | |
| Marketcap | |
| Inspiration | |
| For the past few months I have been searching for a reliable source for historical price information related to cryptocurrencies. I wasn't able to find anything that I could use to my liking, so I built my own data set. | |
| I've written a small script that scrapes historical price information for the top 200 coins by market cap as listed on CoinMarketCap.com. | |
| I plan to run some basic analysis on it to answer questions that I have a ""gut"" feeling about, but no quantitative evidence (yet!). | |
| Questions such as: | |
| What is the correlation between bitcoin and alt coin prices? | |
| What is the average age of the top 10 coins by market cap? | |
| What day of the week is best to buy/sell? | |
| Which coins in the top two hundred are less than 6 months old? | |
| Which currencies are the most volatile? | |
| What the hell happens when we go to bed and Asia starts trading? | |
| Feel free to use this for your own purposes! I just ask that you share your results with the group when complete. Happy hunting!" | |
| The Gravitational Waves Discovery Data,The GW150914 Gravitational Waves event data,Elena Cuoco,50,"Version 1,2016-06-09","physics | |
| space",Other,10 MB,Other,"17,399 views","1,410 downloads",12 kernels,2 topics,https://www.kaggle.com/elenacuoco/the-gravitational-waves-discovery-data,"On February 11th 2016 LIGO-Virgo collaboration gave the announce of the discovery of Gravitational Waves, just 100 years after the Einstein’s paper on their prediction. The LIGO Scientific Collaboration (LSC) and the Virgo Collaboration prepared a web page to inform the broader community about a confirmed astrophysical event observed by the gravitational-wave detectors, and to make the data around that time available for others to analyze: https://losc.ligo.org/events/GW150914/ | |
| You can find much more information on the LOSC web site, and a good starting tutorial at the following link: | |
| https://losc.ligo.org/tutorial00/ | |
| These data sets contain 32 secs of data sampled at 4096Hz an 16384Hz around the GW event detected on 14/09/2015. | |
| Longer sets of data can be downloaded here | |
| https://losc.ligo.org/s/events/GW150914/H-H1_LOSC_4_V1-1126257414-4096.hdf5 | |
| https://losc.ligo.org/s/events/GW150914/L-L1_LOSC_4_V1-1126257414-4096.hdf5 | |
| https://losc.ligo.org/s/events/GW150914/H-H1_LOSC_16_V1-1126257414-4096.hdf5 | |
| https://losc.ligo.org/s/events/GW150914/L-L1_LOSC_16_V1-1126257414-4096.hdf5 | |
| How to acknowledge use of this data: If your research used data from one of the data releases, please cite as: | |
| LIGO Scientific Collaboration, ""LIGO Open Science Center release of S5"", 2014, DOI 10.7935/K5WD3XHR | |
| LIGO Scientific Collaboration, ""LIGO Open Science Center release of S6"", 2015, DOI 10.7935/K5RN35SD | |
| LIGO Scientific Collaboration, ""LIGO Open Science Center release of GW150914"", 2016, DOI10.7935/K5MW2F23 | |
| and please include the statement ""This research has made use of data, software and/or web tools obtained from the LIGO Open Science Center (https://losc.ligo.org), a service of LIGO Laboratory and the LIGO Scientific Collaboration. LIGO is funded by the U.S. National Science Foundation."" | |
| If you would also like to cite a published paper, M Vallisneri et al. ""The LIGO Open Science Center"", proceedings of the 10th LISA Symposium, University of Florida, Gainesville, May 18-23, 2014; also arxiv:1410.4839 | |
| Publications We request that you let the LOSC team know if you publish (or intend to publish) a paper using data released from this site. If you would like, we may be able to review your work prior to publication, as we do for our colleagues in the LIGO Scientific Collaboration. Credits LOSC Development: The LOSC Team and The LIGO Scientific Collaboration | |
| The data products made available through the LOSC web service are created and maintained by LIGO Lab and the LIGO Scientific Collaboration. The development of this web page was a team effort, with all members of the LOSC team making contributions in most areas. In addition to the team members listed below, a large number of individuals in the LIGO Scientific Collaboration have contributed content and advice. The LOSC team includes: | |
| Alan Weinstein: LOSC Director | |
| Roy Williams: LOSC Developer, web services and data base architecture | |
| Jonah Kanner: LOSC Developer, tutorials, documentation, data set curation | |
| Michele Vallisneri: LOSC Developer, data quality curation | |
| Branson Stephens: LOSC Developer, event database and web site architecture | |
| Please send any comments, questions, or concerns to: losc@ligo.org" | |
| Drosophila Melanogaster Genome,Explore the annotated genome of the common fruit fly,Myles O'Neill,50,"Version 1,2016-05-26","biology | |
| medicine",CSV,460 MB,CC0,"12,357 views",668 downloads,21 kernels,3 topics,https://www.kaggle.com/mylesoneill/drosophila-melanogaster-genome,"Drosophila Melanogaster | |
| Drosophila Melanogaster, the common fruit fly, is a model organism which has been extensively used in entymological research. It is one of the most studied organisms in biological research, particularly in genetics and developmental biology. | |
| When its not being used for scientific research, D. melanogaster is a common pest in homes, restaurants, and anywhere else that serves food. They are not to be confused with Tephritidae flys (also known as fruit flys). | |
| https://en.wikipedia.org/wiki/Drosophila_melanogaster | |
| About the Genome | |
| This genome was first sequenced in 2000. It contains four pairs of chromosomes (2,3,4 and X/Y). More than 60% of the genome appears to be functional non-protein-coding DNA. | |
| The genome is maintained and frequently updated at FlyBase. This dataset is sourced from the UCSC Genome Bioinformatics download page. It uses the August 2014 version of the D. melanogaster genome (dm6, BDGP Release 6 + ISO1 MT). http://hgdownload.soe.ucsc.edu/downloads.html#fruitfly | |
| Files were modified by Kaggle to be a better fit for analysis on Scripts. This primarily involved turning files into CSV format, with a header row, as well as converting the genome itself from 2bit format into a FASTA sequence file. | |
| Bioinformatics | |
| Genomic analysis can be daunting to data scientists who haven't had much experience with bioinformatics before. We have tried to give basic explanations to each of the files in this dataset, as well as links to further reading on the biological basis for each. If you haven't had the chance to study much biology before, some light reading (ie wikipedia) on the following topics may be helpful to understand the nuances of the data provided here: Genetics, Genomics (Sequencing/Genome Assembly), Chromosomes, DNA, RNA (mRNA/miRNA), Genes, Alleles, Exons, Introns, Transcription, Translation, Peptides, Proteins, Gene Regulation, Mutation, Phylogenetics, and SNPs. | |
| Of course, if you've got some idea of the basics already - don't be afraid to jump right in! | |
| Learning Bioinformatics | |
| There are a lot of great resources for learning bioinformatics on the web. One cool site is Rosalind - a platform that gives you bioinformatic coding challenges to complete. You can use Kaggle Scripts on this dataset to easily complete the challenges on Rosalind (and see Myles' solutions here if you get stuck). We have set up Biopython on Kaggle's docker image which is a great library to help you with your analyses. Check out their tutorial here and we've also created a python notebook with some of the tutorial applied to this dataset as a reference. | |
| Files in this Dataset | |
| Drosophila Melanogaster Genome | |
| genome.fa | |
| The assembled genome itself is presented here in FASTA format. Each chromosome is a different sequence of nucleotides. Repeats from RepeatMasker and Tandem Repeats Finder (with period of 12 or less) are show in lower case; non-repeating sequence is shown in upper case. | |
| Meta Information | |
| There are 3 additional files with meta information about the genome. | |
| meta-cpg-island-ext-unmasked.csv | |
| This file contains descriptive information about CpG Islands in the genome. | |
| https://en.wikipedia.org/wiki/CpG_site | |
| meta-cytoband.csv | |
| This file describes the positions of cytogenic bands on each chromosome. | |
| https://en.wikipedia.org/wiki/Cytogenetics | |
| meta-simple-repeat.csv | |
| This file describes simple tandem repeats in the genome. | |
| https://en.wikipedia.org/wiki/Repeated_sequence_(DNA) https://en.wikipedia.org/wiki/Tandem_repeat | |
| Drosophila Melanogaster mRNA Sequences | |
| Messenger RNA (mRNA) is an intermediate molecule created as part of the cellular process of converting genomic information into proteins. Some mRNA are never translated into proteins and have functional roles in the cell on their own. Collectively, organism mRNA information is known as a Transcriptome. mRNA files included in this dataset give insight into the activity of genes in the organism. | |
| https://en.wikipedia.org/wiki/Messenger_RNA | |
| mrna-genbank.fa | |
| This file includes all mRNA sequences from GenBank associated with Drosophila Melanogaster. | |
| http://www.ncbi.nlm.nih.gov/genbank/ | |
| mrna-refseq.fa | |
| This file includes all mRNA sequences from RefSeq associated with Drosophila Melanogaster. | |
| http://www.ncbi.nlm.nih.gov/refseq/ | |
| Gene Predictions | |
| A gene is a segment of DNA on the genome which, through mRNA, is used to create proteins in the organism. Knowing which parts of DNA are coding (genes) or non-coding is difficult, and a number of different systems for prediction exist. This dataset includes a number of different gene prediction systems applied to the drosophila melanogaster genome. | |
| https://en.wikipedia.org/wiki/Gene_prediction | |
| genes-augustus.csv | |
| AUGUSTUS is a piece of software that predicts genes ab initio using Hidden Markov Models. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC441517/ | |
| genes-genscan.csv | |
| GENSCAN is an older ab initio software for predicting genes. http://genes.mit.edu/GENSCANinfo.html | |
| genes-ensembl.csv | |
| ensembl-gtp.csv | |
| ensembl-pep.csv | |
| ensembl-source.csv | |
| ensembl-to-gene-name.csv | |
| Ensembl provides gene annotation generated by their software Genebuild. This process combines automatic annotation alongside manual curation. http://uswest.ensembl.org/info/genome/genebuild/genome_annotation.html | |
| We have also included some supplementary files for these, including predicted protein peptide sequences for each predicted gene. | |
| genes-refseq.csv | |
| genes-xeno-refseq.csv | |
| refseq-link.csv | |
| refseq-summary.csv | |
| We have included two RefSeq gene predictions in this dataset. The first is based solely on information from the drosophila melanogaster genome. The second (genes-xeno-refseq.csv) uses genes from other organisms as a basis for predicting genes in drosophila melanogaster. | |
| RefSeq RNAs were aligned against the D. melanogaster genome using blat; those with an alignment of less than 15% were discarded. When a single RNA aligned in multiple places, the alignment having the highest base identity was identified. Only alignments having a base identity level within 0.1% of the best and at least 96% base identity with the genomic sequence were kept. | |
| We have also included supplementary files for these which include information about the genes that have been identified. | |
| http://www.ncbi.nlm.nih.gov/refseq/ | |
| What can you do with this data? | |
| Genomic data is the foundation of bioinformatics, and there is an incredible array of things you can do with this data. A good place to start is to look at some of the meta supplementary files alongside the genomic sequence itself. | |
| We have a number of different gene prediction systems in the dataset, how do they compare to each other? How do they compare to the mRNA data? | |
| Working back from the refseq-summary.csv file, you can look at genes that code for particular proteins - can you find these genes in the genome? | |
| How much of the genome codes for the mRNA's found in our mRNA data? Of the mRNA's we have, how many map to the predicted genes and the predicted peptided sequence data? How much of the mRNA seems to be protein-coding vs how much looks like it is miRNA? Can you find pre-mRNA or splice variants within the mRNA data? Does meta information like cytogenic bands or CpG sites correspond with splice variants or a lack of mRNA altogether? | |
| Those are just some of many ideas that could get you started. | |
| Looking for Feedback | |
| This is the first genomic dataset on Kaggle and we are looking for feedback from our community about how interesting this dataset is to them, or if there are ways we could improve it to better suit analysis. Please post suggestions for supplementary data, future genomes we could host, bioinformatics packages we should include on scripts, and any other feedback on the dataset forum." | |
| All the news,"143,000 articles from 15 American publications",Andrew Thompson,50,"Version 4,2017-08-20|Version 3,2017-08-17|Version 2,2017-08-17|Version 1,2017-08-16",journalism,CSV,639 MB,Other,"9,357 views","1,170 downloads",5 kernels,5 topics,https://www.kaggle.com/snapcrack/all-the-news,"Context | |
| I wanted to see how articles clustered together if the articles were rendered into document-term matrices---would there be greater affinity among political affiliations, or medium, subject matter, etc. The data was scraped using BeautifulSoup and stored in Sqlite, but I've chopped it up into three separate CSVs here, because the entire Sqlite database came out to about 1.2 gb, beyond Kaggle's max. | |
| Content | |
| Each row contains: | |
| an id for the Sqlite database | |
| author name | |
| full date | |
| month | |
| year | |
| title | |
| publication name | |
| article url (not available for all articles) | |
| full article content | |
| The publications include the New York Times, Breitbart, CNN, Business Insider, the Atlantic, Fox News, Talking Points Memo, Buzzfeed News, National Review, New York Post, the Guardian, NPR, Reuters, Vox, and the Washington Post. Sampling wasn't quite scientific; I chose publications based on my familiarity of the domain and tried to get a range of political alignments, as well as a mix of print and digital publications. By count, the publications break down accordingly: | |
| It's not entirely even---this was something of a collect-it-all approach, and some sites are more prolific than others, and some have data that maintains integrity after scraping more easily than others. | |
| For each publication, I used archive.org to grab the past year-and-a-half of either home-page headlines or RSS feeds and ran those links through the scraper. That is, the articles are not the product of scraping an entire site, but rather their more prominently placed articles. For example, CNN's articles from 5/6/16 were what appeared on the homepage of CNN.com proper, not everything within the CNN.com domain. Vox's articles from 5/6/16 were everything that appeared in the Vox RSS reader. on 5/6/16, and so on. RSS readers are a breeze to scrape, and so I used them when possible, but not every publication uses them or makes them easy to find. | |
| The data primarily falls between the years of 2016 and July 2017, although there is a not-insignificant number of articles from 2015, and a possibly insignificant number from before then. | |
| A note: there are some stray spaces between non-word characters at times as well as some other minor blemishes and imperfections here and there, the result of cleaning a very messy dataset. | |
| Acknowledgements | |
| Thanks mostly go to the maesters of Stack Overflow." | |
| Zoo Animal Classification,Use Machine Learning Methods to Correctly Classify Animals Based Upon Attributes,UCI Machine Learning,50,"Version 1,2016-12-25",animals,CSV,5 KB,ODbL,"21,525 views","2,660 downloads",41 kernels,,https://www.kaggle.com/uciml/zoo-animal-classification,"This dataset consists of 101 animals from a zoo. There are 16 variables with various traits to describe the animals. The 7 Class Types are: Mammal, Bird, Reptile, Fish, Amphibian, Bug and Invertebrate | |
| The purpose for this dataset is to be able to predict the classification of the animals, based upon the variables. It is the perfect dataset for those who are new to learning Machine Learning. | |
| zoo.csv | |
| Attribute Information: (name of attribute and type of value domain) | |
| animal_name: Unique for each instance | |
| hair Boolean | |
| feathers Boolean | |
| eggs Boolean | |
| milk Boolean | |
| airborne Boolean | |
| aquatic Boolean | |
| predator Boolean | |
| toothed Boolean | |
| backbone Boolean | |
| breathes Boolean | |
| venomous Boolean | |
| fins Boolean | |
| legs Numeric (set of values: {0,2,4,5,6,8}) | |
| tail Boolean | |
| domestic Boolean | |
| catsize Boolean | |
| class_type Numeric (integer values in range [1,7]) | |
| class.csv | |
| This csv describes the dataset | |
| Class_Number Numeric (integer values in range [1,7]) | |
| Number_Of_Animal_Species_In_Class Numeric | |
| Class_Type character -- The actual word description of the class | |
| Animal_Names character -- The list of the animals that fall in the category of the class | |
| Acknowledgements | |
| UCI Machine Learning: https://archive.ics.uci.edu/ml/datasets/Zoo | |
| Source Information -- Creator: Richard Forsyth -- Donor: Richard S. Forsyth 8 Grosvenor Avenue Mapperley Park Nottingham NG3 5DX 0602-621676 -- Date: 5/15/1990 | |
| Inspiration | |
| What are the best machine learning ensembles/methods for classifying these animals based upon the variables given?" | |
| 515K Hotel Reviews Data in Europe,Can you make your trip more cozy by using data science?,Jason Liu,49,"Version 1,2017-08-22","life | |
| leisure | |
| hotels | |
| + 2 more...",CSV,227 MB,CC0,"9,762 views","1,312 downloads",10 kernels,3 topics,https://www.kaggle.com/jiashenliu/515k-hotel-reviews-data-in-europe,"Acknowledgements | |
| The data was scraped from Booking.com. All data in the file is publicly available to everyone already. Data is originally owned by Booking.com. Please contact me through my profile if you want to use this dataset somewhere else. | |
| Data Context | |
| This dataset contains 515,000 customer reviews and scoring of 1493 luxury hotels across Europe. Meanwhile, the geographical location of hotels are also provided for further analysis. | |
| Data Content | |
| The csv file contains 17 fields. The description of each field is as below: | |
| Hotel_Address: Address of hotel. | |
| Review_Date: Date when reviewer posted the corresponding review. | |
| Average_Score: Average Score of the hotel, calculated based on the latest comment in the last year. | |
| Hotel_Name: Name of Hotel | |
| Reviewer_Nationality: Nationality of Reviewer | |
| Negative_Review: Negative Review the reviewer gave to the hotel. If the reviewer does not give the negative review, then it should be: 'No Negative' | |
| Review_Total_Negative_Word_Counts: Total number of words in the negative review. | |
| Positive_Review: Positive Review the reviewer gave to the hotel. If the reviewer does not give the negative review, then it should be: 'No Positive' | |
| Review_Total_Positive_Word_Counts: Total number of words in the positive review. | |
| Reviewer_Score: Score the reviewer has given to the hotel, based on his/her experience | |
| Total_Number_of_Reviews_Reviewer_Has_Given: Number of Reviews the reviewers has given in the past. | |
| Total_Number_of_Reviews: Total number of valid reviews the hotel has. | |
| Tags: Tags reviewer gave the hotel. | |
| days_since_review: Duration between the review date and scrape date. | |
| Additional_Number_of_Scoring: There are also some guests who just made a scoring on the service rather than a review. This number indicates how many valid scores without review in there. | |
| lat: Latitude of the hotel | |
| lng: longtitude of the hotel | |
| In order to keep the text data clean, I removed unicode and punctuation in the text data and transform text into lower case. No other preprocessing was performed. | |
| Inspiration | |
| The dataset is large and informative, I believe you can have a lot of fun with it! Let me put some ideas below to futher inspire kagglers! | |
| Fit a regression model on reviews and score to see which words are more indicative to a higher/lower score | |
| Perform a sentiment analysis on the reviews | |
| Find correlation between reviewer's nationality and scores. | |
| Beautiful and informative visualization on the dataset. | |
| Clustering hotels based on reviews | |
| Simple recommendation engine to the guest who is fond of a special characteristic of hotel. | |
| The idea is unlimited! Please, have a look into data, generate some ideas and leave a master kernel here! I am ready to upvote your ideas and kernels! Cheers!" | |
| Top 100 Cryptocurrency Historical Data,Historical prices for each of the leading cryptocurrencies,Nate,49,"Version 3,2017-10-10|Version 2,2017-10-08|Version 1,2017-10-06","finance | |
| internet",Other,5 MB,Other,"21,916 views","1,837 downloads",,,https://www.kaggle.com/natehenderson/top-100-cryptocurrency-historical-data,"This dataset contains historical prices as tracked by www.coinmarketcap.com for the top 100 cryptocurrencies by market capitalization as of September 22, 2017, and is current to that date. | |
| Each CSV file is named by its cryptocurrency as named on www.coinmarketcap.com, with the sole exception of ""I-O Coin"" in place of I/O Coin for ease of importing. | |
| Also accompanying the zip of the top 100 CSVs is a CSV named ""Top 100.csv"", which is a list of the top 100, ordered 1 to 100 with Bitcoin at the beginning and GridCoin at the end. The second row of this CSV is the Market Cap as of September 22, 2017. | |
| Row descriptions - Date, string, e.g. ""Sep 22, 2017"" - Open, float (2 decimal places), e.g. 1234.00 - High, float (2 decimal places), e.g. 1234.00 - Low, float (2 decimal places), e.g. 1234.00 - Close, float (2 decimal places), e.g. 1234.00 - Volume [traded in 24 hours], string, e.g. ""1,234,567,890"" - Market Cap [Market capitalization], string, e.g. ""1,234,567,890"" | |
| This is my first dataset and I would greatly appreciate your feedback. Thanks and enjoy!" | |
| US county-level mortality,United States Mortality Rates by County 1980-2014,IHME,49,"Version 6,2016-12-28|Version 5,2016-12-28|Version 4,2016-12-27|Version 3,2016-12-27|Version 2,2016-12-27|Version 1,2016-12-27",demographics,Other,24 MB,Other,"11,632 views","1,353 downloads",38 kernels,3 topics,https://www.kaggle.com/IHME/us-countylevel-mortality,"Context | |
| IHME United States Mortality Rates by County 1980-2014: National - All. (Deaths per 100,000 population) | |
| To quickly get started creating maps, like the one below, see the Quick Start R kernel. | |
| How the Dataset was Created | |
| This Dataset was created from the Excel Spreadsheet, which can be found in the download. Or, you can view the source here. If you take a look at the row for United States, for the column Mortality Rate, 1980*, you'll see the set of numbers 1.52 (1.44, 1.61). Numbers in parentheses are 95% uncertainty. The 1.52 is an age-standardized mortality rate for both sexes combined (deaths per 100,000 population). | |
| In this Dataset 1.44 will be placed in the named column Mortality Rage, 1989 (Min)* and 1.61 is in column named Mortality Rate, 1980 (Max)* . For information on how these Age-standardized mortality rates were calculated, see the December JAMA 2016 article, which you can download for free. | |
| Reference | |
| JAMA Full Article | |
| Video Describing this Study (Short and this is worth viewing) | |
| Data Resources | |
| How Americans Die May Depend On Where They Live, by Anna Maria Barry-Jester (FiveThirtyEight) | |
| Interactive Map from healthdata.org | |
| IHME Data | |
| Acknowledgements | |
| This Dataset was provided by IHME | |
| Institute for Health Metrics and Evaluation 2301 Fifth Ave., Suite 600, Seattle, WA 98121, USA Tel: +1.206.897.2800 Fax: +1.206.897.2899 © 2016 University of Washington" | |
| Health Analytics,26 health indicators (642 variables) from 9 states and 284 districts of India,Rajanand Ilangovan / இராஜ்ஆனந்த் இளங்கோவன்,48,"Version 1,2017-08-10","india | |
| public health | |
| health",CSV,2 MB,CC4,"16,860 views","1,787 downloads",,,https://www.kaggle.com/rajanand/key-indicators-of-annual-health-survey,"Connect/Follow me on LinkedIn for more updates on interesting dataset like this. Thanks. | |
| Context | |
| India - Annual Health Survey(AHS) 2012-13: | |
| The survey was conducted in Empowered Action Group (EAG) states Uttarakhand, Rajasthan, Uttar Pradesh, Bihar, Jharkhand, Odisha, Chhattisgarh & Madhya Pradesh and Assam. These nine states, which account for about 48 percent of the total population, 59 percent of Births, 70 percent of Infant Deaths, 75 percent of Under 5 Deaths and 62 percent of Maternal Deaths in the country, are the high focus States in view of their relatively higher fertility and mortality. | |
| A representative sample of about 21 million population and 4.32 million households were covered 20k+ sample units which is spread across rural and urban area of these 9 states. | |
| The objective of the AHS is to yield a comprehensive, representative and reliable dataset on core vital indicators including composite ones like Infant Mortality Rate, Maternal Mortality Ratio and Total Fertility Rate along with their co-variates (process and outcome indicators) at the district level and map the changes therein on an annual basis. These benchmarks would help in better and holistic understanding and timely monitoring of various determinants on well-being and health of population particularly Reproductive and Child Health. Source | |
| Content | |
| This dataset contains the data about the below 26 key indicators. | |
| AA. Sample Particulars | |
| Sample Units | |
| Households | |
| Population | |
| Ever Married Women (aged 15-49 years) | |
| Currently Married Women (aged 15-49 years) | |
| Children 12-23 months | |
| BB. Household Characteristics | |
| Average Household Size | |
| SC | |
| ST | |
| All | |
| Population below age 15 years (%) | |
| Dependency Ratio | |
| Currently Married Illiterate Women aged 15-49 years (%) | |
| CC. Sex Ratio | |
| Sex Ratio at Birth | |
| Sex Ratio (0- 4 years) | |
| Sex Ratio (All ages) | |
| DD. Effective Literacy Rate | |
| EE. Marriage | |
| Marriages among Females below legal age (18 years) (%) | |
| Marriages among Males below legal age (21 years) (%) | |
| Currently Married Women aged 20-24 years married before legal age (18 years) (%) | |
| Currently Married Men aged 25-29 years married before legal age (21 years) (%) | |
| Mean age at Marriage# - Male | |
| Mean age at Marriage# - Female | |
| FF. Schooling Status | |
| Children currently attending school (Age 6-17 years) (%) | |
| Children attended before / Drop out (Age 6-17 years) (%) | |
| GG. Work Status | |
| Children aged 5-14 years engaged in work (%) | |
| Work Participation Rate (15 years and above) | |
| HH. Disability | |
| Prevalence of any type of Disability (Per 100,000 Population) | |
| II. Injury | |
| Number of Injured Persons by type of Treatment received (Per 100,000 Population) | |
| Severe | |
| Major | |
| Minor | |
| JJ. Acute Illness | |
| Persons suffering from Acute Illness (Per 100,000 Population) | |
| Diarrhoea/Dysentery | |
| Acute Respiratory Infection (ARI) | |
| Fever (All Types) | |
| Any type of Acute Illness | |
| Persons suffering from Acute Illness and taking treatment from Any Source (%) | |
| Persons suffering from Acute Illness and taking treatment from Government Source (%) | |
| KK. Chronic Illness | |
| Having any kind of Symptoms of Chronic Illness (Per 100,000 Population) | |
| Having any kind of Symptoms of Chronic Illness and sought Medical Care (%) | |
| Having diagnosed for Chronic Illness (Per 100,000 Population) | |
| Diabetes | |
| Hypertension | |
| Tuberculosis (TB) | |
| Asthma / Chronic Respiratory Disease | |
| Arthritis | |
| Any kind of Chronic Illness | |
| Having diagnosed for any kind of Chronic Illness and getting Regular Treatment (%) | |
| Having diagnosed for any kind of Chronic Illness and getting Regular Treatment from Government Source (%) | |
| LL. Fertility | |
| Crude Birth Rate (CBR) | |
| Natural Growth Rate | |
| Total Fertility Rate | |
| Women aged 20-24 reporting birth of order 2 & above (%) | |
| Women reporting birth of order 3 & above (%) | |
| Women with two children wanting no more children (%) | |
| Women aged 15-19 years who were already mothers or pregnant at the time of survey (%) | |
| Median age at first live birth of Women aged 15-49 years | |
| Median age at first live birth of Women aged 25-49 years | |
| Live Births taking place after an interval of 36 months (%) | |
| Mean number of children ever born to Women aged 15-49 years | |
| Mean number of children surviving to Women aged 15-49 years | |
| Mean number of children ever born to Women aged 45-49 years | |
| MM. Abortion | |
| Pregnancy to Women aged 15-49 years resulting in abortion (%) | |
| Women who received any ANC before abortion (%) | |
| Women who went for Ultrasound before abortion (%) | |
| Average Month of pregnancy at the time of abortion | |
| Abortion performed by skilled health personnel (%) | |
| Abortion taking place in Institution (%) | |
| NN. Family Planning Practices (Cmw Aged 15-49 Years) | |
| Current Usage | |
| Any Method (%) | |
| Any Modern Method (%) | |
| Female Sterilization (%) | |
| Male Sterilization (%) | |
| Copper-T/IUD (%) | |
| Pills (%) | |
| Condom/Nirodh (%) | |
| Emergency Contraceptive Pills (%) | |
| Any Traditional Method (%) | |
| Periodic Abstinence (%) | |
| Withdrawal (%) | |
| LAM (%) | |
| OO. Unmet Need For Family Planning | |
| Unmet need for Spacing (%) | |
| Unmet need for Limiting (%) | |
| Total Unmet need (%) | |
| PP. Ante Natal Care | |
| Currently Married Pregnant Women aged 15-49 years registered for ANC (%) | |
| Mothers who received any Antenatal Check-up (%) | |
| Mothers who had Antenatal Check-up in First Trimester (%) | |
| Mothers who received 3 or more Antenatal Care (%) | |
| Mothers who received at least one Tetanus Toxoid (TT) injection (%) | |
| Mothers who consumed IFA for 100 days or more (%) | |
| Mothers who had Full Antenatal Check-up (%) | |
| Mothers who received ANC from Govt. Source (%) | |
| Mothers whose Blood Pressure (BP) taken (%) | |
| Mothers whose Blood taken for Hb (%) | |
| Mothers who underwent Ultrasound (%) | |
| QQ. Delivery Care | |
| Institutional Delivery (%) | |
| Delivery at Government Institution (%) | |
| Delivery at Private Institution (%) | |
| Delivery at Home (%) | |
| Delivery at home conducted by skilled health personnel (%) | |
| Safe Delivery (%) | |
| Caesarean out of total delivery taken place in Government Institutions (%) | |
| Caesarean out of total delivery taken place in Private Institutions (%) | |
| RR. Post Natal Care | |
| Less than 24 hrs. stay in institution after delivery (%) | |
| Mothers who received Post-natal Check-up within 48 hrs. of delivery (%) | |
| Mothers who received Post-natal Check-up within 1 week of delivery (%) | |
| Mothers who did not receive any Post-natal Check-up (%) | |
| New borns who were checked up within 24 hrs. of birth (%) | |
| SS. Janani Suraksha Yojana (JSY) | |
| Mothers who availed financial assistance for delivery under JSY (%) | |
| Mothers who availed financial assistance for institutional delivery under JSY (%) | |
| Mothers who availed financial assistance for Government Institutional delivery under JSY (%) | |
| TT. Immunization, Vitamin A & Iron Supplement And Birth Weight | |
| Children aged 12-23 months having Immunization Card (%) | |
| Children aged 12-23 months who have received BCG (%) | |
| Children aged 12-23 months who have received 3 doses of Polio vaccine (%) | |
| Children aged 12-23 months who have received 3 doses of DPT vaccine (%) | |
| Children aged 12-23 months who have received Measles vaccine (%) | |
| Children aged 12-23 months Fully Immunized (%) | |
| Children who have received Polio dose at birth (%) | |
| Children who did not receive any vaccination (%) | |
| Children (aged 6-35 months) who received at least one Vitamin A dose during last six months (%) | |
| Children (aged 6-35 months) who received IFA tablets/syrup during last 3 months (%) | |
| Children whose birth weight was taken (%) | |
| Children with birth weight less than 2.5 Kg. (%) | |
| UU. Childhood Diseases | |
| Children suffering from Diarrhoea (%) | |
| Children suffering from Diarrhoea who received HAF/ORS/ORT (%) | |
| Children suffering from Acute Respiratory Infection (%) | |
| Children suffering from Acute Respiratory Infection who sought treatment (%) | |
| Children suffering from Fever (%) | |
| Children suffering from Fever who sought treatment (%) | |
| VV. Breastfeeding And Supplementation | |
| Children breastfed within one hour of birth (%) | |
| Children (aged 6-35 months) exclusively breastfed for at least six months (%) | |
| Children Who Received Foods Other Than Breast Milk During First 6 Months | |
| Water (%) | |
| Animal/Formula Milk (%) | |
| Semi-Solid Mashed Food (%) | |
| Solid (Adult) Food (%) | |
| Vegetables/Fruits (%) | |
| Average Month By Which Children Received Foods Other Than Breast Milk | |
| Water (%) | |
| Animal/Formula Milk (%) | |
| Semi-Solid Mashed Food (%) | |
| Solid (Adult) Food (%) | |
| Vegetables/Fruits (%) | |
| WW. Birth Registration | |
| Birth Registered (%) | |
| Children Whose Birth Was Registered And Received Birth Certificate (%) | |
| XX. Awareness On Hiv/Aids, Rti/Sti, Haf/Ors/Ort/Zinc And Ari/Pneumonia | |
| Women Who Are Aware of: | |
| HIV/AIDS | |
| RTI/STI | |
| HAF/ORS/ORT/ZINC | |
| Danger Signs Of ARI/Pneumonia (%) | |
| YY. Mortality (unit level data of mortality is available here) | |
| Crude Death Rate (CDR) | |
| Infant Mortality Rate (IMR) | |
| Neo-Natal Mortality Rate | |
| Under Five Mortality Rate (U5MR) | |
| ZZ. Confidence Interval (95%) For Some Important Indicators | |
| Crude Birth Rate - (Lower and Upper Limit) | |
| Crude Death Rate - (Lower and Upper Limit) | |
| Infant Mortality Rate - (Lower and Upper Limit) | |
| Under Five Mortality Rate (U5MR) - (Lower and Upper Limit) | |
| Sex Ratio At Birth - (Lower and Upper Limit) | |
| Acknowledgements | |
| Department of Health and Family Welfare, Govt. of India has published this data in Open Govt Data Platform India portal under Govt. Open Data License - India." | |
| Paradise-Panama-Papers,Data Scientists United Against Corruption,Zeeshan-ul-hassan Usmani,48,"Version 2,2017-11-21|Version 1,2017-11-08","money | |
| banking",CSV,134 MB,CC3,"7,055 views",520 downloads,5 kernels,3 topics,https://www.kaggle.com/zusmani/paradisepanamapapers,"Context | |
| The Paradise Papers is a cache of some 13GB of data that contains 13.4 million confidential records of offshore investment by 120,000 people and companies in 19 tax jurisdictions (Tax Heavens - an awesome video to understand this); that was published by the International Consortium of Investigative Journalists (ICIJ) on November 5, 2017. Here is a brief video about the leak. The people include Queen Elizabeth II, the President of Columbia (Juan Manuel Santos), Former Prime Minister of Pakistan (Shaukat Aziz), U.S Secretary of Commerce (Wilbur Ross) and many more. According to an estimate by the Boston Consulting Group, the amount of money involved is around $10 trillion. The leak contains many famous companies, including Facebook, Apple, Uber, Nike, Walmart, Allianz, Siemens, McDonald’s and Yahoo. | |
| It also contains a lot of U. S President Donald Trump allies including Rax Tillerson, Wilbur Ross, Koch Brothers, Paul Singer, Sheldon Adelson, Stephen Schwarzman, Thomas Barrack and Steve Wynn etc. The complete list of Politicians involve is avaiable here. | |
| The Panama Papers in the cache of 38GB of data from the national corporate registry of Bahamas. It contains world’s top politicians and influential persons as head and director of offshore companies registered in Bahamas. | |
| Offshore Leaks details 13,000 offshore accounts in a report. | |
| I am calling all data scientists to help me stop the corruption and reveal the patterns and linkages invisible for the untrained eye. | |
| Content | |
| The data is the effort of more than 100 journalists from 60+ countries | |
| The original data is available under creative common license and can be downloaded from this link. | |
| I will keep updating the datasets with more leaks and data as it’s available | |
| Acknowledgements | |
| International Consortium of Investigative Journalists (ICIJ) | |
| Paradise Papers Update | |
| Paradise Papers data has been uploaded as released by ICIJ on Nov 21, 2017. You can find Paradise Papers zip file and six extracted files in CSV format, all starting with a prefix of Paradise. Happy Coding! | |
| Inspiration | |
| Some ideas worth exploring: | |
| How many companies and individuals are there in all of the leaks data | |
| How many countries involved | |
| Total money involved | |
| What is the biggest best tax heaven | |
| Can we compare the corruption with human development index and make an argument that would correlate corruption with bad conditions in that country | |
| Who are the biggest cheaters and where they live | |
| What role Fortune 500 companies play in this game | |
| I need your help to make this world corruption free in the age of NLP and Big Data" | |
| U.S. Incomes by Occupation and Gender,Analyze gender gap and differences in industry's incomes,Jean-Phillipe,48,"Version 1,2016-09-23","gender | |
| employment | |
| income",CSV,31 KB,CC0,"14,922 views","2,591 downloads",21 kernels,0 topics,https://www.kaggle.com/jonavery/incomes-by-career-and-gender,"Many people say the gender gap in income levels is overstated in the United States, where some say that inequality in the labor force is a thing of the past. Is there a gender gap at all? Is it stronger in some industries than in others? | |
| This dataset, retrieved from the Bureau of Labor Statistics, shows the median weekly incomes for 535 different occupations. The data encompasses information for all working American citizens as of January 2015. The incomes are broken into male and female statistics, preceded by the total median income when including both genders. The data has been re-formatted from the original PDF-friendly arrangement to make it easier to clean and analyze. | |
| Analysis thus far has found that there is indeed a sizable gender gap between male and female incomes. Use of this dataset should cite the Bureau of Labor Statistics as per their copyright information: | |
| The Bureau of Labor Statistics (BLS) is a Federal government agency and everything that we publish, both in hard copy and electronically, is in the public domain, except for previously copyrighted photographs and illustrations. You are free to use our public domain material without specific permission, although we do ask that you cite the Bureau of Labor Statistics as the source." | |
| OSMI Mental Health in Tech Survey 2016,Data on prevalence and attitudes towards mental health among tech workers,"Open Sourcing Mental Illness, LTD",48,"Version 1,2016-11-14","mental health | |
| employment",Other,80 MB,CC4,"12,698 views","2,090 downloads",24 kernels,0 topics,https://www.kaggle.com/osmi/mental-health-in-tech-2016,"OSMI Mental Health in Tech Survey 2016 | |
| Currently over 1400 responses, the ongoing 2016 survey aims to measure attitudes towards mental health in the tech workplace, and examine the frequency of mental health disorders among tech workers. | |
| How Will This Data Be Used? | |
| We are interested in gauging how mental health is viewed within the tech/IT workplace, and the prevalence of certain mental health disorders within the tech industry. The Open Sourcing Mental Illness team of volunteers will use this data to drive our work in raising awareness and improving conditions for those with mental health disorders in the IT workplace." | |
| National Health and Nutrition Examination Survey,NHANES datasets from 2013-2014,Centers for Disease Control and Prevention,48,"Version 1,2017-01-27","healthcare | |
| health",CSV,31 MB,Other,"14,654 views","2,146 downloads",14 kernels,2 topics,https://www.kaggle.com/cdc/national-health-and-nutrition-examination-survey,"Context | |
| The National Health and Nutrition Examination Survey (NHANES) is a program of studies designed to assess the health and nutritional status of adults and children in the United States. The survey is unique in that it combines interviews and physical examinations. NHANES is a major program of the National Center for Health Statistics (NCHS). NCHS is part of the Centers for Disease Control and Prevention (CDC) and has the responsibility for producing vital and health statistics for the Nation. | |
| The NHANES program began in the early 1960s and has been conducted as a series of surveys focusing on different population groups or health topics. In 1999, the survey became a continuous program that has a changing focus on a variety of health and nutrition measurements to meet emerging needs. The survey examines a nationally representative sample of about 5,000 persons each year. These persons are located in counties across the country, 15 of which are visited each year. | |
| The NHANES interview includes demographic, socioeconomic, dietary, and health-related questions. The examination component consists of medical, dental, and physiological measurements, as well as laboratory tests administered by highly trained medical personnel. | |
| To date, thousands of research findings have been published using the NHANES data. | |
| Content | |
| The 2013-2014 NHANES datasets include the following components: | |
| Demographics dataset: | |
| A complete variable dictionary can be found here | |
| Examinations dataset, which contains: | |
| Blood pressure | |
| Body measures | |
| Muscle strength - grip test | |
| Oral health - dentition | |
| Taste & smell | |
| A complete variable dictionary can be found here | |
| Dietary data - total nutrient intake, first day: | |
| A complete variable dictionary can be found here | |
| Laboratory dataset, which includes: | |
| Albumin & Creatinine - Urine | |
| Apolipoprotein B | |
| Blood Lead, Cadmium, Total Mercury, Selenium, and Manganese | |
| Blood mercury: inorganic, ethyl and methyl | |
| Cholesterol - HDL | |
| Cholesterol - LDL & Triglycerides | |
| Cholesterol - Total | |
| Complete Blood Count with 5-part Differential - Whole Blood | |
| Copper, Selenium & Zinc - Serum | |
| Fasting Questionnaire | |
| Fluoride - Plasma | |
| Fluoride - Water | |
| Glycohemoglobin | |
| Hepatitis A | |
| Hepatitis B Surface Antibody | |
| Hepatitis B: core antibody, surface antigen, and Hepatitis D antibody | |
| Hepatitis C RNA (HCV-RNA) and Hepatitis C Genotype | |
| Hepatitis E: IgG & IgM Antibodies | |
| Herpes Simplex Virus Type-1 & Type-2 | |
| HIV Antibody Test | |
| Human Papillomavirus (HPV) - Oral Rinse | |
| Human Papillomavirus (HPV) DNA - Vaginal Swab: Roche Cobas & Roche Linear Array | |
| Human Papillomavirus (HPV) DNA Results from Penile Swab Samples: Roche Linear Array | |
| Insulin | |
| Iodine - Urine | |
| Perchlorate, Nitrate & Thiocyanate - Urine | |
| Perfluoroalkyl and Polyfluoroalkyl Substances (formerly Polyfluoroalkyl Chemicals - PFC) | |
| Personal Care and Consumer Product Chemicals and Metabolites | |
| Phthalates and Plasticizers Metabolites - Urine | |
| Plasma Fasting Glucose | |
| Polycyclic Aromatic Hydrocarbons (PAH) - Urine | |
| Standard Biochemistry Profile | |
| Tissue Transglutaminase Assay (IgA-TTG) & IgA Endomyseal Antibody Assay (IgA EMA) | |
| Trichomonas - Urine | |
| Two-hour Oral Glucose Tolerance Test | |
| Urinary Chlamydia | |
| Urinary Mercury | |
| Urinary Speciated Arsenics | |
| Urinary Total Arsenic | |
| Urine Flow Rate | |
| Urine Metals | |
| Urine Pregnancy Test | |
| Vitamin B12 | |
| A complete data dictionary can be found here | |
| Questionnaire dataset, which includes information on: | |
| Acculturation | |
| Alcohol Use | |
| Blood Pressure & Cholesterol | |
| Cardiovascular Health | |
| Consumer Behavior | |
| Current Health Status | |
| Dermatology | |
| Diabetes | |
| Diet Behavior & Nutrition | |
| Disability | |
| Drug Use | |
| Early Childhood | |
| Food Security | |
| Health Insurance | |
| Hepatitis | |
| Hospital Utilization & Access to Care | |
| Housing Characteristics | |
| Immunization | |
| Income | |
| Medical Conditions | |
| Mental Health - Depression Screener | |
| Occupation | |
| Oral Health | |
| Osteoporosis | |
| Pesticide Use | |
| Physical Activity | |
| Physical Functioning | |
| Preventive Aspirin Use | |
| Reproductive Health | |
| Sexual Behavior | |
| Sleep Disorders | |
| Smoking - Cigarette Use | |
| Smoking - Household Smokers | |
| Smoking - Recent Tobacco Use | |
| Smoking - Secondhand Smoke Exposure | |
| Taste & Smell | |
| Weight History | |
| Weight History - Youth | |
| A complete variable dictionary can be found here | |
| Medication dataset, which includes prescription medications: | |
| A complete variable dictionary can be found here | |
| Acknowledgements | |
| Original data and additional documents related to the datasets or NHANES can be found here." | |
| Hillary Clinton and Donald Trump Tweets,Tweets from the major party candidates for the 2016 US Presidential Election,Ben Hamner,47,"Version 1,2016-09-28","politics | |
| internet",CSV,5 MB,Other,"25,143 views","2,923 downloads",81 kernels,3 topics,https://www.kaggle.com/benhamner/clinton-trump-tweets,"Twitter has played an increasingly prominent role in the 2016 US Presidential Election. Debates have raged and candidates have risen and fallen based on tweets. | |
| This dataset provides ~3000 recent tweets from Hillary Clinton and Donald Trump, the two major-party presidential nominees." | |
| 2016 March ML Mania Predictions,Forecasting the 2016 NCAA Basketball Tournament,William Cukierski,47,"Version 2,2017-11-16|Version 1,2016-03-16","basketball | |
| artificial intelligence",CSV,27 MB,CC4,"12,482 views","1,485 downloads",73 kernels,2 topics,https://www.kaggle.com/wcukierski/2016-march-ml-mania,"Kaggle’s March Machine Learning Mania competition challenged data scientists to predict winners and losers of the men's 2016 NCAA basketball tournament. This dataset contains the 1070 selected predictions of all Kaggle participants. These predictions were collected and locked in prior to the start of the tournament. | |
| How can this data be used? You can pivot it to look at both Kaggle and NCAA teams alike. You can look at who will win games, which games will be close, which games are hardest to forecast, or which Kaggle teams are gambling vs. sticking to the data. | |
| The NCAA tournament is a single-elimination tournament that begins with 68 teams. There are four games, usually called the “play-in round,” before the traditional bracket action starts. Due to competition timing, these games are included in the prediction files but should not be used in analysis, as it’s possible that the prediction was submitted after the play-in round games were over. | |
| Data Description | |
| Each Kaggle team could submit up to two prediction files. The prediction files in the dataset are in the 'predictions' folder and named according to: | |
| TeamName_TeamId_SubmissionId.csv | |
| The file format contains a probability prediction for every possible game between the 68 teams. This is necessary to cover every possible tournament outcome. Each team has a unique numerical Id (given in Teams.csv). Each game has a unique Id column created by concatenating the year and the two team Ids. The format is the following: | |
| Id,Pred | |
| 2016_1112_1114,0.6 | |
| 2016_1112_1122,0 | |
| ... | |
| The team with the lower numerical Id is always listed first. “Pred” represents the probability that the team with the lower Id beats the team with the higher Id. For example, ""2016_1112_1114,0.6"" indicates team 1112 has a 0.6 probability of beating team 1114. | |
| For convenience, we have included the data files from the 2016 March Mania competition dataset in the Scripts environment (you may find TourneySlots.csv and TourneySeeds.csv useful for determining matchups, see the documentation). However, the focus of this dataset is on Kagglers' predictions." | |
| Annotated Corpus for Named Entity Recognition,Corpus (CoNLL 2002) annotated with IOB and POS tags,Abhinav Walia,47,"Version 4,2017-09-21|Version 3,2017-07-11|Version 2,2017-04-02|Version 1,2017-03-20",linguistics,CSV,164 MB,ODbL,"16,016 views","1,705 downloads",6 kernels,4 topics,https://www.kaggle.com/abhinavwalia95/entity-annotated-corpus,"Context: Annotated Corpus for Named Entity Recognition using GMB(Groningen Meaning Bank) corpus for entity classification with enhanced and popular features by Natural Language Processing applied to the data set. | |
| Tip: Use Pandas Dataframe to load dataset if using Python for convenience. | |
| Content: This is the extract from GMB corpus which is tagged, annotated and built specifically to train the classifier to predict named entities such as name, location, etc. | |
| Number of tagged entities: | |
| 'O': 1146068', geo-nam': 58388, 'org-nam': 48034, 'per-nam': 23790, 'gpe-nam': 20680, 'tim-dat': 12786, 'tim-dow': 11404, 'per-tit': 9800, 'per-fam': 8152, 'tim-yoc': 5290, 'tim-moy': 4262, 'per-giv': 2413, 'tim-clo': 891, 'art-nam': 866, 'eve-nam': 602, 'nat-nam': 300, 'tim-nam': 146, 'eve-ord': 107, 'per-ini': 60, 'org-leg': 60, 'per-ord': 38, 'tim-dom': 10, 'per-mid': 1, 'art-add': 1 | |
| Essential info about entities: | |
| geo = Geographical Entity | |
| org = Organization | |
| per = Person | |
| gpe = Geopolitical Entity | |
| tim = Time indicator | |
| art = Artifact | |
| eve = Event | |
| nat = Natural Phenomenon | |
| Total Words Count = 1354149 Target Data Column: ""tag"" | |
| Inspiration: This dataset is getting more interested because of more features added to the recent version of this dataset. Also, it helps to create a broad view of Feature Engineering with respect to this dataset. | |
| Why this dataset is helpful or playful? | |
| It might not sound so interested for earlier versions, but when you are able to pick intent and custom named entities from your own sentence with more features then, it is getting interested and helps you solve real business problems(like picking entities from Electronic Medical Records, etc) | |
| Please, feel free to ask questions, do variations and let's play together!" | |
| Computer Parts (CPUs and GPUs),How did computer specifications and performance evolve over time?,ilias sekkaf,47,"Version 7,2017-09-30|Version 6,2017-09-14|Version 5,2017-09-14|Version 4,2017-09-14|Version 3,2017-09-13|Version 2,2017-09-13|Version 1,2017-09-13","computers | |
| computer architecture | |
| internet",CSV,1 MB,Other,"9,959 views","1,530 downloads",6 kernels,,https://www.kaggle.com/iliassekkaf/computerparts,"Contents | |
| This dataset contains detailed specifications, release dates, and release prices of computer parts. | |
| The dataset contains two CSV files: gpus.csv for Graphics Processing Units (GPUs), and cpus.csv for Central Processing Units (CPUs). Each table has its own list of unique entries, but the list of features includes: clock speeds, maximum temperatures, display resolutions, power draws, number of threads, release dates, release prices, die size, virtualization support, and many other similar fields. For more specific column-level metadata refer to the Column Metadata. | |
| Looking for inspiration? Try starting by reading ""Using regression to predict the GPUs of the future"". | |
| Inspiration | |
| How did performance over price ratio evolve over time? | |
| How about general computing power? | |
| Are there any manufacturers that are known for some specific range of performance & price? | |
| Acknowledgements | |
| The data given here belongs mainly to Intel, Game-Debate, and the companies involved in producing the part. I do not own the data I uploaded it solely for informative purposes, under their original license." | |
| Synchronized Brainwave Dataset,Brainwave recordings from a group presented with a shared audio-visual stimulus,BioSENSE @ UC Berkeley School of Information,47,"Version 4,2016-10-27|Version 3,2016-10-25|Version 2,2016-10-25|Version 1,2016-10-22","healthcare | |
| human-computer interaction",CSV,101 MB,CC4,"13,355 views","1,195 downloads",62 kernels,0 topics,https://www.kaggle.com/berkeley-biosense/synchronized-brainwave-dataset,"Context | |
| EEG devices are becoming cheaper and more inconspicuous, but few applications leverage EEG data effectively, in part because there are few large repositories of EEG data. The MIDS class at the UC Berkeley School of Information is sharing a dataset collected using consumer-grade brainwave-sensing headsets, along with the software code and visual stimulus used to collect the data. The dataset includes all subjects' readings during the stimulus presentation, as well as readings from before the start and after the end of the stimulus. | |
| Content | |
| We presented two slightly different stimuli to two different groups. Stimuli 1 is available here, and stimuli 2 is available here. | |
| For both stimuli, a group of about 15 people saw the stimuli at the same time, while EEG data was being collected. The stimuli each person saw is available in the session field of subject-metadata.csv. (Subjects who saw stimulus 2 left the room during stimulus 1, and vice versa). | |
| Find the synchronized times for both stimuli in stimulus-timing.csv. | |
| For each participant, we also anonymously collected some other metadata: (1) whether or not they had previously seen the video displayed during the stimulus (a superbowl ad), (2) gender, (3) whether or not they saw hidden icons displayed during the color counting exercise, and (4) their chosen color during the color counting exercise. All of these can be found in subject-metadata.csv. | |
| We also collected the timing (in indra_time) of all stimulus events for both session 1 and session 2. These times are included in stimulus-times.csv. | |
| The server receives one data packet every second from each Mindwave Mobile device, and stores the data in one row entry. | |
| Acknowledgements | |
| Please use the following citation if you publish your research results using this dataset or software code or stimulus file: | |
| John Chuang, Nick Merrill, Thomas Maillart, and Students of the UC Berkeley Spring 2015 MIDS Immersion Class. ""Synchronized Brainwave Recordings from a Group Presented with a Common Audio-Visual Stimulus (May 9, 2015)."" May 2015." | |
| Segmenting Soft Tissue Sarcomas,A challenge to automate tumor segmentation,4Quant,47,"Version 2,2016-12-09|Version 1,2016-12-09",healthcare,Other,379 MB,Other,"11,570 views","1,110 downloads",27 kernels,,https://www.kaggle.com/4quant/soft-tissue-sarcoma,"Summary | |
| The data is a preprocessed subset of the TCIA Study named Soft Tissue Sarcoma. The data have been converted from DICOM folders of varying resolution and data types to 3D HDF5 arrays with isotropic voxel size. This should make it easier to get started and test out various approaches (NN, RF, CRF, etc) to improve segmentations. | |
| TCIA Summary | |
| This collection contains FDG-PET/CT and anatomical MR (T1-weighted, T2-weighted with fat-suppression) imaging data from 51 patients with histologically proven soft-tissue sarcomas (STSs) of the extremities. All patients had pre-treatment FDG-PET/CT and MRI scans between November 2004 and November 2011. (Note: date in the TCIA images have been changed in the interest of de-identification; the same change was applied across all images, preserving the time intervals between serial scans). During the follow-up period, 19 patients developed lung metastases. Imaging data and lung metastases development status were used in the following study: | |
| Vallières, M. et al. (2015). A radiomics model from joint FDG-PET and MRI texture features for the prediction of lung metastases in soft-tissue sarcomas of the extremities. Physics in Medicine and Biology, 60(14), 5471-5496. doi:10.1088/0031-9155/60/14/5471. | |
| Imaging data, tumor contours (RTstruct DICOM objects), clinical data and source code is available for this study. See the DOI below for more details and links to access the whole dataset. Please contact Martin Vallières (mart.vallieres@gmail.com) of the Medical Physics Unit of McGill University for any scientific inquiries about this dataset. | |
| Acknowledgements | |
| We would like to acknowledge the individuals and institutions that have provided data for this collection: McGill University, Montreal, Canada - Special thanks to Martin Vallières of the Medical Physics Unit | |
| License | |
| This collection is freely available to browse, download, and use for commercial, scientific and educational purposes as outlined in the Creative Commons Attribution 3.0 Unported License. See TCIA's Data Usage Policies and Restrictions for additional details. Questions may be directed to help@cancerimagingarchive.net. | |
| Citation | |
| Data Citation | |
| Vallières, Martin, Freeman, Carolyn R., Skamene, Sonia R., & El Naqa, Issam. (2015). A radiomics model from joint FDG-PET and MRI texture features for the prediction of lung metastases in soft-tissue sarcomas of the extremities. The Cancer Imaging Archive. http://doi.org/10.7937/K9/TCIA.2015.7GO2GSKS | |
| Publication Citation | |
| Vallières, M., Freeman, C. R., Skamene, S. R., & Naqa, I. El. (2015, June 29). A radiomics model from joint FDG-PET and MRI texture features for the prediction of lung metastases in soft-tissue sarcomas of the extremities. Physics in Medicine and Biology. IOP Publishing. http://doi.org/10.1088/0031-9155/60/14/5471 | |
| TCIA Citation | |
| Clark K, Vendt B, Smith K, Freymann J, Kirby J, Koppel P, Moore S, Phillips S, Maffitt D, Pringle M, Tarbox L, Prior F. The Cancer Imaging Archive (TCIA): Maintaining and Operating a Public Information Repository, Journal of Digital Imaging, Volume 26, Number 6, December, 2013, pp 1045-1057. (paper)" | |
| "Zillow Rent Index, 2010-Present",Which city has the highest median price or price per square foot?,Zillow,47,"Version 1,2017-03-04","cities | |
| home | |
| real estate",CSV,10 MB,Other,"11,687 views","1,794 downloads",16 kernels,,https://www.kaggle.com/zillow/rent-index,"Context | |
| Zillow operates an industry-leading economics and analytics bureau led by Zillow’s Chief Economist, Dr. Stan Humphries. At Zillow, Dr. Humphries and his team of economists and data analysts produce extensive housing data and analysis covering more than 500 markets nationwide. Zillow Research produces various real estate, rental and mortgage-related metrics and publishes unique analyses on current topics and trends affecting the housing market. | |
| At Zillow’s core is our living database of more than 100 million U.S. homes, featuring both public and user-generated information including number of bedrooms and bathrooms, tax assessments, home sales and listing data of homes for sale and for rent. This data allows us to calculate, among other indicators, the Zestimate, a highly accurate, automated, estimated value of almost every home in the country as well as the Zillow Home Value Index and Zillow Rent Index, leading measures of median home values and rents. | |
| Content | |
| The Zillow Rent Index is the median estimated monthly rental price for a given area, and covers multifamily, single family, condominium, and cooperative homes in Zillow’s database, regardless of whether they are currently listed for rent. It is expressed in dollars and is seasonally adjusted. The Zillow Rent Index is published at the national, state, metro, county, city, neighborhood, and zip code levels. | |
| Zillow produces rent estimates (Rent Zestimates) based on proprietary statistical and machine learning models. Within each county or state, the models observe recent rental listings and learn the relative contribution of various home attributes in predicting prevailing rents. These home attributes include physical facts about the home, prior sale transactions, tax assessment information and geographic location as well as the estimated market value of the home (Zestimate). Based on the patterns learned, these models estimate rental prices on all homes, including those not presently for rent. Because of the availability of Zillow rental listing data used to train the models, Rent Zestimates are only available back to November 2010; therefore, each ZRI time series starts on the same date. | |
| Acknowledgements | |
| The rent index data was calculated from Zillow's proprietary Rent Zestimates and published on its website. | |
| Inspiration | |
| What city has the highest and lowest rental prices in the country? Which metropolitan area is the most expensive to live in? Where have rental prices increased in the past five years and where have they remained the same? What city or state has the lowest cost per square foot?" | |
| OpenAQ,Global Air Pollution Measurements,Open AQ,47,"Version 1,2017-12-02","pollution | |
| bigquery",BigQuery,2 MB,CC4,"8,181 views",0 downloads,978 kernels,3 topics,https://www.kaggle.com/open-aq/openaq,"OpenAQ is an open-source project to surface live, real-time air quality data from around the world. Their “mission is to enable previously impossible science, impact policy and empower the public to fight air pollution.” The data includes air quality measurements from 5490 locations in 47 countries. | |
| Scientists, researchers, developers, and citizens can use this data to understand the quality of air near them currently. The dataset only includes the most current measurement available for the location (no historical data). | |
| Update Frequency: Weekly | |
| Querying BigQuery tables | |
| You can use the BigQuery Python client library to query tables in this dataset in Kernels. Note that methods available in Kernels are limited to querying data. Tables are at bigquery-public-data.openaq.[TABLENAME]. Fork this kernel to get started. | |
| Acknowledgements | |
| Dataset Source: openaq.org | |
| Use: This dataset is publicly available for anyone to use under the following terms provided by the Dataset Source and is provided ""AS IS"" without any warranty, express or implied." | |
| News Aggregator Dataset,Headlines and categories of 400k news stories from 2014,UCI Machine Learning,46,"Version 1,2016-11-01","news agencies | |
| linguistics",CSV,98 MB,CC0,"22,506 views","2,284 downloads",45 kernels,2 topics,https://www.kaggle.com/uciml/news-aggregator-dataset,"This dataset contains headlines, URLs, and categories for 422,937 news stories collected by a web aggregator between March 10th, 2014 and August 10th, 2014. | |
| News categories included in this dataset include business; science and technology; entertainment; and health. Different news articles that refer to the same news item (e.g., several articles about recently released employment statistics) are also categorized together. | |
| Content | |
| The columns included in this dataset are: | |
| ID : the numeric ID of the article | |
| TITLE : the headline of the article | |
| URL : the URL of the article | |
| PUBLISHER : the publisher of the article | |
| CATEGORY : the category of the news item; one of: -- b : business -- t : science and technology -- e : entertainment -- m : health | |
| STORY : alphanumeric ID of the news story that the article discusses | |
| HOSTNAME : hostname where the article was posted | |
| TIMESTAMP : approximate timestamp of the article's publication, given in Unix time (seconds since midnight on Jan 1, 1970) | |
| Acknowledgments | |
| This dataset comes from the UCI Machine Learning Repository. Any publications that use this data should cite the repository as follows: | |
| Lichman, M. (2013). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science. | |
| This specific dataset can be found in the UCI ML Repository at this URL | |
| Inspiration | |
| What kinds of questions can we explore using this dataset? Here are a few possibilities: | |
| can we predict the category (business, entertainment, etc.) of a news article given only its headline? | |
| can we predict the specific story that a news article refers to, given only its headline?" | |
| Extinct Languages,"Number of endangered languages in the world, and their likelihood of extinction",The Guardian,46,"Version 1,2016-12-07","languages | |
| linguistics",CSV,737 KB,Other,"12,447 views","1,399 downloads",71 kernels,2 topics,https://www.kaggle.com/the-guardian/extinct-languages,"Context | |
| A recent Guardian blog post asks: ""How many endangered languages are there in the World and what are the chances they will die out completely?"" The United Nations Education, Scientific and Cultural Organisation (UNESCO) regularly publishes a list of endangered languages, using a classification system that describes its danger (or completion) of extinction. | |
| Content | |
| The full detailed dataset includes names of languages, number of speakers, the names of countries where the language is still spoken, and the degree of endangerment. The UNESCO endangerment classification is as follows: | |
| Vulnerable: most children speak the language, but it may be restricted to certain domains (e.g., home) | |
| Definitely endangered: children no longer learn the language as a 'mother tongue' in the home | |
| Severely endangered: language is spoken by grandparents and older generations; while the parent generation may understand it, they do not speak it to children or among themselves | |
| Critically endangered: the youngest speakers are grandparents and older, and they speak the language partially and infrequently | |
| Extinct: there are no speakers left | |
| Acknowledgements | |
| Data was originally organized and published by The Guardian, and can be accessed via this Datablog post. | |
| Inspiration | |
| How can you best visualize this data? | |
| Which rare languages are more isolated (Sicilian, for example) versus more spread out? Can you come up with a hypothesis for why that is the case? | |
| Can you compare the number of rare speakers with more relatable figures? For example, are there more Romani speakers in the world than there are residents in a small city in the United States?" | |
| International Energy Statistics,Global energy trade & production 1990-2014,United Nations,46,"Version 2,2017-11-16|Version 1,2017-11-16","economics | |
| energy",Other,7 MB,Other,"5,960 views",901 downloads,6 kernels,,https://www.kaggle.com/unitednations/international-energy-statistics,"Curious about the growth of wind energy? The extent to which the decline of coal is an American or international trend? Interested in using energy consumption as an alternate method of comparing national economies? This dataset has you covered. | |
| The Energy Statistics Database contains comprehensive energy statistics on the production, trade, conversion and final consumption of primary and secondary; conventional and non-conventional; and new and renewable sources of energy. | |
| Acknowledgements | |
| This dataset was kindly published by the United Nations Statistics Division on the UNData site. You can find the original dataset here. | |
| License | |
| Per the UNData terms of use: all data and metadata provided on UNdata’s website are available free of charge and may be copied freely, duplicated and further distributed provided that UNdata is cited as the reference." | |
| Mobile phone activity in a city,"Hourly phone calls, SMS and Internet communication of an entire city",Marco De Nadai,45,"Version 4,2016-11-11|Version 3,2016-11-11|Version 2,2016-11-10|Version 1,2016-11-10","cities | |
| internet | |
| telecommunications",CSV,1 GB,ODbL,"25,619 views","2,596 downloads",39 kernels,2 topics,https://www.kaggle.com/marcodena/mobile-phone-activity,"Introduction | |
| The Mobile phone activity dataset is composed by one week of Call Details Records (CDRs) from the city of Milan and the Province of Trentino (Italy). | |
| Description of the dataset | |
| Every time a user engages a telecommunication interaction, a Radio Base Station (RBS) is assigned by the operator and delivers the communication through the network. Then, a new CDR is created recording the time of the interaction and the RBS which handled it. The following activities are present in the dataset: | |
| received SMS | |
| sent SMS | |
| incoming calls | |
| outgoing calls | |
| Internet activity | |
| In particular, Internet activity is generated each time a user starts an Internet connection or ends an Internet connection. Moreover, during the same connection a CDR is generated if the connection lasts for more than 15 min or the user transferred more than 5 MB. | |
| The datasets is spatially aggregated in a square cells grid. The area of Milan is composed of a grid overlay of 1,000 (squares with size of about 235×235 meters. This grid is projected with the WGS84 (EPSG:4326) standard. For more details we link the original paper http://go.nature.com/2fcOX5E | |
| The data provides CellID, CountryCode and all the aforementioned telecommunication activities aggregated every 60 minutes. | |
| Original datasource | |
| The Mobile phone activity dataset is a part of the Telecom Italia Big Data Challenge 2014, which is a rich and open multi-source aggregation of telecommunications, weather, news, social networks and electricity data from the city of Milan and the Province of Trentino (Italy). The original dataset has been created by Telecom Italia in association with EIT ICT Labs, SpazioDati, MIT Media Lab, Northeastern University, Polytechnic University of Milan, Fondazione Bruno Kessler, University of Trento and Trento RISE. In order to make it easy-to-use, here we provide a subset of telecommunications data that allows researchers to design algorithms able to exploit an enormous number of behavioral and social indicators. The complete version of the dataset is available at the following link: http://go.nature.com/2fz4AFr | |
| Relevant, external, data sources | |
| The presented datasets can be enriched by using census data provided by the Italian National Institute of Statistics (ISTAT) (http://www.istat.it/en/), a public research organization and the main provider of official statistics in Italy. The census data have been released for 1999, 2001 and 2011. The dataset (http://www.istat.it/it/archivio/104317), released in Italian, is composed of four parts: Territorial Bases (Basi Territoriali), Administrative Boundaries (Confini Amministrativi), Census Variables (Variabili Censuarie) and data about Toponymy (Dati Toponomastici). | |
| Motivational video: https://www.youtube.com/watch?v=_d2_RWMsUKc | |
| Relevant papers | |
| Blondel, Vincent D., Adeline Decuyper, and Gautier Krings. ""A survey of results on mobile phone datasets analysis."" EPJ Data Science 4, no. 1 (2015): 1. | |
| Francesco Calabrese, Laura Ferrari, and Vincent D. Blondel. 2014. Urban Sensing Using Mobile Phone Network Data: A Survey of Research. ACM Comput. Surv. 47, 2, Article 25 (November 2014), 20 pages. | |
| Eagle, Nathan, Michael Macy, and Rob Claxton. ""Network diversity and economic development."" Science 328, no. 5981 (2010): 1029-1031. | |
| Lenormand, Maxime, Miguel Picornell, Oliva G. Cantú-Ros, Thomas Louail, Ricardo Herranz, Marc Barthelemy, Enrique Frías-Martínez, Maxi San Miguel, and José J. Ramasco. ""Comparing and modelling land use organization in cities."" Royal Society open science 2, no. 12 (2015): 150449. | |
| Louail, Thomas, Maxime Lenormand, Oliva G. Cantu Ros, Miguel Picornell, Ricardo Herranz, Enrique Frias-Martinez, José J. Ramasco, and Marc Barthelemy. ""From mobile phone data to the spatial structure of cities."" Scientific reports 4 (2014). | |
| De Nadai, Marco, Jacopo Staiano, Roberto Larcher, Nicu Sebe, Daniele Quercia, and Bruno Lepri. ""The Death and Life of Great Italian Cities: A Mobile Phone Data Perspective."" WWW, 2016. | |
| Citation | |
| We kindly ask people who use this dataset to cite the following paper, where this aggregation comes from: | |
| Barlacchi, Gianni, Marco De Nadai, Roberto Larcher, Antonio Casella, Cristiana Chitic, Giovanni Torrisi, Fabrizio Antonelli, Alessandro Vespignani, Alex Pentland, and Bruno Lepri. ""A multi-source dataset of urban life in the city of Milan and the Province of Trentino."" Scientific data 2 (2015)." | |
| Magic The Gathering Cards,Analyze cards from this classic trading card game,Myles O'Neill,45,"Version 1,2016-09-27","games and toys | |
| card games",{}JSON,53 MB,Other,"12,840 views",618 downloads,36 kernels,,https://www.kaggle.com/mylesoneill/magic-the-gathering-cards,"Magic The Gathering (MTG, or just Magic) is a trading card game first published in 1993 by Wizards of the Coast. This game has seen immense popularity and new cards are still released every few months. The strength of different cards in the game can vary wildly and as a result some cards now sell on secondary markets for as high as thousands of dollars. | |
| MTG JSON has an excellent collection of every single Magic Card - stored in JSON data. Version 3.6 (collected September 21, 2016) of their database is provided here. | |
| Full documentation for the data is provided here: http://mtgjson.com/documentation.html | |
| Also, if you want to include images of the cards in your writeups, you can grab them from the official Wizards of the Coast website using the following URL: | |
| http://gatherer.wizards.com/Handlers/Image.ashx?multiverseid=180607&type=card | |
| Just replace the multiverse ID with the one provided in the mtgjson file." | |
| Epicurious - Recipes with Rating and Nutrition,"Recipes from Epicurious by rating, nutritional content, and categories",HugoDarwood,45,"Version 2,2017-02-21|Version 1,2016-12-25","food and drink | |
| nutrition",Other,86 MB,Other,"16,850 views","2,477 downloads",268 kernels,,https://www.kaggle.com/hugodarwood/epirecipes,"Context | |
| I created this dataset to explore different factors affecting people's enjoyment of food and/or cooking! | |
| Content | |
| Over 20k recipes listed by recipe rating, nutritional information and assigned category (sparse). I may later upload a version binned by recipe creation date and also including recipe ingredients. | |
| Use the 'full_format_recipes.json' file to interact with all recipe data, 'epi_r.csv' drops ingredients and directions in favour of sparse category dummies. | |
| Acknowledgements | |
| Recipe information lifted from: http://www.epicurious.com/recipes-menus" | |
| Funding Successful Projects on Kickstarter,Predict if a project will get successfully funded or not using labeled data,Ashok Lathwal,45,"Version 1,2017-06-21","business | |
| finance",CSV,57 MB,Other,"8,273 views",999 downloads,6 kernels,5 topics,https://www.kaggle.com/codename007/funding-successful-projects,"Problem Statement | |
| Kickstarter is a community of more than 10 million people comprising of creative, tech enthusiasts who help in bringing creative project to life. Till now, more than $3 billion dollars have been contributed by the members in fueling creative projects. The projects can be literally anything – a device, a game, an app, a film etc. | |
| Kickstarter works on all or nothing basis i.e if a project doesn’t meet it goal, the project owner gets nothing. For example: if a projects’s goal is $500. Even if it gets funded till $499, the project won’t be a success. | |
| Recently, Kickstarter released its public data repository to allow researchers and enthusiasts like us to help them solve a problem. Will a project get fully funded ? | |
| In this challenge, you have to predict if a project will get successfully funded or not. | |
| Data Description | |
| There are three files given to download: train.csv, test.csv and sample_submission.csv The train data consists of sample projects from the May 2009 to May 2015. The test data consists of projects from June 2015 to March 2017." | |
| 2016 Election Polls,Collection of Presidential Election Polls from 2015-2016,FiveThirtyEight,45,"Version 1,2016-11-03","news agencies | |
| politics",CSV,3 MB,Other,"18,154 views","2,251 downloads",83 kernels,2 topics,https://www.kaggle.com/fivethirtyeight/2016-election-polls,"Dataset Information | |
| This dataset is a collection of state and national polls conducted from November 2015-November 2016 on the 2016 presidential election. Data on the raw and weighted poll results by state, date, pollster, and pollster ratings are included. | |
| Content | |
| There are 27 variables: | |
| cycle | |
| branch | |
| type | |
| matchup | |
| forecastdate | |
| state: | |
| startdate | |
| enddate | |
| pollster | |
| grade | |
| samplesize | |
| populaion | |
| poll_wt | |
| rawpoll_clinton | |
| rawpoll_trump | |
| rawpoll_johnson | |
| rawpoll_mcmullin | |
| adjpoll_clinton | |
| adjpoll_trump | |
| adjpoll_johnson | |
| adjpoll_mcmullin | |
| multiversions | |
| url | |
| poll_id | |
| question_id | |
| createddate | |
| timestamp | |
| Inspiration | |
| Some questions for exploring this dataset are: | |
| What are the trends of the polls over time (by day/week/month)? | |
| How do the trends vary by state? | |
| What is the probability that Trump/Clinton will win the 2016 election? | |
| Acknowledgements | |
| The original dataset is from the FiveThirtyEight 2016 Election Forecast and can be downloaded from here. Poll results were aggregated from HuffPost Pollster, RealClearPolitics, polling firms and news reports." | |
| MovieLens 20M Dataset,Over 20 Million Movie Ratings and Tagging Activities Since 1995,GroupLens,45,"Version 2,2016-11-08|Version 1,2016-11-07",film,CSV,885 MB,Other,"15,733 views","2,146 downloads",28 kernels,0 topics,https://www.kaggle.com/grouplens/movielens-20m-dataset,"Context | |
| The datasets describe ratings and free-text tagging activities from MovieLens, a movie recommendation service. It contains 20000263 ratings and 465564 tag applications across 27278 movies. These data were created by 138493 users between January 09, 1995 and March 31, 2015. This dataset was generated on October 17, 2016. | |
| Users were selected at random for inclusion. All selected users had rated at least 20 movies. | |
| Content | |
| No demographic information is included. Each user is represented by an id, and no other information is provided. | |
| The data are contained in six files. | |
| tag.csv that contains tags applied to movies by users: | |
| userId | |
| movieId | |
| tag | |
| timestamp | |
| rating.csv that contains ratings of movies by users: | |
| userId | |
| movieId | |
| rating | |
| timestamp | |
| movie.csv that contains movie information: | |
| movieId | |
| title | |
| genres | |
| link.csv that contains identifiers that can be used to link to other sources: | |
| movieId | |
| imdbId | |
| tmbdId | |
| genome_scores.csv that contains movie-tag relevance data: | |
| movieId | |
| tagId | |
| relevance | |
| genome_tags.csv that contains tag descriptions: | |
| tagId | |
| tag | |
| Acknowledgements | |
| The original datasets can be found here. To acknowledge use of the dataset in publications, please cite the following paper: | |
| F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets: History and Context. ACM Transactions on Interactive Intelligent Systems (TiiS) 5, 4, Article 19 (December 2015), 19 pages. DOI=http://dx.doi.org/10.1145/2827872 | |
| Inspiration | |
| Some ideas worth exploring: | |
| Which genres receive the highest ratings? How does this change over time? | |
| Determine the temporal trends in the genres/tagging activity of the movies released" | |
| Stanford Question Answering Dataset,"New Reading Comprehension Dataset on 100,000+ Question-Answer Pairs",Stanford University,45,"Version 1,2016-11-15","languages | |
| linguistics",{}JSON,34 MB,CC4,"12,777 views",884 downloads,13 kernels,0 topics,https://www.kaggle.com/stanfordu/stanford-question-answering-dataset,"Context | |
| The Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset consisting of questions posed by crowdworkers on a set of Wikipedia articles. The answer to every question is a segment of text, or span, from the corresponding reading passage. There are 100,000+ question-answer pairs on 500+ articles. | |
| Content | |
| There are two files to help you get started with the dataset and evaluate your models: | |
| *train-v1.1.json*fasdf | |
| dev-v1.1.json | |
| Acknowledgements | |
| The original datasets can be found here. | |
| Inspiration | |
| Can you build a prediction model that can accurately predict answers to different types of questions? | |
| You can also explore SQuAD here" | |
| Amazon Reviews: Unlocked Mobile Phones,"More than 400,000 reviews from Amazon's unlocked mobile phone category",PromptCloud,45,"Version 1,2017-01-11","business | |
| internet | |
| telecommunications",CSV,126 MB,CC0,"19,623 views","2,350 downloads",93 kernels,4 topics,https://www.kaggle.com/PromptCloudHQ/amazon-reviews-unlocked-mobile-phones,"Context | |
| PromptCloud extracted 400 thousand reviews of unlocked mobile phones sold on Amazon.com to find out insights with respect to reviews, ratings, price and their relationships. | |
| Content | |
| Given below are the fields: | |
| Product Title | |
| Brand | |
| Price | |
| Rating | |
| Review text | |
| Number of people who found the review helpful | |
| Data was acquired in December, 2016 by the crawlers build to deliver our data extraction services. | |
| Initial Analysis | |
| It can be accessed here: http://www.kdnuggets.com/2017/01/data-mining-amazon-mobile-phone-reviews-interesting-insights.html" | |
| US ACS Financial Hedging Features,+30 Features To Build Models & Make Money,Golden Oak Research Group,45,"Version 4,2018-02-06|Version 3,2017-08-28|Version 2,2017-08-26|Version 1,2017-08-26","finance | |
| demographics",Other,2 MB,Other,"4,826 views",605 downloads,3 kernels,5 topics,https://www.kaggle.com/goldenoakresearch/us-acs-mortgage-equity-loans-rent-statistics,"Background: | |
| The underlying concept behind hedging strategies is simple, create a model, and make money doing it. The hardest part is finding the features that matter. For a more in-depth look at hedging strategies, I have attached one of my graduate papers to get you started. | |
| Mortgage-Backed Securities | |
| Geographic Business Investment | |
| Real Estate Analysis | |
| For any questions, you may reach us at research_development@goldenoakresearch.com. For immediate assistance, you may reach me on at 585-626-2965. Please Note: the number is my personal number and email is preferred | |
| Statistical Fields: | |
| Note: All interpolated statistical data include Mean, Median, and Standard Deviation Statistics. For more information view the variable definitions document. | |
| Monthly Mortgage & Owner Costs: Sum of mortgage payments, home equity loans, utilities, property taxes | |
| Monthly Owner Costs: Sum of utilities, property taxes | |
| Gross Rent: contract rent plus the estimated average monthly cost of utilities | |
| Household Income: sum of the householder and all other individuals +15 years who reside in the household | |
| Family Income: Sum of incomes of all members +15 years of age related to the householder. | |
| Location Fields: | |
| Note: The location fields were derived from a variety of sources. The zip code, area code, and city were derived using a heuristic. All other locations used the census Geo ID to cross reference data between multiple datasets. | |
| State Name, Abbreviation, Number: reported by the U.S. Census Bureau | |
| County Name: reported by the U.S. Census Bureau | |
| Location Type: Specifies Classification of location {City, Village, Town, CPD, ..., etc.} | |
| Area Code: Defined via heuristic. | |
| Zip Code: Defined via heuristic. | |
| City: Defined via heuristic. | |
| Access All 325,258 Location of Our Most Complete Database Ever: | |
| Monetize Risk and Optimize your portfolio instantly at an unbeatable price. Don't settle. Go big and win big. Access all gross rent records and more on a scale roughly equivalent to a neighborhood, see link below: | |
| Full Dataset: View Full Dataset | |
| Real Estate Research: View Research" | |
| Every song you have heard (almost)!,"Over 500,000 song lyrics, urls for over a million artists",Soumitra Agarwal,44,"Version 1,2017-08-20","languages | |
| music | |
| internet",CSV,601 MB,ODbL,"5,629 views",621 downloads,3 kernels,,https://www.kaggle.com/artimous/every-song-you-have-heard-almost,"Dataset for people who double on Music and Data Science | |
| How it began | |
| One fine day, when someone with the idea of self sufficient AI capable of writing it's own poems wanted data, he stumbled upon the idea of using songs as a source. The journey wasn't easy, since each song has it's own page with the lyrics and scraping pages one at a time (when there are over a million of them) is a slow task. I worked up some optimisations here and there. For people interested in going through the process: | |
| GITHUB PROJECT. | |
| Content | |
| There is a lot to play with here, though all of it vertically. There isn't a lot of variation in the types of fields until you look deep enough. | |
| The main dataset is split up into 2 files, each containing ~250,000 songs with their artists and lyrics. The files are titled Lyrics1 and Lyrics2. The fields include band name, song name and lyrics. | |
| Go through the exploration scripts for how to use the lyrics data set and other interesting observations. | |
| Another file containing urls of description pages of different artists is also provided, titled ArtistUrls. | |
| Acknowledgements | |
| After seeing the response for the FIFA PLAYER DATASET it was exciting as ever to get this one off. | |
| Soon it was realised, not a lot of places exist where extracting data is straightforward. | |
| Then one stumbles upon the perfectly indexed page : https://www.lyrics.com/. They deserve major credit for the existence of this dataset. | |
| Inspiration | |
| This started off as a source for some kind of intelligent poet which writes poems on it's own. It would be great to see what the artificially intelligent world has to express once it knows enough, and beautifully if at all?" | |
| Pokemon Sun and Moon (Gen 7) Stats,Explore all 802 Pokemon from the newly released 7th Generation,Myles O'Neill,44,"Version 3,2016-11-19|Version 2,2016-11-18|Version 1,2016-11-17","games and toys | |
| video games",CSV,2 MB,Other,"13,528 views","1,311 downloads",36 kernels,0 topics,https://www.kaggle.com/mylesoneill/pokemon-sun-and-moon-gen-7-stats,"Pokemon Sun and Moon (released November 18th, 2016) are the latest games in the widely popular Pokemon video game franchise. Pokemon games are usually released in pairs (red and blue, gold and silver, x and y, etc.) and collectively each pair that introduces new pokemon to the game is known as a Generation. Pokemon Sun and Moon are the 7th Generation, adding new pokemon to the franchise as well as adjusting the stats and movesets of some of the older pokemon. (Please note that the recently popular PokemonGo games are unrelated to the Pokemon Video Games). | |
| This dataset contains a full set of in-game statistics for all 802 pokemon in the Sun and Moon. It also includes full information on which pokemon can learn which moves (movesets.csv), what moves can do (moves.csv), and how damage is modified by pokemon type (type-chart.csv). | |
| Pokemon Battle Simulation | |
| With the level of detail in the data provided here it is possible to almost fully simulate pokemon battles using the information provided (status effects and some other nuances are still missing). If you are interested in simulating how Pokemon battles would pan out between different opponents make sure to read up on the math behind how Pokemon battles work." | |
| "French employment, salaries, population per town",Some data to show equality and inequalities in France,Etienne LQ,44,"Version 7,2017-10-27|Version 6,2017-10-27|Version 5,2017-10-25|Version 4,2017-10-24|Version 3,2017-10-18|Version 2,2017-10-13|Version 1,2017-10-13","employment | |
| demographics",CSV,344 MB,CC0,"8,148 views","1,070 downloads",7 kernels,3 topics,https://www.kaggle.com/etiennelq/french-employment-by-town,"Context | |
| INSEE is the official french institute gathering data of many types around France. It can be demographic (Births, Deaths, Population Density...), Economic (Salary, Firms by activity / size...) and more. | |
| It can be a great help to observe and measure inequality in the french population. | |
| Content | |
| Four files are in the dataset : | |
| base_etablissement_par_tranche_effectif : give information on the number of firms in every french town, categorized by size , come from INSEE. | |
| CODGEO : geographique code for the town (can be joined with *code_insee* column from ""name_geographic_information.csv') | |
| LIBGEO : name of the town (in french) | |
| REG : region number | |
| DEP : depatment number | |
| E14TST : total number of firms in the town | |
| E14TS0ND : number of unknown or null size firms in the town | |
| E14TS1 : number of firms with 1 to 5 employees in the town | |
| E14TS6 : number of firms with 6 to 9 employees in the town | |
| E14TS10 : number of firms with 10 to 19 employees in the town | |
| E14TS20 : number of firms with 20 to 49 employees in the town | |
| E14TS50 : number of firms with 50 to 99 employees in the town | |
| E14TS100 : number of firms with 100 to 199 employees in the town | |
| E14TS200 : number of firms with 200 to 499 employees in the town | |
| E14TS500 : number of firms with more than 500 employees in the town | |
| name_geographic_information : give geographic data on french town (mainly latitude and longitude, but also region / department codes and names ) | |
| EU_circo : name of the European Union Circonscription | |
| code_région : code of the region attached to the town | |
| nom_région : name of the region attached to the town | |
| chef.lieu_région : name the administrative center around the town | |
| numéro_département : code of the department attached to the town | |
| nom_département : name of the department attached to the town | |
| préfecture : name of the local administrative division around the town | |
| numéro_circonscription : number of the circumpscription | |
| nom_commune : name of the town | |
| codes_postaux : post-codes relative to the town | |
| code_insee : unique code for the town | |
| latitude : GPS latitude | |
| longitude : GPS longitude | |
| éloignement : i couldn't manage to figure out what was the meaning of this number | |
| net_salary_per_town_per_category : salaries around french town per job categories, age and sex | |
| CODGEO : unique code of the town | |
| LIBGEO : name of the town | |
| SNHM14 : mean net salary | |
| SNHMC14 : mean net salary per hour for executive | |
| SNHMP14 : mean net salary per hour for middle manager | |
| SNHME14 : mean net salary per hour for employee | |
| SNHMO14 : mean net salary per hour for worker | |
| SNHMF14 : mean net salary for women | |
| SNHMFC14 : mean net salary per hour for feminin executive | |
| SNHMFP14 : mean net salary per hour for feminin middle manager | |
| SNHMFE14 : mean net salary per hour for feminin employee | |
| SNHMFO14 : mean net salary per hour for feminin worker | |
| SNHMH14 : mean net salary for man | |
| SNHMHC14 : mean net salary per hour for masculin executive | |
| SNHMHP14 : mean net salary per hour for masculin middle manager | |
| SNHMHE14 : mean net salary per hour for masculin employee | |
| SNHMHO14 : mean net salary per hour for masculin worker | |
| SNHM1814 : mean net salary per hour for 18-25 years old | |
| SNHM2614 : mean net salary per hour for 26-50 years old | |
| SNHM5014 : mean net salary per hour for >50 years old | |
| SNHMF1814 : mean net salary per hour for women between 18-25 years old | |
| SNHMF2614 : mean net salary per hour for women between 26-50 years old | |
| SNHMF5014 : mean net salary per hour for women >50 years old | |
| SNHMH1814 : mean net salary per hour for men between 18-25 years old | |
| SNHMH2614 : mean net salary per hour for men between 26-50 years old | |
| SNHMH5014 : mean net salary per hour for men >50 years old | |
| population : demographic information in France per town, age, sex and living mode | |
| NIVGEO : geographic level (arrondissement, communes...) | |
| CODGEO : unique code for the town | |
| LIBGEO : name of the town (might contain some utf-8 errors, this information has better quality name_geographic_information) | |
| MOCO : cohabitation mode : [list and meaning available in Data description] | |
| AGE80_17 : age category (slice of 5 years) | ex : 0 -> people between 0 and 4 years old | |
| SEXE : sex, 1 for men | 2 for women | |
| NB : Number of people in the category | |
| departments.geojson : contains the borders of french departments. From Gregoire David (github) | |
| These datasets can be merged by : CODGEO = code_insee | |
| Acknowledgements | |
| The entire dataset has been created (and actualized) by INSEE, I just uploaded it on Kaggle after doing some jobs and checks on it. I haven't seen INSEE on Kaggle yet but I think it would be great to bring the organization in as a Kaggle actor. | |
| Inspiration | |
| First aim I had creating that dataset was to provide a map of french towns with the number of firm that are settled in by size. | |
| Now my goal is to explore inequality between men and women, youngsters and elders, working / social classes. | |
| Population can also be a great filter to explain some phenomenons on the maps." | |
| 2015 Notebook UX Survey,Understand user perspectives on Jupyter Notebooks,Project Jupyter,44,"Version 2,2017-05-02|Version 1,2016-01-21",human-computer interaction,CSV,748 KB,CC4,"12,070 views",813 downloads,20 kernels,0 topics,https://www.kaggle.com/jupyter/2015-notebook-ux-survey,"At the end of 2015, the Jupyter Project conducted a UX Survey for Jupyter Notebook users. This dataset, Survey.csv, contains the raw responses. | |
| See the Google Group Thread for more context around this dataset." | |
| My Complete Genome,"6,000 Base-Pairs of Phenotype SNPs - Complete Raw Data",Zeeshan-ul-hassan Usmani,44,"Version 4,2017-01-30|Version 3,2017-01-28|Version 2,2017-01-28|Version 1,2017-01-26",human genetics,CSV,15 MB,CC0,"12,160 views",749 downloads,8 kernels,2 topics,https://www.kaggle.com/zusmani/mygenome,"Context | |
| Zeeshan-ul-hassan Usmani’s Genome Phenotype SNPs Raw Data | |
| Genomics is a branch of molecular biology that involves structure, function, variation, evolution and mapping of genomes. There are several companies offering next generation sequencing of human genomes from complete 3 billion base-pairs to a few thousand Phenotype SNPs. I’ve used 23andMe (using Illumina HumanOmniExpress-24) for my DNA’s Phenotype SNPs. I am sharing the entire raw dataset here for the international research community for following reasons: | |
| I am a firm believer in open dataset, transparency, and the right to learn, research, explores, and educate. I do not want to restrict the knowledge flow for mere privacy concerns. Hence, I am offering my entire DNA raw data for the world to use for research without worrying about privacy. I call it copyleft dataset. | |
| Most of available test datasets for research come from western world and we don’t see much from under-developing countries. I thought to share my data to bridge the gap and I expect others to follow the trend. | |
| I would be the happiest man on earth, if a life can be saved, knowledge can be learned, an idea can be explore, or a fact can be found using my DNA data. Please use it the way you will | |
| Content | |
| Name: Zeeshan-ul-hassan Usmani | |
| Age: 38 Years | |
| Country of Birth: Pakistan | |
| Country of Ancestors: India (Utter Pradesh - UP) | |
| File: GenomeZeeshanUsmani.csv | |
| Size: 15 MB | |
| Sources: 23andMe Personalized Genome Report | |
| The research community is still progressively working in this domain and it is agreed upon by professionals that genomics is still in its infancy. You now have the chance to explore this novel domain via the dataset and become one of the few genomics early adopters. | |
| The data-set is a complete genome extracted from www.23andme.com and is represented as a sequence of SNPs represented by the following symbols: A (adenine), C (cytosine), G (guanine), T (thymine), D (base deletions), I (base insertions), and '_' or '-' if the SNP for particular location is not accessible. It contains Chromosomes 1-22, X, Y, and mitochondrial DNA. | |
| A complete list of the exact SNPs (base pairs) available and their data-set index can be found at https://api.23andme.com/res/txt/snps.b4e00fe1db50.data | |
| For more information about how the data-set was extracted follow https://api.23andme.com/docs/reference/#genomes | |
| Moreover, for a more detailed understanding of the data-set content please acquaint yourself with the description of https://api.23andme.com/docs/reference/#genotypes | |
| Acknowledgements | |
| Users are allowed to use, copy, distribute and cite the dataset as follows: “Zeeshan-ul-hassan Usmani, Genome Phenotype SNPS Raw Data File by 23andMe, Kaggle Dataset Repository, Jan 25, 2017.” | |
| Useful Links | |
| You may use the following human genome database sites for help: | |
| GenBank - https://www.ncbi.nlm.nih.gov/genbank/ | |
| The Human Genome Project - https://www.genome.gov/hgp/ | |
| Genomes OnLine Database (GOLD) - https://gold.jgi.doe.gov | |
| Complete Genomics - http://www.completegenomics.com/public-data/ | |
| Inspiration | |
| Some ideas worth exploring: | |
| Is the individual in question more susceptible to cancer? | |
| Does he tend to gain weight? | |
| Where is his place of origin? | |
| Which gene determines certain biological feature (cancer susceptibility, fat generation rate, hair color etc. | |
| How does this phenotype SNPs compare with other similar datasets from the western-world? | |
| What would be the likely cause of death for this person? | |
| What are the most likely diseases/illnesses this person is going to face in lifetime? | |
| What is unique about this dataset? | |
| What else you can extract from this dataset when it comes to personal trait, intelligence level, ancestry and body makeup? | |
| Sample Reports | |
| Please check out following reports to understand what can be done with this data | |
| Ancestry – https://www.23andme.com/published-report/eeb4f9bbd6b5474f/?share_id=f6c5562848e84586 | |
| Weight Report - https://you.23andme.com/published/reports/65c9af9f8223456d/?share_id=0126f129e4f3458b" | |
| "United States Energy, Census, and GDP 2010-2014",Examining the relationships between various data collected by the US government,Lislejoem,44,"Version 4,2017-03-25|Version 3,2017-03-21|Version 2,2017-02-03|Version 1,2016-11-20",energy,CSV,74 KB,CC0,"12,191 views","1,587 downloads",26 kernels,,https://www.kaggle.com/lislejoem/us_energy_census_gdp_10-14,"The purpose of this data set is to allow exploration between various types of data that is commonly collected by the US government across the states and the USA as a whole. The data set consists of three different types of data: | |
| Census and Geographic Data; | |
| Energy Data; and | |
| Economic Data. | |
| When creating the data set, I combined data from many different types of sources, all of which are cited below. I have also provided the fields included in the data set and what they represent below. I have not performed any research on the data yet, but am going to dive in soon. I am particularly interested in the relationships between various types of data (i.e. GDP or birth rate) in prediction algorithms. Given that I have compiled 5 years’ worth of data, this data set was primarily constructed with predictive algorithms in mind. | |
| An additional note before you delve into the fields: * There could have been many more variables added across many different fields of metrics. I have stopped here, but it could potentially be beneficial to observe the interaction of these variables with others (i.e. the GDP of certain industries, the average age in a state, the male/female gender ratio, etc.) to attempt to find additional trends. | |
| Census and Geographic Data | |
| StateCodes: The state 2-letter abbreviations. Note that I added ""US"" for the United States. | |
| Region: The number corresponding to the region the state lies within, according to the 2010 census. (1 = Northeast, 2 = Midwest, 3 = South, 4 = West) | |
| Division: The number corresponding to the division the state lies within, according to the 2010 census. (1 = New England, 2 = Middle Atlantic, 3 = East North Central, 4 = West North Central, 5 = South Atlantic, 6 = East South Central, 7 = West South Central, 8 = Mountain, 9 = Pacific) | |
| Coast: Whether the state shares a border with an ocean. (1 = Yes, 0 = No) | |
| Great Lakes: Whether the state shares a border with a great lake. (1 = Yes, 0 = No | |
| CENSUS2010POP: 4/1/2010 resident total Census 2010 population | |
| POPESTIMATE{year}: 7/1/{year} resident total population estimate | |
| RBIRTH{year}: Birth rate in period 7/1/{year - 1} to 6/30/{year} | |
| RDEATH{year}: Death rate in period 7/1/{year - 1} to 6/30/{year} | |
| RNATURALINC{year}: Natural increase rate in period 7/1/{year - 1} to 6/30/{year} | |
| RINTERNATIONALMIG{year}: Net international migration rate in period 7/1/{year - 1} to 6/30/{year} | |
| RDOMESTICMIG{year}: Net domestic migration rate in period 7/1/{year - 1} to 6/30/{year} | |
| RNETMIG{year}: Net migration rate in period 7/1/{year - 1} to 6/30/{year} | |
| As noted from the census: | |
| Net international migration for the United States includes the international migration of both native and foreign-born populations. Specifically, it includes: (a) the net international migration of the foreign born, (b) the net migration between the United States and Puerto Rico, (c) the net migration of natives to and from the United States, and (d) the net movement of the Armed Forces population between the United States and overseas. Net international migration for Puerto Rico includes the migration of native and foreign-born populations between the United States and Puerto Rico. | |
| Codes for most of the data, information about the geographic terms and coditions, and more information about the methodology behind the population estimates can be found on the US Census website. | |
| Energy Data | |
| TotalC{year}: Total energy consumption in billion BTU in given year. | |
| TotalP{year}: Total energy production in billion BTU in given year. | |
| TotalE{year}: Total Energy expenditures in million USD in given year. | |
| TotalPrice{year}: Total energy average price in USD/million BTU in given year. | |
| TotalC{first year}–{second year}: The first year’s total energy consumption divided by the second year’s total energy consumption, times 100. (The percent change between years in total energy consumption.) | |
| TotalP{first year}–{second year}: The first year’s total energy production divided by the second year’s total energy production, times 100. (The percent change between years in total energy production.) | |
| TotalE{first year}–{second year}: The first year’s total energy expenditure divided by the second year’s total energy expenditure, times 100. (The percent change between years in total energy expenditure.) | |
| TotalPrice{first year}–{second year}: The first year’s total energy average price divided by the second year’s total energy average price, times 100. (The percent change between years in total energy average price.) | |
| BiomassC{year}: Biomass total consumption in billion BTU in given year. | |
| CoalC{year}: Coal total consumption in billion BTU in given year. | |
| CoalP{year}: Coal total production in billion BTU in given year. | |
| CoalE{year}: Coal total expenditures in million USD in given year. | |
| CoalPrice{year}: Coal average price in USD per million BTU in given year. | |
| ElecC{year}: Electricity total consumption in billion BTU in given year. | |
| ElecE{year}: Electricity total expenditures in million USD in given year. | |
| ElecPrice{year}: Electricity average price in USD per million BTU in given year. | |
| FossFuelC{year}: Fossil fuels total consumption in billion BTU in given year. | |
| GeoC{year}: Geothermal energy total consumption in billion BTU in given year. | |
| GeoP{year}: Geothermal energy net generation in the electric power sector in million kilowatt hours in given year. | |
| HydroC{year}: Hydropower total consumption in billion BTU in given year. | |
| HydroP{year}: Hydropower total net generation in million kilowatt hours in given year. | |
| NatGasC{year}: Natural gas total consumption (including supplemental gaseous fuels) in billion BTU in given year. | |
| NatGasE{year}: Natural gas total expenditures in million USD in given year. | |
| NatGasPrice{year}: Natural gas average price in USD per million BTU in given year. | |
| LPGC{year}: LPG total consumption in billion BTU in given year. | |
| LPGE{year}: LPG total expenditures in million USD in given year. | |
| LPGPrice{year}: LPG average price in USD per million BTU in given year. | |
| Notes: | |
| BTU stands for British Thermal Unit and is a unit of measurement for energy. One BTU is equal to the amount of energy used to raise the temperature of one pound of water on degree Fahrenheit. | |
| Many other types of energy and their associated consumption, production, expenditure, and price totals can be found from the EIA; this is where I received the data I used in compiling this dataset. | |
| Economic Data | |
| GDP{year}{quarter}: The GDP in the provided quarter of the given year (in million USD). | |
| GDP{year}: The average GDP throughout the given year (in million USD). | |
| Notes: | |
| The GDP is reported by the Bureau of Economic Analysis from the U.S. Department of Commerce and measures the value of the goods and services produced by the economy in a given period. | |
| The quarterly GDP data can be downloaded from the BEA. | |
| The yearly GDP data can be downloaded from the BEA. | |
| Image credit: http://www.freelargeimages.com/wp-content/uploads/2014/11/Map_of_united_states-3.jpg" | |
| Crowdedness at the Campus Gym,Number of attendees every 10 minutes from the last year in the gym,Nick Rose,44,"Version 2,2017-03-19|Version 1,2016-12-14","health | |
| demographics | |
| sociology",CSV,3 MB,ODbL,"16,879 views","1,609 downloads",78 kernels,7 topics,https://www.kaggle.com/nsrose7224/crowdedness-at-the-campus-gym,"Background | |
| When is my university campus gym least crowded, so I know when to work out? We measured how many people were in this gym once every 10 minutes over the last year. We want to be able to predict how crowded the gym will be in the future. | |
| Goals | |
| Given a time of day (and maybe some other features, including weather), predict how crowded the gym will be. | |
| Figure out which features are actually important, which are redundant, and what features could be added to make the predictions more accurate. | |
| Data | |
| The dataset consists of 26,000 people counts (about every 10 minutes) over the last year. In addition, I gathered extra info including weather and semester-specific information that might affect how crowded it is. The label is the number of people, which I'd like to predict given some subset of the features. | |
| Label: | |
| Number of people | |
| Features: | |
| date (string; datetime of data) | |
| timestamp (int; number of seconds since beginning of day) | |
| day_of_week (int; 0 [monday] - 6 [sunday]) | |
| is_weekend (int; 0 or 1) [boolean, if 1, it's either saturday or sunday, otherwise 0] | |
| is_holiday (int; 0 or 1) [boolean, if 1 it's a federal holiday, 0 otherwise] | |
| temperature (float; degrees fahrenheit) | |
| is_start_of_semester (int; 0 or 1) [boolean, if 1 it's the beginning of a school semester, 0 otherwise] | |
| month (int; 1 [jan] - 12 [dec]) | |
| hour (int; 0 - 23) | |
| Acknowledgements | |
| This data was collected with the consent of the university and the gym in question." | |
| ATP Men's Tour,Results of the ATP tour competitions since 2000,Jordan Goblet,43,"Version 2,2016-09-27|Version 1,2016-09-26",tennis,CSV,9 MB,Other,"15,319 views","1,794 downloads",72 kernels,3 topics,https://www.kaggle.com/jordangoblet/atp-tour-20002016,"Results for the men's ATP tour date back to January 2000, including Grand Slams, Masters Series, Masters Cup and International Series competitions. Historical head-to-head betting odds go back to 2001. | |
| See here for more details about the metadata : http://www.tennis-data.co.uk/notes.txt | |
| Source - http://www.tennis-data.co.uk/data.php | |
| There is a lot you can do with this data set. The ultimate goal is obviously to predict the outcome of the game or to build an efficient betting strategy based on your model(s). | |
| You can also | |
| compare the betting agencies (Bet365, Bet&Win, Ladbrokes, Pinnacles...) in terms of predictions quality or measure the progress of these betting agencies over the years | |
| discover if it's possible to predict specific events (3 sets match, retirement, walk over...) or the evolution of players." | |
| 3D MNIST,A 3D version of the MNIST database of handwritten digits,David de la Iglesia Castro,43,"Version 13,2016-11-10|Version 12,2016-11-05|Version 11,2016-11-05|Version 10,2016-10-26|Version 9,2016-10-26|Version 8,2016-10-25|Version 7,2016-10-25|Version 6,2016-10-20|Version 5,2016-10-18|Version 4,2016-10-18|Version 3,2016-10-18|Version 2,2016-10-18|Version 1,2016-10-13","writing | |
| artificial intelligence",Other,244 MB,Other,"14,269 views",830 downloads,62 kernels,2 topics,https://www.kaggle.com/daavoo/3d-mnist,"Context | |
| The aim of this dataset is to provide a simple way to get started with 3D computer vision problems such as 3D shape recognition. | |
| Accurate 3D point clouds can (easily and cheaply) be adquired nowdays from different sources: | |
| RGB-D devices: Google Tango, Microsoft Kinect, etc. | |
| Lidar. | |
| 3D reconstruction from multiple images. | |
| However there is a lack of large 3D datasets (you can find a good one here based on triangular meshes); it's especially hard to find datasets based on point clouds (wich is the raw output from every 3D sensing device). | |
| This dataset contains 3D point clouds generated from the original images of the MNIST dataset to bring a familiar introduction to 3D to people used to work with 2D datasets (images). | |
| In the 3D_from_2D notebook you can find the code used to generate the dataset. | |
| You can use the code in the notebook to generate a bigger 3D dataset from the original. | |
| Content | |
| full_dataset_vectors.h5 | |
| The entire dataset stored as 4096-D vectors obtained from the voxelization (x:16, y:16, z:16) of all the 3D point clouds. | |
| In adition to the original point clouds, it contains randomly rotated copies with noise. | |
| The full dataset is splitted into arrays: | |
| X_train (10000, 4096) | |
| y_train (10000) | |
| X_test(2000, 4096) | |
| y_test (2000) | |
| Example python code reading the full dataset: | |
| with h5py.File(""../input/train_point_clouds.h5"", ""r"") as hf: | |
| X_train = hf[""X_train""][:] | |
| y_train = hf[""y_train""][:] | |
| X_test = hf[""X_test""][:] | |
| y_test = hf[""y_test""][:] | |
| train_point_clouds.h5 & test_point_clouds.h5 | |
| 5000 (train), and 1000 (test) 3D point clouds stored in HDF5 file format. The point clouds have zero mean and a maximum dimension range of 1. | |
| Each file is divided into HDF5 groups | |
| Each group is named as its corresponding array index in the original mnist dataset and it contains: | |
| ""points"" dataset: x, y, z coordinates of each 3D point in the point cloud. | |
| ""normals"" dataset: nx, ny, nz components of the unit normal associate to each point. | |
| ""img"" dataset: the original mnist image. | |
| ""label"" attribute: the original mnist label. | |
| Example python code reading 2 digits and storing some of the group content in tuples: | |
| with h5py.File(""../input/train_point_clouds.h5"", ""r"") as hf: | |
| a = hf[""0""] | |
| b = hf[""1""] | |
| digit_a = (a[""img""][:], a[""points""][:], a.attrs[""label""]) | |
| digit_b = (b[""img""][:], b[""points""][:], b.attrs[""label""]) | |
| voxelgrid.py | |
| Simple Python class that generates a grid of voxels from the 3D point cloud. Check kernel for use. | |
| plot3D.py | |
| Module with functions to plot point clouds and voxelgrid inside jupyter notebook. You have to run this locally due to Kaggle's notebook lack of support to rendering Iframes. See github issue here | |
| Functions included: | |
| array_to_color Converts 1D array to rgb values use as kwarg color in plot_points() | |
| plot_points(xyz, colors=None, size=0.1, axis=False) | |
| plot_voxelgrid(v_grid, cmap=""Oranges"", axis=False) | |
| Acknowledgements | |
| Website of the original MNIST dataset | |
| Website of the 3D MNIST dataset | |
| Have fun!" | |
| US Household Income Statistics,"+32,000 records, with grandularity on a neighborhood scale (mean, median, Stdev)",Golden Oak Research Group,43,"Version 6,2017-08-21|Version 5,2017-08-14|Version 4,2017-08-14|Version 3,2017-08-14|Version 2,2017-08-10|Version 1,2017-08-09","geography | |
| income | |
| finance | |
| + 2 more...",Other,6 MB,Other,"6,803 views","1,210 downloads",3 kernels,2 topics,https://www.kaggle.com/goldenoakresearch/us-household-income-stats-geo-locations,"New Upload: | |
| Added +32,000 more locations. For information on data calculations please refer to the methodology pdf document. Information on how to calculate the data your self is also provided as well as how to buy data for $1.29 dollars. | |
| What you get: | |
| The database contains 32,000 records on US Household Income Statistics & Geo Locations. The field description of the database is documented in the attached pdf file. To access, all 348,893 records on a scale roughly equivalent to a neighborhood (census tract) see link below and make sure to up vote. Up vote right now, please. Enjoy! | |
| Household & Geographic Statistics: | |
| Mean Household Income (double) | |
| Median Household Income (double) | |
| Standard Deviation of Household Income (double) | |
| Number of Households (double) | |
| Square area of land at location (double) | |
| Square area of water at location (double) | |
| Geographic Location: | |
| Longitude (double) | |
| Latitude (double) | |
| State Name (character) | |
| State abbreviated (character) | |
| State_Code (character) | |
| County Name (character) | |
| City Name (character) | |
| Name of city, town, village or CPD (character) | |
| Primary, Defines if the location is a track and block group. | |
| Zip Code (character) | |
| Area Code (character) | |
| Abstract | |
| The dataset originally developed for real estate and business investment research. Income is a vital element when determining both quality and socioeconomic features of a given geographic location. The following data was derived from over +36,000 files and covers 348,893 location records. | |
| License | |
| Only proper citing is required please see the documentation for details. Have Fun!!! | |
| Golden Oak Research Group, LLC. “U.S. Income Database Kaggle”. Publication: 5, August 2017. Accessed, day, month year. | |
| Sources, don't have 2 dollars? Get the full information yourself! | |
| 2011-2015 ACS 5-Year Documentation was provided by the U.S. Census Reports. Retrieved August 2, 2017, from https://www2.census.gov/programs-surveys/acs/summary_file/2015/data/5_year_by_state/ | |
| Found Errors? | |
| Please tell us so we may provide you the most accurate data possible. You may reach us at: research_development@goldenoakresearch.com | |
| for any questions you can reach me on at 585-626-2965 | |
| please note: it is my personal number and email is preferred | |
| Check our data's accuracy: Census Fact Checker | |
| Access all 348,893 location records and more: | |
| Don't settle. Go big and win big. Optimize your potential. Overcome limitation and outperform expectation. Access all household income records on a scale roughly equivalent to a neighborhood, see link below: | |
| Website: Golden Oak Research Kaggle Deals all databases $1.29 Limited time only | |
| A small startup with big dreams, giving the every day, up and coming data scientist professional grade data at affordable prices It's what we do." | |
| Crime Statistics for South Africa,A history of crime statistics from 2004 to 2015 per province and station,Stephan Wessels,42,"Version 8,2016-11-02|Version 7,2016-11-02|Version 6,2016-10-27|Version 5,2016-10-27|Version 4,2016-10-27|Version 3,2016-10-26|Version 2,2016-10-26|Version 1,2016-10-19",crime,Other,23 MB,Other,"15,787 views","2,050 downloads",85 kernels,6 topics,https://www.kaggle.com/slwessels/crime-statistics-for-south-africa,"Context | |
| CRIME STATISTICS: INTEGRITY | |
| The South African Police Service (SAPS) has accepted a new and challeging objective of ensuring that its crime statistics are in line with international best practice. This will be achieved through a Memorandum of Understanding with Statistics South Africa (Stats SA), aimed at further enhancing the quality and integrity of the South African crime statistics. | |
| The crime statistics generated by SAPS are an important link in the value chain of the statistics system informs policy development and planning in the criminal justice system. The collaboration with StatsSA will go a long way in enhancing the integrity of the SAPS crime statistics and ensuring that policy-makers have quality data to assist them with making policy decisions. | |
| Content | |
| The dataset contains South African crime statistics, broken down per province, station and crime type. | |
| Acknowledgements | |
| Data as published from: | |
| http://www.saps.gov.za/resource_centre/publications/statistics/crimestats/2015/crime_stats.php | |
| Further sources: | |
| http://www.saps.gov.za/services/crimestats.php | |
| An overview presentation: | |
| http://www.saps.gov.za/services/final-crime-stats-release-02september2016.pdf" | |
| CS:GO Competitive Matchmaking Data,"Damage/Grenade entries on over 35,000 rounds played in competitive matchmaking",KP,42,"Version 5,2017-10-12|Version 4,2017-09-29|Version 3,2017-09-26|Version 2,2017-09-26|Version 1,2017-09-23","video games | |
| internet",Other,366 MB,CC4,"7,863 views",628 downloads,5 kernels,3 topics,https://www.kaggle.com/skihikingkevin/csgo-matchmaking-damage,"Introduction | |
| Video games are a rich area for data extraction due to their digital nature. Notable examples such as the complex EVE Online economy, World of Warcraft corrupted blood incident and even Grand Theft Auto self-driving cars tells us that fiction is closer to reality than we really think. Data scientists can gain insight on the logic and decision-making that the players face when put in hypothetical and virtual scenarios. | |
| In this Kaggle Dataset, I provide just over 1400 competitive matchmaking matches from Valve's game Counter-strike: Global Offensive (CS:GO). The data was extracted from competitive matchmaking replays submitted to csgo-stats. I intend for this data-set to be purely exploratory, however users are free to create their own predictive models they see fit. | |
| About Counter-Strike: Global Offensive | |
| Counter-Strike: Global Offensive is a first-person shooter game pitting two teams of 5 players against each other. Within a maximum of 30 rounds, the two teams find themselves on either side as a Counter Terrorist or Terrorist. Both sides are tasked with eliminating the opposition or, as the terrorist team, planting the C4 bomb at a bomb site and allowing it to explode. Rounds are played out until either of those two objectives or if the maximum time is reached (in which the counter terrorists then win by default). At the end of the 15th round, the two teams switch sides and continue until one team reaches 16 round wins first. CS:GO is widely known for its competitive aspect of technical skill, teamwork and in-game strategies. Players are constantly rewarded with the efforts they put it in training and learning through advancing in rank. | |
| Click here to read more about the competitive mechanics of CS:GO. | |
| Content | |
| This dataset within the 1400 matches provides every successful entry of duels (or battle) that took place for a player. That is, each row documents an event when a player is hurt by another player (or World e.g fall damage). There are over 900,000 entries within more than 31500 rounds. | |
| mm_master_demos.csv contains information on rounds fired, while mm_grenades_demos.csv contains information on grenades thrown. The fields in the two datasets are similar: highlights include shooters and victims, event coordinates, and timestamps. The datasets also includes static information on the match winner, player ranks before and after the match, and other miscellaneous match-level metadata. | |
| For further information on individual fields in the dataset, refer to the Column Metadata. | |
| Interpreting Positional Data | |
| This dataset also includes a selection of the game's official radar maps, as well as a table, map_data.csv, to aid in mapping data over them. The X,Y coordinates included in the dataset are all in in-game coordinates and need to be linearly scaled to be plotted on any official radar maps. See converting to map coordinates for more information. | |
| Acknowledgements | |
| Definitely the guys from csgo-stats, without them, this wouldn't have been possible! :) | |
| /r/globaloffensive for many years of lulz | |
| Akiver of CSGO Demo Manager for spending so much time perfecting his demo parser." | |
| US Census Demographic Data,Demographic and Economic Data for Tracts and Counties,MuonNeutrino,42,"Version 1,2017-12-14","united states | |
| demographics",CSV,6 MB,CC0,"7,173 views","1,517 downloads",3 kernels,,https://www.kaggle.com/muonneutrino/us-census-demographic-data,"Context | |
| This dataset expands on my earlier New York City Census Data dataset. It includes data from the entire country instead of just New York City. The expanded data will allow for much more interesting analyses and will also be much more useful at supporting other data sets. | |
| Content | |
| The data here are taken from the DP03 and DP05 tables of the 2015 American Community Survey 5-year estimates. The full datasets and much more can be found at the American Factfinder website. Currently, I include two data files: | |
| acs2015_census_tract_data.csv: Data for each census tract in the US, including DC and Puerto Rico. | |
| acs2015_county_data.csv: Data for each county or county equivalent in the US, including DC and Puerto Rico. | |
| The two files have the same structure, with just a small difference in the name of the id column. Counties are political subdivisions, and the boundaries of some have been set for centuries. Census tracts, however, are defined by the census bureau and will have a much more consistent size. A typical census tract has around 5000 or so residents. | |
| The Census Bureau updates the estimates approximately every year. At least some of the 2016 data is already available, so I will likely update this in the near future. | |
| Acknowledgements | |
| The data here were collected by the US Census Bureau. As a product of the US federal government, this is not subject to copyright within the US. | |
| Inspiration | |
| There are many questions that we could try to answer with the data here. Can we predict things such as the state (classification) or household income (regression)? What kinds of clusters can we find in the data? What other datasets can be improved by the addition of census data?" | |
| Aviation Accident Database & Synopses,The NTSB aviation accident dataset,Kheirallah Samaha,41,"Version 6,2018-01-15|Version 5,2017-01-10|Version 4,2017-01-09|Version 3,2017-01-05|Version 2,2016-12-12|Version 1,2016-12-11",aviation,CSV,4 MB,CC0,"18,163 views","2,487 downloads",87 kernels,,https://www.kaggle.com/khsamaha/aviation-accident-database-synopses,"Content | |
| The NTSB aviation accident database contains information from 1962 and later about civil aviation accidents and selected incidents within the United States, its territories and possessions, and in international waters. | |
| Acknowledgements | |
| Generally, a preliminary report is available online within a few days of an accident. Factual information is added when available, and when the investigation is completed, the preliminary report is replaced with a final description of the accident and its probable cause. Full narrative descriptions may not be available for dates before 1993, cases under revision, or where NTSB did not have primary investigative responsibility. | |
| Inspiration | |
| Hope it will teach us how to improve the quality and safety of traveling by Airplane." | |
| Forest Cover Type Dataset,Tree types found in the Roosevelt National Forest in Colorado,UCI Machine Learning,41,"Version 1,2016-11-03","botany | |
| ecology | |
| plants",CSV,72 MB,Other,"13,386 views","1,303 downloads",66 kernels,0 topics,https://www.kaggle.com/uciml/forest-cover-type-dataset,"Context | |
| This dataset contains tree observations from four areas of the Roosevelt National Forest in Colorado. All observations are cartographic variables (no remote sensing) from 30 meter x 30 meter sections of forest. There are over half a million measurements total! | |
| Content | |
| This dataset includes information on tree type, shadow coverage, distance to nearby landmarks (roads etcetera), soil type, and local topography. | |
| Acknowledgement | |
| This dataset is part of the UCI Machine Learning Repository, and the original source can be found here. The original database owners are Jock A. Blackard, Dr. Denis J. Dean, and Dr. Charles W. Anderson of the Remote Sensing and GIS Program at Colorado State University. | |
| Inspiration | |
| Can you build a model that predicts what types of trees grow in an area based on the surrounding characteristics? A past Kaggle competition project on this topic can be found here. | |
| What kinds of trees are most common in the Roosevelt National Forest? | |
| Which tree types can grow in more diverse environments? Are there certain tree types that are sensitive to an environmental factor, such as elevation or soil type?" | |
| Cryptocoins Historical Prices,A collection of prices for cryptocurrencies,ValerioVaccaro,40,"Version 8,2018-01-12|Version 7,2018-01-06|Version 6,2017-12-01|Version 5,2017-11-13|Version 4,2017-11-08|Version 3,2017-10-22|Version 2,2017-10-22|Version 1,2017-10-18",,CSV,20 MB,CC0,"6,216 views",856 downloads,10 kernels,,https://www.kaggle.com/valeriovaccaro/cryptocoinshistoricalprices,"Introduction | |
| This file contains the values of the price for more than 1000 different cryptocurrencies (including scams) recorded on daily base, I decide to include all coins in order to analyze exotic coins and compare with well knows cryptocurrencies. All this dataset come from coinmarketcap historical pages, grabbed using just an R script. Thanks coinmarketcap to making this data available for free (and for every kind of usage). | |
| The dataset | |
| Available columns in the dataset: | |
| Date - the day of recorded values | |
| Open - the opening price (in USD) | |
| High - the highest price (in USD) | |
| Low - the lowest price (in USD) | |
| Close - the closing price (in USD) | |
| Volume - total exchanged volume (in USD) | |
| Market.Cap - the total market capitalization for the coin (in USD) | |
| coin - the name of the coin | |
| Delta - calculated as (Close - Open) / Open" | |
| 2012 and 2016 Presidential Elections,"Election results with county information on race, income and education",Joel Wilson,40,"Version 2,2016-11-28|Version 1,2016-11-23",politics,CSV,3 MB,Other,"15,634 views","2,711 downloads",39 kernels,3 topics,https://www.kaggle.com/joelwilson/2012-2016-presidential-elections,"These data files contain election results for both the 2012 and 2016 US Presidential Elections, include proportions of votes cast for Romney, Obama (2012) and Trump, Clinton (2016). | |
| The election results were obtained from this Git repository: https://github.com/tonmcg/County_Level_Election_Results_12-16 | |
| The county facts data was obtained from another Kaggle election data set: https://www.kaggle.com/benhamner/2016-us-election" | |
| Emoji sentiment,Are people that use emoji happier?,Jose Berengueres,39,"Version 2,2017-10-01|Version 1,2017-10-01","linguistics | |
| internet",CSV,152 MB,GPL,"7,772 views",771 downloads,,0 topics,https://www.kaggle.com/harriken/emoji-sentiment,"Are people that use emoji happier? | |
| paper --> https://arxiv.org/abs/1710.00888 | |
| At ASONAM2017, PydataDubai vol 1.0 @ AWOK, PyDataBCN2017 @ EASDE we have presented the paper Happiness inside a job?... Many people in the various audiences asked why we avoid using emojis to predict and profile employees. The answer is that we prefer to use links of likes because they are more authentic than words or emojis. In the same way that google page rank is more effective when it looks at links between pages rather than content inside the pages. ... Still people keep asking about it. But there is one thing emoji are good at estimating: author sentiment and that is just possible thanks to the unique characteristics of the dataset at hand. | |
| Previous research has traditionally analyzed emoji sentiment from the point of view of the reader of the content not the author. Here, we analyze emoji sentiment from the author point of view and present a benchmark that was built from an employee happiness dataset where emoji happen to be annotated with daily happiness of the author of the comment. We also found out that people that use emoji are happier!?, muuch happier... But the question is, what did we miss? | |
| Content | |
| The main table contains columns named after emoji hex codes, a 1 means the emoji appears one time in the comment (row). This dataset is an expanded version of this one, but has different formats, columns and one different table, that is why we decided to release it as separate dataset. as he scripts are not compatible. | |
| Other stuff | |
| The R script written on MAC OS does not work in the kaggle platform (because numbers become factors and other little changes in how the code is interpreted...), the full working script (tested on R studio MAC OS) can be found at https://github.com/orioli/emoji-writer-sentiment | |
| Thank you to Lewis Michel" | |
| Human Resources Data Set,Dataset used for learning data visualization and basic regression,Dr. Rich,39,"Version 4,2018-02-15|Version 3,2017-07-24|Version 2,2017-07-21|Version 1,2017-07-20","employment | |
| business",CSV,176 KB,CC4,"12,756 views","2,047 downloads",5 kernels,2 topics,https://www.kaggle.com/rhuebner/human-resources-data-set,"Context | |
| HR data can be hard to come by, and HR professionals generally lag behind with respect to analytics and data visualization competency. Thus, Dr. Carla Patalano and I set out to create our own HR-related dataset, which is used in one of our graduate MSHRM courses called HR Metrics and Analytics, at New England College of Business. We created this data set ourselves. | |
| Content | |
| There are multiple worksheets within the Excel workbook. These include | |
| Core data set | |
| Production staff | |
| Sales analysis | |
| Salaries | |
| Recruiting sources | |
| The Excel workbook revolves around a fictitious company, called Dental Magic, and the core data set contains names, DOBs, age, gender, marital status, date of hire, reasons for termination, department, whether they are active or terminated, position title, pay rate, manager name, and performance score. | |
| Acknowledgements | |
| Dr. Carla Patalano provided many suggestions for creating this synthetic data set, which has been used now by over 30 Human Resource Management students at the college. Students in the course learn data visualization techniques with Tableau Desktop and use this data set to complete a series of assignments. | |
| Inspiration | |
| Is there any relationship between who a person works for and their performance score? | |
| What is the overall diversity profile of the organization? | |
| What are our best recruiting sources if we want to ensure a diverse organization? | |
| There are so many other interesting questions that could be addressed through this interesting data set. Dr. Patalano and I look forward to seeing what we can come up with." | |
| Text Similarity,Natural Language Processing on Stock data,Rishi Sankineni,39,"Version 1,2017-03-19","languages | |
| finance | |
| linguistics",CSV,202 KB,ODbL,"11,418 views",866 downloads,13 kernels,,https://www.kaggle.com/rishisankineni/text-similarity,"Context: | |
| Natural Language Processing(NLP), Text Similarity(lexical and semantic) | |
| Content: | |
| In each row of the included datasets(train.csv and test.csv), products X(description_x) and Y(description_y) are considered to refer to the same security(same_security) if they have the same ticker(ticker_x,ticker_y), even if the descriptions don't exactly match. You can make use of these descriptions to predict whether each pair in the test set also refers to the same security. | |
| Dataset info: | |
| Train - description_x, description_y, ticker_x, ticker_y, same_security. Test - description_x, description_y, same_security(to be predicted) | |
| Past Research: | |
| This dataset is pretty similar to the Quora Question Pairs . You can also check out my kernel for dataset exploration and n-gram analysis N-gram analysis on stock data. | |
| How to Approach: | |
| There are several good ways to approach this, check out this algorithm, and see how far you can go with it: https://en.wikipedia.org/wiki/Tf–idf http://scikit learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html. You can also try doing n-gram analysis(check out my kernel). I would suggest using log-loss as your evaluation metric since it gives you a number between 0 and 1 instead of binary classification, which is not so effective in this case. | |
| Acknowledgements: | |
| Quovo stock data." | |
| 2016 U.S. Presidential Election Memes,"Analyze 45,000+ U.S. Presidential Election Memes",SIZZLE,39,"Version 4,2016-11-02|Version 3,2016-10-28|Version 2,2016-10-27|Version 1,2016-10-26","popular culture | |
| politics | |
| internet",CSV,26 MB,CC4,"21,453 views","1,353 downloads",53 kernels,3 topics,https://www.kaggle.com/SIZZLE/2016electionmemes,"We’re releasing 30,000+ OCR’d political memes and their captions . With the election just days away, we hope to contribute to the pre and post-election analysis of the most meme-orable election in modern history. | |
| v2 of the dataset includes 8 csv’s: | |
| Bern.csv: 146 rows | |
| Bernie.csv: 2100 rows | |
| Clinton.csv: 4362 rows | |
| Donald.csv: 6499 rows | |
| Gary_Johnston.csv: 140 rows | |
| Hillary.csv: 7398 rows | |
| Jill Stein.csv: 96 rows | |
| Trump.csv: 12139 rows | |
| with the following columns: | |
| timestamp (date published) | |
| id (our unique identifier) | |
| link (post url) | |
| caption (meme caption via ocr) | |
| author | |
| network | |
| likes/upvotes | |
| Let us know any revisions you'd like to see. And of course, if you find errors in the data, do let us know." | |
| Cycle Share Dataset,Bicycle Trip Data from Seattle's Cycle Share System,Pronto Cycle Share,39,"Version 1,2016-11-07","cycling | |
| road transport",CSV,46 MB,Other,"14,437 views","2,009 downloads",41 kernels,3 topics,https://www.kaggle.com/pronto/cycle-share-dataset,"Context | |
| The Pronto Cycle Share system consists of 500 bikes and 54 stations located in Seattle. Pronto provides open data on individual trips, stations, and daily weather. | |
| Content | |
| There are 3 datasets that provide data on the stations, trips, and weather from 2014-2016. | |
| Station dataset | |
| station_id: station ID number | |
| name: name of station | |
| lat: station latitude | |
| long: station longitude | |
| install_date: date that station was placed in service | |
| install_dockcount: number of docks at each station on the installation date | |
| modification_date: date that station was modified, resulting in a change in location or dock count | |
| current_dockcount: number of docks at each station on 8/31/2016 | |
| decommission_date: date that station was placed out of service | |
| Trip dataset | |
| trip_id: numeric ID of bike trip taken | |
| starttime: day and time trip started, in PST | |
| stoptime: day and time trip ended, in PST | |
| bikeid: ID attached to each bike | |
| tripduration: time of trip in seconds | |
| from_station_name: name of station where trip originated | |
| to_station_name: name of station where trip terminated | |
| from_station_id: ID of station where trip originated | |
| to_station_id: ID of station where trip terminated | |
| usertype: ""Short-Term Pass Holder"" is a rider who purchased a 24-Hour or 3-Day Pass; ""Member"" is a rider who purchased a Monthly or an Annual Membership | |
| gender: gender of rider | |
| birthyear: birth year of rider | |
| Weather dataset contains daily weather information in the service area | |
| Acknowledgements | |
| The original datasets can be downloaded here. | |
| Inspiration | |
| Some ideas worth exploring: | |
| What is the most popular bike route? | |
| How are bike uses or routes affected by user characteristics, station features, and weather?" | |
| Daily Sea Ice Extent Data,Total sea ice extent from 1978 to present,National Snow and Ice Data Center,39,"Version 2,2017-06-09|Version 1,2016-12-07","environment | |
| climate",CSV,4 MB,Other,"10,577 views","1,224 downloads",27 kernels,3 topics,https://www.kaggle.com/nsidcorg/daily-sea-ice-extent-data,"Context | |
| The National Snow and Ice Data Center (NSIDC) supports research into our world’s frozen realms: the snow, ice, glaciers, frozen ground, and climate interactions that make up Earth’s cryosphere. NSIDC manages and distributes scientific data, creates tools for data access, supports data users, performs scientific research, and educates the public about the cryosphere. | |
| Content | |
| The dataset provides the total extent for each day for the entire time period (1978-2015). There are 7 variables: | |
| Year | |
| Month | |
| Day | |
| Extent: unit is 10^6 sq km | |
| Missing: unit is 10^6 sq km | |
| Source: Source data product web site: http://nsidc.org/data/nsidc-0051.html | |
| hemisphere | |
| Acknowledgements | |
| The original datasets can be found here and here. | |
| Inspiration | |
| Can you visualize the change in sea ice over time? | |
| Do changes in sea ice differ between the two hemispheres?" | |
| World Bank Youth Unemployment Rates,Youth Unemployment rates by country from 2010-2014,SovBoc2018,38,"Version 1,2016-11-05","employment | |
| finance",CSV,17 KB,Other,"16,609 views","2,443 downloads",129 kernels,0 topics,https://www.kaggle.com/sovannt/world-bank-youth-unemployment,"Context A data driven look into answering the common question while travelling overseas: ""how easy is it to get a job in your country?"" | |
| Content This dataset contains youth unemployment rates (% of total labor force ages 15-24) (modeled ILO estimate) Latest data available from 2010 to 2014. | |
| Acknowledgements International Labour Organization. | |
| http://data.worldbank.org/indicator/SL.UEM.TOTL.ZS | |
| Released under Open license." | |
| Crime in India,State-wise data from 2001 is classified according to 40+factors. (75+ csv files),Rajanand Ilangovan / இராஜ்ஆனந்த் இளங்கோவன்,38,"Version 4,2017-09-01|Version 3,2017-08-24|Version 2,2017-08-07|Version 1,2017-08-07","india | |
| crime",CSV,12 MB,CC4,"9,932 views","1,669 downloads",5 kernels,,https://www.kaggle.com/rajanand/crime-in-india,"Connect/Follow me on LinkedIn for more updates on interesting dataset like this. Thanks. | |
| Context | |
| This dataset contains complete information about various aspects of crimes happened in India from 2001. There are many factors that can be analysed from this dataset. Over all, I hope this dataset helps us to understand better about India. | |
| Content | |
| I : Cases Reported and their Disposal by Police and Court | |
| Indian Penal Code | |
| Special & Local Laws | |
| IA : SC/ST Cases Reported and their Disposal by Police and Court | |
| Crime against SCs | |
| Crime against STs | |
| IB : Children Cases Reported and their Disposal by Police and Court | |
| Abetment of Suicide (Section 305 IPC) | |
| Buying of Girls for Prostitution (Section 373 IPC) | |
| Child Marriage Restraint Act, 1929 | |
| Exposure and Abandonment (Section 317 IPC) | |
| Foeticide (Section 315 and 316 IPC) | |
| Infanticide (Section 315 IPC) | |
| Kidnapping & Abduction (Section 360,361,363,363-A, 363 read with Section 384, 366, 367 & 369 IPC) | |
| Murder (Section 302, 315 IPC) | |
| Other Crimes against Children | |
| Other Murder of Children (Section 302 IPC) | |
| Procuration of Minor Girls (Section 366-A IPC) | |
| Rape (Section 376 IPC) | |
| Selling of Girls for Prostitution (Section 372 IPC) | |
| Total Crimes against Children | |
| II : Persons Arrested and their Disposal by Police and Court | |
| Indian Penal Code | |
| Special and Local Laws | |
| IIA : SC/ST Persons Arrested and their Disposal by Police and Court | |
| Crime against SCs | |
| Crime against STs | |
| IIB : Children Persons Arrested and their Disposal by Police and Court | |
| Abetment of suicide (Section 305 IPC) | |
| Buying of girls for prostitution (Section 373 IPC) | |
| Child Marriage Restraint Act, 1929 | |
| Exposure and Abandonment (Section 317 IPC) | |
| Foeticide (Section 315 and 316 IPC) | |
| Kidnapping & Abduction (Section 360,361,363,363-A, 366, 367 & 369 IPC) | |
| Murder - Infanticide (Section 315 IPC) | |
| Murder - Other Murder of Children | |
| Murder (Section 302, 315 IPC) | |
| Other Crimes against Children | |
| Procuration of minor girls (Section 366-A IPC) | |
| Rape (Section 376 IPC) | |
| Selling of girls for prostitution (Section 372 IPC) | |
| Total Crimes against Children | |
| IV : Persons Arrested by Sex and Age Group | |
| Indian Penal Code | |
| Special & Local Laws | |
| V : Juveniles Apprehended | |
| Indian Penal Code | |
| Special & Local Laws | |
| VI : Juveniles Arrested and their Disposal | |
| VII : Property Stolen & Recovered (Crime Head) | |
| Dacoity | |
| Robbery | |
| Burglary | |
| Theft | |
| Criminal Breach of Trust | |
| Other Property | |
| Total Property Stolen & Recovered | |
| VIII : Property Stolen & Recovered (Nature of Property) | |
| Communation and Electricity Wire | |
| Cattle | |
| Cycle | |
| Motor Vehicles | |
| Motor Vehicles - Motor Cycle/Scooters | |
| Motor Vehicles - Motor Car/Taxi/Jeep | |
| Motor Vehicles - Other Motor Vehicles | |
| Fire Arms | |
| Explosives/Explosive Substances | |
| Electronic Components | |
| Cultural Property including Antiques | |
| Other kinds of Property | |
| Total Property Stolen & Recovered | |
| IX : Police Strength (Actual & Sanctioned) | |
| A) Actual Civil Police (Incl. District Armed Police and Women Police) | |
| A) Acual Armed Police (Incl. Women Police) | |
| A) Actual Police Strength (Incl. Women) | |
| B) Acual Women Civil Police (Incl. District Armed Force) | |
| B) Actual Women Armed Police | |
| B) Actual Women Police Strength | |
| C) Sanctioned Civil Police (Incl. District Armed Police) | |
| C) Santioned Armed Police (Incl. Women Police) | |
| C) Santioned Police Strength (Incl. Women) | |
| D) Sanctioned Women Civil Police (Incl. District Armed Police) | |
| D) Sanctioned Women Armed Police | |
| D) Sanctioned Women Police Strength | |
| X : Police Personnel Killed or Injured on duty | |
| Constables | |
| Head Constables | |
| Assistant Sub-Inspectos | |
| Sub-Inspectors | |
| Inspectors | |
| Gazetted Officers | |
| Total Police Killed or Injured | |
| X-B : Age Profile of Police Personnel Killed on Duty | |
| X-C : Natural Deaths and Suicides of Police Personnel | |
| Natural Deaths of Police Personnel (while in service) | |
| Police Personnel Committed Suicide | |
| XI : Casualties under Police Firing and LathiCharge | |
| Riot Control | |
| Anti Dacoity Operations | |
| Against Extremists & Terrorists | |
| Against Others | |
| Total Casualties | |
| XII : Cases Reported Value of Property Stolen under Dacoity, Robbery, Burglary and Theft by Place of Occurance | |
| Residential Premises | |
| Highways | |
| River and Sea | |
| Railways 4.1 In Running Trains 4.2 Others | |
| Banks | |
| Commercial Establishments (Shops etc.) | |
| Other Places | |
| Total | |
| XIII : Particulars of Juveniles Arrested | |
| Education | |
| Economic Setup | |
| Family Background | |
| Recidivism | |
| XIV : Motive/Cause of Murder and Culpable Homicide not Amounting to Murder | |
| XV : Victims of Rape(Age Group-wise) | |
| Incest Rape Cases | |
| Other Rape Cases (Otherthan Incest) | |
| Total Rape Cases | |
| XV-A : Rape Offenders relation, nearness to Rape Victims | |
| XVI : Persons Arrested under Recidivism | |
| XVII : Anti Corruption - Cases | |
| XVIII : Anti Corruption - Arrests | |
| XIX : Complaints/Cases Against Police Personnel | |
| Complaints Received/Cases Registered | |
| Police Personnel Involved/Action Taken | |
| Departmental Action/Punishments | |
| *XX : Police Budget and Infrastructure | |
| Equipments and Transport Support | |
| Distribution of Police Stations by Crime Incidences | |
| Distribution of Police Stations by Police Strength | |
| Organisational Set Up | |
| SCs/STs and Muslims in Police Force (Actual) | |
| XXI : 1. Nature of Complaints Received by Police | |
| XXI : 2. Trial of Violent Crimes by Courts | |
| Murder | |
| Attempt to Murder | |
| C H Not Amounting to Murder | |
| Rape | |
| Kidnapping & Abduction 5.1 Kidnapping & Abduction of Women & Girls 5.2 Kidnapping & Abduction of Others | |
| Dacoity | |
| Preparation & Assembly for Dacoity | |
| Robbery | |
| Riots | |
| Arson | |
| Dowry Deaths | |
| Total Trials (Sum of 1-11 Above) | |
| XXI : 3. Period of Trials by Courts | |
| District/Session Judge | |
| Additional Session Judge | |
| Chief Judicial Magistrate | |
| Judicial Magistrate (I) | |
| Judicial Magistrate (II) | |
| Special Judicial Magistrate | |
| Other courts | |
| Total Trials (Sum of 1-7 Above) | |
| XXI : 4.1 Autho Theft (Stolen & Recovered) | |
| Motor Cycles/ Scooters | |
| Motor Car/Taxi/Jeep | |
| Buses | |
| Goods carrying vehicles (Trucks/Tempo etc) | |
| Other Motor vehicles | |
| Total (Sum of 1-5 Above) | |
| XXI : 4.2 Serious Fraud | |
| Criminal Breach of Trust | |
| Cheating | |
| XXI : 5.1 Victims of Murder (Age & Sex-Wise) | |
| Male Victims | |
| Female Victims | |
| Total | |
| XXI : 5.2 Victims of CH not Amounting to Murder (Age & Sex-wise) | |
| Male Victims | |
| Female Victims | |
| Total | |
| XXI : 5.3 Use of FireArms in Murder Cases | |
| XXI : 6. Human Rights Violation by Police | |
| Disappearance of Persons | |
| Illegal Detention/Arrests | |
| Fake Encounter Killings | |
| Violation Against Terrorists/Extremists | |
| Extortion | |
| Torture | |
| False Implication | |
| Failure in Taking Action | |
| Indignity to Women | |
| Atrocities on SC/ST | |
| Others | |
| Total (Sum of 1-11 Above) | |
| XXI : 7. Police Housing | |
| For Officers (Dy.SP & Above) | |
| Upper SubOrdinates (ASI to Inspectos) | |
| Lower SubOrdinates (Constables, Head Constables & Class-IV Subordinate Staff) | |
| XXI : 8. Home Guards and Auxilliary force | |
| XXI : 9. Unidentified Deadbodies Recovered & Inquest conducted | |
| XXI : 10. Victims of Kidnapping & Abduction for Specific Purpose | |
| For Adoption | |
| For Begging | |
| for Camel Racing | |
| For Illicit Intercourse | |
| For Marriage | |
| For Prostitution | |
| For Ransom | |
| For Revenge | |
| For Sale | |
| For Selling Bodyparts | |
| For Slavery | |
| For Unlawful Activity | |
| Other Purposes | |
| Total (Sum of 1-13 Above) | |
| XXI : 11. Custodial Deaths | |
| Deaths in Custody/Lockup of Persons Remanded to Police Custody by Court | |
| Deaths in Custody/Lockup of Persons Not Remanded to Police Custody by Court | |
| Deaths in Custody during production/process in courts/journey connected with investigation | |
| Deaths during Hospitalisation/Treatment | |
| Deaths due to Other Reasons | |
| XXI : 12. Escapes from Police Custody | |
| Cases under Crime Against Women | |
| Rape | |
| Kidnapping & Abduction of Women & Girls | |
| Dowry Deaths | |
| Molestation | |
| Sexual Harassment | |
| Cruelty by Husband and Relatives | |
| Importation of Girls | |
| Immoral Traffic Prevention Act, 1956 | |
| Dowry Prohibition Act, 1961 | |
| Indecent Representation of Women(Prohibition) Act, 1986 | |
| Sati Prevention Act, 1987 | |
| Total Crimes Against Women | |
| Arrests under Crime Against Women | |
| Rape | |
| Kidnapping & Abduction of Women & Girls | |
| Dowry Deaths | |
| Molestation | |
| Sexual Harassment | |
| Cruelty by Husband and Relatives | |
| Importation of Girls | |
| Immoral Traffic Prevention Act, 1956 | |
| Dowry Prohibition, 1961 | |
| Indecent Representation of Women(Prohibition) Act, 1986 | |
| Sati Prevention Act, 1987 | |
| Total Crimes Against Women | |
| Some of the data contains district level data. The districts are police districts and also include special police unit. Therefore these may be different from revenue districts. Most of the data is from 2001 to 2010. But there are few files which has data only from 2011 and few are having 2001-14. | |
| Inspiration | |
| There could be many things one can understand by analyzing this dataset. Few inspirations for you to start with. | |
| What is the major reason people being kidnapped in each and every state? | |
| Offenders relation to the rape victim | |
| Juveniles family background, education and economic setup. | |
| Which state has more crime against children and women? | |
| Age group wise murder victim | |
| Crime by place of occurrence. | |
| Anti corruption cases vs arrests. | |
| Which state has more number of complaints against police? | |
| Which state is the safest for foreigners? | |
| Acknowledgements | |
| National Crime Records Bureau (NCRB), Govt of India has published this dataset on their website and also has shared on Open Govt Data Platform India portal under Govt. Open Data License - India." | |
| 80 Cereals,Nutrition data on 80 cereal products,Chris Crawford,38,"Version 2,2017-10-25|Version 1,2017-08-19",food and drink,CSV,5 KB,CC3,"11,553 views","1,995 downloads",286 kernels,,https://www.kaggle.com/crawford/80-cereals,"Context | |
| If you like to eat cereal, do yourself a favor and avoid this dataset at all costs. After seeing these data it will never be the same for me to eat Fruity Pebbles again. | |
| Content | |
| Fields in the dataset: | |
| Name: Name of cereal | |
| mfr: Manufacturer of cereal | |
| A = American Home Food Products; | |
| G = General Mills | |
| K = Kelloggs | |
| N = Nabisco | |
| P = Post | |
| Q = Quaker Oats | |
| R = Ralston Purina | |
| type: | |
| cold | |
| hot | |
| calories: calories per serving | |
| protein: grams of protein | |
| fat: grams of fat | |
| sodium: milligrams of sodium | |
| fiber: grams of dietary fiber | |
| carbo: grams of complex carbohydrates | |
| sugars: grams of sugars | |
| potass: milligrams of potassium | |
| vitamins: vitamins and minerals - 0, 25, or 100, indicating the typical percentage of FDA recommended | |
| shelf: display shelf (1, 2, or 3, counting from the floor) | |
| weight: weight in ounces of one serving | |
| cups: number of cups in one serving | |
| rating: a rating of the cereals (Possibly from Consumer Reports?) | |
| Acknowledgements | |
| These datasets have been gathered and cleaned up by Petra Isenberg, Pierre Dragicevic and Yvonne Jansen. The original source can be found here | |
| This dataset has been converted to CSV | |
| Inspiration | |
| Eat too much sugary cereal? Ruin your appetite with this dataset!" | |
| 1000 Netflix Shows,Understand the rating distributions of Netflix shows,Chase Willden,38,"Version 1,2017-06-11","film | |
| internet",CSV,87 KB,CC0,"11,602 views","1,660 downloads",13 kernels,2 topics,https://www.kaggle.com/chasewillden/netflix-shows,"Context | |
| Netflix in the past 5-10 years has captured a large populate of viewers. With more viewers, there most likely an increase of show variety. However, do people understand the distribution of ratings on Netflix shows? | |
| Content | |
| Because of the vast amount of time it would take to gather 1,000 shows one by one, the gathering method took advantage of the Netflix’s suggestion engine. The suggestion engine recommends shows similar to the selected show. As part of this data set, I took 4 videos from 4 ratings (totaling 16 unique shows), then pulled 53 suggested shows per video. The ratings include: G, PG, TV-14, TV-MA. I chose not to pull from every rating (e.g. TV-G, TV-Y, etc.). | |
| Acknowledgements | |
| The data set and the research article can be found at The Concept Center | |
| Inspiration | |
| I was watching Netflix with my wife and we asked ourselves, why are there so many R and TV-MA rating shows?" | |
| Who eats the food we grow?,"Worldwide food\feed production and distribution, 1961-2013",Dor Oppenheim,38,"Version 7,2017-11-30|Version 6,2017-11-21|Version 5,2017-11-21|Version 4,2017-11-21|Version 3,2017-11-17|Version 2,2017-11-17|Version 1,2017-11-17","animals | |
| agriculture",CSV,874 KB,CC0,"4,114 views",630 downloads,2 kernels,3 topics,https://www.kaggle.com/dorbicycle/world-foodfeed-production,"Context | |
| Our world population is expected to grow from 7.3 billion today to 9.7 billion in the year 2050. Finding solutions for feeding the growing world population has become a hot topic for food and agriculture organizations, entrepreneurs and philanthropists. These solutions range from changing the way we grow our food to changing the way we eat. To make things harder, the world's climate is changing and it is both affecting and affected by the way we grow our food – agriculture. This dataset provides an insight on our worldwide food production - focusing on a comparison between food produced for human consumption and feed produced for animals. | |
| Content | |
| The Food and Agriculture Organization of the United Nations provides free access to food and agriculture data for over 245 countries and territories, from the year 1961 to the most recent update (depends on the dataset). One dataset from the FAO's database is the Food Balance Sheets. It presents a comprehensive picture of the pattern of a country's food supply during a specified reference period, the last time an update was loaded to the FAO database was in 2013. The food balance sheet shows for each food item the sources of supply and its utilization. This chunk of the dataset is focused on two utilizations of each food item available: | |
| Food - refers to the total amount of the food item available as human food during the reference period. | |
| Feed - refers to the quantity of the food item available for feeding to the livestock and poultry during the reference period. | |
| Dataset's attributes: | |
| Area code - Country name abbreviation | |
| Area - County name | |
| Item - Food item | |
| Element - Food or Feed | |
| Latitude - geographic coordinate that specifies the north–south position of a point on the Earth's surface | |
| Longitude - geographic coordinate that specifies the east-west position of a point on the Earth's surface | |
| Production per year - Amount of food item produced in 1000 tonnes | |
| Acknowledgements | |
| This dataset was meticulously gathered, organized and published by the Food and Agriculture Organization of the United Nations. | |
| Inspiration | |
| Animal agriculture and factory farming is a a growing interest of the public and of world leaders. | |
| Can you find interesting outliers in the data? | |
| What are the fastest growing countries in terms of food production\consumption? | |
| Compare between food and feed consumption." | |
| Did it rain in Seattle? (1948-2017),"More than 25,000 consecutive days of Seattle weather data",Rachael Tatman,38,"Version 1,2017-12-21","united states | |
| north america | |
| climate | |
| weather",CSV,744 KB,CC0,"4,636 views",921 downloads,16 kernels,0 topics,https://www.kaggle.com/rtatman/did-it-rain-in-seattle-19482017,"Context: | |
| Besides coffee, grunge and technology companies, one of the things that Seattle is most famous for is how often it rains. This dataset contains complete records of daily rainfall patterns from January 1st, 1948 to December 12, 2017. | |
| Content | |
| This data was collected at the Seattle-Tacoma International Airport. The dataset contains five columns: | |
| DATE = the date of the observation | |
| PRCP = the amount of precipitation, in inches | |
| TMAX = the maximum temperature for that day, in degrees Fahrenheit | |
| TMIN = the minimum temperature for that day, in degrees Fahrenheit | |
| RAIN = TRUE if rain was observed on that day, FALSE if it was not | |
| Acknowledgements: | |
| This dataset was compiled by NOAA and is in the public domain. | |
| Inspiration: | |
| Can you use this dataset to build a model of whether it will rain on a specific day given information on the previous days? | |
| Is there a correlation between the minimum and maximum temperature? Can you predict one given the other? | |
| Can you model changes in the amount of precipitation over time? Is there seasonality?" | |
| Boston Airbnb Open Data,"A sneak peek into the Airbnb activity in Boston, MA, USA",Airbnb,38,"Version 1,2016-11-17","united states | |
| home | |
| hotels",CSV,72 MB,CC0,"26,681 views","3,426 downloads",70 kernels,2 topics,https://www.kaggle.com/airbnb/boston,"Context | |
| Since 2008, guests and hosts have used Airbnb to travel in a more unique, personalized way. As part of the Airbnb Inside initiative, this dataset describes the listing activity of homestays in Boston, MA. | |
| Content | |
| The following Airbnb activity is included in this Boston dataset: * Listings, including full descriptions and average review score * Reviews, including unique id for each reviewer and detailed comments * Calendar, including listing id and the price and availability for that day | |
| Inspiration | |
| Can you describe the vibe of each Boston neighborhood using listing descriptions? | |
| What are the busiest times of the year to visit Boston? By how much do prices spike? | |
| Is there a general upward trend of both new Airbnb listings and total Airbnb visitors to Boston? | |
| For more ideas, visualizations of all Boston datasets can be found here. | |
| Acknowledgement | |
| This dataset is part of Airbnb Inside, and the original source can be found here." | |
| Random Sample of NIH Chest X-ray Dataset,"5,606 images and labels sampled from the NIH Chest X-ray Dataset",National Institutes of Health Chest X-Ray Dataset,38,"Version 4,2017-11-23|Version 3,2017-11-17|Version 2,2017-11-16|Version 1,2017-11-16","healthcare | |
| health | |
| biotechnology | |
| + 2 more...",Other,2 GB,CC0,"5,282 views",707 downloads,4 kernels,,https://www.kaggle.com/nih-chest-xrays/sample,"NIH Chest X-ray Dataset Sample | |
| National Institutes of Health Chest X-Ray Dataset | |
| Chest X-ray exams are one of the most frequent and cost-effective medical imaging examinations available. However, clinical diagnosis of a chest X-ray can be challenging and sometimes more difficult than diagnosis via chest CT imaging. The lack of large publicly available datasets with annotations means it is still very difficult, if not impossible, to achieve clinically relevant computer-aided detection and diagnosis (CAD) in real world medical sites with chest X-rays. One major hurdle in creating large X-ray image datasets is the lack resources for labeling so many images. Prior to the release of this dataset, Openi was the largest publicly available source of chest X-ray images with 4,143 images available. | |
| This NIH Chest X-ray Dataset is comprised of 112,120 X-ray images with disease labels from 30,805 unique patients. To create these labels, the authors used Natural Language Processing to text-mine disease classifications from the associated radiological reports. The labels are expected to be >90% accurate and suitable for weakly-supervised learning. The original radiology reports are not publicly available but you can find more details on the labeling process in this Open Access paper: ""ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases."" (Wang et al.) | |
| Link to paper | |
| File contents - This is a random sample (5%) of the full dataset: | |
| sample.zip: Contains 5,606 images with size 1024 x 1024 | |
| sample_labels.csv: Class labels and patient data for the entire dataset | |
| Image Index: File name | |
| Finding Labels: Disease type (Class label) | |
| Follow-up # | |
| Patient ID | |
| Patient Age | |
| Patient Gender | |
| View Position: X-ray orientation | |
| OriginalImageWidth | |
| OriginalImageHeight | |
| OriginalImagePixelSpacing_x | |
| OriginalImagePixelSpacing_y | |
| Class descriptions | |
| There are 15 classes (14 diseases, and one for ""No findings"") in the full dataset, but since this is drastically reduced version of the full dataset, some of the classes are sparse with the labeled as ""No findings"" | |
| Hernia - 13 images | |
| Pneumonia - 62 images | |
| Fibrosis - 84 images | |
| Edema - 118 images | |
| Emphysema - 127 images | |
| Cardiomegaly - 141 images | |
| Pleural_Thickening - 176 images | |
| Consolidation - 226 images | |
| Pneumothorax - 271 images | |
| Mass - 284 images | |
| Nodule - 313 images | |
| Atelectasis - 508 images | |
| Effusion - 644 images | |
| Infiltration - 967 images | |
| No Finding - 3044 images | |
| Full Dataset Content | |
| The full dataset can be found here. There are 12 zip files in total and range from ~2 gb to 4 gb in size. | |
| Data limitations: | |
| The image labels are NLP extracted so there could be some erroneous labels but the NLP labeling accuracy is estimated to be >90%. | |
| Very limited numbers of disease region bounding boxes (See BBox_list_2017.csv) | |
| Chest x-ray radiology reports are not anticipated to be publicly shared. Parties who use this public dataset are encouraged to share their “updated” image labels and/or new bounding boxes in their own studied later, maybe through manual annotation | |
| Modifications to original data | |
| Original TAR archives were converted to ZIP archives to be compatible with the Kaggle platform | |
| CSV headers slightly modified to be more explicit in comma separation and also to allow fields to be self-explanatory | |
| Citations | |
| Wang X, Peng Y, Lu L, Lu Z, Bagheri M, Summers RM. ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases. IEEE CVPR 2017, ChestX-ray8_Hospital-Scale_Chest_CVPR_2017_paper.pdf | |
| NIH News release: NIH Clinical Center provides one of the largest publicly available chest x-ray datasets to scientific community | |
| Original source files and documents: https://nihcc.app.box.com/v/ChestXray-NIHCC/folder/36938765345 | |
| Acknowledgements | |
| This work was supported by the Intramural Research Program of the NClinical Center (clinicalcenter.nih.gov) and National Library of Medicine (www.nlm.nih.gov)." | |
| History of Hearthstone,"346,242 decks representing more than 3 years of gameplay!",romainvincent,37,"Version 3,2017-07-06|Version 2,2017-07-03|Version 1,2017-06-27",video games,CSV,78 MB,Other,"7,991 views",481 downloads,5 kernels,,https://www.kaggle.com/romainvincent/history-of-hearthstone,"Context | |
| Hearthstone is a very popular collectible card game published by Blizzard Entertainment in 2014. The goal of the game consists in building a 30 cards deck in order to beat your opponent. A few weeks ago, I decided to download all the decks posted by players at Hearthpwn. The code to download the data is available here. | |
| Content | |
| This upload is composed of two files : | |
| data.json / data.csv | |
| Contains the actual Hearthstone deck records. Each record features : | |
| date (str) : the date of publication (or last update) of the deck. | |
| user (str) : the user who uploaded the deck. | |
| deck_class (str) : one of the nine character class in Hearthstone (Druid, Priest, ...). | |
| deck_archetype (str) : the theme of deck labelled by the user (Aggro Druid, Dragon Priest, ...). | |
| deck_format (str) : the game format of the deck on the day data was recorded (W for ""Wild"" or S for ""Standard""). | |
| deck_set (str) : the latest expansion published prior the deck publication (Naxxramas, TGT Launch, ...). | |
| deck_id (int) : the ID of the deck. | |
| deck_type (str) : the type of the deck labelled by the user : | |
| Ranked Deck : a deck played on ladder. | |
| Theorycraft : a deck built with unreleased cards to get a gist of the future metagame. | |
| PvE Adventure : a deck built to beat the bosses in adventure mode. | |
| Arena : a deck built in arena mode. | |
| Tavern Brawl : a deck built for the weekly tavern brawl mode. | |
| Tournament : a deck brought at tournament by a pro-player. | |
| None : the game type was not mentioned. | |
| rating (int) : the number of upvotes received by that deck. | |
| title (str) : the name of the deck. | |
| craft_cost (int) : the amount of dust (in-game craft material) required to craft the deck. | |
| cards (list) : a list of 30 card ids. Each ID can be mapped to the card description using the reference file. | |
| refs.json | |
| Contains the reference to the cards played in Hearthstone. This file was originally proposed on HearthstoneJSON. Each record features a lot of informations about the cards, I'll list the most important : | |
| dbfId (int) : the id of the card (the one used in data.json). | |
| rarity (str) : the rarity of the card (EPIC, RARE, ...). | |
| cardClass (str) : the character class (WARLOCK, PRIEST, ...). | |
| artist (str) : the artist behind the card's art. | |
| collectible (bool) : whether or not the card can be collected. | |
| cost (int) : the card play cost. | |
| health (int) : the card health (if it's a minion). | |
| attack (int) : the card attack (if it's a minion). | |
| name (str) : the card name. | |
| flavor (str) : the card's flavor text. | |
| set (str) : the set / expansion which featured this card. | |
| text (int) : the card's text. | |
| type (str) : the card's type (MINION, SPELL, ...). | |
| race (str) : the card's race (if it's a minion). | |
| set (str) : the set / expansion which featured this card. | |
| ... | |
| If you need help cleaning the data take a look at my start over kernel! | |
| What you could do : | |
| Try to predict the deck archetype based on the cards features in the deck. | |
| Seek relationships between the cost of the deck and it's popularity. | |
| Describe the evolution of the meta-game over-time. | |
| Find out unbalanced (overplayed) cards" | |
| Pakistan Suicide Bombing Attacks,Most Authentic Count of Suicide Bombing Attacks in Pakistan (1995-2016),Zeeshan-ul-hassan Usmani,37,"Version 6,2017-12-01|Version 5,2017-10-10|Version 4,2017-07-19|Version 3,2017-03-23|Version 2,2017-01-29|Version 1,2017-01-26",crime,CSV,226 KB,CC0,"11,927 views",943 downloads,27 kernels,,https://www.kaggle.com/zusmani/pakistansuicideattacks,"Context | |
| Pakistan Suicide Bombing Attacks (1995-2016) | |
| Suicide bombing is an operational method in which the very act of the attack is dependent upon the death of the perpetrator. Though only 3% of all terrorist attacks around the world can be classified as suicide bombing attacks these account for 48% of the casualties. Explosions and suicide bombings have become the modus operandi of terrorist organizations throughout the world. The world is full of unwanted explosives, brutal bombings, accidents, and violent conflicts, and there is a need to understand the impact of these explosions on one’s surroundings, the environment, and most importantly on human bodies. From 1980 to 2001 (excluding 9/11/01) the average number of deaths per incident for suicide bombing attacks was 13. This number is far above the average of less than one death per incident across all types of terrorist attacks over the same time period. Suicide bombers, unlike any other device or means of destruction, can think and therefore detonate the charge at an optimal location with perfect timing to cause maximum carnage and destruction. Suicide bombers are adaptive and can quickly change targets if forced by security risk or the availability of better targets. Suicide attacks are relatively inexpensive to fund and technologically primitive, as IEDs can be readily constructed. | |
| World has seen more than 3,600 suicide bombing attacks in over 40 countries since 1982. Suicide Bombing has wreaked havoc in Pakistan in the last decade or so. From only a couple of attacks before 2000, it kept escalating after the US Operation Enduring Freedom in Afghanistan, promiscuously killing hundreds of people each year, towering as one of the most prominent security threats that every single Pakistani faces today. The conundrum of suicide bombing in Pakistan has obliterated 6,982 clean-handed civilians and injured another 17,624 in a total of 475 attacks since 1995. More than 94% of these attacks have taken place after year 2006. From 2007 to 2013 the country witnessed a suicide bombing attack on every 6th day that increased to every 4th day in 2013. Counting the dead and the injured, each attack victimizes 48 people in Pakistan. | |
| Pakistan Body Count (www.PakistanBodyCount.org) is the oldest and most accurate running tally of suicide bombings in Pakistan. The given database (PakistanSuicideAttacks.CSV) has been populated by using majority of the data from Pakistan Body Count, and building up on it by canvassing open source newspapers, media reports, think tank analyses, and personal contacts in media and law enforcement agencies. We provide a count of the people killed and injured in suicide attacks, including the ones who died later in hospitals or homes due to injuries caused or aggravated by these attacks (second and tertiary blast injuries), making it the most authentic source for suicide attacks related data in this region. | |
| We will keep releasing the updates every quarter at this page. | |
| Content | |
| Geography: Pakistan | |
| Time period: 1995-2016 | |
| Unit of analysis: Attack | |
| Dataset: The dataset contains detailed information of 475 suicide bombing attacks in Pakistan that killed an estimated 6,982 and injured 17,624 people. | |
| Variables: The dataset contains Serial No, Incident Date, Islamic Date (based on Islamic lunar calendar), approximate Time, Long-Lat, City, Province, Location, Location Sensitivity & Type, Target type and Sect, Open/Close Space (as it will change the impact of blast waves due to reflection), min and max number of people killed and injured, number of suicide bombers, amount of explosive being used and the name of hospitals where victims went for treatment. | |
| Sources: Unclassified media articles, hospital reports, think tank analysis and reports, and government official press releases. | |
| Acknowledgements & References | |
| Pakistan Body Count has been leveraged extensively in scholarly publications, reports, media articles and books. The website and the dataset has been collected and curated by the founder Zeeshan-ul-hassan Usmani. | |
| Users are allowed to use, copy, distribute and cite the dataset as follows: “Zeeshan-ul-hassan Usmani, Pakistan Body Count, Pakistan Suicide Bombing Attacks Dataset, Kaggle Dataset Repository, Jan 25, 2017.” | |
| Past Work | |
| Zeeshan-ul-hassan Usmani and Daniel Kirk, “Simulation of Suicide Bombing – Using Computers to Save Lives”, I-Universe, New York, NY, April 2011 | |
| Zeeshan-ul-hassan Usmani and Daniel Kirk, “Modeling and Simulation of Explosion Effectiveness as a Function of Blast and Crowd Characteristics”, The Journal of Defense Modeling and Simulation: Applications, Methodology, Technology, Sage Publications with Society of Simulation, Vol. 6, No. 2, pp. 79-95, Vista, CA, USA, October 2009 | |
| Muhammad Irfan and Zeeshan-ul-hassan Usmani, “Suicide Terrorism and its New Target –Pakistan”, in Wars, Insurgencies and Terrorist Attacks: A Psychosocial Perspective from The Muslim World, by Unaiza Niaz, Oxford University Press, Canada, July 2010 | |
| Sana Rasheed, Data Science for Suicide Bombings, I-Universe, New York, NY, December 2016 | |
| Zeeshan-ul-hassan Usmani and Sana Rasheed, “Terrorism: What Data Sciences Can Do?”, 20th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD 2014, Data Framework Track) at Bloomberg, New York, NY, USA (August 24-27, 2014) | |
| Zeeshan-ul-hassan Usmani, “Suicide Bombing Forecaster – Novel Techniques to Predict Patterns of Suicide Bombing in Pakistan”, 2012 Conference on Homeland Security, part of 2012 Autumn Simulation Multi-Conference, San Diego, CA, USA, October 28 – 31, 2012 | |
| Zeeshan-ul-hassan Usmani, “BlastSim – Simulation to Save Lives”, IEEE/SIC Winter Simulation Conference, PhD Colloquium, Austin, Texas, December 13-16, 2009 | |
| Zeeshan-ul-hassan Usmani, Fawzi Alghamdi, and Daniel Kirk, “BlastSim – Multi-agent Simulation of Suicide Bombing“, IEEE Symposium: Computational Intelligence for Security and Defense Applications (CISDA), Ottawa, Canada, July 8-10, 2009 | |
| Zeeshan-ul-hassan Usmani, Eyosias Imana and Daniel Kirk, “Virtual Iraq – Simulation of Insurgent Attacks”, IEEE Workshop on Computational Intelligence in Virtual Environments (CIVE), March 30-April 2, 2009 | |
| Zeeshan-ul-hassan Usmani, Eyosias Imana, and Daniel Kirk, “Random Walk in Extreme Conditions – An Agent Based Simulation of Suicide Bombing”, IEEE Symposium on Intelligent Agents, March, 2009 | |
| Zeeshan-ul-hassan Usmani, Eyosias Imana and Daniel Kirk, “Escaping Death – Geometrical Recommendations for High Value Targets”, IEEE International Joint Conferences on Computer, Information and Systems Sciences and Engineering (CIS2E 08), Bridgeport, CT, December 5–13, 2008 | |
| Zeeshan-ul-hassan Usmani and Daniel Kirk, “Extreme Conditions for Intelligent Agents”, IEEE 2008 WI-IAT Doctoral Workshop, Sydney, Australia, December 9-12, 2008 | |
| Zeeshan-ul-hassan Usmani, Andrew English & Richard Griffith, “The Effects of a Suicide Bombing: Crowd Formations”, Inter-service/Industry Training, Simulation, and Education Conference (I/ITSEC), Orlando, FL, Nov 26-29 2007 | |
| Inspiration | |
| Some ideas worth exploring: | |
| How many people got killed and injured per year? | |
| Visualize suicide attacks on timeline | |
| Find out any correlation with number of suicide bombing attacks with drone attacks | |
| Find out any correlation with suicide bombing attacks with influencing events given in the dataset | |
| Can we predict the next suicide bombing attack? | |
| Find the correlation between blast/explosive weight and number of people killed and injured | |
| Find the impact of holiday type on number of blast victims | |
| Find the correlation between Islamic date and blast day/time/size/number of victims | |
| Find the Top 10 locations of blasts | |
| Find the names of hospitals sorted by number of victims | |
| Questions? | |
| For detailed visit www.PakistanBodyCount.org | |
| Or contact Pakistan Body Count staff at info@pakistanbodycount.org" | |
| R Questions from Stack Overflow,Full text of Stack Overflow Q&A about the R statistical programming language,Stack Overflow,37,"Version 3,2017-09-26|Version 2,2016-10-21|Version 1,2016-10-20","linguistics | |
| internet | |
| programming languages",CSV,516 MB,Other,"15,609 views",956 downloads,39 kernels,,https://www.kaggle.com/stackoverflow/rquestions,"Full text of questions and answers from Stack Overflow that are tagged with the r tag, useful for natural language processing and community analysis. | |
| This is organized as three tables: | |
| Questions contains the title, body, creation date, score, and owner ID for each R question. | |
| Answers contains the body, creation date, score, and owner ID for each of the answers to these questions. The ParentId column links back to the Questions table. | |
| Tags contains the tags on each question besides the R tag. | |
| For space reasons only non-deleted and non-closed content are included in the dataset. The dataset contains questions up to 24 September 2017 (UTC). | |
| License | |
| All Stack Overflow user contributions are licensed under CC-BY-SA 3.0 with attribution required." | |
| Eating & Health Module Dataset,American Time Use Survey (ATUS) Eating & Health Module Files from 2014,US Bureau of Labor Statistics,37,"Version 2,2016-11-10|Version 1,2016-11-09","nutrition | |
| demographics",CSV,19 MB,Other,"16,422 views","1,820 downloads",88 kernels,2 topics,https://www.kaggle.com/bls/eating-health-module-dataset,"Context | |
| The ATUS Eating & Health (EH) Module was fielded from 2006 to 2008 and again in 2014 to 2016. The EH Module data files contain additional information related to eating, meal preparation and health. | |
| Data for 2015 currently are being processed and have not yet been released. Data collection is planned to run through December 2016. | |
| Content | |
| There are 3 datasets from 2014: | |
| The EH Respondent file, which contains information about EH respondents, including general health and body mass index. There are 37 variables. | |
| The EH Activity file, which contains information such as the activity number, whether secondary eating occurred during the activity, and the duration of secondary eating. There are 5 variables. | |
| The EH Replicate weights file, which contains miscellaneous EH weights. There are 161 variables. | |
| The data dictionary can be found here. | |
| Acknowledgements | |
| The original datasets can be found here. | |
| Inspiration | |
| Some ideas for exploring the datasets are: | |
| What is the relationship between weight or BMI and meal preparation patterns, consumption of fresh/fast food, or snacking patterns? | |
| Do grocery shopping patterns differ by income?" | |
| "The Academy Awards, 1927-2015",What actors and films have received the most Oscars?,Academy of Motion Picture Arts and Sciences,37,"Version 1,2017-02-14",film,CSV,775 KB,Other,"12,072 views","1,835 downloads",31 kernels,0 topics,https://www.kaggle.com/theacademy/academy-awards,"Context | |
| Each January, the entertainment community and film fans around the world turn their attention to the Academy Awards. Interest and anticipation builds to a fevered pitch leading up to the Oscar telecast in February, when hundreds of millions of movie lovers tune in to watch the glamorous ceremony and learn who will receive the highest honors in filmmaking. | |
| Achievements in up to 25 regular categories will be honored on February 26, 2017, at the 89th Academy Awards presentation at the Dolby Theatre at Hollywood & Highland Center. | |
| Content | |
| The Academy Awards Database contains the official record of past Academy Award winners and nominees. The data is complete through the 2015 (88th) Academy Awards, presented on February 28, 2016. | |
| Acknowledgements | |
| The awards data was scraped from the Official Academy Awards Database; nominees were listed with their name first and film following in some categories, such as Best Actor/Actress, and in the reverse for others. | |
| Inspiration | |
| Do the Academy Awards reflect the diversity of American films or are the #OscarsSoWhite? Which actor/actress has received the most awards overall or in a single year? Which film has received the most awards in a ceremony? Can you predict who will receive the 2016 awards?" | |
| UK Car Accidents 2005-2015,Data from the UK Department for Transport,silicon99,36,"Version 3,2017-02-21|Version 2,2017-02-01|Version 1,2017-02-01",road transport,CSV,534 MB,CC0,"19,343 views","3,315 downloads",29 kernels,2 topics,https://www.kaggle.com/silicon99/dft-accident-data,"Context | |
| UK police forces collect data on every vehicle collision in the uk on a form called Stats19. Data from this form ends up at the DfT and is published at https://data.gov.uk/dataset/road-accidents-safety-data | |
| Content | |
| There are 3 CSVs in this set. Accidents is the primary one and has references by Accident_Index to the casualties and vehicles tables. This might be better done as a database. | |
| Inspiration | |
| Questions to ask of this data - | |
| combined with population data, how do different areas compare? | |
| what trends are there for accidents involving different road users eg motorcycles, peds, cyclists | |
| are road safety campaigns effective? | |
| likelihood of accidents for different groups / vehicles | |
| many more.. | |
| Manifest | |
| dft05-15.tgz - tar of Accidents0515.csv, Casualties0515.csv and Vehicles0515.csv tidydata.sh - script to get and tidy data." | |
| Board Game Data,Data is a collection of board game information from Board Game Geek,mrpantherson,36,"Version 5,2018-02-16|Version 4,2017-06-16|Version 3,2017-06-08|Version 2,2017-04-29|Version 1,2017-04-07",board games,CSV,818 KB,CC0,"11,059 views","1,193 downloads",39 kernels,6 topics,https://www.kaggle.com/mrpantherson/board-game-data,"Context | |
| Being a fan of board games, I wanted to see if there was any correlation with a games rating and any particular quality, the first step was to collect of this data. | |
| Content | |
| The data was collected in March of 2017 from the website https://boardgamegeek.com/, this site has an API to retrieve game information (though sadly XML not JSON). | |
| Acknowledgements | |
| Mainly I want to thank the people who run the board game geek website for maintaining such a great resource for those of us in the hobby. | |
| Inspiration | |
| I wish I had some better questions to ask of the data, perhaps somebody else can think of some good ways to get some insight of this dataset." | |
| San Francisco Library Usage,"Anonymized library usage data by over 420,000 patrons",DataSF,36,"Version 2,2017-01-07|Version 1,2017-01-05",libraries,CSV,33 MB,Other,"7,141 views",771 downloads,28 kernels,,https://www.kaggle.com/datasf/sf-library-usage-data,"Context | |
| San Francisco's Integrated Library System (ILS) is composed of bibliographic records including inventoried items, patron records, and circulation data. The data is used in the daily operation of the library, including circulation, online public catalog, cataloging, acquisitions, collection development, processing, and serials control. This dataset represents the usage of inventoried items by patrons (~420K records). | |
| Content | |
| The dataset includes approximately 420,000 records, with each record representing an anonymized library patron. Individual columns include statistics on the type code and age of the patron, the year the patron registered (only since 2003), and how heavily the patron has been utilizing the library system (in terms of number of checkouts) since first registering. | |
| For more information on specific columns refer to the official data dictionary and the information in the Column Metadata on the /Data tab. | |
| Acknowledgements | |
| The data is provided by SF Public Library via the San Francisco Open Data Portal, under the PDDL 1.0 ODC Public Domain Dedication and Licence (PDDL). | |
| Photo via Flickr Kolya Miller (CC BY-NC-SA 2.0). | |
| Inspiration | |
| What attributes are most associated with library activity (# of checkouts, # of renewals)? | |
| Can you group the data into type of patrons? What classifiers would you use to predict patron type?" | |
| Solar and Lunar Eclipses,"Date, time, and location of every eclipse in five thousand years",NASA,36,"Version 1,2017-02-09","astronomy | |
| space",CSV,2 MB,CC0,"5,630 views",684 downloads,15 kernels,,https://www.kaggle.com/nasa/solar-eclipses,"Context | |
| Eclipses of the sun can only occur when the moon is near one of its two orbital nodes during the new moon phase. It is then possible for the Moon's penumbral, umbral, or antumbral shadows to sweep across Earth's surface thereby producing an eclipse. There are four types of solar eclipses: a partial eclipse, during which the moon's penumbral shadow traverses Earth and umbral and antumbral shadows completely miss Earth; an annular eclipse, during which the moon's antumbral shadow traverses Earth but does not completely cover the sun; a total eclipse, during which the moon's umbral shadow traverses Earth and completely covers the sun; and a hybrid eclipse, during which the moon's umbral and antumbral shadows traverse Earth and annular and total eclipses are visible in different locations. Earth will experience 11898 solar eclipses during the five millennium period -1999 to +3000 (2000 BCE to 3000 CE). | |
| Eclipses of the moon can occur when the moon is near one of its two orbital nodes during the full moon phase. It is then possible for the moon to pass through Earth's penumbral or umbral shadows thereby producing an eclipse. There are three types of lunar eclipses: a penumbral eclipse, during which the moon traverses Earth's penumbral shadow but misses its umbral shadow; a partial eclipse, during which the moon traverses Earth's penumbral and umbral shadows; and a total eclipse, during which the moon traverses Earth's penumbral and umbral shadows and passes completely into Earth's umbra. Earth will experience 12064 lunar eclipses during the five millennium period -1999 to +3000 (2000 BCE to 3000 CE). | |
| Acknowledgements | |
| Lunar eclipse predictions were produced by Fred Espenak from NASA's Goddard Space Flight Center." | |
| "Fatal Police Shootings, 2015-Present",Civilians shot and killed by on-duty police officers in United States,The Washington Post,36,"Version 3,2017-03-11|Version 2,2017-01-03|Version 1,2016-11-26","crime | |
| demographics",CSV,192 KB,CC4,"10,025 views","1,737 downloads",48 kernels,,https://www.kaggle.com/washingtonpost/police-shootings,"The Washington Post is compiling a database of every fatal shooting in the United States by a police officer in the line of duty since January 1, 2015. | |
| In 2015, The Post began tracking more than a dozen details about each killing — including the race of the deceased, the circumstances of the shooting, whether the person was armed and whether the victim was experiencing a mental-health crisis — by culling local news reports, law enforcement websites and social media and by monitoring independent databases such as Killed by Police and Fatal Encounters. | |
| The Post is documenting only those shootings in which a police officer, in the line of duty, shot and killed a civilian — the circumstances that most closely parallel the 2014 killing of Michael Brown in Ferguson, Missouri, which began the protest movement culminating in Black Lives Matter and an increased focus on police accountability nationwide. The Post is not tracking deaths of people in police custody, fatal shootings by off-duty officers or non-shooting deaths. | |
| The FBI and the Centers for Disease Control and Prevention log fatal shootings by police, but officials acknowledge that their data is incomplete. In 2015, The Post documented more than two times more fatal shootings by police than had been recorded by the FBI. | |
| The Post’s database is updated regularly as fatal shootings are reported and as facts emerge about individual cases. The Post is seeking assistance in making the database as comprehensive as possible. To provide information about fatal police shootings, send us an email at policeshootingsfeedback@washpost.com. | |
| CREDITS | |
| Research and Reporting: Julie Tate, Jennifer Jenkins and Steven Rich | |
| Production and Presentation: John Muyskens, Kennedy Elliott and Ted Mellnik" | |
| "Olympic Sports and Medals, 1896-2014",Which countries and athletes have won the most medals at the Olympic games?,The Guardian,36,"Version 1,2017-01-24",olympic games,CSV,3 MB,CC4,"15,684 views","3,899 downloads",27 kernels,,https://www.kaggle.com/the-guardian/olympic-games,"Content | |
| Which Olympic athletes have the most gold medals? Which countries are they from and how has it changed over time? | |
| More than 35,000 medals have been awarded at the Olympics since 1896. The first two Olympiads awarded silver medals and an olive wreath for the winner, and the IOC retrospectively awarded gold, silver, and bronze to athletes based on their rankings. This dataset includes a row for every Olympic athlete that has won a medal since the first games. | |
| Acknowledgements | |
| Data was provided by the IOC Research and Reference Service and published by The Guardian's Datablog." | |
| Google Text Normalization Challenge,"Text-to-speech synthesis text normalization data, from Sproat & Jaitly 2016",Google Natural Language Understanding Research,36,"Version 1,2017-04-27","languages | |
| linguistics",CSV,9 GB,CC4,"10,298 views","1,539 downloads",7 kernels,0 topics,https://www.kaggle.com/google-nlu/text-normalization,"Challenge Description | |
| This dataset and accompanying paper present a challenge to the community: given a large corpus of written text aligned to its normalized spoken form, train an RNN to learn the correct normalization function. That is, a date written ""31 May 2014"" is spoken as ""the thirty first of may twenty fourteen."" We present a dataset of general text where the normalizations were generated using an existing text normalization component of a text-to-speech (TTS) system. This dataset was originally released open-source here and is reproduced on Kaggle for the community. | |
| The Data | |
| The data in this directory are the English language training, development and test data used in Sproat and Jaitly (2016). | |
| The following divisions of data were used: | |
| Training: output_1 through output_21 (corresponding to output-000[0-8]?-of-00100 in the original dataset) | |
| Runtime eval: output_91 (corresponding to output-0009[0-4]-of-00100 in the original dataset) | |
| Test data: output_96 (corresponding to output-0009[5-9]-of-00100 in the original dataset) | |
| In practice for the results reported in the paper only the first 100,002 lines of output-00099-of-00100 were used (for English). | |
| Lines with """" in two columns are the end of sentence marker, otherwise there are three columns, the first of which is the ""semiotic class"" (Taylor, 2009), the second is the input token and the third is the output, following the paper cited above. | |
| All text is from Wikipedia. All data were extracted on 2016/04/08, and run through the Google Kestrel TTS text normalization system (Ebden and Sproat, 2015), so that the notion of ""token"", ""semiotic class"" and reference output are all Kestrel's notion. | |
| Our Research | |
| In this paper, we present our own experiments with this data set with a variety of different RNN architectures. While some of the architectures do in fact produce very good results when measured in terms of overall accuracy, the errors that are produced are problematic, since they would convey completely the wrong message if such a system were deployed in a speech application. On the other hand, we show that a simple FST-based filter can mitigate those errors, and achieve a level of accuracy not achievable by the RNN alone. | |
| Though our conclusions are largely negative on this point, we are actually not arguing that the text normalization problem is intractable using an pure RNN approach, merely that it is not going to be something that can be solved merely by having huge amounts of annotated text data and feeding that to a general RNN model. And with open-source data, we provide a novel data set for sequence-to-sequence modeling in the hopes that the the community can find better solutions. | |
| Disclaimer | |
| This is not an official Google product. | |
| References | |
| Ebden, Peter and Sproat, Richard. 2015. The Kestrel TTS text normalization system. Natural Language Engineering. 21(3). | |
| Richard Sproat and Navdeep Jaitly. 2016. RNN Approaches to Text Normalization: A Challenge. Released on arXiv.org: https://arxiv.org/abs/1611.00068 | |
| Taylor, Paul. 2009. Text-to-Speech Synthesis. Cambridge University Press, Cambridge." | |
| NFL Draft Outcomes,All players selected in the NFL Draft from 1985 to 2015 with outcome statistics,Ron Graf,35,"Version 1,2016-12-19",american football,CSV,780 KB,Other,"7,327 views",975 downloads,23 kernels,0 topics,https://www.kaggle.com/ronaldjgrafjr/nfl-draft-outcomes,"Context | |
| Consolidated draft data from http://www.pro-football-reference.com/ for all drafts from 1985 to 2015. | |
| Content | |
| Pro-Football-Reference AV: Approximate Value is PFR's attempt to attach a single number to every player-season since 1960. Methodology can be found here: http://www.pro-football-reference.com/blog/indexd961.html?page_id=8061 | |
| Player_Id Pro Football Reference Player Id | |
| Year Draft Year | |
| Rnd Draft Round | |
| Pick Draft Pick | |
| Tm Team | |
| Player Player first and last name | |
| Pos Position unfiltered | |
| Position Standard Position standardized to one of the following QB, LB, WR, T, DE, RB, DB, DT, C, C, G, TE, FB, P, LS, K | |
| First4AV AV accumulated for this player's first four seasons | |
| Age Age at time of draft | |
| To Year of last season played | |
| AP1 # of first team all-pro selections | |
| PB # of pro-bowl selections | |
| St # of years as a primary starter in their primary position | |
| CarAV Weighted Career AV - 100% of best season, 95% of second best season, 90% of third best season, and so on | |
| DrAV AV accumulated for team that drafted this player | |
| G Games played | |
| Cmp Pass completions | |
| Pass_Att Pass attempts | |
| Pass_Yds Yards gained by passing | |
| Pass_TD Passing touchdowns | |
| Pass_Int Interceptions thrown | |
| Rush_Att Rushing attempts | |
| Rush_Yds Rushing yards gained | |
| Rush_TDs Rushing touchdowns | |
| Rec Receptions | |
| Rec_Yds Receiving yards gained | |
| Rec_Tds Receiving touchdowns | |
| Tkl Tackles | |
| Def_Int Defensive interceptions | |
| Sk Sacks | |
| College/Univ College/University attended by player | |
| Acknowledgements | |
| http://www.pro-football-reference.com/" | |
| Horse Racing - Tipster Bets,Thirty nine thousand bets from thirty one tipsters,gunner38,35,"Version 3,2016-09-14|Version 2,2016-09-14|Version 1,2016-09-13",horse racing,CSV,3 MB,Other,"16,607 views","1,060 downloads",19 kernels,4 topics,https://www.kaggle.com/gunner38/horseracing,"Horse Racing - A different and profitable approach | |
| The traditional approach in attempting to make a profit from horse-racing, using machine learning techniques, is to use systems involving dozens and dozens of variables. These systems include the following types of variables: | |
| Horse - Name, Sex, Age, Pedigree, Weight, Speed over various distances, race data with finishing times and positions - etc. Trainer info. Jockey info. Track info - Track, track conditions - etc. | |
| And a whole lot more. | |
| Finding, compiling, maintaining and updating this data is a massive task for the individual. Unless you have access to a database of such data - where would you even start? | |
| We have a different approach. | |
| We collect, maintain and use data from various 'Tipsters'. The tipsters use their skill to study the horses and make a prediction - that they think a particular horse will win a particular race. We take those tipsters predictions and put them through a machine learning algorithm (microsoft azure) asking it to predict a 'win' or 'lose' based upon the tipsters performance history. | |
| We have a database of approx. 39,000 bets using 31 odd tipsters. Fifteen tipsters are active and sixteen tipsters are inactive The betting history for the inactive tipsters is used in the dataset as it appears to add 'weight' to the system when considering active tips. | |
| We have been using this system live for three months now and although it has it's ups and downs - it makes money! One bookmaker has already closed our account. | |
| We are looking to further optimize the system to reach it's maximum efficiency coupled with a betting strategy to increase profitability. We ask for your help. If you can produce an 'Azure' system more efficient than ours - then further information will be shared with you. | |
| Questions | |
| Are bets from inactive tipsters critical to performance? | |
| Is it better to have all the tipsters 'stacked on top of each other' in one large dataset or is the system better served by separating them out? | |
| Predicting Bets | |
| When we ask the system to predict if a bet will Win or Lose for say Tipster A - we take the last ID number for that Tipster and add one to it - making it a new ID - outside the systems experience. That ID is used for all that Tipsters bets until the system is updated. The system is updated once a week. | |
| Good hunting. | |
| Gunner38" | |
| NFL Statistics,"Basic NFL statistics, career statistics, and game logs",KendallGillies,35,"Version 1,2017-06-09",american football,CSV,93 MB,Other,"10,675 views","1,587 downloads",3 kernels,3 topics,https://www.kaggle.com/kendallgillies/nflstatistics,"NFL-Statistics-Scrape | |
| Here are the basic statistics, career statistics and game logs provided by the NFL on their website (http://www.nfl.com) for all players past and present. | |
| Summary | |
| The data was scraped using a Python code. The code can be located at Github: https://github.com/kendallgillies/NFL-Statistics-Scrape | |
| Explanation of Data | |
| The first main group of statistics is the basic statistics provided for each player. This data is stored in the CSV file titled Basic_Stats.csv along with the player’s name and URL identifier. If available the data pulled for each player is as follows: | |
| Number | |
| Position | |
| Current Team | |
| Height | |
| Weight | |
| Age | |
| Birthday | |
| Birth Place | |
| College Attended | |
| High School Attended | |
| High School Location | |
| Experience | |
| The second main group of statistics gathered for each player are their career statistics. While each player has a main position they play, they will have statistics in other areas; therefore, the career statistics are divided into statistics types. The statistics are then stored in CSV files based on statistic type along with the player name, URL identifier and position (if available). The following are the career statistics types and accompanying CSV file names: | |
| Defensive Statistics – Career_Stats_Defensive.csv | |
| Field Goal Kickers - Career_Stats_Field_Goal_Kickers.csv | |
| Fumbles - Career_Stats_Fumbles.csv | |
| Kick Return - Career_Stats_Kick_Return.csv | |
| Kickoff - Career_Stats_Kickoff.csv | |
| Offensive Line - Career_Stats_Offensive_Line.csv | |
| Passing - Career_Stats_Passing.csv | |
| Punt Return - Career_Stats_Punt_Return.csv | |
| Punting - Career_Stats_Punting.csv | |
| Receiving - Career_Stats_Receiving.csv | |
| Rushing - Career_Stats_Rushing.csv | |
| The final group of statistics is the game logs for each player. The game logs are stored by position and have the player name, URL identifier and position (if available). The following are the game log types and accompanying CSV file names: | |
| Quarterback – Game_Logs_Quarterback.csv | |
| Running back – Game_Logs_Runningback.csv | |
| Wide Receiver and Tight End – Game_Logs_Wide_Receiver_and_Tight_End.csv | |
| Offensive Line – Game_Logs_Offensive_Line.csv | |
| Defensive Lineman – Game_Logs_Defensive_Lineman.csv | |
| Kickers – Game_Logs_Kickers.csv | |
| Punters – Game_Logs_Punters.csv | |
| Glossary | |
| While most of the abbreviations used by the NFL have been translated in the table headers in the data files, there are still a couple of abbreviations used. | |
| FG: Field Goal | |
| TD: Touchdown | |
| Int: Interception" | |
| Tree Census in New York City,What tree species are thriving on the streets of each NYC borough?,NYC Parks and Recreation,35,"Version 2,2017-07-17|Version 1,2017-02-07","plants | |
| forestry",CSV,475 MB,CC0,"5,243 views",657 downloads,7 kernels,0 topics,https://www.kaggle.com/nycparks/tree-census,"Context | |
| New York City’s trees shade us in the summer, beautify our neighborhoods, help reduce noise, and support urban wildlife. Beyond these priceless benefits, our urban forest provides us a concrete return on the financial investment we put into it. This return includes stormwater interception, energy conservation, air pollutant removal, and carbon dioxide storage. Our publicly owned trees are as much of an asset to us as our streets, sewers, bridges, and public buildings. | |
| Content | |
| This dataset includes a record for every tree in New York City and includes the tree's location by borough and latitude/longitude, species by Latin name and common names, size, health, and issues with the tree's roots, trunk, and branches. | |
| Acknowledgements | |
| The 2015, 2005, and 1995 tree censuses were conducted by NYC Parks and Recreation staff, TreesCount! program staff, and hundreds of volunteers." | |
| Daily Happiness & Employee Turnover,Is There a Relationship Between Employee Happiness and Job Turnover?,Jose Berengueres,35,"Version 3,2017-08-07|Version 2,2017-08-07|Version 1,2017-08-05",economics,CSV,49 MB,ODbL,"8,454 views",927 downloads,9 kernels,3 topics,https://www.kaggle.com/harriken/employeeturnover,"Can Happiness Predict Employee Turnover, or is it the Other Way Around? | |
| It is the summer of 2016. I am in Barcelona and it is hot and humid. By chance I go to a talk where Alex Rios - the ceo of myhappyforce.com explains his product. He has built an app where employees report daily happiness levels at work. This app is used by companies to track happiness of the workforce. After the talk I ask him if he would opensource the (anonymized data) so we can better understad the phenomenon of employee turnover. Here is what we did, we developed a model that predicts which employees will churn. Then we looked at the features (used by the model) that are common to employees that churn. The top feautures of employees that churn are: | |
| low ratio of likes received (likeability) | |
| low posting frequency (engagement), | |
| low relative happiness (employee happiness normalized by company mean). | |
| Surprisingly, a priori expected explanatory features such as mean happiness level and the ratio of likes (positivity), were not significant. Precision@50 = 80% out of a test set with 116 churns, sample size N=2k. Another surprise was that raw happiness is a bad predictor of churn. But, the question is, What did we miss? Can you find more insights? | |
| Starter script | |
| R starter script https://www.kaggle.com/harriken/how-many-unlikes-it-takes-to-get-fired | |
| Content | |
| The data consists of four tables: votes, comments, interactions and churn. A vote was obtained when an employee opened the app and answered the question: How happy are you at work today? To vote the employee indicates their feeling by touching one of four icons that appeared on the screen. After the employee indicates their happiness level, a second screen appears where they can input a text explanation (usually a complaint, suggestion or comment), this is the comments table. Out of 4,356 employees, 2,638 employees commented at least once. Finally, in a third screen the employee can see their peers’ comments and like or dislike them, this data is stored in the interactions table. 3,516 employees liked or disliked at least one of their peers’ comments. The churn table contains when an employee churned (quit or was fired). | |
| Acknowledgements | |
| Python script version with social graph features: http://bit.ly/2v2sEZg | |
| More detailed R scripts: https://github.com/orioli/e3 | |
| The paper which was presented at ASONAM 2017 Sydney | |
| Slides https://www.slideshare.net/harriken/ieee-happiness-an-inside-job-asoman-2017 | |
| Inspiration | |
| The cost of employee turnover has been pointed out extensively in the literature. A high turnover rate not only increases human resource costs, which can reach up to 150% of the annual salary per replaced employee, but it also has social costs, as it is correlated with lower wages, lower productivity per employee, and not surprisingly, a less loyal workforce 1. For reference, in 2006, turnover at Walmart’s Sam’s Club was 44% with an average hourly pay of $10.11, while at Costco it was a much lower 17% with a higher $17.0 hourly wage 2. In addition, a more recent study correlated companies with low turnover with a series of socially positive characteristics dubbed high-involvement work practices 3. On the other hand, research on employee turnover (churn) is not a prolific topic in the engineering community. In IEEE publications, one can find just over 278 publications with titles containing the keyword churn, and the bulk of those focus on customer churn, and specifically churn in the telecommunications industry, while on the topic of employee churn there is just one title indexed 4. The goal is to clarify the characteristics of employees that will churn (or that are at risk of churning), to help companies understand the causes so they can reduce the turnover rate." | |
| RSNA Bone Age,Predict Age from X-Rays,Kevin Mader,35,"Version 2,2018-01-24|Version 1,2018-01-24","healthcare | |
| orthopedic surgery | |
| pediatrics | |
| image data",Other,9 GB,Other,"3,306 views",431 downloads,5 kernels,,https://www.kaggle.com/kmader/rsna-bone-age,"Context | |
| At RSNA 2017 there was a contest to correctly identify the age of a child from an X-ray of their hand. This is the dataset on Kaggle making it easier to experiment with and do educational demos. Additionally maybe there are some new ideas for building smarter models for handling X-ray images. | |
| Content | |
| A number of folders full of images (digital and scanned) with a CSV containing the age (what is to be predicted) and the gender (useful additional information) | |
| Acknowledgements | |
| The dataset was originally published on CloudApp as an RSNA challenge. | |
| Original Dataset Acknowledgements | |
| The Radiological Society of North America (RSNA) Radiology Informatics Committee (RIC) Pediatric Bone Age Machine Learning Challenge Organizing Committee: | |
| Kathy Andriole, Massachusetts General Hospital | |
| Brad Erickson, Mayo Clinic | |
| Adam Flanders, Thomas Jefferson University | |
| Safwan Halabi, Stanford University | |
| Jayashree Kalpathy-Cramer, Massachusetts General Hospital | |
| Marc Kohli, University of California - San Francisco | |
| Luciano Prevedello, The Ohio State University | |
| Data sets used in the Pediatric Bone Age Challenge have been contributed by Stanford University, the University of Colorado and the University of California - Los Angeles. | |
| The MedICI platform (built CodaLab) used for the challenge is provided by Jayashree Kalpathy-Cramer, supported through NIH grants (U24CA180927) and a contract from Leidos. | |
| Inspiration | |
| Can you predict with better than 4.2 months accuracy? | |
| Is identifying the joints an important step? | |
| What algorithms work best? | |
| What do the algorithms focus on? | |
| Is gender a necessary piece of information or can it be automatically derived from the image?" | |
| GloVe: Global Vectors for Word Representation,Pre-trained word vectors from Wikipedia 2014 + Gigaword 5,Rachael Tatman,35,"Version 1,2017-08-05","languages | |
| linguistics",Other,1 GB,Other,"5,055 views",591 downloads,15 kernels,0 topics,https://www.kaggle.com/rtatman/glove-global-vectors-for-word-representation,"Context | |
| GloVe is an unsupervised learning algorithm for obtaining vector representations for words. Training is performed on aggregated global word-word co-occurrence statistics from a corpus, and the resulting representations showcase interesting linear substructures of the word vector space. | |
| Content | |
| This dataset contains English word vectors pre-trained on the combined Wikipedia 2014 + Gigaword 5th Edition corpora (6B tokens, 400K vocab). All tokens are in lowercase. This dataset contains 50-dimensional, 100-dimensional and 200-dimensional pre trained word vectors. For 300-dimensional word vectors and additional information, please see the project website. | |
| Acknowledgements | |
| This data has been released under the Open Data Commons Public Domain Dedication and License. If you use this dataset in your work, please cite the following paper: | |
| Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. GloVe: Global Vectors for Word Representation. URL: https://nlp.stanford.edu/pubs/glove.pdf | |
| Inspiration | |
| GloVe embeddings have been used in more than 2100 papers, and counting! You can use these pre-trained embeddings whenever you need a way to quantify word co-occurrence (which also captures some aspects of word meaning.)" | |
| An Open Dataset for Human Activity Analysis,"Data collected using Smartphone, Smartwatch and Smartglasses",Sasan Jafarnejad,35,"Version 3,2017-09-01|Version 2,2017-08-31|Version 1,2017-08-31",consumer electronics,CSV,434 MB,ODbL,"7,367 views",665 downloads,,2 topics,https://www.kaggle.com/sasanj/human-activity-smart-devices,"Context | |
| The study of human mobility and activities has opened up to an incredible number of studies in the past, most of which included the use of sensors distributed on the body of the subject. More recently, the use of smart devices has been particularly relevant because they are already everywhere and they come with accurate miniaturized sensors. Whether it is smartphones, smartwatches or smartglasses, each device can be used to describe complementary information such as emotions, precise movements, or environmental conditions. | |
| Content | |
| First of all, a smartphone is used to capture mainly contextual data. Two applications are used: a simple data collection application based on the SWIPE open-source sensing system (SWIPE), and a logbook application for obtaining real data on user activity (TimeLogger). SWIPE is a platform for sensing, recording and processing human dynamics using smartwatches and smartphones. | |
| Then, a smartwatch is used primarily to capture the user's heart rate. Motion data is also collected, without being at the heart of the dataset due to its need to be configured with a low sampling frequency, which would drastically increase the dataset and drain the battery as well. An application based on SWIPE is used. | |
| Finally, JINS MEME smartglasses are used. This model has the advantage of being compact and simple to carry. It does not have a camera or a screen; it simply has three types of sensors: an accelerometer (for detecting steps or activities), a gyroscope (for head movements) and an occulographic sensor (eye blinking, eye orientation). The official DataLogger application from JINS MEME is used. | |
| For more information on the dataset please refer to the corresponding publication, available at An Open Dataset for Human Activity Analysis using Smart Devices. | |
| The current dataset on Kaggle contains smartglasses data with 20ms interval (due to storage limitations), same data with 10ms interval is also available on demand. Contact sasan.jafarnejad [at] uni [dot] lu to receive the 10ms version. | |
| Acknowledgements | |
| This work was performed within the eGLASSES project, which is partially funded by NCBiR, FWF, SNSF, ANR and FNR under the ERA-NET CHIST-ERAII framework." | |
| (LoL) League of Legends Ranked Games,"Details from over 50,000 ranked games of LoL",Mitchell J,35,"Version 9,2017-09-23|Version 8,2017-09-22|Version 7,2017-09-22|Version 6,2017-09-07|Version 5,2017-09-07|Version 4,2017-09-07|Version 3,2017-09-07|Version 2,2017-09-07|Version 1,2017-09-06","video games | |
| internet",{}JSON,9 MB,CC0,"8,996 views","1,042 downloads",3 kernels,6 topics,https://www.kaggle.com/datasnaek/league-of-legends,"General Info | |
| This is a collection of over 50,000 ranked EUW games from the game League of Legends, as well as json files containing a way to convert between champion and summoner spell IDs and their names. For each game, there are fields for: | |
| Game ID | |
| Creation Time (in Epoch format) | |
| Game Duration (in seconds) | |
| Season ID | |
| Winner (1 = team1, 2 = team2) | |
| First Baron, dragon, tower, blood, inhibitor and Rift Herald (1 = team1, 2 = team2, 0 = none) | |
| Champions and summoner spells for each team (Stored as Riot's champion and summoner spell IDs) | |
| The number of tower, inhibitor, Baron, dragon and Rift Herald kills each team has | |
| The 5 bans of each team (Again, champion IDs are used) | |
| This dataset was collected using the Riot Games API, which makes it easy to lookup and collect information on a users ranked history and collect their games. However finding a list of usernames is the hard part, in this case I am using a list of usernames scraped from 3rd party LoL sites. | |
| Possible Uses | |
| There is a vast amount of data in just a single LoL game. This dataset takes the most relevant information and makes it available easily for use in things such as attempting to predict the outcome of a LoL game, analysing which in-game events are most likely to lead to victory, understanding how big of an effect bans of a specific champion have, and more." | |
| Meetups data from meetup.com,Organized data for various meetup.com entities along with relational schema,Sumit Kumar,35,"Version 1,2017-12-15","hobbies | |
| social groups | |
| internet",CSV,197 MB,CC0,"2,974 views",399 downloads,,0 topics,https://www.kaggle.com/sirpunch/meetups-data-from-meetupcom,"Summary | |
| “Meetup is a social networking website that aims to brings people together to do, explore, teach and learn the things that help them come alive.” | |
| Meetup allows members to find and join groups unified by a common interest. As of 2017, there are 32 million users with 280 thousand groups available across 182 countries. | |
| A member needs to be able to identify groups and activities which interest them the most to be able to use this platform to network effectively. | |
| The aim of our team was to use this dataset to build a recommender system which will identify and suggest groups and activities to a member based on their interest and additional interests of similar members. Furthermore, a social network analysis was done to identify the relationship between groups and people. | |
| Database EER diagram | |
| Data Collection Method | |
| Data was collected using Meetup API. | |
| Python script was used to ping meetup API and collect responses as JSON objects. | |
| Logical chunks of data were exported and saved as csv files. | |
| Data Cleaning | |
| Data is filtered to include only 3 cities' information (New York, San Francisco, Chicago). | |
| Character encoding is normalized to ASCII characters across tables. | |
| Example Visualizations" | |
| Global Commodity Trade Statistics,Three decades of global trade flows,United Nations,35,"Version 3,2017-11-15|Version 2,2017-11-15|Version 1,2017-11-15","supply chain | |
| economics | |
| shipping",CSV,121 MB,Other,"3,732 views",588 downloads,4 kernels,0 topics,https://www.kaggle.com/unitednations/global-commodity-trade-statistics,"Are you curious about fertilizer use in developing economies? The growth of Chinese steel exports? American chocolate consumption? Which parts of the world still use typewriters? You'll find all of that and more here. This dataset covers import and export volumes for 5,000 commodities across most countries on Earth over the last 30 years. | |
| Acknowledgements | |
| This dataset was kindly published by the United Nations Statistics Division on the UNData site. You can find the original dataset here. | |
| Inspiration | |
| Some of these numbers are more trustworthy than others. I'd expect that British tea imports are fairly accurate, but doubt that Afghanistan exported exactly 51 sheep in 2016. Can you identify which nations appear to have the most trustworthy data? Which industries? | |
| License | |
| Per the UNData terms of use: all data and metadata provided on UNdata’s website are available free of charge and may be copied freely, duplicated and further distributed provided that UNdata is cited as the reference." | |
| Online Job Postings,"Dataset of 19,000 online job posts from 2004 to 2015",Mad Hab,34,"Version 1,2017-04-22","employment | |
| linguistics",CSV,92 MB,Other,"11,063 views","1,795 downloads",5 kernels,0 topics,https://www.kaggle.com/madhab/jobposts,"Job Posts dataset | |
| The dataset consists of 19,000 job postings that were posted through the Armenian human resource portal CareerCenter. The data was extracted from the Yahoo! mailing group https://groups.yahoo.com/neo/groups/careercenter-am. This was the only online human resource portal in the early 2000s. A job posting usually has some structure, although some fields of the posting are not necessarily filled out by the client (poster). The data was cleaned by removing posts that were not job related or had no structure. The data consists of job posts from 2004-2015 | |
| Content | |
| jobpost – The original job post | |
| date – Date it was posted in the group | |
| Title – Job title | |
| Company - employer | |
| AnnouncementCode – Announcement code (some internal code, is usually missing) | |
| Term – Full-Time, Part-time, etc | |
| Eligibility -- Eligibility of the candidates | |
| Audience --- Who can apply? | |
| StartDate – Start date of work | |
| Duration - Duration of the employment | |
| Location – Employment location | |
| JobDescription – Job Description | |
| JobRequirment - Job requirements | |
| RequiredQual -Required Qualification | |
| Salary - Salary | |
| ApplicationP – Application Procedure | |
| OpeningDate – Opening date of the job announcement | |
| Deadline – Deadline for the job announcement | |
| Notes - Additional Notes | |
| AboutC - About the company | |
| Attach - Attachments | |
| Year - Year of the announcement (derived from the field date) | |
| Month - Month of the announcement (derived from the field date) | |
| IT – TRUE if the job is an IT job. This variable is created by a simple search of IT job titles within column “Title” | |
| Acknowledgements | |
| The data collection and initial research was funded by the American University of Armenia’s research grant (2015). | |
| Inspiration | |
| The online job market is a good indicator of overall demand for labor in the local economy. In addition, online job postings data are easier and quicker to collect, and they can be a richer source of information than more traditional job postings, such as those found in printed newspapers. The data can be used in the following ways: -Understand the demand for certain professions, job titles, or industries -Help universities with curriculum development -Identify skills that are most frequently required by employers, and how the distribution of necessary skills changes over time -Make recommendations to job seekers and employers | |
| Past research | |
| We have used association rules mining and simple text mining techniques to analyze the data. Some results can be found here (https://www.slideshare.net/HabetMadoyan/it-skills-analysis-63686238)." | |
| PUBG Match Deaths and Statistics,Over 65 million death entries for PlayerUnknown Battleground's matches,KP,34,"Version 3,2018-01-12|Version 2,2018-01-12|Version 1,2018-01-11","video games | |
| geography | |
| demographics",Other,4 GB,CC0,"8,572 views",604 downloads,2 kernels,4 topics,https://www.kaggle.com/skihikingkevin/pubg-match-deaths,"Introduction | |
| Video games are a rich area for data extraction due to its digital nature. Notable examples such as the complex EVE Online economy, World of Warcraft corrupted blood incident and even Grand Theft Auto self-driving cars tells us that fiction is closer to reality than we really think. Data scientists can gain insight on the logic and decision-making that the players face when put in hypothetical and virtual scenarios. | |
| In this Kaggle Dataset, I provide over 720,000 competitive matches from the popular game PlayerUnknown's Battlegrounds. The data was extracted from pubg.op.gg, a game tracker website. I intend for this data-set to be purely exploratory, however users are free to create their own predictive models they see fit. | |
| PlayerUnknown's Battlegrounds | |
| PUBG is a first/third-person shooter battle royale style game that matches over 90 players on a large island where teams and players fight to the death until one remains. Players are airdropped from an airplane onto the island where they are to scavenge towns and buildings for weapons, ammo, armor and first-aid. Players will then decide to either fight or hide with the ultimate goal of being the last one standing. A bluezone (see below) will appear a few minutes into the game to corral players closer and closer together by dealing damage to anyone that stands within the bluezone and sparing whoever is within the safe zone. | |
| Read more about PUBG here | |
| The Dataset | |
| This dataset provides two zips: aggregate and deaths. | |
| In deaths, the files record every death that occurred within the 720k matches. That is, each row documents an event where a player has died in the match. | |
| In aggregate, each match's meta information and player statistics are summarized (as provided by pubg). It includes various aggregate statistics such as player kills, damage, distance walked, etc as well as metadata on the match itself such as queue size, fpp/tpp, date, etc. | |
| The uncompressed data is divided into 5 chunks of approximately 2gb each. For details on columns, please see the file descriptions. | |
| Interpreting Positional Data | |
| The X,Y coordinates are all in in-game coordinates and need to be linearly scaled to be plotted on square erangel and miramar maps. The coordinate min,max are 0,800000 respectively. | |
| Potential Bias in the Data | |
| The scraping methodology first starts with an initial seed player, I chose this to be my own account (a rather low rank individual). I then use the seed player to scrape for all players that it has encountered in its historical matches. I then take a random subset of 5000 players from this and then scrape for their historical games for the final dataset. What this could produce is an unrepresentative sample of all games played as it is more likely that I queued and matched with lower rated players and those players more than likely also got matched against lower rated players as well. Thus, the matches and deaths are more representative of lower tier gameplay but given the simplicity of the dataset, this shouldn't be an issue. | |
| Acknowledgements | |
| Pubg.op.gg, if this is against the TOS, please let me know and I will take it down" | |
| Audio Cats and Dogs,Classify raw sound events,marc moreaux,34,"Version 5,2017-10-05|Version 4,2017-10-04|Version 3,2017-10-03|Version 2,2017-09-29|Version 1,2017-09-29","animals | |
| acoustics",Other,59 MB,CC3,"6,389 views",486 downloads,9 kernels,,https://www.kaggle.com/mmoreaux/audio-cats-and-dogs,"Context | |
| With this dataset we hope to do a nice cheeky wink to the ""cats and dogs"" image dataset. In fact, this dataset is aimed to be the audio counterpart of the famous ""cats and dogs"" image classification task, here available on Kaggle. | |
| Content | |
| The dataset consists in many ""wav"" files for both the cat and dog classes : | |
| cat has 164 WAV files to which corresponds 1323 sec of audio | |
| dog has 113 WAV files to which corresponds 598 sec of audio | |
| You can have an visual description of the Wav here : Visualizing woofs & meows 🐱. In Accessing the Dataset 2 we propose a train / test split which can be used. | |
| All the WAV files contains 16KHz audio and have variable length. | |
| Acknowledgements | |
| We have not much credit in proposing the dataset here. Much of the work have been done by the AE-Dataset creator (From which we extracted the two classes) and by the humans behind FreeSound From which was extracted the AE-Dataset. | |
| Inspiration | |
| You might use this dataset to test raw audio classification challenge ;) | |
| A more challenging dataset is available here" | |
| Religious and philosophical texts,"5 texts from Project Gutenberg to encourage text-mining, sentiment analysis, etc",Rob Harrand,34,"Version 1,2016-09-07","languages | |
| faith and traditions | |
| linguistics",Other,8 MB,Other,"7,162 views",910 downloads,23 kernels,,https://www.kaggle.com/tentotheminus9/religious-and-philosophical-texts,"These are 5 texts taken from Project Gutenberg, uploaded to Kaggle to encourage things like text-mining and sentiment analysis. These are fun skills to develop and many existing datasets on Kaggle don't lend themselves to these sorts of analyses. | |
| The 5 books are, | |
| The King James Bible | |
| The Quran | |
| The Book Of Mormon | |
| The Gospel of Buddha | |
| Meditations, by Marcus Aurelius | |
| Project Gutenberg is an online archive of books that are free to download and distribute. These files are taken without alteration (filenames or contents) from the Project Gutenberg website" | |
| Blood Cell Images,"12,500 images: 4 different cell types",paultimothymooney,34,"Version 5,2018-01-20|Version 4,2018-01-18|Version 3,2018-01-11|Version 2,2018-01-11|Version 1,2018-01-10",medicine,Other,293 MB,CC0,"2,482 views",349 downloads,,,https://www.kaggle.com/paultimothymooney/blood-cells,"Context | |
| The diagnosis of blood-based diseases often involves identifying and characterizing patient blood samples. Automated methods to detect and classify blood cell subtypes have important medical applications. | |
| Content | |
| This dataset contains 12,500 augmented images of blood cells (JPEG) with accompanying cell type labels (CSV). There are approximately 3,000 images for each of 4 different cell types grouped into 4 different folders (according to cell type). The cell types are Eosinophil, Lymphocyte, Monocyte, and Neutrophil. This dataset is accompanied by an additional dataset containing the original 410 images (pre-augmentation) as well as two additional subtype labels (WBC vs WBC) and also bounding boxes for each cell in each of these 410 images (JPEG + XML metadata). More specifically, the folder 'dataset-master' contains 410 images of blood cells with subtype labels and bounding boxes (JPEG + XML), while the folder 'dataset2-master' contains 2,500 augmented images as well as 4 additional subtype labels (JPEG + CSV). There are approximately 3,000 augmented images for each class of the 4 classes as compared to 88, 33, 21, and 207 images of each in folder 'dataset-master'. | |
| Acknowledgements | |
| https://github.com/Shenggan/BCCD_Dataset MIT License | |
| Inspiration | |
| The diagnosis of blood-based diseases often involves identifying and characterizing patient blood samples. Automated methods to detect and classify blood cell subtypes have important medical applications." | |
| Restaurant Data with Consumer Ratings,Data from a restaurant recommender prototype,UCI Machine Learning,34,"Version 1,2017-09-28",business,CSV,203 KB,CC0,"11,132 views","1,709 downloads",5 kernels,,https://www.kaggle.com/uciml/restaurant-data-with-consumer-ratings,"Context | |
| This dataset was used for a study where the task was to generate a top-n list of restaurants according to the consumer preferences and finding the significant features. Two approaches were tested: a collaborative filter technique and a contextual approach: (i) The collaborative filter technique used only one file i.e., rating_final.csv that comprises the user, item and rating attributes. (ii) The contextual approach generated the recommendations using the remaining eight data files. | |
| Content | |
| There are 9 data files and a README, and are grouped like this: | |
| Restaurants | |
| 1 chefmozaccepts.csv | |
| 2 chefmozcuisine.csv | |
| 3 chefmozhours4.csv | |
| 4 chefmozparking.csv | |
| 5 geoplaces2.csv | |
| Consumers | |
| 6 usercuisine.csv | |
| 7 userpayment.csv | |
| 8 userprofile.csv | |
| User-Item-Rating | |
| 9 rating_final.csv | |
| More detailed file descriptions can also be found in the README: | |
| 1 chefmozaccepts.csv | |
| Instances: 1314 | |
| Attributes: 2 | |
| placeID: Nominal | |
| Rpayment: Nominal, 12 | |
| 2 chefmozcuisine.csv | |
| Instances: 916 | |
| Attributes: 2 | |
| placeID: Nominal | |
| Rcuisine: Nominal, 59 | |
| 3 chefmozhours4.csv | |
| Instances: 2339 | |
| Attributes: 3 | |
| placeID: Nominal | |
| hours: Nominal, Range:00:00-23:30 | |
| days: Nominal, 7 | |
| 4 chefmozparking.csv | |
| Instances: 702 | |
| Attributes: 2 | |
| placeID: Nominal | |
| parking_lot: Nominal, 7 | |
| 5 geoplaces2.csv | |
| Instances: 130 | |
| Attributes: 21 | |
| placeID: Nominal | |
| latitude: Numeric | |
| longitude: Numeric | |
| the_geom_meter: Nominal (Geospatial) | |
| name: Nominal | |
| address: Nominal,Missing: 27 | |
| city: Nominal, Missing: 18 | |
| state: Nominal, Missing: 18 | |
| country: Nominal, Missing: 28 | |
| fax: Numeric, Missing: 130 | |
| zip: Nominal,Missing: 74 | |
| alcohol: Nominal, Values: 3 | |
| smoking_area: Nominal, 5 | |
| dress_code: Nominal, 3 | |
| accessibility: Nominal, 3 | |
| price: Nominal, 3 | |
| url: Nominal, Missing: 116 | |
| Rambience: Nominal, 2 | |
| franchise: Nominal, 2 | |
| area: Nominal, 2 | |
| other_services: Nominal, 3 | |
| 6 rating_final.csv | |
| Instances: 1161 | |
| Attributes: 5 | |
| userID: Nominal | |
| placeID: Nominal | |
| rating: Numeric, 3 | |
| food_rating: Numeric, 3 | |
| service_rating: Numeric, 3 | |
| 7 usercuisine.csv | |
| Instances: 330 | |
| Attributes: 2 | |
| userID: Nominal | |
| Rcuisine: Nominal, 103 | |
| 8 userpayment.csv | |
| Instances: 177 | |
| Attributes: 2 | |
| userID: Nominal | |
| Upayment: Nominal, 5 | |
| 9 userprofile | |
| Instances: 138 | |
| Attributes: 19 | |
| userID: Nominal | |
| latitude: Numeric | |
| longitude: Numeric | |
| the_geom_meter: Nominal (Geospatial) | |
| smoker: Nominal | |
| drink_level: Nominal, 3 | |
| dress_preference:Nominal, 4 | |
| ambience: Nominal, 3 | |
| transport: Nominal, 3 | |
| marital_status: Nominal, 3 | |
| hijos: Nominal, 3 | |
| birth_year: Nominal | |
| interest: Nominal, 5 | |
| personality: Nominal, 4 | |
| religion: Nominal, 5 | |
| activity: Nominal, 4 | |
| color: Nominal, 8 | |
| weight: Numeric | |
| budget: Nominal, 3 | |
| height: Numeric | |
| Acknowledgements | |
| This dataset was originally downloaded from the UCI ML Repository: UCI ML | |
| Creators: Rafael Ponce Medellín and Juan Gabriel González Serna rafaponce@cenidet.edu.mx, gabriel@cenidet.edu.mx Department of Computer Science. National Center for Research and Technological Development CENIDET, México | |
| Donors of database: Blanca Vargas-Govea and Juan Gabriel González Serna blanca.vargas@cenidet.edu.mx/blanca.vg@gmail.com, gabriel@cenidet.edu.mx Department of Computer Science. National Center for Research and Technological Development CENIDET, México | |
| Inspiration | |
| Use this data to create a restaurant recommender or determine which restaurants a person is most likely to visit." | |
| International Datasets,International health and population metrics,US Census Bureau,34,"Version 1,2017-06-28","demographics | |
| international relations",CSV,2 GB,Other,"7,151 views","1,184 downloads",4 kernels,0 topics,https://www.kaggle.com/census/international-data,"Content | |
| The United States Census Bureau’s International Dataset provides estimates of country populations since 1950 and projections through 2050. Specifically, the data set includes midyear population figures broken down by age and gender assignment at birth. Additionally, they provide time-series data for attributes including fertility rates, birth rates, death rates, and migration rates. | |
| The full documentation is available here. For basic field details, please see the data dictionary. | |
| Note: The U.S. Census Bureau provides estimates and projections for countries and areas that are recognized by the U.S. Department of State that have a population of at least 5,000. | |
| Acknowledgements | |
| This dataset was created by the United States Census Bureau. | |
| Inspiration | |
| Which countries have made the largest improvements in life expectancy? Based on current trends, how long will it take each country to catch up to today’s best performers? | |
| Use this dataset with BigQuery | |
| You can use Kernels to analyze, share, and discuss this data on Kaggle, but if you’re looking for real-time updates and bigger data, check out the data on BigQuery, too: https://cloud.google.com/bigquery/public-data/international-census." | |
| "Hurricanes and Typhoons, 1851-2014","Location, wind, and pressure of tropical cyclones in Atlantic and Pacific Oceans",NOAA,34,"Version 1,2017-01-21",weather,CSV,9 MB,CC0,"7,344 views","1,309 downloads",7 kernels,0 topics,https://www.kaggle.com/noaa/hurricane-database,"Context | |
| The National Hurricane Center (NHC) conducts a post-storm analysis of each tropical cyclone in the Atlantic basin (i.e., North Atlantic Ocean, Gulf of Mexico, and Caribbean Sea) and and the North Pacific Ocean to determine the official assessment of the cyclone's history. This analysis makes use of all available observations, including those that may not have been available in real time. In addition, NHC conducts ongoing reviews of any retrospective tropical cyclone analyses brought to its attention and on a regular basis updates the historical record to reflect changes introduced. | |
| Content | |
| The NHC publishes the tropical cyclone historical database in a format known as HURDAT, short for HURricane DATabase. These databases (Atlantic HURDAT2 and NE/NC Pacific HURDAT2) contain six-hourly information on the location, maximum winds, central pressure, and (starting in 2004) size of all known tropical cyclones and subtropical cyclones." | |
| Employee Attrition,Can you forecast employee attrition?,People HR Analytics Repository,34,"Version 2,2017-04-27|Version 1,2017-01-14","employment | |
| business",CSV,7 MB,CC0,"22,008 views","2,429 downloads",20 kernels,3 topics,https://www.kaggle.com/HRAnalyticRepository/employee-attrition-data,"Context | |
| This data was originally posted on my personal oneDrive account. | |
| It represent fictitious/fake data on terminations. For each of 10 years it show employees that are active and those that terminated. | |
| The intent is to see if individual terminations can be predicted from the data provided. | |
| The thing to be predicted is status of active or terminated | |
| Content | |
| The data contains | |
| employee id employee record date ( year of data) birth date hire date termination date age length of service city department job title store number gender termination reason termination type status year status business unit | |
| These might be typical types of data in hris | |
| Acknowledgements | |
| None- its fake data | |
| Inspiration | |
| A lot of turnover analyses occur at an aggregate level-such as turnover rates. But few analyses concentrate on trying to identify exactly which individuals might leave based on patterns that might be present in existing data. | |
| Machine learning algorithms often showcase customer churn examples for telcos or product marketing. Those algorithms equally apply to employee churn." | |
| IMDB data from 2006 to 2016,"A data set of 1,000 popular movies on IMDB in the last 10 years",PromptCloud,34,"Version 1,2017-06-26",film,CSV,303 KB,Other,"15,077 views","2,108 downloads",8 kernels,,https://www.kaggle.com/PromptCloudHQ/imdb-data,"Here's a data set of 1,000 most popular movies on IMDB in the last 10 years. The data points included are: | |
| Title, Genre, Description, Director, Actors, Year, Runtime, Rating, Votes, Revenue, Metascrore | |
| Feel free to tinker with it and derive interesting insights." | |
| US Gross Rent ACS Statistics,"+40,000 Samples: Real Estate Application (Mean, Median, Standard Deviation)",Golden Oak Research Group,34,"Version 3,2017-08-23|Version 2,2017-08-21|Version 1,2017-08-21","finance | |
| real estate | |
| demographics",Other,5 MB,Other,"2,632 views",444 downloads,3 kernels,,https://www.kaggle.com/goldenoakresearch/acs-gross-rent-us-statistics,"What you get: | |
| Upvote! The database contains +40,000 records on US Gross Rent & Geo Locations. The field description of the database is documented in the attached pdf file. To access, all 325,272 records on a scale roughly equivalent to a neighborhood (census tract) see link below and make sure to upvote. Upvote right now, please. Enjoy! | |
| Get the full free database with coupon code: FreeDatabase, See directions at the bottom of the description... And make sure to upvote :) coupon ends at 2:00 pm 8-23-2017 | |
| Gross Rent & Geographic Statistics: | |
| Mean Gross Rent (double) | |
| Median Gross Rent (double) | |
| Standard Deviation of Gross Rent (double) | |
| Number of Samples (double) | |
| Square area of land at location (double) | |
| Square area of water at location (double) | |
| Geographic Location: | |
| Longitude (double) | |
| Latitude (double) | |
| State Name (character) | |
| State abbreviated (character) | |
| State_Code (character) | |
| County Name (character) | |
| City Name (character) | |
| Name of city, town, village or CPD (character) | |
| Primary, Defines if the location is a track and block group. | |
| Zip Code (character) | |
| Area Code (character) | |
| Abstract | |
| The data set originally developed for real estate and business investment research. Income is a vital element when determining both quality and socioeconomic features of a given geographic location. The following data was derived from over +36,000 files and covers 348,893 location records. | |
| License | |
| Only proper citing is required please see the documentation for details. Have Fun!!! | |
| Golden Oak Research Group, LLC. “U.S. Income Database Kaggle”. Publication: 5, August 2017. Accessed, day, month year. | |
| For any questions, you may reach us at research_development@goldenoakresearch.com. For immediate assistance, you may reach me on at 585-626-2965 | |
| please note: it is my personal number and email is preferred | |
| Check our data's accuracy: Census Fact Checker | |
| Access all 325,272 location for Free Database Coupon Code: | |
| Don't settle. Go big and win big. Optimize your potential**. Access all gross rent records and more on a scale roughly equivalent to a neighborhood, see link below: | |
| Website: Golden Oak Research make sure to upvote | |
| A small startup with big dreams, giving the every day, up and coming data scientist professional grade data at affordable prices It's what we do." | |
| Tweets Targeting Isis,General tweets about Isis & related words,ActiveGalaXy,33,"Version 3,2016-07-30|Version 2,2016-07-29|Version 1,2016-07-27","crime | |
| internet",CSV,29 MB,CC0,"11,563 views","1,016 downloads",26 kernels,3 topics,https://www.kaggle.com/activegalaxy/isis-related-tweets,"Context | |
| The image at the top of the page is a frame from today's (7/26/2016) Isis #TweetMovie from twitter, a ""normal"" day when two Isis operatives murdered a priest saying mass in a French church. (You can see this in the center left). A selection of data from this site is being made available here to Kaggle users. | |
| UPDATE: An excellent study by Audrey Alexander titled Digital Decay? is now available which traces the ""change over time among English-language Islamic State sympathizers on Twitter. | |
| Intent | |
| This data set is intended to be a counterpoise to the How Isis Uses Twitter data set. That data set contains 17k tweets alleged to originate with ""100+ pro-ISIS fanboys"". This new set contains 122k tweets collected on two separate days, 7/4/2016 and 7/11/2016, which contained any of the following terms, with no further editing or selection: | |
| isis | |
| isil | |
| daesh | |
| islamicstate | |
| raqqa | |
| Mosul | |
| ""islamic state"" | |
| This is not a perfect counterpoise as it almost surely contains a small number of pro-Isis fanboy tweets. However, unless some entity, such as Kaggle, is willing to expend significant resources on a service something like an expert level Mechanical Turk or Zooniverse, a high quality counterpoise is out of reach. | |
| A counterpoise provides a balance or backdrop against which to measure a primary object, in this case the original pro-Isis data. So if anyone wants to discriminate between pro-Isis tweets and other tweets concerning Isis you will need to model the original pro-Isis data or signal against the counterpoise which is signal + noise. Further background and some analysis can be found in this forum thread. | |
| This data comes from postmodernnews.com/token-tv.aspx which daily collects about 25MB of Isis tweets for the purposes of graphical display. PLEASE NOTE: This server is not currently active. | |
| Data Details | |
| There are several differences between the format of this data set and the pro-ISIS fanboy dataset. 1. All the twitter t.co tags have been expanded where possible 2. There are no ""description, location, followers, numberstatuses"" data columns. | |
| I have also included my version of the original pro-ISIS fanboy set. This version has all the t.co links expanded where possible." | |
| Datascience Universities across US,Contains datascience programs offered and location data by university.,SrihariRao,33,"Version 1,2016-09-25","education | |
| information technology",CSV,342 KB,ODbL,"9,630 views","1,466 downloads",8 kernels,,https://www.kaggle.com/sriharirao/datascience-universities-across-us,"Contains DataScience programs offered by universities along with program details, world ranking and a lot lot more. Happy exploring !!!" | |
| TechCrunch Posts Compilation,40k compiled posts with a rich set of features will boost your visualizations,Thiago Balbo,33,"Version 1,2016-10-19","linguistics | |
| internet",CSV,136 MB,CC0,"5,740 views",505 downloads,34 kernels,,https://www.kaggle.com/thibalbo/techcrunch-posts-compilation,"Inspiration | |
| I'm a big fan of TechCrunch for a while now. Kind of because I get to know about new startups that's coming up or maybe just because I find Tito Hamze videos fun. But TechCrunch got plenty of good content. And where we find good content we produce great exploratory analysis. | |
| This dataset is a great opportunity for you to boost your skills as an EDA expert! It provides several features that make you able to create different analyses such as time series, clustering, predictive, segmenting, classification and tons of others. Let's not forget about word2vec for that. It would be awesome to see that in action here! | |
| I've made the scraper available on github, if you want to check it out, here is the link: techcrunch scraper repo | |
| Content | |
| This dataset comes with a rich set of features. You will have: | |
| authors: authors of the post - can be one or multiple authors | |
| category: post category | |
| content: post content - each paragraph can be extracted by splitting on the \n | |
| date: post date | |
| id: post id - the same id used on techcrunch website | |
| img_src: post main image url | |
| section: post section - each section is one of the options on the main page dropdown menu | |
| tags: post tags - can be zero or multiple tags | |
| title: post title | |
| topics: post topics | |
| url: post url | |
| Acknowledgements | |
| All posts were scraped from the TechCrunch website on mid oct-16. Each line contains information about one post and each post appear in no more than one line." | |
| Short Jokes,"Collection of over 200,000 short jokes for humour research",Abhinav Moudgil,33,"Version 1,2017-02-07","humor | |
| linguistics",CSV,23 MB,ODbL,"9,366 views",916 downloads,5 kernels,0 topics,https://www.kaggle.com/abhinavmoudgil95/short-jokes,"Context | |
| Generating humor is a complex task in the domain of machine learning, and it requires the models to understand the deep semantic meaning of a joke in order to generate new ones. Such problems, however, are difficult to solve due to a number of reasons, one of which is the lack of a database that gives an elaborate list of jokes. Thus, a large corpus of over 0.2 million jokes has been collected by scraping several websites containing funny and short jokes. | |
| Visit my Github repository for more information regarding collection of data and the scripts used. | |
| Content | |
| This dataset is in the form of a csv file containing 231,657 jokes. Length of jokes ranges from 10 to 200 characters. Each line in the file contains a unique ID and joke. | |
| Disclaimer | |
| It has been attempted to keep the jokes as clean as possible. Since the data has been collected by scraping websites, it is possible that there may be a few jokes that are inappropriate or offensive to some people." | |
| Spotify Song Attributes,An attempt to build a classifier that can predict whether or not I like a song,GeorgeMcIntire,33,"Version 1,2017-08-05",,CSV,217 KB,Other,"13,177 views","1,517 downloads",7 kernels,,https://www.kaggle.com/geomack/spotifyclassification,"Context | |
| A dataset of 2017 songs with attributes from Spotify's API. Each song is labeled ""1"" meaning I like it and ""0"" for songs I don't like. I used this to data to see if I could build a classifier that could predict whether or not I would like a song. | |
| I wrote an article about the project I used this data for. It includes code on how to grab this data from the Spotipy API wrapper and the methods behind my modeling. https://opendatascience.com/blog/a-machine-learning-deep-dive-into-my-spotify-data/ | |
| Content | |
| Each row represents a song. | |
| There are 16 columns. 13 of which are song attributes, one column for song name, one for artist, and a column called ""target"" which is the label for the song. | |
| Here are the 13 track attributes: acousticness, danceability, duration_ms, energy, instrumentalness, key, liveness, loudness, mode, speechiness, tempo, time_signature, valence. | |
| Information on what those traits mean can be found here: https://developer.spotify.com/web-api/get-audio-features/ | |
| Acknowledgements | |
| I would like to thank Spotify for providing this readily accessible data. | |
| Inspiration | |
| I'm a music lover who's curious about why I love the music that I love." | |
| PLAYERUNKNOWN'S BATTLEGROUNDS Player Statistics,Player statistics for PUBG. 150 features per player.,JustinMoore,33,"Version 1,2017-08-01","games and toys | |
| video games",CSV,62 MB,CC0,"7,833 views",711 downloads,8 kernels,2 topics,https://www.kaggle.com/lazyjustin/pubgplayerstats,"Context | |
| Grab your pans... | |
| Player statistics for approximately 85,000 of the top PUBG players (as tracked by https://pubgtracker.com/). All statistics were gathered using aggregate region filters (all regions) and feature labels are subdivided by server type: solo, duo, and squad. | |
| 87,898 players with 150 numerical game-play features per player (+2 for player name and PUBG Tracker ID). | |
| Content | |
| Features include KD ratios, wins, losses, damage, wins, top 10's, and movement characteristics (walking/riding distance etc...) | |
| Acknowledgements | |
| Special thanks to pubgtracker.com for their support and aid with gathering this data. More information can be found here: https://pubgtracker.com/ | |
| PLAYERUNKNOWN'S BATTLEGROUNDS is a registered trademark, trademark or service mark of Bluehole, Inc. and its affiliates https://www.playbattlegrounds.com/main.pu | |
| Inspiration | |
| As a gamer addicted to PUBG, it was a blast putting this data set together. Some great project ideas include: | |
| a. Visualizations of player skill vs. specific strategies | |
| b. Unsupervised clustering of players based on strategy (for matchmaking or team building) | |
| c. Prediction of features based on player skill and/or strategies" | |
| NBA Enhanced Box Score and Standings Stats,Box Scores and Standings with advanced calculations applied,Paul Rossotti,33,"Version 19,2018-02-26|Version 18,2018-02-18|Version 17,2018-02-11|Version 16,2018-02-03|Version 15,2018-01-28|Version 14,2018-01-21|Version 13,2018-01-13|Version 12,2018-01-06|Version 11,2018-01-06|Version 10,2018-01-06|Version 9,2017-12-31|Version 8,2017-12-24|Version 7,2017-12-16|Version 6,2017-12-14|Version 5,2017-11-18|Version 4,2017-11-13|Version 3,2017-11-12|Version 2,2017-11-11|Version 1,2017-11-11","basketball | |
| sports",CSV,6 MB,CC4,"3,648 views",556 downloads,6 kernels,0 topics,https://www.kaggle.com/pablote/nba-enhanced-stats,"Context | |
| Dataset is based on box score and standing statistics from the NBA. | |
| Calculations such as number of possessions, floor impact counter, strength of schedule, and simple rating system are performed. | |
| Finally, extracts are created based on a perspective: | |
| teamBoxScore.csv communicates game data from each teams perspective | |
| officialBoxScore.csv communicates game data from each officials perspective | |
| playerBoxScore.csv communicates game data from each players perspective | |
| standing.csv communicates standings data for each team every day during the season | |
| Content | |
| Data Sources | |
| Box score and standing statistics were obtained by a Java application using RESTful APIs provided by xmlstats. | |
| Calculation Sources | |
| Another Java application performs advanced calculations on the box score and standing data. | |
| Formulas for these calculations were primarily obtained from these sources: | |
| https://basketball.realgm.com/info/glossary | |
| https://www.nbastuffer.com/team-evaluation-metrics/ | |
| https://www.basketball-reference.com/about/glossary.html | |
| Inspiration | |
| Favoritism | |
| Does a referee impact the number of fouls made against a player or the pace of a game? | |
| Forcasting | |
| Can the aggregated points scored by and against a team along with their strength of schedule be used to determine their projected winning percentage for the season? | |
| Predicting the Past | |
| For a given game, can games played earlier in the season help determine how a team will perform? | |
| Lots of data elements and possibilities. Let your imagination roam!" | |
| 350 000+ movies from themoviedb.org,More than 350k movies and main casting/crew up to Aug17,Stephanerappeneau,33,"Version 8,2017-10-13|Version 7,2017-10-13|Version 6,2017-10-13|Version 5,2017-10-13|Version 4,2017-10-05|Version 3,2017-09-23|Version 2,2017-09-20|Version 1,2017-09-17","arts and entertainment | |
| film | |
| actors",CSV,192 MB,Other,"6,906 views",942 downloads,5 kernels,6 topics,https://www.kaggle.com/stephanerappeneau/350-000-movies-from-themoviedborg,"Context | |
| I love movies. | |
| I tend to avoid marvel-transformers-standardized products, and prefer a mix of classic hollywood-golden-age and obscure polish artsy movies. Throw in an occasional japanese-zombie-slasher-giallo as an alibi. Good movies don't exist without bad movies. | |
| On average I watch 200+ movies each year, with peaks at more than 500 movies. Nine years ago I started to log my movies to avoid watching the same movie twice, and also assign scores. Over the years, it gave me a couple insights on my viewing habits but nothing more than what a tenth-grader would learn at school. | |
| I've recently suscribed to Netflix and it pains me to see the global inefficiency of recommendation systems for people like me, who mostly swear by ""La politique des auteurs"". It's a term coined by famous new-wave french movie critic André Bazin, meaning that the quality of a movie is essentially linked to the director and it's capacity to execute his vision with his crew. We could debate it depends on movie production pipeline, but let's not for now. Practically, what it means, is that I essentially watch movies from directors who made films I've liked. | |
| I suspect Neflix calibrate their recommandation models taking into account the way the ""average-joe"" chooses a movie. A few months ago I had read a study based on a survey, showing that people chose a movie mostly based on genre (55%), then by leading actors (45%). Director or Release Date were far behind around 10% each. It is not surprising, since most people I know don't care who the director is. Lots of US blockbusters don't even mention it on the movie poster. I am aware that collaborative filtering is based on user proximity , which I believe decreases (or even eliminates) the need to characterize a movie. So here I'm more interested in content based filtering which is based on product proximity for several reasons : | |
| Users tastes are not easily accessible. It is, after all, Netflix treasure chest | |
| Movie offer on Netflix is so bad for someone who likes author's movies that it wouldn't help | |
| Modeling a movie intrinsic qualities is a nice challenge | |
| Enough. | |
| ""The secret of getting ahead is getting started"" (Mark Twain) | |
| Content | |
| The primary source is www.themoviedb.org. If you watch obscure artsy romanian homemade movies you may find only 95% of your movies referenced...but for anyone else it should be in the 98%+ range. | |
| movies details are from www.themoviedb.org API : movies/details | |
| movies crew & casting are from www.themoviedb.org API : movies/credits | |
| both can be joined by id | |
| they contain all 350k movies up, from end of 19th century to august 2017. If you remove short movies from imdb you get similar amounts of movies. | |
| I uploaded the program to retrieve incremental movie details on github : https://github.com/stephanerappeneau/scienceofmovies/tree/master/PycharmProjects/GetAllMovies (need a dev API key from themoviedb.org though) | |
| I have tried various supervised (decision tree) / unsupervised (clustering, NLP) approaches described in the discussions, source code is on github : https://github.com/stephanerappeneau/scienceofmovies | |
| As a bonus I've uploaded the bio summary from top 500 critically-acclaimed directors from wikipedia, for some interesting NLTK analysis | |
| Here is overview of the available sources that I've tried : | |
| • Imdb.com free csv dumps (ftp://ftp.funet.fi/pub/mirrors/ftp.imdb.com/pub/temporaryaccess/) are badly documented, incomplete, loosely structured and impossible to join/merge. There's an API hosted by Amazon Web Service : 1€ every 100 000 requests. With around 1 million movies, it could become expensive also features are bare. So I've searched for other sources. | |
| • www.themoviedb.org is based on crowdsourcing and has an excellent API, limited to 40 requests every 10 seconds. It is quite generous, well documented, and enough to sweep the 450 000 movies in a few days. For my purpose, data quality is not significantly worse than imdb, and as imdb key is also included there's always the possibility to complete my dataset later (I actually did it) | |
| • www.Boxofficemojo.com has some interesting budget/revenue figures (which are sorely lacking in both imdb & tmdb), but it actually tracks only a few thousand movies, mainly blockbusters. There are other professional sources that are used by film industry to get better predictive / marketing insights but that's beyond my reach for this experiment. • www.wikipedia.com is an interesting source with no real cap on API calls, however it requires a bit of webscraping and for movies or directors the layout and quality varies a lot, so I suspected it'd get a lot of work to get insights so I put this source in lower priority. | |
| • www.google.com will ban you after a few minutes of web scraping because their job is to scrap data from others, than sell it, duh. • It's worth mentionning that there are a few dumps of Netflix anonymized user tastes on kaggle, because they've organised a few competitions to improve their recommendation models. https://www.kaggle.com/netflix-inc/netflix-prize-data | |
| • Online databases are largely white anglo-saxon centric, meaning bollywood (India is the 2nd bigger producer of movies) offer is mostly absent from datasets. I'm fine with that, as it's not my cup of tea plus I lack domain knowledge. The sheer amount of indian movies would probably skew my results anyway (I don't want to have too many martial-arts-musicals in my recommendations ;-)). I have, however, tremendous respect for indian movie industry so I'd love to collaborate with an indian cinephile ! | |
| Inspiration | |
| Starting from there, I had multiple problem statements for both supervised / unsupervised machine learning | |
| Can I program a tailored-recommendation system based on my own criteria ? | |
| What are the characteristics of movies/directors I like the most ? | |
| What is the probability that I will like my next movie ? | |
| Can I find the data ? | |
| One of the objectives of sharing my work here is to find cinephile data-scientists who might be interested and, hopefully, contribute or share insights :) Other interesting leads : use tagline for NLP/Clustering/Genre guessing, leverage on budget/revenue, link with other data sources using the imdb normalized title, etc. | |
| Motivation, Disclaimer and Acknowledgements | |
| I've graduated from an french engineering school, majoring in artificial intelligence, but that was 17 years ago right in the middle of A.I-winter. Like a lot of white male rocket scientists, I've ended up in one of the leading european investment bank, quickly abandonning IT development to specialize in trading/risk project management and internal politics. My recent appointment in the Data Office made me aware of recent breakthroughts in datascience, and I thought that developing a side project would be an excellent occasion to learn something new. Plus it'd give me a well-needed credibility which too often lack decision makers when it comes to datascience. | |
| I've worked on some of the features with Cédric Paternotte, a fellow friend of mine who is a professor of philosophy of sciences in La Sorbonne. Working with someone with a different background seem a good idea for motivation, creativity and rigor. | |
| Kudos to www.themoviedb.org or www.wikipedia.com sites, who really have a great attitude towards open data. This is typically NOT the case of modern-bigdata companies who mostly keep data to themselves to try to monetize it. Such a huge contrast with imdb or instagram API, which generously let you grab your last 3 comments at a miserable rate. Even if 15 years ago this seemed a mandatory path to get services for free, I predict one day governments will need to break this data monopoly. | |
| [Disclaimer : I apologize in advance for my engrish (I'm french ^-^), any bad-code I've written (there are probably hundreds of way to do it better and faster), any pseudo-scientific assumption I've made, I'm slowly getting back in statistics and lack senior guidance, one day I regress a non-stationary time series and the day after I'll discover I shouldn't have, and any incorrect use of machine-learning models]" | |
| Crime in Los Angeles,Crime data from 2010 through September 2017,City of Los Angeles,33,"Version 7,2017-09-18|Version 6,2017-09-13|Version 5,2017-09-13|Version 4,2017-09-13|Version 3,2017-09-12|Version 2,2017-09-12|Version 1,2017-09-07",crime,Other,360 MB,CC0,"10,038 views","1,343 downloads",9 kernels,4 topics,https://www.kaggle.com/cityofLA/crime-in-los-angeles,"This dataset reflects incidents of crime in the City of Los Angeles dating back to 2010. This data is transcribed from original crime reports that are typed on paper and therefore there may be some inaccuracies within the data. Some location fields with missing data are noted as (0°, 0°). Address fields are only provided to the nearest hundred block in order to maintain privacy. | |
| Reporting District Shapefile Attributes | |
| REPDIST, Number, min: 101 max: 2,199 avg: 1,162 count: 1,135 | |
| PREC, Number, min: 1 max: 21 avg: 11 count: 1,135 | |
| APREC, Text, PACIFIC (74), DEVONSHIRE (70), WEST LOS ANGELES (69), NORTHEAST (64), HOLLENBECK (63), MISSION (62)... (15 more) | |
| BUREAU, Text, VALLEY BUREAU (399), WEST BUREAU (288), CENTRAL BUREAU (267), SOUTH BUREAU (181) | |
| BASICCAR, Text, 8A29 (17), 17A35 (17), 1A1 (15), 17A49 (14), 16A35 (14), 14A73 (14), 19A43 (13), 8A95 (12), 19A7 (12)... (160 more) | |
| TOOLTIP, Text, Bureau: SOUTH BUREAU\nDistrict: 562\nDivision: HARBOR (1)... (1134 more) | |
| OBJECTID | |
| Unique ID | |
| Acknowledgements | |
| This dataset was kindly released by the City of Los Angeles. You can find the original dataset, updated weekly, here. | |
| Inspiration | |
| Some of the MO codes seem unlikely or unrelated to crime. Can you find out what would lead to the use of code 0107 God or 1021 Repair?" | |
| Y Combinator Companies,Publicly launched YC companies funded from summer 2005 to summer 2016,Ben Hamner,32,"Version 2,2016-09-08|Version 1,2016-09-08","business | |
| finance",CSV,122 KB,Other,"9,461 views",833 downloads,27 kernels,0 topics,https://www.kaggle.com/benhamner/y-combinator-companies,"Data scraped from www.ycombinator.com/companies on September 8, 2016." | |
| My Settlers of Catan Games,"Starting positions, roll distributions, postgame statistics of 50 4-player games",Lumin,32,"Version 1,2016-08-30",board games,CSV,16 KB,CC4,"9,045 views",555 downloads,24 kernels,,https://www.kaggle.com/lumins/settlers-of-catan-games,"The strategic board game The Settlers of Catan is a modern classic. Introduced in 1995, it has sold over 22 million copies worldwide. Learning how to play the game well requires an inherent understanding of probability, economics, game theory, and social interactions. | |
| This is my personal dataset of 50 4-player games I played on playcatan.com in 2014. Using the ingame statistics page and a spreadsheet, I logged starting position choices, the distribution of dice rolls, and how each player spent the resources they acquired by the end of the game. Note, of course, because this dataset only consists of my games, any analysis done is most relevant for games involving me... | |
| My personal analysis of this dataset consisted of a best subsets regression, and resulted a 4-variable model that likely overfitted, but managed to ascertain the winner correctly, in 40 of 50 games. | |
| Questions to possibly consider: | |
| How much luck is involved in winning a Catan game? | |
| Does starting position matter? If so, what starting settlements lead to success from each position? | |
| How much information on the eventual winner can be gained from starting position/settlements alone? | |
| By looking at postgame stats, what leads to a win? Can these statistics be a guide for ingame strategy? | |
| Data details/guide: | |
| gameNum - each game I played has 4 corresponding rows, 1 per player. | |
| player - the starting position corresponding to each row | |
| points - how many points the player ended the game with (the game is won with 10 or more) | |
| me - the position I played during the game | |
| 2, 3, ..., 12 - how many rolls of each value occurred during the game (game is played with 2 dice) | |
| settlement1, settlement2 - each starting settlement is logged as 3 pairs of [number, resource]: | |
| L = lumber | |
| C = clay | |
| S = sheep | |
| W = wheat | |
| O = ore | |
| 3G = 3:1 general port | |
| 2(X) = 2:1 port for resource X | |
| D = desert | |
| EX: in game 1, player 1's first settlement was on a 6-lumber, 3-clay, and 11-clay. | |
| production - total cards gained from settlements and cities during game | |
| tradeGain - total cards gained from peer AND bank trades during game | |
| robberCardsGain - total cards gained from stealing with the robber, plus cards gained with non-knight development cards. A road building card is +4 resources. | |
| totalGain - sum of previous 3 columns. | |
| tradeLoss - total cards lost from peer AND bank trades during game | |
| robberCardsLoss - total cards lost from robbers, knights, and other players' monopoly cards | |
| tribute - total cards lost when player had to discard on a 7 roll (separate from previous column.) | |
| totalLoss - sum of previous 3 columns. | |
| totalAvailable - totalGain minus totalLoss. | |
| I only ask that if you produce a good model, you share it with me! Please don't hesitate to ask any clarifying questions." | |
| Twitter Friends,40k full Twitter user profile data (including who they follow!),Hubert Wassner,32,"Version 4,2016-09-02|Version 3,2016-09-02|Version 2,2016-09-02|Version 1,2016-08-31","twitter | |
| internet",CSV,428 MB,CC4,"5,924 views",578 downloads,5 kernels,,https://www.kaggle.com/hwassner/TwitterFriends,"Twitter Friends and hashtags | |
| Context | |
| This datasets is an extract of a wider database aimed at collecting Twitter user's friends (other accound one follows). The global goal is to study user's interest thru who they follow and connection to the hashtag they've used. | |
| Content | |
| It's a list of Twitter user's informations. In the JSON format one twitter user is stored in one object of this more that 40.000 objects list. Each object holds : | |
| avatar : URL to the profile picture | |
| followerCount : the number of followers of this user | |
| friendsCount : the number of people following this user. | |
| friendName : stores the @name (without the '@') of the user (beware this name can be changed by the user) | |
| id : user ID, this number can not change (you can retrieve screen name with this service : https://tweeterid.com/) | |
| friends : the list of IDs the user follows (data stored is IDs of users followed by this user) | |
| lang : the language declared by the user (in this dataset there is only ""en"" (english)) | |
| lastSeen : the time stamp of the date when this user have post his last tweet. | |
| tags : the hashtags (whith or without #) used by the user. It's the ""trending topic"" the user tweeted about. | |
| tweetID : Id of the last tweet posted by this user. | |
| You also have the CSV format which uses the same naming convention. | |
| These users are selected because they tweeted on Twitter trending topics, I've selected users that have at least 100 followers and following at least 100 other account (in order to filter out spam and non-informative/empty accounts). | |
| Acknowledgements | |
| This data set is build by Hubert Wassner (me) using the Twitter public API. More data can be obtained on request (hubert.wassner AT gmail.com), at this time I've collected over 5 milions in different languages. Some more information can be found here (in french only) : http://wassner.blogspot.fr/2016/06/recuperer-des-profils-twitter-par.html | |
| Past Research | |
| No public research have been done (until now) on this dataset. I made a private application which is described here : http://wassner.blogspot.fr/2016/09/twitter-profiling.html (in French) which uses the full dataset (Millions of full profiles). | |
| Inspiration | |
| On can analyse a lot of stuff with this datasets : | |
| stats about followers & followings | |
| manyfold learning or unsupervised learning from friend list | |
| hashtag prediction from friend list | |
| Contact | |
| Feel free to ask any question (or help request) via Twitter : @hwassner | |
| Enjoy! ;)" | |
| What is a note?,A collection of notes played on a guitar,Melvyn Drag,32,"Version 4,2016-11-23|Version 3,2016-11-20|Version 2,2016-11-19|Version 1,2016-11-17",music,Other,14 MB,CC0,"5,866 views",382 downloads,15 kernels,,https://www.kaggle.com/juliancienfuegos/what-is-a-note,"Overview | |
| We could take a music theory class to understand what a note is, but why don't we just find out for ourselves? In this data set we have the notes on a guitar on open strings, and on the 1st-8th frets on every string. The notes were recorded on a nice but low-end guitar called the Yamaha C-40. The guitar is in standard tuning (from the top string to the bottom on we have E low, A, D, G, B, E high). | |
| What to do with this dataset | |
| I didn't label any notes (except the open ones - an open A string is an A). If you want a challenge you can cluster the notes and see if your clustering lumps all the same notes together. If you want labels so you can do some inspection of notes that are the same you can look on google for a guitar fretboard diagram. I have done both of these experiments and learned a bit about music which I 'm hoping to verify soon when I find a good music theory book. | |
| This data set might be interesting to use to be able to write sheet music from an audio sample of some finger-picked music. Identifying chords is a difficult computational task, but finger-style guitar, with clear, individual notes, might be easier. If this is the case, a simple script could be written to write sheet music from an audio sample. | |
| One draw back about the data set is that some non-plucked strings were vibrating when I played a note. I tried various techniques to muffle them, but there is still some noise in the background. I don't know if this is because of my technique or something that happens to all players on all guitars. In any event, this noise didn't hurt my analysis. | |
| About the data | |
| They were recorded by me, Melvyn, using a program called audacity. | |
| There is a directory with the name of the string. Inside the directory you will find .wav files named either open, 1, 2, ....8 for the fingering of the string. There is also a directory called ""scale"" I recorded some notes that make a ""do-re-mi..."" scale. You can use these for a number of things. | |
| I use the GuitarTuner app to tune the Guitar - I'm just learning so I don't have an ear for notes yet. After some initial analysis it looks like the guitar might be a bit out of tune, so the resonant frequencies are a bit off from what they should be. Another thing that is interesting to think about it is how far a frequency must be from the proper on until it becomes distinguishable as a different note." | |
| "Emotion, Aging, and Sentiment Over Time",Data from the Chronist project,Chris Roth,32,"Version 1,2017-02-13","gerontology | |
| linguistics",CSV,39 MB,Other,"7,811 views",757 downloads,13 kernels,0 topics,https://www.kaggle.com/cjroth/chronist,"Chronist is a project to quantitatively monitor the emotional and physical changes of an individual over periods of time. My thesis is that if you can accurately show emotional or physical change over time, you can objectively pinpoint how an environmental change such as a career change, moving to a new city, starting or ending a relationship, or starting a new habit like going to the gym affected your physical and emotional health. This can lead to important insights on an individual level and for a population as a whole. | |
| If you are interested in hearing more about this project, contributing your data, or collaborating, contact me at chris@cjroth.com. | |
| See the GitHub repository to read more about the tools that were used to generate the dataset." | |
| Solar Radiation Prediction,Task from NASA Hackathon,Andrey,32,"Version 1,2017-05-22","space | |
| energy",CSV,3 MB,ODbL,"7,082 views",881 downloads,38 kernels,3 topics,https://www.kaggle.com/dronio/SolarEnergy,"Context | |
| Space Apps Moscow was held on April 29th & 30th. Thank you to the 175 people who joined the International Space Apps Challenge at this location! | |
| Content | |
| The dataset contains such columns as: ""wind direction"", ""wind speed"", ""humidity"" and temperature. The response parameter that is to be predicted is: ""Solar_radiation"". It contains measurements for the past 4 months and you have to predict the level of solar radiation. Just imagine that you've got solar energy batteries and you want to know will it be reasonable to use them in future? | |
| Acknowledgements | |
| Thanks NASA for the dataset. | |
| Inspiration | |
| Predict the level of solar radiation. Here are some intersecting dependences that i have figured out: 1. Humidity & Solar_radiation. 2.Temeperature & Solar_radiation. | |
| The best result of accuracy I could get using cross-validation was only 55%." | |
| "Structural MRI Datasets (T1, T2, FLAIR etc.)",Coursera NeuroHacking in R course datasets,IlknurIcke,32,"Version 2,2017-01-05|Version 1,2017-01-04",healthcare,Other,212 MB,Other,"11,885 views","1,552 downloads",14 kernels,2 topics,https://www.kaggle.com/ilknuricke/neurohackinginrimages,"Context | |
| Data from the Coursera Course: Neurohacking In R taught by Dr. Elizabeth Sweeney , Rice Academy Postdoctoral Fellow, Ciprian M. Crainiceanu, Professor and John Muschelli III , Assistant Scientist | |
| Please see https://www.coursera.org/learn/neurohacking for the lecture notes and example code from the instructors. | |
| Content | |
| Structural MRI images for visualization and image processing | |
| From the instructors: | |
| About this course: Neurohacking describes how to use the R programming language (https://cran.r-project.org/) and its associated packages to perform manipulation, processing, and analysis of neuroimaging data. We focus on publicly-available structural magnetic resonance imaging (MRI). We discuss concepts such as inhomogeneity correction, image registration, and image visualization. By the end of this course, you will be able to: Read/write images of the brain in the NIfTI (Neuroimaging Informatics Technology Initiative) format Visualize and explore these images, perform inhomogeneity correction, brain extraction, and image registration (within a subject and to a template). | |
| Acknowledgements | |
| Dataset is public domain and was originally posted for the Coursera online course NeuroHacking in R. | |
| Notes | |
| A. When you download the zip archive, double clicking might try to compress the file instead of extracting it. Unzipping on terminal (mac) correctly decompresses the archive. | |
| B. The zip file contains a directory structure: | |
| BRAINIX | |
| -----DICOM | |
| ----FLAIR | |
| ----ROI | |
| ----T1 | |
| ----T2 | |
| -----NIFTI | |
| kirby21 | |
| -----visit 1 | |
| ----113 | |
| -----visit 2 | |
| ----113 | |
| Template | |
| However, when it is unzipped here on Kaggle environment, somehow the directory structure is not maintained, therefore files with the same names are being overwritten. As a workaround, I added the directory names to the files ie. BRAINIX_DICOM_T1_IM_0001_0011.dcm instead of just IM_0001_0011.dcm. | |
| Check out script https://www.kaggle.com/ilknuricke/d/ilknuricke/neurohackinginrimages/structural-mri-visualization/code for example use." | |
| World Tennis Odds Database,"+139K tennis matches, 15 bookies, worldwide.",DMPierre,32,"Version 1,2017-09-11","tennis | |
| sports",CSV,49 MB,CC4,"4,259 views",368 downloads,3 kernels,0 topics,https://www.kaggle.com/dmpierre/world-tennis-odds-database,"Context | |
| How good of an arbitrageur would you be? | |
| Find it out in the World Tennis Database which gathers more than 139K matches with odds from 15 different bookies (49 MB). | |
| If you are looking to predict the outcome of a tennis match, to find arbitrage opportunities, inspecting variations in a particular player odds or simply searching to improve your Machine Learning or visualisation skills, then this dataset might be looking for you too. | |
| Content | |
| Data is packed in CSV format, ready to spit out some interesting statistics. It is composed of the following 72 columns: | |
| Url, string | |
| Country, string | |
| Date, (yyyy-mm-dd hh:mm), to ease date-time transformations. | |
| Day, string | |
| Tournament name, string | |
| Doubles, either 0 or 1, when it is not a single player match | |
| Player 1(2) name, string | |
| Player 1(2) score, int, number of sets won | |
| Player 1(2) set 0 score, int, up until set 4 - indexing of sets starts at 0, ask why to python ;) | |
| No set info, either 0 or 1, when there is no informations about the final set scores | |
| Missing bookies, either 0 or 1, when there is no informations about any bookies odds | |
| Retired player, either 0 or 1, when one player retired | |
| Cancelled game, either 0 or 1, when the game got cancelled | |
| Comments, string used to insert any comments during the scraping process | |
| Walkover, either 0 or 1, when one player chose to walkover | |
| Awarded player, either 0 or 1, when one player got awarded | |
| Fifteen bookies were then taken into account, each having three type of infos: Player 1 odd, Player 2 odd, Payout. This result in adding to the preceding 27 columns, 45 others. | |
| Bookies were sorted alphabetically: | |
| 10Bet | |
| 18Bet | |
| 5Dimes | |
| Bet At Home | |
| Bet365 | |
| BetHard | |
| BetOlimp | |
| BetRally | |
| BWin | |
| JetBull | |
| MarathonBet | |
| Pinnacle | |
| TempoBet | |
| TonyBet | |
| Unibet | |
| Acknowledgements | |
| Huge kudos to the OddsPortal Website for their wonderful archiving job. | |
| Cover photo by Jeremy Galliani on Unsplash. | |
| Inspiration | |
| Various interesting infos and predictions can be made out of this dataset. | |
| Individual players trajectories and their respective odds movements. | |
| Bookies respective strategies. Who sets the pace? | |
| Detecting patterns in arbitrage situations (arbitrageur perspective). | |
| And of course, predicting the winner of a game, as draws are not allowed. | |
| Of course, I got inspired by the European Soccer Database. | |
| Finally, for details about the scraping process, visit https://dmpierre.github.io/." | |
| German Credit Risk,Credit classification,UCI Machine Learning,32,"Version 1,2016-12-15",,CSV,49 KB,ODbL,"24,058 views","2,456 downloads",16 kernels,3 topics,https://www.kaggle.com/uciml/german-credit,"Context | |
| The original dataset contains 1000 entries with 20 categorial/symbolic attributes prepared by Prof. Hofmann. In this dataset, each entry represents a person who takes a credit by a bank. Each person is classified as good or bad credit risks according to the set of attributes. The link to the original dataset can be found below. | |
| Content | |
| It is almost impossible to understand the original dataset due to its complicated system of categories and symbols. Thus, I wrote a small Python script to convert it into a readable CSV file. Several columns are simply ignored, because in my opinion either they are not important or their descriptions are obscure. The selected attributes are: | |
| Age (numeric) | |
| Sex (text: male, female) | |
| Job (numeric: 0 - unskilled and non-resident, 1 - unskilled and resident, 2 - skilled, 3 - highly skilled) | |
| Housing (text: own, rent, or free) | |
| Saving accounts (text - little, moderate, quite rich, rich) | |
| Checking account (numeric, in DM - Deutsch Mark) | |
| Credit amount (numeric, in DM) | |
| Duration (numeric, in month) | |
| Purpose (text: car, furniture/equipment, radio/TV, domestic appliances, repairs, education, business, vacation/others) | |
| Acknowledgements | |
| Source: UCI" | |
| Hospital General Information,General information & quality ratings for almost all US hospitals,Centers for Medicare & Medicaid Services,32,"Version 2,2017-08-10|Version 1,2017-01-07","healthcare | |
| hospitals | |
| public health",CSV,3 MB,Other,"11,210 views","1,372 downloads",36 kernels,4 topics,https://www.kaggle.com/cms/hospital-general-information,"Context | |
| There are all sorts of reasons why you'd want to know a hospital's quality rating. | |
| Your mom is having her second hip replacement. Her first one went terribly and you're nervous about how she'll do. Which hospital would you suggest she have her surgery? | |
| You're selecting a health plan on your state's Exchange, but your top two choices partner with different hospitals. How will you decide which plan to pick? | |
| Your brother has Cystic Fibrosis and has to go to the ER frequently. He hates waiting. Which hospitals/states provide care in the timeliest manner? | |
| Your in-laws moved to Florida recently to retire, and have been trying to convince you to move there too. You're looking for any way possible to show them that your state is better. Does your state have better hospitals? | |
| Every hospital in the United States of America that accepts publicly insured patients (Medicaid or MediCare) is required to submit quality data, quarterly, to the Centers for Medicare & Medicaid Services (CMS). There are very few hospitals that do not accept publicly insured patients, so this is quite a comprehensive list. | |
| Content | |
| This file contains general information about all hospitals that have been registered with Medicare, including their addresses, type of hospital, and ownership structure. It also contains information about the quality of each hospital, in the form of an overall rating (1-5, where 5 is the best possible rating & 1 is the worst), and whether the hospital scored above, same as, or below the national average for a variety of measures. | |
| This data was updated by CMS on July 25, 2017. CMS' overall rating includes 60 of the 100 measures for which data is collected & reported on Hospital Compare website (https://www.medicare.gov/hospitalcompare/search.html). Each of the measures have different collection/reporting dates, so it is impossible to specify exactly which time period this dataset covers. For more information about the timeframes for each measure, see: https://www.medicare.gov/hospitalcompare/Data/Data-Updated.html# For more information about the data itself, APIs and a variety of formats, see: https://data.medicare.gov/Hospital-Compare | |
| Acknowledgements | |
| Attention: Works of the U.S. Government are in the public domain and permission is not required to reuse them. An attribution to the agency as the source is appreciated. Your materials, however, should not give the false impression of government endorsement of your commercial products or services. See 42 U.S.C. 1320b-10. | |
| Inspiration | |
| Which hospital types & hospital ownerships are most common? | |
| Which hospital types & ownerships are associated with better than average ratings/mortality/readmission/etc? | |
| What is the average hospital rating, by state? | |
| Which hospital types & hospital ownerships are more likely to have not submitted proper data (""Not Available"" & ""Results are not available for this reporting period"")? | |
| Which parts of the country have the highest & lowest density of religious hospitals?" | |
| Women's Shoe Prices,"A list of 10,000 women's shoes and the prices at which they are sold.",Datafiniti,32,"Version 2,2017-05-15|Version 1,2017-04-25","clothing | |
| business",CSV,102 MB,CC4,"13,544 views","2,002 downloads",59 kernels,3 topics,https://www.kaggle.com/datafiniti/womens-shoes-prices,"About This Data | |
| This is a list of 10,000 women's shoes and their product information provided by Datafiniti's Product Database. | |
| The dataset includes shoe name, brand, price, and more. Note that each shoe will have an entry for each price found for it and some shoes may have multiple entries. | |
| What You Can Do with This Data | |
| You can use this data to determine brand markups, pricing strategies, and trends for luxury shoes. E.g.: | |
| What is the average price of each distinct brand listed? | |
| Which brands have the highest prices? | |
| Which ones have the widest distribution of prices? | |
| Is there a typical price distribution (e.g., normal) across brands or within specific brands? | |
| Further processing data would also let you: | |
| Correlate specific product features with changes in price. | |
| You can cross-reference this data with a sample of our Men's Shoe Prices to see if there are any differences between women's brands and men's brands. | |
| Data Schema | |
| A full schema for the data is available in our support documentation. | |
| About Datafiniti | |
| Datafiniti provides instant access to web data. We compile data from thousands of websites to create standardized databases of business, product, and property information. Learn more. | |
| Want More? | |
| You can get more data like this by joining Datafiniti or requesting a demo." | |
| "SpaceX Missions, 2006-Present","Location, date/time, and outcome for every rocket launch",SpaceX,32,"Version 1,2017-03-01","space | |
| business",CSV,7 KB,Other,"6,597 views",751 downloads,19 kernels,2 topics,https://www.kaggle.com/spacex/spacex-missions,"Context | |
| SpaceX has gained worldwide attention for a series of historic milestones. It is the only private company ever to return a spacecraft from low-Earth orbit, which it first accomplished in December 2010. The company made history again in May 2012 when its Dragon spacecraft attached to the International Space Station, exchanged cargo payloads, and returned safely to Earth — a technically challenging feat previously accomplished only by governments. Since then Dragon has delivered cargo to and from the space station multiple times, providing regular cargo resupply missions for NASA. | |
| Under a $1.6 billion contract with NASA, SpaceX is flying numerous cargo resupply missions to the International Space Station, for a total of at least 20 flights under the Commercial Resupply Services contract. In 2016, NASA awarded SpaceX a second version of that contract that will cover a minimum of 6 additional flights from 2019 onward. In the near future, SpaceX will carry crew as part of NASA’s Commercial Crew Program as well. Dragon was designed from the outset to carry astronauts and SpaceX is in the process of upgrading Dragon to make it crew-ready. SpaceX is the world’s fastest-growing provider of launch services and has over 70 future missions on its manifest, representing over $10 billion in contracts. These include commercial satellite launches as well as NASA and other US Government missions. | |
| Content | |
| This dataset includes a record for each payload carried during a SpaceX mission into outer space. | |
| Acknowledgements | |
| The data was scraped from the SpaceX and NASA website. | |
| Inspiration | |
| Has the rate of SpaceX rocket launches increased in the past decade? How many missions do you predict will be launched in 2018?" | |
| Handwritten math symbols dataset,Over 100 000 image samples.,Xai Nano,31,"Version 2,2017-01-16|Version 1,2017-01-15","writing | |
| mathematics",Other,410 MB,CC0,"9,919 views",938 downloads,8 kernels,3 topics,https://www.kaggle.com/xainano/handwrittenmathsymbols,"Contact | |
| Email me at: itsawesome17@gmail.com | |
| My blog: http://blog.mathocr.com/ | |
| Content | |
| Dataset consists of jpg files(45x45) DISCLAIMER: dataset does not contain Hebrew alphabet at all. It includes basic Greek alphabet symbols like: alpha, beta, gamma, mu, sigma, phi and theta. English alphanumeric symbols are included. All math operators, set operators. Basic pre-defined math functions like: log, lim, cos, sin, tan. Math symbols like: \int, \sum, \sqrt, \delta and more. | |
| Acknowledgements | |
| Original source, that was parsed, extracted and modified is CROHME dataset. | |
| Visit CROHME at http://www.isical.ac.in/~crohme/index.html. | |
| Inspiration | |
| Due to the technological advances in recent years, paper scientific documents are used less and less. Thus, the trend in the scientific community to use digital documents has increased considerably. Among these documents, there are scientific documents and more specifically mathematics documents. So I give a tool, to research recognizing handwritten math language in variety of applications. | |
| Source code | |
| You can find source code responsible for parsing original CROHME dataset here: | |
| https://github.com/XaiNano/CROHME_extractor | |
| This parser allows you not only to extract math symbols into square images of desired size, but also lets you specify categories of classes to be extracted like: digits, greek_letters, lowercase_letters, operators, and more. It also contains visualization tools and histograms showing appearances of each class in the dataset. | |
| Commercial use | |
| Rights for commercial usage cannot be granted." | |
| Food Images (Food-101),Labeled food images in 101 categories from apple pies to waffles,Kevin Mader,31,"Version 4,2017-08-09|Version 3,2017-08-08|Version 2,2017-08-08|Version 1,2017-08-08","food and drink | |
| popular culture | |
| image data | |
| multiclass classification",Other,663 MB,Other,"10,093 views","1,551 downloads",10 kernels,0 topics,https://www.kaggle.com/kmader/food41,"Overview | |
| The dataset contains a number of different subsets of the full food-101 data. The idea is to make a more exciting simple training set for image analysis than CIFAR10 or MNIST. For this reason the data includes massively downscaled versions of the images to enable quick tests. The data has been reformatted as HDF5 and specifically Keras HDF5Matrix which allows them to be easily read in. The file names indicate the contents of the file. For example | |
| food_c101_n1000_r384x384x3.h5 means there are 101 categories represented, with n=1000 images, that have a resolution of 384x384x3 (RGB, uint8) | |
| food_test_c101_n1000_r32x32x1.h5 means the data is part of the validation set, has 101 categories represented, with n=1000 images, that have a resolution of 32x32x1 (float32 from -1 to 1) | |
| Challenge | |
| The first goal is to be able to automatically classify an unknown image using the dataset, but beyond this there are a number of possibilities for looking at what regions / image components are important for making classifications, identify new types of food as combinations of existing tags, build object detectors which can find similar objects in a full scene. | |
| Data Acknowledgement | |
| The data was repackaged from the original source (gzip) available at https://www.vision.ee.ethz.ch/datasets_extra/food-101/ | |
| License | |
| The Food-101 data set consists of images from Foodspotting [1]. Any use beyond scientific fair use must be negotiated with the respective picture owners according to the Foodspotting terms of use [2]. | |
| [1] http://www.foodspotting.com/ [2] http://www.foodspotting.com/terms/" | |
| Historical Hourly Weather Data 2012-2017,Hourly weather data for 30 US & Canadian Cities + 6 Israeli Cities,Selfish Gene,31,"Version 2,2017-12-28|Version 1,2017-12-23","united states | |
| time series | |
| geography | |
| + 2 more...",CSV,12 MB,ODbL,"3,585 views",593 downloads,,2 topics,https://www.kaggle.com/selfishgene/historical-hourly-weather-data,"Historical Hourly Weather Data | |
| Who amongst us doesn't small talk about the weather every once in a while? | |
| The goal of this dataset is to elevate this small talk to medium talk. | |
| Just kidding, I actually originally decided to collect this dataset in order to demonstrate basic signal processing concepts, such as filtering, Fourier transform, auto-correlation, cross-correlation, etc..., (for a data analysis course I'm currently preparing). | |
| I wanted to demonstrate these concepts on signals that we all have intimate familiarity with and hope that this way these concepts will be better understood than with just made up signals. | |
| The weather is excellent for demonstrating these kinds of concepts as it contains periodic temporal structure with two very different periods (daily and yearly). | |
| Content | |
| The dataset contains ~5 years of high temporal resolution (hourly measurements) data of various weather attributes, such as temperature, humidity, air pressure, etc. | |
| This data is available for 30 US and Canadian Cities, as well as 6 Israeli cities. | |
| I've organized the data according to a common time axis for easy use. | |
| Each attribute has it's own file and is organized such that the rows are the time axis (it's the same time axis for all files), and the columns are the different cities (it's the same city ordering for all files as well). | |
| Additionally, for each city we also have the country, latitude and longitude information in a separate file. | |
| Acknowledgements | |
| The dataset was aquired using Weather API on the OpenWeatherMap website, and is available under the ODbL License. | |
| Inspiration | |
| Weather data is both intrinsically interesting, and also potentially useful when correlated with other types of data. | |
| For example, Wildfire spread is potentially related to weather conditions, demand for cabs is famously known to be correlated with weather conditions (here, here and here you can find NYC cab ride data), and use of city bikes is probably also correlated with weather in interesting ways (check out this Austin dataset, this SF dataset, this Montreal dataset, and this NYC dataset). | |
| Traffic is also probably related to weather. | |
| Another potentially interesting source of correlation is between weather and crime. Here are a few crime datasets on kaggle of cities present in this weather dataset: Chicago, Philadelphia, Los Angeles, Vancouver, Austin, NYC | |
| There are many other potentially interesting connections between everyday life and the weather that we can explore together with the help of this dataset. Have fun!" | |
| Rainfall in India,Sub-division wise monthly data for 115 years from 1901-2015.,Rajanand Ilangovan / இராஜ்ஆனந்த் இளங்கோவன்,31,"Version 2,2017-08-05|Version 1,2017-07-23","india | |
| weather",CSV,583 KB,CC4,"6,058 views","1,308 downloads",5 kernels,0 topics,https://www.kaggle.com/rajanand/rainfall-in-india,"Connect/Follow me on LinkedIn for more updates on interesting dataset like this. Thanks. | |
| Context | |
| This data set contains monthly rainfall detail of 36 meteorological sub-divisions of India. | |
| Content | |
| Time Period: 1901 - 2015 | |
| Granularity: Monthly | |
| Location: 36 meteorological sub-divisions in India | |
| Rainfall unit: mm | |
| Acknowledgements | |
| India Meteorological Department(IMD) Govt. of India has shared this dataset under Govt. Open Data License - India." | |
| "English Premier League Players Dataset, 2017/18","A unique dataset containing FPL data, popularity and market values",ShubhamMaurya,31,"Version 1,2017-08-03","popular culture | |
| association football | |
| sports",CSV,34 KB,CC0,"8,498 views","1,156 downloads",7 kernels,0 topics,https://www.kaggle.com/mauryashubham/english-premier-league-players-dataset,"Context | |
| For most football fans, May - July represents a lull period due to the lack of club football. What makes up for it, is the intense transfer speculation that surrounds all major player transfers today. Their market valuations also lead to a few raised eyebrows, lately more than ever. I was curious to see how good a proxy popularity could be for ability, and the predictive power it would have in a model estimating a player's market value. | |
| Content | |
| name: Name of the player | |
| club: Club of the player | |
| age : Age of the player | |
| position : The usual position on the pitch | |
| position_cat : | |
| 1 for attackers | |
| 2 for midfielders | |
| 3 for defenders | |
| 4 for goalkeepers | |
| market_value : As on transfermrkt.com on July 20th, 2017 | |
| page_views : Average daily Wikipedia page views from September 1, 2016 to May 1, 2017 | |
| fpl_value : Value in Fantasy Premier League as on July 20th, 2017 | |
| fpl_sel : % of FPL players who have selected that player in their team | |
| fpl_points : FPL points accumulated over the previous season | |
| region: | |
| 1 for England | |
| 2 for EU | |
| 3 for Americas | |
| 4 for Rest of World | |
| nationality | |
| new_foreign : Whether a new signing from a different league, for 2017/18 (till 20th July) | |
| age_cat | |
| club_id | |
| big_club: Whether one of the Top 6 clubs | |
| new_signing: Whether a new signing for 2017/18 (till 20th July) | |
| Inspiration | |
| To statistically analyse the beautiful game." | |
| My Uber Drives,Complete Details of My Uber Drives in 2016,Zeeshan-ul-hassan Usmani,31,"Version 1,2017-03-23",road transport,CSV,84 KB,ODbL,"18,369 views","1,877 downloads",26 kernels,4 topics,https://www.kaggle.com/zusmani/uberdrives,"Context | |
| My Uber Drives (2016) | |
| Here are the details of my Uber Drives of 2016. I am sharing this dataset for data science community to learn from the behavior of an ordinary Uber customer. | |
| Content | |
| Geography: USA, Sri Lanka and Pakistan | |
| Time period: January - December 2016 | |
| Unit of analysis: Drives | |
| Total Drives: 1,155 | |
| Total Miles: 12,204 | |
| Dataset: The dataset contains Start Date, End Date, Start Location, End Location, Miles Driven and Purpose of drive (Business, Personal, Meals, Errands, Meetings, Customer Support etc.) | |
| Acknowledgements & References | |
| Users are allowed to use, download, copy, distribute and cite the dataset for their pet projects and training. Please cite it as follows: “Zeeshan-ul-hassan Usmani, My Uber Drives Dataset, Kaggle Dataset Repository, March 23, 2017.” | |
| Past Research | |
| Uber TLC FOIL Response - The dataset contains over 4.5 million Uber pickups in New York City from April to September 2014, and 14.3 million more Uber pickups from January to June 2015 https://github.com/fivethirtyeight/uber-tlc-foil-response | |
| 1.1 Billion Taxi Pickups from New York - http://toddwschneider.com/posts/analyzing-1-1-billion-nyc-taxi-and-uber-trips-with-a-vengeance/ | |
| What you can do with this data - a good example by Yao-Jen Kuo - https://yaojenkuo.github.io/uber.html | |
| Inspiration | |
| Some ideas worth exploring: | |
| • What is the average length of the trip? | |
| • Average number of rides per week or per month? | |
| • Total tax savings based on traveled business miles? | |
| • Percentage of business miles vs personal vs. Meals | |
| • How much money can be saved by a typical customer using Uber, Careem, or Lyft versus regular cab service?" | |
| SEPTA - Regional Rail,Predict Arrival Times of Philadelphia's Regional Trains.,SEPTA,31,"Version 26,2016-11-07|Version 25,2016-11-03|Version 24,2016-11-02|Version 23,2016-11-01|Version 22,2016-10-11|Version 21,2016-10-08|Version 20,2016-10-02|Version 19,2016-09-26|Version 18,2016-09-16|Version 17,2016-09-01|Version 16,2016-08-12|Version 15,2016-08-02|Version 14,2016-07-25|Version 13,2016-07-20|Version 12,2016-07-15|Version 11,2016-07-07|Version 10,2016-06-11|Version 9,2016-06-02|Version 8,2016-05-30|Version 7,2016-05-28|Version 6,2016-05-26|Version 5,2016-05-25|Version 4,2016-05-25|Version 3,2016-05-25|Version 2,2016-05-25|Version 1,2016-05-24",rail transport,CSV,775 MB,CC0,"25,033 views","1,803 downloads",89 kernels,11 topics,https://www.kaggle.com/septa/on-time-performance,"SEPTA - Southeastern Pennsylvania Transportation Authority | |
| The SEPTA Regional Rail system consists of commuter rail service on 13 branches to more than 150 active stations in Philadelphia, Pennsylvania, and its suburbs and satellite cities. | |
| SEPTA uses On-Time Performance (OTP) to measure service reliability. OTP identifies the number of trains for all rail lines that arrive at their scheduled destination at the scheduled time. However, by industry standard, a train may arrive up to 5 minutes and 59 seconds after its scheduled time and still be considered on-time. | |
| SEPTA has established an annual goal of 91% for Regional Rail On-Time Performance. How well are they doing? Is it even a meaningful measure? | |
| Data Description | |
| otp.csv | |
| train_id | |
| direction ('N' or 'S' direction is demarcated as either Northbound or Southbound)^1 | |
| origin (See map below - you'll see 'Warminster', 'Glenside',...'Airport Terminal..') | |
| next_station (Think of this as the station stop, at timeStamp) | |
| date | |
| status ('On Time', '5 min', '10 min'. This is a status on train lateness. 999 is a suspended train) | |
| timeStamp | |
| trainView.csv - GPS Train data (early release) | |
| Most GPS coordinates are based on track telemetry; however, cars are being equipped with GPS units. | |
| train_id | |
| status | |
| next_station | |
| service | |
| dest | |
| lon | |
| lat | |
| source | |
| track_change | |
| track | |
| date | |
| timeStamp0 First timeStamp at coordinates. | |
| timeStamp1 Last timeStamp at coordinates. | |
| You can look at the example here on how to track a single train. | |
| What To Look For... | |
| Ah, as a commuter you care more about the performance of the train(s) you plan to take. OTP maybe 91% or above; but, if the train you take runs late, you'll spend extra time on the tracks. If it consistently runs late, maybe the schedule should be changed. | |
| Look for patterns. For example, during the Tuesday morning rush do some trains run consistently late? How long does it take to get to from point A to point B in the system? Performance is VERY important to SEPTA. | |
| Below is a map of the system and station stops. This dataset contains data on the Regional Rail Lines. | |
| SEPTA train schedules can be found here. Note different Weekday, Saturday, and Sunday schedules." | |
| French presidential election,Extract from Twitter about the French election (with a taste of Google Trends),Jean-Michel D.,31,"Version 21,2017-07-05|Version 20,2017-05-12|Version 19,2017-05-06|Version 18,2017-04-29|Version 17,2017-04-25|Version 16,2017-04-21|Version 15,2017-04-17|Version 14,2017-04-14|Version 13,2017-04-11|Version 12,2017-04-07|Version 11,2017-04-05|Version 10,2017-04-03|Version 9,2017-04-03|Version 8,2017-03-30|Version 7,2017-03-27|Version 6,2017-03-24|Version 5,2017-03-23|Version 4,2017-03-22|Version 3,2017-02-28|Version 2,2017-02-28|Version 1,2017-02-28",politics,SQLite,3 GB,ODbL,"11,027 views",909 downloads,39 kernels,2 topics,https://www.kaggle.com/jeanmidev/french-presidential-election,"Context | |
| This dataset is born from a test with the twitter streaming api to filter and collect data from this flow on a specific topic, in this case the French election.The script used to make this data collection is available on this Github repository. | |
| Since the 18th of March, the French election enter in the final straight line until the first poll the 23 April 2017 , the candidates for the position are: | |
| M. Nicolas DUPONT-AIGNAN | |
| Mme Marine LE PEN | |
| M. Emmanuel MACRON | |
| M. Benoît HAMON | |
| Mme Nathalie ARTHAUD | |
| M. Philippe POUTOU | |
| M. Jacques CHEMINADE | |
| M. Jean LASSALLE | |
| M. Jean-Luc MÉLENCHON | |
| M. François ASSELINEAU | |
| M. François FILLON | |
| The idea was to collect the data from the Twitter API periodically. The acquisition process evolved as follows: | |
| Versions 1, 2 and 3 Every hour a python script listens to the twitter api stream for 10 minutes during 3 weeks. | |
| Version 4+ The new versions will be based on a new data structure, and start after the validation by the French constitutional council on 18 March 2017 of the candidates. | |
| The data will be stored in a dbsqlite files(database_number of the week_number_block_weekday.sqlite format) and will be updated as often as I can (at least every week). | |
| After the first round (version 18+) i had to readjust the number of files per week and the 20 files kaggle limitation push me to reduce the number of files to upload (but you can join for your local analytics the version 17 + version 18+) | |
| Example : Illustration of the number of mentions of the different candidates | |
| I add to these databases a sqlite database that contains the informations from the google trends about the top 5 candidates.In thid database there is : | |
| A table that contains the overall interests by region | |
| A table that contains the interests by region for each candidate | |
| A table with the top25 associated queries for each candidate in top and rising ranking | |
| Content | |
| In this dbsqlite file, you will find a data table that contains for every row: | |
| ===============Common=============== | |
| the index of the line | |
| the language of the tweet | |
| for each candidate :mention_candidatename, if the candidate or his associated account has been called (0 or 1) | |
| the tweet | |
| the timestamp in milliseconds | |
| ===============Version 4+=============== | |
| the day | |
| the hour (London timezone) | |
| the username of the user that made the tweet | |
| the username location (that he gives with his profile) | |
| if the tweet is a retweet or a quote (0 or 1) | |
| the username that has been retweeted | |
| the original tweet (the one retweeted or quoted) | |
| Acknowledgements | |
| This election is gonna be intense. | |
| Inspiration | |
| The first version of the dataset was just a test to collect the data and see the first pieces of work that the community can do with this dataset.The new versions are (I think and hope) adapted to do deep text analytics." | |
| "Oil Pipeline Accidents, 2010-Present","Causes, injuries/fatalities, and costs of pipeline leaks and spills",Department of Transportation,31,"Version 1,2017-02-09","environment | |
| energy",CSV,887 KB,CC0,"8,446 views","1,218 downloads",40 kernels,0 topics,https://www.kaggle.com/usdot/pipeline-accidents,"Content | |
| This database includes a record for each oil pipeline leak or spill reported to the Pipeline and Hazardous Materials Safety Administration since 2010. These records include the incident date and time, operator and pipeline, cause of incident, type of hazardous liquid and quantity lost, injuries and fatalities, and associated costs. | |
| Acknowledgements | |
| The oil pipeline accident reports were collected and published by the DOT's Pipeline and Hazardous Materials Safety Administration." | |
| Gene expression dataset (Golub et al.),Molecular Classification of Cancer by Gene Expression Monitoring,Chris Crawford,30,"Version 3,2017-08-09|Version 2,2017-08-09|Version 1,2017-08-09","human genetics | |
| biology | |
| health sciences | |
| biotechnology",CSV,4 MB,CC0,"7,855 views",971 downloads,28 kernels,0 topics,https://www.kaggle.com/crawford/gene-expression,"Context | |
| This dataset comes from a proof-of-concept study published in 1999 by Golub et al. It showed how new cases of cancer could be classified by gene expression monitoring (via DNA microarray) and thereby provided a general approach for identifying new cancer classes and assigning tumors to known classes. These data were used to classify patients with acute myeloid leukemia (AML) and acute lymphoblastic leukemia (ALL). | |
| Content | |
| Golub et al ""Molecular Classification of Cancer: Class Discovery and Class Prediction by Gene Expression Monitoring"" | |
| There are two datasets containing the initial (training, 38 samples) and independent (test, 34 samples) datasets used in the paper. These datasets contain measurements corresponding to ALL and AML samples from Bone Marrow and Peripheral Blood. Intensity values have been re-scaled such that overall intensities for each chip are equivalent. | |
| Acknowledgements | |
| Molecular Classification of Cancer: Class Discovery and Class Prediction by Gene Expression | |
| Science 286:531-537. (1999). Published: 1999.10.14 | |
| T.R. Golub, D.K. Slonim, P. Tamayo, C. Huard, M. Gaasenbeek, J.P. Mesirov, H. Coller, M. Loh, J.R. Downing, M.A. Caligiuri, C.D. Bloomfield, and E.S. Lander | |
| These datasets have been converted to a comma separated value files (CSV). | |
| Inspiration | |
| These datasets are great for classification problems. The original authors used the data to classify the type of cancer in each patient by their gene expressions." | |
| Brazil's House of Deputies Reimbursements,Refunds Spendings from 2009 to 2017,epattaro,30,"Version 4,2017-11-04|Version 3,2017-11-04|Version 2,2017-01-27|Version 1,2017-01-22","brazil | |
| finance | |
| politics",CSV,394 MB,CC0,"5,539 views",465 downloads,23 kernels,6 topics,https://www.kaggle.com/epattaro/brazils-house-of-deputies-reimbursements,"Context | |
| Brazilian politicians are entitled to refunds if they spend any of their money on an activity that is enabling them to ""better serve the people"". | |
| Those expenses are public data, however, there is little monitoring/data analysis of it. A quick look at it shows a deputy with over 800 flights in one year. Another deputy has over 140.000R$ expenses on mailing (old fashion mail) in one year. | |
| There are a lot of very suspicious data regarding the deputies expending behavior. Can you help spot outliers and companies charging unusual amounts of money for a service? | |
| Content | |
| Data is public. It was collected from the official government website: | |
| http://www2.camara.leg.br/transparencia/cota-para-exercicio-da-atividade-parlamentar/dados-abertos-cota-parlamentar | |
| It was converted from xml to csv, filtered out irrelevant columns, and translated a few of the features to English. | |
| The uploaded data contains: | |
| u'deputy_name', u'political_party', u'deputy_state', u'company_name' u'company_id' u'refund_date' | |
| Inspiration | |
| Brazil is currently passing through thriving times. | |
| Its political group has always been public involved in many scandals, but it is just now that a few brave men and women have started doing something about it. In 2016 we have had senators, former ministers and many others formally charged and arrested for their crimes." | |
| NEWS SUMMARY,Generating short length descriptions of news articles.,Kondalarao Vonteru,30,"Version 1,2017-08-10","journalism | |
| india | |
| linguistics",CSV,11 MB,GPL,"6,163 views",562 downloads,,2 topics,https://www.kaggle.com/sunnysai12345/news-summary,"Context | |
| I am currently working on summarizing chat context where it helps an agent in understanding previous context quickly. It interests me to apply the deep learning models to existing datasets and how they perform on them. I believe news articles are rich in grammar and vocabulary which allows us to gain greater insights. | |
| Content | |
| The dataset consists of 4515 examples and contains Author_name, Headlines, Url of Article, Short text, Complete Article. I gathered the summarized news from Inshorts and only scraped the news articles from Hindu, Indian times and Guardian. Time period ranges from febrauary to august 2017. | |
| Acknowledgements | |
| I would like to thank the authors of Inshorts for their amazing work | |
| Inspiration | |
| Generating short length descriptions(headlines) from text(news articles). | |
| Summarizing large amount of information which can be represented in compressed space | |
| Purpose | |
| When I was working on the summarization task I didn't find any open source data-sets to work on, I believe there are people just like me who are working on these tasks and I hope it helps them. | |
| Contributions | |
| It will be really helpful if anyone found nice insights from this data and can share their work. Thankyou...!!! | |
| For those who are interested here is the link for the github code which includes the scripts for scraping. https://github.com/sunnysai12345/News_Summary" | |
| Douban Movie Short Comments Dataset,DMSCD producted by utmhikari,utmhikari,30,"Version 7,2017-03-06|Version 6,2017-01-16|Version 5,2017-01-16|Version 4,2017-01-16|Version 3,2017-01-01|Version 2,2016-12-29|Version 1,2016-12-29","film | |
| internet",CSV,387 MB,CC0,"7,193 views",517 downloads,3 kernels,2 topics,https://www.kaggle.com/utmhikari/doubanmovieshortcomments,"Douban Movie Short Comments Dataset V2 | |
| Introduction | |
| Douban Movie is a Chinese website that allows Internet users to share their comments and viewpoints about movies. Users are able to post short or long comments on movies and give them marks. This dataset contains more than 2 million short comments of 28 movies in Douban Movie website. It can be used on text classification, text clustering, sentiment analysis, semantic web construction and some other fields that relate to web mining or NLP (of Chinese lol). | |
| Metadata | |
| ID the ID of the comment (start from 0) | |
| Movie_Name_EN the English name of the movie | |
| Movie_Name_CN the Chinese name of the movie | |
| Crawl_Date the date that the data are crawled | |
| Number the number of the comment | |
| Username the username of the account | |
| Date the date that the comment posted | |
| Star the star that users give to the movie (from 1 to 5, 5 grades) | |
| Comment the content of the comment | |
| Like the count of ""like"" on the comment" | |
| GoodReads Book reviews,Dataset of book Reviews,Gnanesh,30,"Version 2,2017-10-21|Version 1,2017-10-21",,CSV,305 MB,CC4,"9,980 views",374 downloads,,,https://www.kaggle.com/gnanesh/goodreads-book-reviews,"Context | |
| Goodreads Book Reviews dataset. | |
| Content | |
| This data was collected duing oct (12-21) '17 | |
| Schema: root |-- bookID: string (nullable = false) |-- title: string (nullable = false) |-- author: string (nullable = false) |-- rating: string (nullable = false) |-- ratingsCount: string (nullable = false) |-- reviewsCount: string (nullable = false) |-- reviewerName: string (nullable = true) |-- reviewerRatings: string (nullable = true) |-- review: string (nullable = true) | |
| Acknowledgements | |
| Thank you, Goodreads for the data. All the data here belongs to Goodreads. | |
| Inspiration | |
| This data was actually scrapped for WebMining Project." | |
| Recommendation System for Angers Smart City,Global Smart City Survey for Recommendation Systems,ASSO PAVIC - Angers Smart City,30,"Version 1,2017-11-29","learning | |
| artificial intelligence | |
| mobile web",Other,757 KB,ODbL,313 views,14 downloads,,0 topics,https://www.kaggle.com/assopavic/recommendation-system-for-angers-smart-city,"Context | |
| In January 2017, PAVIC submitted a survey focused on Smart City and collected data from 1076 people. This survey was fully anonymous and was aimed at improving the citizens' lives in the future Smart City | |
| Content | |
| The idea of the survey is to obtain a precise insight concerning the citizens' reactions to different recommendations in two different contexts. In clear, respondents were asked to choose among a set of 18 recommendations those that they would be most interested in if it were proposed in two different contexts: on a sunny and warm (20°C) Saturday afternoon in Spring (referred to as the ""Sun"" context) and on a rainy and cold (8°C) Saturday afternoon in Winter (referred to as the ""Rain"" context). The recommendations concerned various subjects : social or cultural events, discounts in restaurants, useful city information, etc. and people were asked in each context which they would like to receive as push notifications on their phones. For each context, respondents could give several or no responses. | |
| The following are the precise text of the questions submitted to the respondents : - for the ""Sun"" dataset : A Saturday in spring around 4pm with a comfy temperature of 20°C or 68°F. You are downtown in the city for the afternoon and your mobile application can send you personalized services/activities notifications in real-time. Which of the following activities would you want to receive in your notifications ? (Several may be chosen). - for the ""Rain"" dataset : A rainy Saturday in the winter around 4pm with a cool temperature of 8°C or 46°F. You are downtown in the city for the afternoon and your mobile application can send you personalized services/activities notifications in real-time. Which of the following activities would you want to receive in your notifications ? (Several may be chosen). | |
| Inspiration | |
| These dataset could allow future applications both to simulate recommendation system algorithms, and to deduce clusters from the collected profiles." | |
| Hacker News Posts,Hacker News posts from the past 12 months (including # of votes and comments),Hacker News,30,"Version 1,2016-09-27","news agencies | |
| internet",CSV,45 MB,CC0,"6,415 views",650 downloads,34 kernels,,https://www.kaggle.com/hacker-news/hacker-news-posts,"This data set is Hacker News posts from the last 12 months (up to September 26 2016). | |
| It includes the following columns: | |
| title: title of the post (self explanatory) | |
| url: the url of the item being linked to | |
| num_points: the number of upvotes the post received | |
| num_comments: the number of comments the post received | |
| author: the name of the account that made the post | |
| created_at: the date and time the post was made (the time zone is Eastern Time in the US) | |
| One fun project suggestion is a model to predict the number of votes a post will attract. | |
| The scraper is written, so I can keep this up-to-date and add more historical data. I can also scrape the comments. Just make the request in this dataset's forum. | |
| The is a fork of minimaxir's HN scraper (thanks minimaxir): https://github.com/minimaxir/get-all-hacker-news-submissions-comments" | |
| Education Statistics,Studies of education status indicators around the world,World Bank,30,"Version 1,2016-11-18","countries | |
| education",CSV,296 MB,Other,"16,223 views","2,151 downloads",10 kernels,0 topics,https://www.kaggle.com/theworldbank/education-statistics,"Context | |
| The World Bank EdStats All Indicator Query holds over 4,000 internationally comparable indicators that describe education access, progression, completion, literacy, teachers, population, and expenditures. The indicators cover the education cycle from pre-primary to vocational and tertiary education. | |
| Content | |
| In addition to the above mentioned indicators, this dataset also holds learning outcome data from international and regional learning assessments (e.g. PISA, TIMSS, PIRLS), equity data from household surveys, and projection/attainment data to 2050. For further information, please visit the EdStats website. | |
| Inspiration | |
| In your opinion, what are some of the more surprising indicators? Are there any you would consider adding? | |
| Which countries have the highest adult illiteracy rates? Have they changed over time, and do rates vary based on age bracket? | |
| Do school enrollment rates have an impact on adult illiteracy rates? If so, can you determine approximately how long a change in enrollment takes in order to impact illiteracy? Does this change vary among countries, and if so, can you point to changes in policies or NGO efforts that might play a role? | |
| Acknowledgements | |
| Data was acquired from the World Bank, and can be accessed in multiple formats here." | |
| Illegal Immigrants Arrested by US Border Patrol,Has the number of Mexican citizens crossing the border increased or decreased?,US Customs and Border Protection,30,"Version 1,2017-01-28","crime | |
| international relations",CSV,6 KB,CC0,"7,911 views",940 downloads,55 kernels,,https://www.kaggle.com/cbp/illegal-immigrants,"Content | |
| This report provides statistics for the number of illegal immigrants arrested or apprehended by the border patrol in each division (or sector) of the United States borders with Canada, Mexico, and Caribbean islands; this data is a partial measure of the flow of people illegally entering the United States. | |
| Acknowledgements | |
| Data was compiled and published by the US Border Patrol on the Customs and Border Protection webpage." | |
| Road Accidents Incidence,Road Accidents Data Great Britain 1979-2015,Akshay Babbar,29,"Version 1,2017-01-23",road transport,CSV,68 MB,Other,"9,609 views","1,611 downloads",10 kernels,3 topics,https://www.kaggle.com/akshay4/road-accidents-incidence,"Context | |
| Road Accidents | |
| Content | |
| Dataset has been fetched from here and the files have been merged and cleaned to reach the final data attached. Primarily Captures Road Accidents in UK between 1979 and 2015 and has 70 features/columns and about 250K rows. Also attached with it is an excel file with Multiple Tabs that can help one to understand the Data. | |
| Acknowledgements | |
| Data has been fetched from Open Data Platform UK and is being shared under Open Government Licence. For more details refer to Open Data UK" | |
| Tourists Visiting Brazil,Who visited Brazil between 1989 and 2015?,Luiza Fontana,29,"Version 1,2016-11-27","brazil | |
| leisure",CSV,33 MB,Other,"6,142 views",750 downloads,18 kernels,,https://www.kaggle.com/zafontana/touristsinbrazil,"Context: | |
| This dataset contains the number of international tourists arriving in Brazil each month from 1989 to 2015. | |
| Content: | |
| Continent | |
| Country | |
| State of arrival | |
| Way in (by land, sea, river or air) | |
| Year | |
| Month | |
| Count | |
| Acknowledgements: | |
| I've downloaded this dataset from dados.gov.br, the Brazilian open data portal, and tried to tidy it up a bit." | |
| International Debt Statistics,Major Financial Indicators from Developing and Advanced Economies,World Bank,29,"Version 1,2016-11-25",finance,CSV,14 MB,Other,"6,979 views",999 downloads,10 kernels,0 topics,https://www.kaggle.com/theworldbank/international-debt-statistics,"Context | |
| Focuses on financial flows, trends in external debt, and other major financial indicators for developing and advanced economies (data from Quarterly External Debt Statistics and Quarterly Public Sector Debt databases). Includes over 200 time series indicators from 1970 to 2014, for most reporting countries, and pipeline data for scheduled debt service payments on existing commitments to 2022. | |
| Content | |
| This dataset contains country names and indicator variables from 1970 until 2024. Additional materials and detailed descriptions of the datasets can be downloaded from here. | |
| Acknowledgement | |
| The original datasets and data dictionaries can be found here. | |
| Inspiration | |
| Few ideas for exploring the dataset: | |
| Compare the current account balance across countries. Is there a pattern associated with developing vs. advanced economies? | |
| How have the debt-related indicators changed over time? Are these strongly associated with other financial indicators?" | |
| Storm Prediction Center,"Historic tornado data, 1950 to 2015",JeffTennis,29,"Version 3,2016-11-22|Version 2,2016-11-06|Version 1,2016-11-06",climate,CSV,5 MB,Other,"10,219 views","1,212 downloads",11 kernels,5 topics,https://www.kaggle.com/jtennis/spctornado,"Database of tornado activity from 1950 to 2015 | |
| Created by National Weather service and available at http://www.spc.noaa.gov/gis/svrgis/ | |
| Enhance understanding of where tornados happen, indicators of damage, and weather conditions associated with tornados (temp/El Nino, La Nina) | |
| Metadata available at http://www.spc.noaa.gov/wcm/data/SPC_severe_database_description.pdf" | |
| Suicides in India,Sucides in each state is classified according to various parameters from 2001-12,Rajanand Ilangovan / இராஜ்ஆனந்த் இளங்கோவன்,29,"Version 1,2017-07-23","india | |
| mental health | |
| death | |
| health",CSV,15 MB,CC4,"11,599 views","1,940 downloads",26 kernels,5 topics,https://www.kaggle.com/rajanand/suicides-in-india,"Connect/Follow me on LinkedIn for more updates on interesting dataset like this. Thanks. | |
| Context | |
| This data set contains yearly suicide detail of all the states/u.t of India by various parameters from 2001 to 2012. | |
| Content | |
| Time Period: 2001 - 2012 Granularity: Yearly Location: States and U.T's of India | |
| Parameters: | |
| a) Suicide causes b) Education status c) By means adopted d) Professional profile e) Social status | |
| Acknowledgements | |
| National Crime Records Bureau (NCRB), Govt of India has shared this dataset under Govt. Open Data License - India. NCRB has also shared the historical data on their website" | |
| One Week of Global News Feeds,7 days of tracking 20k news feeds worldwide,Rohk,29,"Version 2,2017-09-30|Version 1,2017-09-25","news agencies | |
| historiography | |
| linguistics | |
| internet",CSV,280 MB,CC4,"6,201 views",377 downloads,6 kernels,2 topics,https://www.kaggle.com/therohk/global-news-week,"Context | |
| This dataset is a snapshot of most of the new news content published online over one week (August 24, 2017 through August 30, 2017). | |
| Prepared by Rohit Kulkarni | |
| It includes approximately 1.4 million articles, with 20,000 news sources and 20+ languages. | |
| This dataset has just four fields (as per the column metadata): | |
| publish_time - earliest known time of the url appearing online in yyyyMMddHHmm format, IST timezone | |
| feed_code - unique identifier for the publisher or domain | |
| source_url - url of the article | |
| headline_text - Headline of the article (UTF8, 20 possible languages) | |
| See the ""Basic Data Exploration"" notebook for a quick look at the dataset contents. | |
| Inspiration | |
| The sources include news feeds, news websites, government agencies, tech journals, company websites, blogs and wikipedia updates. The data has been collected by polling RSS feeds and by crawling other large news aggregators. | |
| This 7 day slice was selected as there wasn't any downtime or internet outage during the interval. New news content is produced at this rate by publishers everyday, throughout the year. | |
| Several other News datasets exploring other attributes, countries and topics can be seen on my profile. | |
| Acknowledgements | |
| This dataset is free to use with the following citation: | |
| Rohit Kulkarni (2017), One Week of Global Feeds [News CSV Dataset], doi:10.7910/DVN/ILAT5B, Retrieved from: [this url]" | |
| Tsunami Causes and Waves,"Cause, magnitude, and intensity of every tsunami since 2000 BC",NOAA,29,"Version 1,2017-02-03",oceans,CSV,3 MB,CC0,"6,660 views",898 downloads,13 kernels,2 topics,https://www.kaggle.com/noaa/seismic-waves,"Context | |
| Tsunami is a Japanese word that translates to ""harbor wave"". It is a wave or a series of waves generated by an impulsive vertical displacement of the surface of the ocean or other body of water. Tsunamis have been responsible for over 500,000 fatalities throughout the world — almost half from the 2004 Indian Ocean earthquake and tsunami! | |
| Content | |
| The NOAA/WDS tsunami database is a listing of historical tsunami source events and runup locations throughout the world from 2000 B.C. to the present. The events were gathered from scientific and scholarly sources, regional and worldwide catalogs, tide gauge data, deep ocean sensor data, individual event reports, and unpublished works. There are currently over 2,000 source events in the database with event validities greater than one and over 13,000 runup locations where tsunami effects were observed. | |
| Acknowledgements | |
| NOAA's National Centers for Environmental Information (NCEI) and the World Data Service for Geophysics compiled and published this tsunami database for tsunami warning centers, engineers, oceanographers, seismologists, and the general public." | |
| "NASA Astronauts, 1959-Present",Which American astronaut has spent the most time in space?,NASA,29,"Version 1,2017-03-08","space | |
| astronauts | |
| employment",CSV,80 KB,CC0,"4,751 views",560 downloads,17 kernels,0 topics,https://www.kaggle.com/nasa/astronaut-yearbook,"Context | |
| The term ""astronaut"" derives from the Greek words meaning ""space sailor"" and refers to all who have been launched as crew members aboard NASA spacecraft bound for orbit and beyond. | |
| Content | |
| The National Aeronautics and Space Administration (NASA) selected the first group of astronauts in 1959. From 500 candidates with the required jet aircraft flight experience and engineering training in addition to a height below 5 feet 11 inches, seven military men became the nation's first astronauts. The second and third groups chosen included civilians with extensive flying experience. By 1964, requirements had changed, and emphasis was placed on academic qualifications; in 1965, six scientist astronauts were selected from a group of 400 applicants who had a doctorate or equivalent experience in the natural sciences, medicine, or engineering. The group named in 1978 was the first of space shuttle flight crews and fourteen groups have been selected since then with a mix of pilots and mission specialists. | |
| There are currently 50 active astronauts and 35 management astronauts in the program; 196 astronauts have retired or resigned and 49 are deceased (as of April 2013). | |
| Acknowledgements | |
| This dataset was published by the National Aeronautics and Space Administration as the ""Astronaut Fact Book"" (April 2013 edition). Active astronauts' mission names and flight statistics were updated from the NASA website. | |
| Inspiration | |
| Which American astronaut has spent the most time in space? What university has produced the most astronauts? What subject did the most astronauts major in at college? Have most astronauts served in the military? Which branch? What rank did they achieve?" | |
| Toy Products on Amazon,"10,000 toy products on Amazon.com",PromptCloud,29,"Version 1,2017-09-16",internet,CSV,34 MB,CC4,"8,759 views","1,253 downloads",3 kernels,0 topics,https://www.kaggle.com/PromptCloudHQ/toy-products-on-amazon,"Context | |
| This is a pre-crawled dataset, taken as subset of a bigger dataset (more than 7 million products) that was created by extracting data from Amazon.com. | |
| Content | |
| This dataset has following fields: | |
| product_name | |
| manufacturer - The item manufacturer, as reported on Amazon. Some common ""manufacturers"", like Disney, actually outsource their assembly line. | |
| price | |
| number_available_in_stock | |
| number_of_reviews | |
| number_of_answered_questions - Amazon includes a Question and Answer service on all or most of its products. This field is a count of how many questions that were asked actually got answered. | |
| average_review_rating | |
| amazon_category_and_sub_category - A tree-based, >>-delimited categorization for the item in question. | |
| customers_who_bought_this_item_also_bought - References to other items that similar users bought. This is a recommendation engine component that played a big role in making Amazon popular initially. | |
| description | |
| product_information | |
| product_description | |
| items_customers_buy_after_viewing_this_item | |
| customer_questions_and_answers - A string entry with all of the product's JSON question and answer pairs. | |
| customer_reviews - A string entry with all of the product's JSON reviews. | |
| sellers - A string entry with all of the product's JSON seller information (many products on Amazon are sold by third parties). | |
| Acknowledgements | |
| This dataset was created by PromptCloud's in-house web-crawling service. | |
| Inspiration | |
| This detailed dataset can be used to answer questions like: | |
| What types of toys are most popular on Amazon? | |
| How dominant are brands in the Amazon toy market? | |
| Can you break down reviews to analyze their sentiment and contents?" | |
| NYC Property Sales,A year's worth of properties sold on the NYC real estate market,City of New York,29,"Version 1,2017-09-23","cities | |
| real estate",CSV,13 MB,CC0,"9,350 views","1,362 downloads",8 kernels,3 topics,https://www.kaggle.com/new-york-city/nyc-property-sales,"Context | |
| This dataset is a record of every building or building unit (apartment, etc.) sold in the New York City property market over a 12-month period. | |
| Content | |
| This dataset contains the location, address, type, sale price, and sale date of building units sold. A reference on the trickier fields: | |
| BOROUGH: A digit code for the borough the property is located in; in order these are Manhattan (1), Bronx (2), Brooklyn (3), Queens (4), and Staten Island (5). | |
| BLOCK; LOT: The combination of borough, block, and lot forms a unique key for property in New York City. Commonly called a BBL. | |
| BUILDING CLASS AT PRESENT and BUILDING CLASS AT TIME OF SALE: The type of building at various points in time. See the glossary linked to below. | |
| For further reference on individual fields see the Glossary of Terms. For the building classification codes see the Building Classifications Glossary. | |
| Note that because this is a financial transaction dataset, there are some points that need to be kept in mind: | |
| Many sales occur with a nonsensically small dollar amount: $0 most commonly. These sales are actually transfers of deeds between parties: for example, parents transferring ownership to their home to a child after moving out for retirement. | |
| This dataset uses the financial definition of a building/building unit, for tax purposes. In case a single entity owns the building in question, a sale covers the value of the entire building. In case a building is owned piecemeal by its residents (a condominium), a sale refers to a single apartment (or group of apartments) owned by some individual. | |
| Acknowledgements | |
| This dataset is a concatenated and slightly cleaned-up version of the New York City Department of Finance's Rolling Sales dataset. | |
| Inspiration | |
| What can you discover about New York City real estate by looking at a year's worth of raw transaction records? Can you spot trends in the market, or build a model that predicts sale value in the future?" | |
| Arabic Handwritten Characters Dataset,Arabic Handwritten Characters Data-set,Mohamed Loey,28,"Version 1,2017-06-23",,CSV,73 MB,ODbL,"4,195 views",488 downloads,3 kernels,,https://www.kaggle.com/mloey1/ahcd1,"Arabic Handwritten Characters Dataset | |
| Astract | |
| Handwritten Arabic character recognition systems face several challenges, including the unlimited variation in human handwriting and large public databases. In this work, we model a deep learning architecture that can be effectively apply to recognizing Arabic handwritten characters. A Convolutional Neural Network (CNN) is a special type of feed-forward multilayer trained in supervised mode. The CNN trained and tested our database that contain 16800 of handwritten Arabic characters. In this paper, the optimization methods implemented to increase the performance of CNN. Common machine learning methods usually apply a combination of feature extractor and trainable classifier. The use of CNN leads to significant improvements across different machine-learning classification algorithms. Our proposed CNN is giving an average 5.1% misclassification error on testing data. | |
| Context | |
| The motivation of this study is to use cross knowledge learned from multiple works to enhancement the performance of Arabic handwritten character recognition. In recent years, Arabic handwritten characters recognition with different handwriting styles as well, making it important to find and work on a new and advanced solution for handwriting recognition. A deep learning systems needs a huge number of data (images) to be able to make a good decisions. | |
| Content | |
| The data-set is composed of 16,800 characters written by 60 participants, the age range is between 19 to 40 years, and 90% of participants are right-hand. Each participant wrote each character (from ’alef’ to ’yeh’) ten times on two forms as shown in Fig. 7(a) & 7(b). The forms were scanned at the resolution of 300 dpi. Each block is segmented automatically using Matlab 2016a to determining the coordinates for each block. The database is partitioned into two sets: a training set (13,440 characters to 480 images per class) and a test set (3,360 characters to 120 images per class). Writers of training set and test set are exclusive. Ordering of including writers to test set are randomized to make sure that writers of test set are not from a single institution (to ensure variability of the test set). | |
| In an experimental section we showed that the results were promising with 94.9% classification accuracy rate on testing images. In future work, we plan to work on improving the performance of handwritten Arabic character recognition. | |
| Acknowledgements | |
| Ahmed El-Sawy, Mohamed Loey, Hazem EL-Bakry, Arabic Handwritten Characters Recognition using Convolutional Neural Network, WSEAS, 2017 Our proposed CNN is giving an average 5.1% misclassification error on testing data. | |
| Inspiration | |
| Creating the proposed database presents more challenges because it deals with many issues such as style of writing, thickness, dots number and position. Some characters have different shapes while written in the same position. For example the teh character has different shapes in isolated position. | |
| Benha University | |
| http://bu.edu.eg/staff/mloey | |
| https://mloey.github.io/" | |
| Delhi Weather Data,Delhi Weather Data from 1997 to 2016 december,MahirKukreja,28,"Version 2,2017-04-25|Version 1,2017-04-14",climate,CSV,6 MB,Other,"7,647 views","1,284 downloads",26 kernels,,https://www.kaggle.com/mahirkukreja/delhi-weather-data,"Context | |
| This dataset contains weather data for New Delhi, India. | |
| Content | |
| This data was taken out from wunderground with the help of their easy to use api. It contains various features such as temperature, pressure, humidity, rain, precipitation,etc. | |
| Acknowledgements | |
| This data is owned by wunderground and although I ended up using noaa's data for my research, i thought that i'd share this data here as I haven't worked on hourly data yet and this might be of huge importance. | |
| Inspiration | |
| The main target is to develop a prediction model accurate enough for predicting the weather. We can try something like predicting the weather in the next 24 hours like microsoft tried some time back. | |
| https://blogs.microsoft.com/next/2015/08/10/hows-the-weather-using-artificial-intelligence-for-better-answers/#sm.018l60051a9neka10is1m5qpi6u5y" | |
| Indian Premier League CSV dataset,577 matches up to season 9,HarshaVardhan,28,"Version 2,2016-12-30|Version 1,2016-12-27",cricket,CSV,6 MB,Other,"15,661 views","3,225 downloads",12 kernels,,https://www.kaggle.com/harsha547/indian-premier-league-csv-dataset,"Upon request from some users, I am uploading CSV Version. | |
| Yes, There is already a dataset from manas | |
| However, I thought this dataset is different than that one.which includes player metadata, information about all the 11 players who participated in the match. | |
| Thoughts : | |
| ■ who are the valuable players for respective teams. | |
| ■ who are more effective bowlers to bowl in the slog overs , is it spinners ? | |
| ■ Dhoni's strike rate against left-arm spinners in last five overs | |
| Have fun with this dataset. Files in the dataset include: | |
| 1. Ball_by_Ball : Includes ball by ball details of all the 577 matches. | |
| 2. Match : Match metadata | |
| 3. Player : Player metadata | |
| 4. Player_Match : to know , who is the captain and keeper of the match , Includes every player who take part in match even If player haven't get a chance to either bat or bowl. | |
| 5. Season : Season wise details , Orange cap , Purple cap , Man_Of_The_Series | |
| 6. Team : Team Name | |
| Diagram" | |
| Air quality in northern Taiwan,"Air quality monitoring data from northern Taiwan, 2015",Nelson Chu,28,"Version 5,2016-07-29|Version 4,2016-07-27|Version 3,2016-07-27|Version 2,2016-07-27|Version 1,2016-07-27",environment,CSV,18 MB,Other,"7,300 views",840 downloads,13 kernels,2 topics,https://www.kaggle.com/nelsonchu/air-quality-in-northern-taiwan,"Context | |
| This data is from Environmental Protection Administration, Executive Yuan, R.O.C. (Taiwan). | |
| There is air quality data and meteorological monitoring data for research and analysis (only include northern Taiwan 2015). | |
| Content | |
| 25 observation stations data in the 2015_Air_quality_in_northern_Taiwan.csv | |
| The columns in csv file are: | |
| time - The first column is observation time of 2015 | |
| station - The second column is station name, there is 25 observation stations | |
| [Banqiao, Cailiao, Datong, Dayuan, Guanyin, Guting, Keelung, Linkou, Longtan, Pingzhen, Sanchong, Shilin, Songshan, Tamsui, Taoyuan, Tucheng, Wanhua, Wanli, Xindian, Xinzhuang, Xizhi, Yangming, Yonghe, Zhongli, Zhongshan] | |
| items - From the third column to the last one | |
| item - unit - description | |
| SO2 - ppb - Sulfur dioxide | |
| CO - ppm - Carbon monoxide | |
| O3 - ppb - ozone | |
| PM10 - μg/m3 - Particulate matter | |
| PM2.5 - μg/m3 - Particulate matter | |
| NOx - ppb - Nitrogen oxides | |
| NO - ppb - Nitric oxide | |
| NO2 - ppb - Nitrogen dioxide | |
| THC - ppm - Total Hydrocarbons | |
| NMHC - ppm - Non-Methane Hydrocarbon | |
| CH4 - ppm - Methane | |
| UVB - UVI - Ultraviolet index | |
| AMB_TEMP - Celsius - Ambient air temperature | |
| RAINFALL - mm | |
| RH - % - Relative humidity | |
| WIND_SPEED - m/sec - The average of last ten minutes per hour | |
| WIND_DIREC - degress - The average of last ten minutes per hour | |
| WS_HR - m/sec - The average of hour | |
| WD_HR - degress - The average of hour | |
| PH_RAIN - PH - Acid rain | |
| RAIN_COND - μS/cm - Conductivity of acid rain | |
| Data mark | |
| # indicates invalid value by equipment inspection | |
| * indicates invalid value by program inspection | |
| x indicates invalid value by human inspection | |
| NR indicates no rainfall | |
| blank indicates no data | |
| License | |
| Open Government Data License, version 1.0 http://data.gov.tw/license" | |
| Billboard 1964-2015 Songs + Lyrics,50 years of pop music lyrics,RakanNimer,28,"Version 1,2017-04-16","music | |
| linguistics",CSV,8 MB,Other,"7,244 views",856 downloads,10 kernels,2 topics,https://www.kaggle.com/rakannimer/billboard-lyrics,"Original Dataset Author : https://github.com/walkerkq | |
| From https://github.com/walkerkq/musiclyrics : | |
| 50 Years of Pop Music Lyrics | |
| Billboard has published a Year-End Hot 100 every December since 1958. The chart measures the performance of singles in the U.S. throughout the year. Using R, I’ve combined the lyrics from 50 years of Billboard Year-End Hot 100 (1965-2015) into one dataset for analysis. You can download that dataset here. | |
| The songs used for analysis were scraped from Wikipedia’s entry for each Billboard Year-End Hot 100 Songs (e.g., 2014). This is the year-end chart, not weekly rankings. Many artists have made the weekly chart but not the final year end chart. The final chart is calculated using an inverse point system based on the weekly Billboard charts (100 points for a week at number one, 1 point for a week at number 100, etc). | |
| I used the xml and RCurl packages to scrape song and artist names from each Wikipedia entry. I then used that list to scrape lyrics from sites that had predictable URL strings (for example, metrolyrics.com uses metrolyrics.com/SONG-NAME-lyrics-ARTIST-NAME.html). If the first site scrape failed, I moved onto the second, and so on. About 78.9% of the lyrics were scraped from metrolyics.com, 15.7% from songlyrics.com, 1.8% from lyricsmode.com. About 3.6% (187/5100) were unavailable. | |
| The dataset features 5100 observations with the features rank (1-100), song, artist, year, lyrics, and source. The artist feature is fairly standardized thanks to Wikipedia, but there is still quite a bit of noise when it comes to artist collaborations (Justin Timberlake featuring Timbaland, for example). If there were any errors in the lyrics that were scraped, such as spelling errors or derivatives like ""nite"" instead of ""night,"" they haven't been corrected. | |
| Full analysis can be found here. | |
| walkerkq | |
| Acknowledgements | |
| Dataset is a mirror of : https://github.com/walkerkq/musiclyrics All credits to gathering it goes to https://github.com/walkerkq | |
| Inspiration | |
| What makes a song's lyrics popular ?" | |
| Silicon Valley Diversity Data,What’s diversity like for 23 top tech companies?,Rachael Tatman,28,"Version 1,2017-11-09","united states | |
| employment | |
| demographics",CSV,217 KB,ODbL,"4,654 views",553 downloads,7 kernels,2 topics,https://www.kaggle.com/rtatman/silicon-valley-diversity-data,"Context | |
| There has been a lot of discussion of the ways in which the workforce for Silicon Valley tech companies differs from that of the United States as a whole. In particular, a lot of evidence suggests that tech workers (who tend to be more highly paid than workers in many other professions) are more likely to be white and male. This dataset will allow you to investigate the demographics for 23 Silicon Valley tech companies for yourself. | |
| Contents | |
| This database contains EEO-1 reports filed by Silicon Valley tech companies. It was compiled by Reveal from The Center for Investigative Reporting. | |
| There are six columns in this dataset: | |
| company: Company name | |
| year: For now, 2016 only | |
| race: Possible values: ""American_Indian_Alaskan_Native"", ""Asian"", ""Black_or_African_American"", ""Latino"", ""Native_Hawaiian_or_Pacific_Islander"", ""Two_or_more_races"", ""White"", ""Overall_totals"" | |
| gender: Possible values: ""male"", ""female"". Non-binary gender is not counted in EEO-1 reports. | |
| job_category: Possible values: ""Administrative support"", ""Craft workers"", ""Executive/Senior officials & Mgrs"", ""First/Mid officials & Mgrs"", ""laborers and helpers"", ""operatives"", ""Professionals"", ""Sales workers"", ""Service workers"", ""Technicians"", ""Previous_totals"", ""Totals"" | |
| count: Mostly integer values, but contains ""na"" for a no-data variable. | |
| Acknowledgements: | |
| The EEO-1 database is licensed under the Open Database License (ODbL) by Reveal from The Center for Investigative Reporting. | |
| You are free to copy, distribute, transmit and adapt the spreadsheet, so long as you: | |
| credit Reveal (including this link if it’s distributed online); | |
| inform Reveal that you are using the data in your work by emailing Sinduja Rangarajan at srangarajan@revealnews.org; and | |
| offer any new work under the same license. | |
| Inspiration: | |
| How does each company’s workforce compare to the United States population as a whole? You can find county level diversity information here. | |
| Which company is the most diverse? Least diverse?" | |
| Global Food Prices,743k Rows of Monthly Market Food Prices Across Developing Countries,Jacob Boysen,28,"Version 1,2017-08-04","food and drink | |
| economics",CSV,83 MB,Other,"7,664 views","1,261 downloads",3 kernels,0 topics,https://www.kaggle.com/jboysen/global-food-prices,"Context: | |
| Global food price fluctuations can cause famine and large population shifts. Price changes are increasingly critical to policymakers as global warming threatens to destabilize the food supply. | |
| Content: | |
| Over 740k rows of prices obtained in developing world markets for various goods. Data includes information on country, market, price of good in local currency, quantity of good, and month recorded. | |
| Acknowledgements: | |
| Compiled by the World Food Program and distributed by HDX. | |
| Inspiration: | |
| This data would be particularly interesting to pair with currency fluctuations, weather patterns, and/or refugee movements--do any price changes in certain staples predict population upheaval? Do certain weather conditions influence market prices? | |
| License: | |
| Released under CC BY-IGO." | |
| Fatal Police Shootings in the US,Fatal police shootings in the US since 2015 with additional US census data,Karolina Wullum,28,"Version 1,2017-09-23","united states | |
| death | |
| crime | |
| violence",CSV,3 MB,CC4,"4,738 views","1,560 downloads",7 kernels,2 topics,https://www.kaggle.com/kwullum/fatal-police-shootings-in-the-us,"The 2014 killing of Michael Brown in Ferguson, Missouri, began the protest movement culminating in Black Lives Matter and an increased focus on police accountability nationwide. | |
| Since Jan. 1, 2015, The Washington Post has been compiling a database of every fatal shooting in the US by a police officer in the line of duty. It's difficult to find reliable data from before this period, as police killings haven't been comprehensively documented, and the statistics on police brutality are much less available. As a result, a vast number of cases go unreported. | |
| The Washington Post is tracking more than a dozen details about each killing - including the race, age and gender of the deceased, whether the person was armed, and whether the victim was experiencing a mental-health crisis. They have gathered this information from law enforcement websites, local new reports, social media, and by monitoring independent databases such as ""Killed by police"" and ""Fatal Encounters"". The Post has also conducted additional reporting in many cases. | |
| There are four additional datasets. These are US census data on poverty rate, high school graduation rate, median household income, and racial demographics. | |
| Source of census data: https://factfinder.census.gov/faces/nav/jsf/pages/community_facts.xhtml" | |
| Biomechanical features of orthopedic patients,Classifying patients based on six features,UCI Machine Learning,28,"Version 1,2017-09-07",,CSV,50 KB,CC0,"5,074 views","1,588 downloads",9 kernels,0 topics,https://www.kaggle.com/uciml/biomechanical-features-of-orthopedic-patients,"Context | |
| The data have been organized in two different but related classification tasks. | |
| column_3C_weka.csv (file with three class labels) | |
| The first task consists in classifying patients as belonging to one out of three categories: Normal (100 patients), Disk Hernia (60 patients) or Spondylolisthesis (150 patients). | |
| column_2C_weka.csv (file with two class labels) | |
| For the second task, the categories Disk Hernia and Spondylolisthesis were merged into a single category labelled as 'abnormal'. Thus, the second task consists in classifying patients as belonging to one out of two categories: Normal (100 patients) or Abnormal (210 patients). | |
| Content | |
| Field Descriptions: | |
| Each patient is represented in the data set by six biomechanical attributes derived from the shape and orientation of the pelvis and lumbar spine (each one is a column): | |
| pelvic incidence | |
| pelvic tilt | |
| lumbar lordosis angle | |
| sacral slope | |
| pelvic radius | |
| grade of spondylolisthesis | |
| Acknowledgements | |
| The original dataset was downloaded from UCI ML repository: | |
| Lichman, M. (2013). UCI Machine Learning Repository [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California, School of Information and Computer Science | |
| Files were converted to CSV | |
| Inspiration | |
| Use these biomechanical features to classify patients according to their labels" | |
| Global Historical Climatology Network,Land and ocean temperature anomalies,NOAA,28,"Version 1,2016-10-24","climate | |
| history",CSV,20 MB,Other,"9,696 views",871 downloads,28 kernels,,https://www.kaggle.com/noaa/global-historical-climatology-network,"Global Historical Climatology Network-Monthly (GHCN-M) | |
| Context | |
| The Global Historical Climatology Network (GHCN) is an integrated database of climate summaries from land surface stations across the globe. This data set contains gridded mean temperature anomalies, or departures from a reference value or long-term average, from the Global Historical Climatology Network-Monthly (GHCN-M) version 3.2.1 temperature data set. The gridded anomalies were produced from GHCN-M bias corrected data. Each month of data consists of 2,592 gridded data points produced on a 5° by 5° basis for the entire globe (72 longitude by 36 latitude grid boxes). | |
| Frequency: Monthly | |
| Period: 1880 to 2016 | |
| Content | |
| Gridded data for every month from January 1880 to the most recent month is available. The data are temperature anomalies in degrees Celsius. Each gridded value was multiplied by 100 and written to file as an integer. Missing values are represented by the value -9999. The data are formatted by year, month, latitude and longitude. There are 72 longitude grid values per line -- each grid is labeled as a concatenation of ""lon"", ""w"" or ""e"", then the degree. The latitude is captured in the ""lat"" field where the value indicates the lower bound of a grid cell (e.g. 85 indicates 85-90N whereas -90 indicates 85-90S). Longitude values are written from 180°W to 180°E, and latitude values from 90°N to 90°S. | |
| This dataset permits the quantification of changes in the mean monthly temperature and precipitation for the earth's surface. Changes in the observing system itself have been carefully removed to allow for the true climate variability at the earth's surface to be represented in the data. | |
| Many surface weather stations undergo minor relocations through their history of observation. Stations may also be subject to changes in instrumentation as measurement technology evolves. Further, the land use/land cover in the vicinity of an observing site may also change with time. Such modifications to an observing site have the potential to alter a thermometer's microclimate exposure characteristics and/or change the bias of measurements, the impact of which can be a systematic shift in the mean level of temperature readings that is unrelated to true climate variations. The process of removing such ""non-climatic"" artifacts in a climate time series is called homogenization. In version 3 of the GHCN-Monthly temperature data, the apparent impacts of documented and undocumented inhomogeneities are detected and removed through automated pairwise comparisons of mean monthly temperature series as detailed in Menne and Williams [2009]. | |
| Inspiration | |
| This granular dataset permits extensive historical analysis of the earth’s climate to answer questions about climate change, including how different regions of the planet have been affected by changes in temperature over time. Get started by forking the kernel Mapping Historical Temperature Anomalies with R. | |
| Acknowledgements | |
| This data is a product of NOAA's National Centers for Environmental Information (NCEI). It was compiled through the aggregation and analysis of many thousands of weather station records. The compete description of the processes and methods used may be found at https://www.ncdc.noaa.gov/ghcnm/v3.php. | |
| The Global Historical Climatology Network-Monthly (GHCN-M) temperature dataset was first developed in the early 1990s (Vose et al. 1992). A second version was released in 1997 following extensive efforts to increase the number of stations and length of the data record (Peterson and Vose, 1997). Methods for removing inhomogeneities from the data record associated with non-climatic influences such as changes in instrumentation, station environment, and observing practices that occur over time were also included in the version 2 release (Peterson and Easterling, 1994; Easterling and Peterson 1995). Since that time efforts have focused on continued improvements in dataset development methods including new quality control processes and advanced techniques for removing data inhomogeneities (Menne and Williams, 2009). | |
| License | |
| Public Domain License" | |
| Restaurants on TripAdvisor,"Data on 18,000 restaurants",PromptCloud,28,"Version 1,2017-09-15",internet,CSV,7 MB,CC4,"8,571 views","1,302 downloads",2 kernels,0 topics,https://www.kaggle.com/PromptCloudHQ/restaurants-on-tripadvisor,"Context | |
| This is a pre-crawled dataset, taken as subset of a bigger dataset (more than 14,92,992 restaurants) that was created by extracting data from TripAdvisor.com. | |
| Content | |
| This dataset has following fields: | |
| Restaurant URL | |
| Name | |
| Address | |
| Phone | |
| City | |
| State | |
| Country | |
| Neighbourhood | |
| Email ID | |
| Menu | |
| Website | |
| Latitude | |
| Longitude | |
| About Restaurant | |
| Cuisine | |
| Good for(suitable) | |
| Price | |
| Currency | |
| Rating | |
| Ranking | |
| Deal(Promotion) | |
| Total Review | |
| Last Reviewed | |
| Recommended | |
| Dining Option | |
| Award | |
| Acknowledgements | |
| This dataset was created by PromptCloud's in-house web-crawling service. | |
| Inspiration | |
| The country-wise analyses of cuisine, rating, ranking, etc. can be performed." | |
| NYC Parking Tickets,"42.3M Rows of Parking Ticket Data, Aug 2013-June 2017",City of New York,28,"Version 2,2017-10-27|Version 1,2017-10-26","cities | |
| government | |
| law | |
| automobiles",CSV,8 GB,CC0,"6,894 views","1,210 downloads",3 kernels,0 topics,https://www.kaggle.com/new-york-city/nyc-parking-tickets,"Context | |
| The NYC Department of Finance collects data on every parking ticket issued in NYC (~10M per year!). This data is made publicly available to aid in ticket resolution and to guide policymakers. | |
| Content | |
| There are four files, covering Aug 2013-June 2017. The files are roughly organized by fiscal year (July 1 - June 30) with the exception of the initial dataset. The initial dataset also lacks 8 columns that are included in the other three datasets (although be warned that these additional data columns are used sparingly). See the dataset descriptions for exact details. Columns include information about the vehicle ticketed, the ticket issued, location, and time. | |
| Acknowledgements | |
| Data was produced by NYC Department of Finance. FY2018 data is found here with updates every third week of the month. | |
| Inspiration | |
| When are tickets most likely to be issued? Any seasonality? | |
| Where are tickets most commonly issued? | |
| What are the most common years and types of cars to be ticketed?" | |
| Electoral Donations in Brazil,Statistical analysis of Brazil in 2014,FelipeLeiteAntunes,27,"Version 5,2017-11-19|Version 4,2017-10-12|Version 3,2017-10-10|Version 2,2017-10-09|Version 1,2017-10-09","crime | |
| finance | |
| politics",CSV,69 MB,ODbL,"2,711 views",196 downloads,4 kernels,,https://www.kaggle.com/felipeleiteantunes/electoral-donations-brazil2014,"Context | |
| Brazil has elections every two years, but alternating between two different types of elections, each type occurring every four years. There are the municipal elections, where mayors and city council members are elected (the last one occurred in 2016) and general elections where president, governors, senators and congressmen (regional and national) are elected (the last one occurred in 2014). Brazil has 26 federal units plus the federal district. Each one of these units (regions) elects its senators, congressmen and governors. | |
| For each federal unit, Brazil's TSE provides information on the donations declared by the three entities: candidates, parties and committees. The data comprises information describing every donation received. The donations can be divided in two categories with respect to the donor: they can come from legal persons (private citizens, identified by the CPF (CPF is an identification number used by the Brazilian tax revenue office. It is roughly the Brazilian analogue to a social security number. With the same purpose, companies are identified with a similar number called CNPJ number) or from legal entities (i.e. companies, identified by the CNPJ number). Also, some entities can make donations among them (the party can give part of the money from a given donation to a candidate). In this type of transaction, the information on the original donor is also specified in the declarations. From now on, these type of donations will be referred to as non-original donations. Apart from information concerning each Brazilian federal unit separately, one can also obtain the information declared by the parties and committees at national level and for the presidential campaign (which has national and not regional scope). | |
| Related paper: | |
| https://arxiv.org/pdf/1707.08826.pdf" | |
| Moscow Ring Roads,Shapefiles for use in Sberbank,Chippy,27,"Version 3,2017-05-27|Version 2,2017-05-22|Version 1,2017-05-22","russia | |
| geography",CSV,3 MB,ODbL,"3,255 views","1,136 downloads",11 kernels,0 topics,https://www.kaggle.com/nigelcarpenter/sberbankmoscowroads,"Context | |
| This data set was created for use in the Sberbank Kaggle competition. | |
| Content | |
| The data consists of three GIS shapefiles one for each of the 3 major Moscow ring roads; the MKAD, TTK (or third ring) and Sadovoe (or garden ring). | |
| Acknowledgements | |
| The road shapefiles have been extracted from OpenStreetMap data, processed in QGiS to extract only the roads of interest. | |
| OpenStreetMap License: https://www.OpenStreetMap.org/copyright | |
| Inspiration | |
| With these files and the distances given in the Sberbank dataset it should be possible to better understand the location of properties. With a better understanding of location it may be possible to improve the quality of the overall dataset which contains material amounts of missing or poorly coded data." | |
| Circadian Rhythm in the Brain,Fluorescence signal from the circadian regulation region of the brain,Kevin Mader,27,"Version 3,2017-03-21|Version 2,2017-03-21|Version 1,2017-03-21",neuroscience,Other,1 GB,Other,"4,722 views",293 downloads,8 kernels,,https://www.kaggle.com/kmader/circadian-rhythm-in-the-brain,"Overview | |
| The data are a time-series of fluorescence images measured of at OHSU Lab of Charles Allen (https://www.ohsu.edu/xd/research/centers-institutes/oregon-institute-occupational-health-sciences/research/allen/). | |
| Introduction | |
| We use a fluorescent protein as a reporter for the circadian clock gene Period1. We are able to follow the expression of this gene in many neurons for several days to understand how the neural network in the suprachiasmatic nucleus synchronizes the circadian clock of individual neurons to produce a precise circadian rhythm. We analyze each image to determine the fluorescence intensity of each neuron over multiple circadian cycles. | |
| FAQ | |
| How where the images obtained: which animal and what staining? | |
| The images were taken from a transgenic mouse in which expression of the fluorescent protein Venus is driven by the promoter for the circadian clock gene Period 1. | |
| What is the anatomy of the images, and how are they oriented? | |
| The bright line is the third ventricle, which resides on the midline of the brain. The two bright regions on either side of the ventricle are the two portions of the Suprachiasmatic nucleus (SCN). Below the ventricle and the SCN is a dark horizontal band that represents the optic chasm. | |
| What is the bright vertical line in the top middle? | |
| The bright line is the third ventricle. Pericytes that line the ventricle express the Venus at very high levels. We don't know the function of the circadian clock in these cells. | |
| Challenge | |
| Currently we have to analyze each experiment by hand to follow an individual through a couple hundred images. This takes several days. This problem is going to get worse because we have just purchased a new automated microscope stage that will allow us to simultaneously image from four suprachiasmatic nuclei. | |
| Preview | |
| Ideas for Analysis | |
| Wavelets (pywavelets) following https://www.ncbi.nlm.nih.gov/pubmed/18931366 | |
| Questions | |
| Do the cells move during the experiment? | |
| How regular is their signal? | |
| Is the period 24 hours? | |
| Do nearby cells oscillate together? | |
| Do they form chunks or groups, over what range do they work? | |
| Are there networks formed from time-precedence?" | |
| Cheltenham's Facebook Groups,Discover How a Community Leverages Facebook,Mike Chirico,27,"Version 44,2017-12-23|Version 43,2017-11-20|Version 42,2017-11-14|Version 41,2017-11-06|Version 40,2017-08-19|Version 39,2017-07-15|Version 38,2017-07-02|Version 37,2017-06-25|Version 36,2017-06-10|Version 35,2017-05-28|Version 34,2017-05-14|Version 33,2017-03-23|Version 32,2017-03-11|Version 31,2017-03-11|Version 30,2017-03-03|Version 29,2017-02-23|Version 28,2016-12-02|Version 27,2016-10-23|Version 26,2016-10-18|Version 25,2016-10-11|Version 24,2016-10-03|Version 23,2016-09-22|Version 22,2016-09-16|Version 21,2016-09-05|Version 20,2016-08-19|Version 19,2016-08-08|Version 18,2016-08-03|Version 17,2016-07-30|Version 16,2016-07-22|Version 15,2016-07-20|Version 14,2016-07-15|Version 13,2016-06-25|Version 12,2016-06-25|Version 11,2016-06-19|Version 10,2016-06-15|Version 9,2016-06-10|Version 8,2016-06-09|Version 7,2016-06-07|Version 6,2016-06-07|Version 5,2016-06-07|Version 4,2016-06-05|Version 3,2016-06-04|Version 2,2016-06-03|Version 1,2016-06-03",internet,CSV,59 MB,GPL,"12,868 views",862 downloads,50 kernels,,https://www.kaggle.com/mchirico/cheltenham-s-facebook-group,"Facebook is becoming an essential tool for more than just family and friends. Discover how Cheltenham Township (USA), a diverse community just outside of Philadelphia, deals with major issues such as the Bill Cosby trial, everyday traffic issues, sewer I/I problems and lost cats and dogs. And yes, theft. | |
| Communities work when they're connected and exchanging information. What and who are the essential forces making a positive impact, and when and how do conversational threads get directed or misdirected? | |
| Use Any Facebook Public Group | |
| You can leverage the examples here for any public Facebook group. For an example of the source code used to collect this data, and a quick start docker image, take a look at the following project: facebook-group-scrape. | |
| Data Sources | |
| There are 4 csv files in the dataset, with data from the following 5 public Facebook groups: | |
| Unofficial Cheltenham Township | |
| Elkins Park Happenings! | |
| Free Speech Zone | |
| Cheltenham Lateral Solutions | |
| Cheltenham Township Residents | |
| post.csv | |
| These are the main posts you will see on the page. It might help to take a quick look at the page. Commas in the msg field have been replaced with {COMMA}, and apostrophes have been replaced with {APOST}. | |
| gid Group id (5 different Facebook groups) | |
| pid Main Post id | |
| id Id of the user posting | |
| name User's name | |
| timeStamp | |
| shares | |
| url | |
| msg Text of the message posted. | |
| likes Number of likes | |
| comment.csv | |
| These are comments to the main post. Note, Facebook postings have comments, and comments on comments. | |
| gid Group id | |
| pid Matches Main Post identifier in post.csv | |
| cid Comment Id. | |
| timeStamp | |
| id Id of user commenting | |
| name Name of user commenting | |
| rid Id of user responding to first comment | |
| msg Message | |
| like.csv | |
| These are likes and responses. The two keys in this file (pid,cid) will join to post and comment respectively. | |
| gid Group id | |
| pid Matches Main Post identifier in post.csv | |
| cid Matches Comments id. | |
| response Response such as LIKE, ANGRY etc. | |
| id The id of user responding | |
| name Name of the user responding | |
| member.csv | |
| These are all the members in the group. Some members never, or rarely, post or comment. You may find multiple entries in this table for the same person. The name of the individual never changes, but they change their profile picture. Each profile picture change is captured in this table. Facebook gives users a new id in this table when they change their profile picture. | |
| gid Group id | |
| id Id of the member | |
| name Name of the member | |
| url URL of the member" | |
| Forecasts for Product Demand,Make Accurate Forecasts for Thousands of Different Products,FelixZhao,27,"Version 1,2017-08-25","business | |
| supply chain | |
| product",CSV,49 MB,GPL,"6,631 views",857 downloads,2 kernels,0 topics,https://www.kaggle.com/felixzhao/productdemandforecasting,"Context | |
| The dataset contains historical product demand for a manufacturing company with footprints globally. The company provides thousands of products within dozens of product categories. There are four central warehouses to ship products within the region it is responsible for. Since the products are manufactured in different locations all over the world, it normally takes more than one month to ship products via ocean to different central warehouses. If forecasts for each product in different central with reasonable accuracy for the monthly demand for month after next can be achieved, it would be beneficial to the company in multiple ways. This dataset is all real-life data and products/warehouse and category information encoded. | |
| Content | |
| Product_Code: The product name encoded. Warehouse: Warehouse name encoded. Product_Category: Product Category for each Product_Code encoded. Date: The date customer needs the product. Order_Demand: single order qty. | |
| Inspiration | |
| Is it possible to make forecasts for thousands of products (some of them are highly variable in terms of monthly demand) for the the month after next?" | |
| "Bad teeth, sugar and government health spending",Thanks to Gapminder Data,churandy,27,"Version 3,2016-08-19|Version 2,2016-08-19|Version 1,2016-08-19","public health | |
| dentistry | |
| health",CSV,304 KB,CC0,"12,681 views","2,035 downloads",16 kernels,,https://www.kaggle.com/angelmm/healthteethsugar,"If you get richer your teeth could get worse (if you eat more sugar foods) or better (because of better health assistance or, even, more education and health-conciousness). These variables can be analysed with these data, downloaded from Gapminder Data: | |
| Bad teeth per child (12 yr, WHO) | |
| GDP/capita (US$, inflation-adjusted, World Bank) | |
| Government health spending per person (US$, WHO) | |
| Sugar comsumption per person (g per day, FAO) | |
| Literacy rate, adult total (% of people ages 15 and above, UNESCO)" | |
| 65 World Indexes,Why are some countries so different?,JoniHoppen,27,"Version 2,2017-02-27|Version 1,2017-02-18",international relations,CSV,120 KB,ODbL,"5,836 views",934 downloads,16 kernels,,https://www.kaggle.com/joniarroba/65-world-indexes-gathered,"Context | |
| Why some countries are so different from the others? | |
| Feel free to upvote :) Autor: Joni Hoppen - linkedin - https://www.linkedin.com/in/joniarroba/ | |
| Content | |
| I have gathered manually most of the information at World Bank, Unicef and so on. Some data were not there so I used K-nn to guess some values and have a full dataset that can be used of our data science community. | |
| Information of each of the 65 variables were made available here http://bit.ly/2l2Hjh3 | |
| Acknowledgements | |
| Thanks www.aquare.la Advanced Analytics that came up with the idea of creating this dataset to test their VORTX tool. Also Thanks to professionals involved in creating Indexes and collecting them, this is such a great valuable work to help better see the world. | |
| Inspiration | |
| What would be the best, way to equalize the world?" | |
| Historical Sales and Active Inventory,Records of sold and unsold products and their characteristics,JamesS,27,"Version 3,2016-12-08|Version 2,2016-12-08|Version 1,2016-12-08","business | |
| product",CSV,13 MB,Other,"18,882 views","2,483 downloads",34 kernels,4 topics,https://www.kaggle.com/flenderson/sales-analysis,"Context | |
| Attached is a set of products in which we are trying to determine which products we should continue to sell, and which products to remove from our inventory. The file contains BOTH historical sales data AND active inventory, which can be discerned with the column titled ""File Type"". | |
| We suspect that data science applied to the set--such as a decision tree analysis or logistic regression, or some other machine learning model---can help us generate a value (i.e., probability score) for each product, that can be used as the main determinant evaluating the inventory. Each row in the file represents one product. | |
| It is important to note that we have MANY products in our inventory, and very few of them tend to sell (only about 10% sell each year) and many of the products only have a single sale in the course of a year. | |
| Content | |
| The file contains historical sales data (identified with the column titled File_Type) along with current active inventory that is in need of evaluation (i.e., File Type = ""Active""). The historical data shows sales for the past 6 months. The binary target (1 = sale, 0 = no sale in past six months) is likely the primary target that should drive the analysis. | |
| The other columns contain numeric and categorical attributes that we deem relevant to sales. | |
| Note that some of the historical sales SKUs are ALSO included in the active inventory. | |
| A few comments about the attributes included, as we realize we may have some attributes that are unnecessary or may need to be explained. | |
| SKU_number: This is the unique identifier for each product. | |
| Order: Just a sequential counter. Can be ignored. | |
| SoldFlag: 1 = sold in past 6 mos. 0 = Not sold | |
| MarketingType = Two categories of how we market the product. This should probably be ignored, or better yet, each type should be considered independently. | |
| New_Release_Flag = Any product that has had a future release (i.e., Release Number > 1) | |
| Inspiration | |
| (1) What is the best model to use that will provide us with a probability estimate of a sale for each SKU? We are mainly interested in a relative unit that we can continuously update based on these attributes (and others that we add, as we are able). | |
| (2) Is it possible to provide a scored file (i.e., a probability score for each SKU in the file), and to provide an evaluation of the accuracy of the selected model? | |
| (3) What are the next steps we should take? | |
| Thanks very much for any suggestions you may provide." | |
| Devanagari Character Set,Over 92 thousand images of characters from devanagari script,Rishi Anand,27,"Version 4,2017-11-21|Version 3,2017-08-25|Version 2,2017-06-02|Version 1,2017-04-25","languages | |
| linguistics | |
| image data | |
| multiclass classification",Other,121 MB,CC0,"5,590 views",433 downloads,14 kernels,2 topics,https://www.kaggle.com/rishianand/devanagari-character-set,"Context | |
| This is a dataset of Devanagari Script Characters. It comprises of 92000 images [32x32 px] corresponding to 46 characters, consonants ""ka"" to ""gya"", and the digits 0 to 9. The vowels are missing. | |
| Content | |
| The CSV file is of the dimension 92000 * 1025. There are 1024 input features of pixel values in grayscale (0 to 255). The column ""character"" represents the Devanagari Character Name corresponding to each image. | |
| Acknowledgements | |
| This dataset was originally created by Computer Vision Research Group, Nepal. [website archive] (https://web.archive.org/web/20160105230017/http://cvresearchnepal.com/wordpress/dhcd/) | |
| Example Script | |
| https://nbviewer.jupyter.org/github/rishianand9/devanagari-character-recognition/blob/master/DCRS.ipynb" | |
| American Time Use Survey,Multi-Year Survey Microdata Files from 2003-2015,US Bureau of Labor Statistics,27,"Version 2,2017-06-15|Version 1,2016-11-10","time series | |
| demographics",CSV,2 GB,Other,"22,868 views","1,159 downloads",10 kernels,0 topics,https://www.kaggle.com/bls/american-time-use-survey,"Context | |
| The American Time Use Survey (ATUS) is the Nation’s first federally administered, continuous survey on time use in the United States. The goal of the survey is to measure how people divide their time among life’s activities. | |
| In ATUS, individuals are randomly selected from a subset of households that have completed their eighth and final month of interviews for the Current Population Survey (CPS). ATUS respondents are interviewed only one time about how they spent their time on the previous day, where they were, and whom they were with. The survey is sponsored by the Bureau of Labor Statistics and is conducted by the U.S. Census Bureau. | |
| The major purpose of ATUS is to develop nationally representative estimates of how people spend their time. Many ATUS users are interested in the amount of time Americans spend doing unpaid, nonmarket work, which could include unpaid childcare, eldercare, housework, and volunteering. The survey also provides information on the amount of time people spend in many other activities, such as religious activities, socializing, exercising, and relaxing. In addition to collecting data about what people did on the day before the interview, ATUS collects information about where and with whom each activity occurred, and whether the activities were done for one’s job or business. Demographic information—including sex, race, age, educational attainment, occupation, income, marital status, and the presence of children in the household—also is available for each respondent. Although some of these variables are updated during the ATUS interview, most of this information comes from earlier CPS interviews, as the ATUS sample is drawn from a subset of households that have completed month 8 of the CPS. | |
| The user guide can be found here. | |
| Content | |
| There are 8 datasets containing microdata from 2003-2015: | |
| Respondent file: The Respondent file contains information about ATUS respondents, including their labor force status and earnings. | |
| Roster file: The Roster file contains information about household members and nonhousehold children (under 18) of ATUS respondents. It includes information such as age and sex. | |
| Activity file: The Activity file contains information about how ATUS respondents spent their diary day. It includes information such as activity codes, activity start and stop times, and locations. Because Activity codes have changed somewhat between 2003 and 2015, this file uses activity codes that appear in the 2003-2015 ATUS Coding Lexicon (PDF). | |
| Activity summary file: The Activity summary file contains information about the total time each ATUS respondent spent doing each activity on the diary day. Because Activity codes have changed somewhat between 2003 and 2015, this file uses activity codes that appear in the 2003-2015 ATUS Coding Lexicon (PDF). | |
| Who file: The Who file includes codes that indicate who was present during each activity. | |
| CPS 2003-2015 file: The ATUS-CPS file contains information about each household member of all individuals selected to participate in ATUS. The information on the ATUS-CPS file was collected 2 to 5 months before the ATUS interview. | |
| Eldercare Roster file: The ATUS Eldercare Roster file contains information about people for whom the respondent provided care. Eldercare data have been collected since 2011. | |
| Replicate weights file: The Replicate weights file contains miscellaneous ATUS weights. | |
| The ATUS interview data dictionary can be found here. | |
| The ATUS Current Population Survey (CPS) data dictionary can be found here. | |
| The ATUS occupation and industry codes can be found here. | |
| The ATUS activity lexicon can be found here. | |
| Acknowledgements | |
| The original datasets can be found here. | |
| Inspiration | |
| How do daily activities differ by: | |
| labor force status | |
| income | |
| household composition | |
| geographical region | |
| disability status" | |
| Journalists Killed Worldwide Since 1992,Journalist Deaths from 1992-2016,Committee to Protect Journalists,27,"Version 1,2016-11-13","news agencies | |
| journalism | |
| death | |
| crime",CSV,313 KB,Other,"6,165 views",910 downloads,17 kernels,,https://www.kaggle.com/cpjournalists/journalists-killed-worldwide-since-1992,"Context | |
| CPJ began compiling detailed records on journalist deaths in 1992. CPJ applies strict journalistic standards when investigating a death. One important aspect of their research is determining whether a death was work-related. As a result, they classify deaths as ""motive confirmed"" or ""motive unconfirmed."" | |
| Content | |
| The dataset contains 18 variables: | |
| Type: CPJ classified deaths as motive confirmed or motive confirmed, as well as Media Workers | |
| Date | |
| Name | |
| Sex | |
| Country_killed | |
| Organization | |
| Nationality | |
| Medium | |
| Job | |
| Coverage | |
| Freelance | |
| Local_Foreign | |
| Source_fire | |
| Type_death | |
| Impunity_for_murder | |
| Taken_captive | |
| Threatened | |
| Tortured | |
| Acknowledgements | |
| The original dataset can be found here. | |
| Inspiration | |
| Some ideas for exploring the dataset: | |
| What is the trend in journalist deaths over time and how does this differ by type of death, job, coverage, and country? | |
| Are there differences by sex and/or nationality?" | |
| Chronic Disease Indicators,"Disease Data Across the US, 2001-2016",Centers for Disease Control and Prevention,27,"Version 1,2017-08-18",healthcare,CSV,117 MB,CC0,"7,215 views",859 downloads,2 kernels,0 topics,https://www.kaggle.com/cdc/chronic-disease,"Context: | |
| CDC's Division of Population Health provides cross-cutting set of 124 indicators that were developed by consensus and that allows states and territories and large metropolitan areas to uniformly define, collect, and report chronic disease data that are important to public health practice and available for states, territories and large metropolitan areas. In addition to providing access to state-specific indicator data, the CDI web site serves as a gateway to additional information and data resources. | |
| Content: | |
| A variety of health-related questions were assessed at various times and places across the US over the past 15 years. Data is provided with confidence intervals and demographic stratification. | |
| Acknowledgements: | |
| Data was compiled by the CDC. | |
| Inspiration: | |
| Any interesting trends in certain groups? | |
| Any correlation between disease indicators and locality hospital spending?" | |
| Correlates of War: World Religions,"World, regional, and national populations by religious beliefs",University of Michigan,27,"Version 1,2017-01-28","faith and traditions | |
| politics | |
| war",CSV,627 KB,Other,"7,744 views","1,208 downloads",66 kernels,0 topics,https://www.kaggle.com/umichigan/world-religions,"Content | |
| The World Religion Project aims to provide detailed information about religious adherence worldwide since 1945. It contains data about the number of adherents by religion in each of the states in the international system for every half-decade period. Some of the religions are divided into religious families, and the breakdown of adherents within a given religion into religious families is provided to the extent data are available. | |
| The project was developed in three stages. The first stage consisted of the formation of a religions tree. A religion tree is a systematic classification of major religions and of religious families within those major religions. To develop the religion tree we prepared a comprehensive literature review, the aim of which was to define a religion, to find tangible indicators of a given religion of religious families within a major religion, and to identify existing efforts at classifying world religions. The second stage consisted of the identification of major data sources of religious adherence and the collection of data from these sources according to the religion tree classification. This created a dataset that included multiple records for some states for a given point in time, yet contained multiple missing data for specific states, specific time periods, and specific religions. The third stage consisted of cleaning the data, reconciling discrepancies of information from different sources, and imputing data for the missing cases. | |
| Acknowledgements | |
| The dataset was created by Zeev Maoz, University of California-Davis, and Errol Henderson, Pennsylvania State University, and published by the Correlates of War Project." | |
| "Chase Bank Branch Deposits, 2010-2016",Where did Chase Bank customers deposit the most money last year?,Chase Bank,27,"Version 1,2017-03-15",finance,CSV,953 KB,CC0,"8,385 views",980 downloads,8 kernels,,https://www.kaggle.com/chasebank/bank-deposits,"Content | |
| This dataset includes a record for every branch of Chase Bank in the United States, including the branch's name and number, date established as a bank office and (if applicable) acquired by JP Morgan Chase, physical location as street address, city, state, zip, and latitude and longitude coordinates, and the amount deposited at the branch (or the institution, for the bank's main office) between July 1 and June 30, 2016, in US dollars. | |
| Acknowledgements | |
| The location data was scraped from the Chase Bank website. The deposit data was compiled from the Federal Deposit Insurance Corporation's annual Summary of Deposits reports. | |
| Inspiration | |
| Where did Chase Bank customers deposit the most money last year? Which bank branch has seen the most growth in deposits? How did the bank network of branch locations grow over the past century? What city has the most bank branches per capita?" | |
| Common Voice,"500 hours of speech recordings, with speaker demographics",Mozilla,27,"Version 2,2017-12-12|Version 1,2017-12-01","languages | |
| acoustics | |
| linguistics",Other,12 GB,CC0,"2,281 views",231 downloads,,2 topics,https://www.kaggle.com/mozillaorg/common-voice,"General Information | |
| Common Voice is a corpus of speech data read by users on the Common Voice website (http://voice.mozilla.org/), and based upon text from a number of public domain sources like user submitted blog posts, old books, movies, and other public speech corpora. Its primary purpose is to enable the training and testing of automatic speech recognition (ASR) systems. | |
| Structure | |
| The corpus is split into several parts for your convenience. The subsets with “valid” in their name are audio clips that have had at least 2 people listen to them, and the majority of those listeners say the audio matches the text. The subsets with “invalid” in their name are clips that have had at least 2 listeners, and the majority say the audio does not match the clip. All other clips, ie. those with fewer than 2 votes, or those that have equal valid and invalid votes, have “other” in their name. | |
| The “valid” and “other” subsets are further divided into 3 groups: | |
| dev - for development and experimentation | |
| train - for use in speech recognition training | |
| test - for testing word error rate | |
| Organization and Conventions | |
| Each subset of data has a corresponding csv file with the following naming convention: | |
| “cv-{type}-{group}.csv” | |
| Here “type” can be one of {valid, invalid, other}, and “group” can be one of {dev, train, test}. Note, the invalid set is not divided into groups. | |
| Each row of a csv file represents a single audio clip, and contains the following information: | |
| filename - relative path of the audio file | |
| text - supposed transcription of the audio | |
| up_votes - number of people who said audio matches the text | |
| down_votes - number of people who said audio does not match text | |
| age - age of the speaker, if the speaker reported it | |
| teens: '< 19' | |
| twenties: '19 - 29' | |
| thirties: '30 - 39' | |
| fourties: '40 - 49' | |
| fifties: '50 - 59' | |
| sixties: '60 - 69' | |
| seventies: '70 - 79' | |
| eighties: '80 - 89' | |
| nineties: '> 89' | |
| gender - gender of the speaker, if the speaker reported it | |
| male | |
| female | |
| other | |
| accent - accent of the speaker, if the speaker reported it | |
| us: 'United States English' | |
| australia: 'Australian English' | |
| england: 'England English' | |
| canada: 'Canadian English' | |
| philippines: 'Filipino' | |
| hongkong: 'Hong Kong English' | |
| indian: 'India and South Asia (India, Pakistan, Sri Lanka)' | |
| ireland: 'Irish English' | |
| malaysia: 'Malaysian English' | |
| newzealand: 'New Zealand English' | |
| scotland: 'Scottish English' | |
| singapore: 'Singaporean English' | |
| southatlandtic: 'South Atlantic (Falkland Islands, Saint Helena)' | |
| african: 'Southern African (South Africa, Zimbabwe, Namibia)' | |
| wales: 'Welsh English' | |
| bermuda: 'West Indies and Bermuda (Bahamas, Bermuda, Jamaica, Trinidad)' | |
| The audio clips for each subset are stored as mp3 files in folders with the same naming conventions as it’s corresponding csv file. So, for instance, all audio data from the valid train set will be kept in the folder “cv-valid-train” alongside the “cv-valid-train.csv” metadata file. | |
| Acknowledgments | |
| This dataset was compiled by Michael Henretty, Tilman Kamp, Kelly Davis & The Common Voice Team, who included the following acknowledgments: | |
| We sincerely thank all of the people who donated their voice on the Common Voice website and app. You are the backbone of this project, and we thank you for making this possible! | |
| We also thank our community on Discourse (https://discourse.mozilla-community.org/c/voice) and Github (https://github.com/mozilla/voice-web), you have made this project better every step of the way. | |
| And special thanks to Mycroft, SNIPS.ai, Mythic, Tatoeba.org, Bangor University, and SAP for joining us on this journey. We look forward to working more with each of you." | |
| Google Job Skills,Find what you need to get a job at Google,Niyamat Ullah,26,"Version 1,2018-01-07","databases | |
| learning | |
| internet",CSV,407 KB,CC4,"5,112 views",605 downloads,5 kernels,0 topics,https://www.kaggle.com/niyamatalmass/google-job-skills,"Context | |
| There is a question in our mind that which language, skills, and experience should we add to our toolbox for getting a job in Google. Well, I think why not we find out the answer by analyzing the Google Jobs Site. Google published all of their jobs at https://careers.google.com/. So I scraped all of the job data from that site by going every job page using Selenium. I only take Job Title, Job Location, Job responsibilities, minimum and preferred qualifications for this dataset. | |
| Content | |
| This dataset is collected using Selenium by scraping all of the jobs text for Google Career site. About the column | |
| Title: The title of the job | |
| Category: Category of the job | |
| Location: Location of the job | |
| Responsibilities: Responsibilities for the job | |
| Minimum Qualifications: Minimum Qualifications for the job | |
| Preferred Qualifications: Preferred Qualifications for the job | |
| Acknowledgements | |
| This dataset is collected using Selenium. This product uses the Google Career site but is not endorsed or certified by Google Career site. | |
| Inspiration | |
| You can find most popular skills for Google Jobs | |
| Create identical job posts | |
| Most popular languages | |
| etc" | |
| US Energy Statistics,All data published by the US Energy Information Administration,Sohier Dane,26,"Version 2,2017-08-22|Version 1,2017-08-22","government agencies | |
| energy",{}JSON,911 MB,Other,"3,855 views",389 downloads,,0 topics,https://www.kaggle.com/sohier/us-energy-statistics,"This dataset contains the entire contents of each major API data set published by the US Energy Information Administration. That's everything from the hourly electricity consumption in the United States to natural gas futures contracts. | |
| This data has been lightly reprocessed from the EIA's bulk download facility by converting each file from a zip of jsons into a single json with the series name as the keys to the specific time series. Please note that there are thousands of time series in here, and many of them may still require additional cleaning to deal with odd date formats and so on. The file preview is unable to show a complete listing. You can usually find full details of a given time series in the 'description' field." | |
| Sentiment Lexicons for 81 Languages,Sentiment Polarity Lexicons (Positive vs. Negative),Rachael Tatman,26,"Version 1,2017-09-14","languages | |
| linguistics",CSV,2 MB,Other,"4,513 views",553 downloads,,3 topics,https://www.kaggle.com/rtatman/sentiment-lexicons-for-81-languages,"Context: | |
| Sentiment analysis, the task of automatically detecting whether a piece of text is positive or negative, generally relies on a hand-curated list of words with positive sentiment (good, great, awesome) and negative sentiment (bad, gross, awful). This dataset contains both positive and negative sentiment lexicons for 81 languages. | |
| Content: | |
| The sentiment lexicons in this dataset were generated via graph propagation based on a knowledge graph--a graphical representation of real-world entities and the links between them. The general intuition is that words which are closely linked on a knowledge graph probably have similar sentiment polarities. For this project, sentiments were generated based on English sentiment lexicons. | |
| This dataset contains sentiment lexicons for the following languages: | |
| Afrikaans | |
| Albanian | |
| Arabic | |
| Aragonese | |
| Armenian | |
| Azerbaijani | |
| Basque | |
| Belarusian | |
| Bengali | |
| Bosnian | |
| Breton | |
| Bulgarian | |
| Catalan | |
| Chinese | |
| Croatian | |
| Czech | |
| Danish | |
| Dutch | |
| Esperanto | |
| Estonian | |
| Faroese | |
| Finnish | |
| French | |
| Galician | |
| Georgian | |
| German | |
| Greek | |
| Gujarati | |
| Haitian Creole | |
| Hebrew | |
| Hindi | |
| Hungarian | |
| Icelandic | |
| Ido | |
| Indonesian | |
| Interlingua | |
| Irish | |
| Italian | |
| Kannada | |
| Khmer | |
| Kirghiz | |
| Korean | |
| Kurdish | |
| Latin | |
| Latvian | |
| Lithuanian | |
| Luxembourgish | |
| Macedonian | |
| Malay | |
| Maltese | |
| Marathi | |
| Norwegian | |
| Norwegian | |
| Persian | |
| Polish | |
| Portuguese | |
| Romanian | |
| Romansh | |
| Russian | |
| Scottish | |
| Serbian | |
| Slovak | |
| Slovene | |
| Spanish | |
| Swahili | |
| Swedish | |
| Tagalog | |
| Tamil | |
| Telugu | |
| Thai | |
| Turkish | |
| Turkmen | |
| Ukrainian | |
| Urdu | |
| Uzbek | |
| Vietnamese | |
| Volapük | |
| Walloon | |
| Welsh | |
| Western Frisian | |
| Yiddish | |
| For more information and additional sentiment lexicons, please visit the project’s website. | |
| Acknowledgements: | |
| This dataset was collected by Yanqing Chen and Steven Skiena. If you use it in your work, please cite the following paper: | |
| Chen, Y., & Skiena, S. (2014). Building Sentiment Lexicons for All Major Languages. In ACL (2) (pp. 383-389). | |
| It is distributed here under the GNU General Public License. Note that this is the full GPL, which allows many free uses, but does not allow its incorporation into any type of distributed proprietary software, even in part or in translation. For commercial applications please contact the dataset creators. | |
| Inspiration: | |
| These word lists contain many words with similar meanings. Can you automatically detect which words are cognates? | |
| Can you use these sentiment lexicons to reverse-engineer the knowledge graphs that generated them?" | |
| Top Spotify Tracks of 2017,Audio features of top Spotify songs,Nadin Tamer,26,"Version 1,2017-12-20","popular culture | |
| music",CSV,13 KB,Other,"3,490 views",567 downloads,,2 topics,https://www.kaggle.com/nadintamer/top-tracks-of-2017,"Top Spotify Tracks of 2017 | |
| At the end of each year, Spotify compiles a playlist of the songs streamed most often over the course of that year. This year's playlist (Top Tracks of 2017) included 100 songs. The question is: What do these top songs have in common? Why do people like them? | |
| Original Data Source: The audio features for each song were extracted using the Spotify Web API and the spotipy Python library. Credit goes to Spotify for calculating the audio feature values. | |
| Data Description: There is one .csv file in the dataset. (featuresdf.csv) This file includes: | |
| Spotify URI for the song | |
| Name of the song | |
| Artist(s) of the song | |
| Audio features for the song (such as danceability, tempo, key etc.) | |
| A more detailed explanation of the audio features can be found in the Metadata tab. | |
| Exploring the Data: Some suggestions for what to do with the data: | |
| Look for patterns in the audio features of the songs. Why do people stream these songs the most? | |
| Try to predict one audio feature based on the others | |
| See which features correlate the most" | |
| Dark Net Marketplace Data (Agora 2014-2015),"Includes over 100,000 unique listings of drugs, weapons and more",philipjames11,26,"Version 1,2017-12-06","crime | |
| illegal drugs | |
| internet",CSV,8 MB,CC0,"3,688 views",374 downloads,,0 topics,https://www.kaggle.com/philipjames11/dark-net-marketplace-drug-data-agora-20142015,"Context | |
| This data set was made from an html rip made by reddit user ""usheep"" who threatened to expose all the vendors on Agora to the police if they did not meet his demands (sending him a small monetary amount ~few hundred dollars in exchange for him not leaking their info). Most information about what happened to ""usheep"" and his threats is nonexistent. He posted the html rip and was never heard from again. Agora shut down a few months after. It is unknown if this was related to ""usheep"" or not, but the raw html data remained. | |
| Content | |
| This is a data parse of marketplace data ripped from Agora (a dark/deep web) marketplace from the years 2014 to 2015. It contains drugs, weapons, books, services, and more. Duplicate listings have been removed and prices have been averaged of any duplicates. All of the data is in a csv file and has over 100,000 unique listings. | |
| It is organized by: | |
| Vendor: The seller | |
| Category: Where in the marketplace the item falls under | |
| Item: The title of the listing | |
| Description: The description of the listing | |
| Price: Cost of the item (averaged across any duplicate listings between 2014 and 2015) | |
| Origin: Where the item is listed to have shipped from | |
| Destination: Where the item is listed to be shipped to (blank means no information was provided, but mostly likely worldwide. I did not enter worldwide for any blanks however as to not make assumptions) | |
| Rating: The rating of the seller (a rating of [0 deals] or anything else with ""deals"" in it means there is not concrete rating as the amount of deals is too small for a rating to be displayed) | |
| Remarks: Only remark options are blank, or ""Average price may be skewed outliar > .5 BTC found"" which is pretty self explanatory. | |
| Acknowledgements | |
| Though I got this data from a 3rd party, it seems as though it originally came from here: https://www.gwern.net/DNM-archives Gwern Branwen seems to have complied all of his dark net marketplace leaks and html rips and has a multitude of possible uses for the data at the link above. It is free for anyone to use as long as proper credit is given to the creator. I would be happy to parse more data if anyone would like to request a specific website and/or format. | |
| Inspiration | |
| This data could be used to track drug dealers across different platforms. Potentially find correlations between different drugs and from where/to they ship in the world to show correlations between types of drugs and where drug dealers that supply them are located. Prices can estimate drug economies in certain regions of the world. Similar listings from 2 different vendors can perhaps point to competition to corner a market, or even show that some vendors may work together to corner a market. There are quite a few opportunities to do some really great stuff to find correlations between illegal drugs, weapons, and more in order to curb the flow of dark net drug trade by identifying high risk regions or vendors. I can potentially do a new parse of other websites so you can find correlations across websites rather than just within Agora." | |
| World Development Indicators,The most accurate global development data available,World Bank,26,"Version 1,2017-08-14","countries | |
| globalization | |
| economics",CSV,234 MB,Other,"3,703 views",490 downloads,,,https://www.kaggle.com/theworldbank/world-development-indicators,"World Development Indicators provides a compilation of relevant, high-quality, and internationally comparable statistics about global development and the fight against poverty. It is intended to help policymakers, students, analysts, professors, program managers, and citizens find and use data related to all aspects of development, including those that help monitor progress toward the World Bank Group’s two goals of ending poverty and promoting shared prosperity. | |
| Content | |
| This dataset includes indicators at both national and regional levels for: -Agriculture & Rural Development -Aid Effectiveness -Climate Change -Economy & Growth -Education -Energy & Mining -Environment -External Debt -Financial Sector -Gender -Health -Infrastructure -Labor & Social Protection -Poverty -Private Sector -Public Sector -Science & Technology -Social Development -Trade, Urban Development | |
| Acknowledgements | |
| This dataset was kindly made available by the World Bank. Please check their instance at http://data.worldbank.org/data-catalog/world-development-indicators for updates and related information." | |
| Match Statistics from top 5 European Leagues,"Italy, Spain, England, Germany, France 2012-2017",Jemilu Mohammed,26,"Version 3,2017-07-07|Version 2,2017-07-07|Version 1,2017-05-16",association football,CSV,6 MB,CC0,"4,376 views",601 downloads,2 kernels,4 topics,https://www.kaggle.com/jangot/ligue1-match-statistics,"Context | |
| I am a student exploring the possibility of making money in football betting. I am currently doing a literature review on modelling association football scores and trying to put together a machine learning system to use for my first betting campaign next season. What I have learned thus far is that outcomes of football events are partly deterministic and partly random. I do not know exactly how to go about implementing this in a machine learning system yet. I am also hoping to find useful features from this dataset. | |
| Content | |
| The data here contains match statistics collected from whoscored.com europes top five leagues from 2012-2013 to 2016-2017 season. It contains just about all match statistics that anyone can ever hope for including but not limited to Goals, Corners, Possession, Ratings, Coaches, LineUps, and other relevent match statistics | |
| The features are simply just self explanatory and have been given long but meaningful names | |
| Acknowledgement | |
| I collected the data from the whoscored.com website. I scraped it using beautifulSoup in python and just extracted the features I thought could have some use. | |
| Inspiration | |
| This is just something I hope could become something but hey, it may be nothing. I am just interested to know the kind of insights that could be generated from this." | |
| "Finishers Boston Marathon 2015, 2016 & 2017","This data has the names, times and general demographics of the finishers",rojour,26,"Version 4,2017-04-30|Version 3,2017-04-30|Version 2,2017-04-30|Version 1,2017-03-25","running | |
| walking",CSV,12 MB,Other,"8,689 views","1,238 downloads",34 kernels,2 topics,https://www.kaggle.com/rojour/boston-results,"Context | |
| This is a list of the finishers of the Boston Marathon of 2015, 2016 and 2017. | |
| It's important to highlight that the Boston Marathon is the oldest marathon run in the US as it is the only marathon (other than olympic trails) that most of the participants have to qualify to participate. | |
| For the professional runners, it's a big accomplishment to win the marathon. For most of the other participants, it's an honor to be part of it. | |
| Content | |
| It contains the name, age, gender, country, city and state (where available), times at 9 different stages of the race, expected time, finish time and pace, overall place, gender place and division place. | |
| Decided to keep every year as a separate file, making it more manageable and easier to deal with it. | |
| Acknowledgements | |
| Data was scrapped from the official marathon website - http://registration.baa.org/2017/cf/Public/iframe_ResultsSearch.cfm | |
| I have found that other people have done this kind of scraping, so, some of those ideas together with things I have learned in my quest to become a data scientist created the set. | |
| You can actually find the scraping notebooks at - https://github.com/rojour/boston_results . Notebook it's not very clean yet, but I will get to it soon... | |
| Inspiration | |
| I was a participant in the marathon 2016 and 2017 edition, as well as a data science student, so it was a natural curiosity. | |
| I have done a preliminary study of some fun facts. You can see the kernel here as well as in the github page listed above. | |
| Already some people have created some fun analysis of the results (mostly of the first part - 2016) that was the first upload, but I am curious of what people may come up with... now that three years are available, it may spark the creative juices of some. | |
| I believe it's a simple, fun dataset that can be used by the new to play with, and by some veterans to get creative." | |
| DC Metro Crime Data,Consolidated set of all registered crimes from crimemaps.dc.gov,LucasVinze,26,"Version 5,2017-11-03|Version 4,2017-04-18|Version 3,2016-08-19|Version 2,2016-08-19|Version 1,2016-08-18",crime,CSV,109 MB,Other,"13,501 views","1,695 downloads",47 kernels,2 topics,https://www.kaggle.com/vinchinzu/dc-metro-crime-data,"Dataset of all of the crimes in the DC metro police system ranging from Theft, Arson, Assault, Homicide, Sex Abuse, Robbery, and Burglary. | |
| Data can be easily geocoded and mapped, trends can be extracted, and predictions can be made. | |
| Would be interesting to combine with other datasets, i.e. changes in housing prices, history of construction sites etc. j An informal hypothesis would be: If the local government invests in fixing the sidewalks in a neighborhood, how much would the investment decrease crime levels on a block by block basis. | |
| Raw Data can be accessed from: http://crimemap.dc.gov/CrimeMapSearch.aspx#tabs-GeoOther | |
| The data is most easily accessed by downloading 1 ward at a time for the specific data range." | |
| UFO Sightings around the world,"80,000+ documented close encounters from the past 70 years",Cam Nugent,26,"Version 1,2017-08-15","history | |
| linguistics",CSV,13 MB,CC0,"5,323 views",677 downloads,5 kernels,0 topics,https://www.kaggle.com/camnugent/ufo-sightings-around-the-world,"Context | |
| Extraterrestrials, visitors, little green men, UFOs, swap gas. What do they want? Where do they come from? Do they like cheeseburgers? This dataset will likely not help you answer these questions. It does contain over 80,000 records of UFO sightings dating back as far as 1949. With the latitude and longitude data it is possible to assess the global distribution of UFO sightings (patterns could aid in planetary defence if invasion proves to be imminent). The dates and times, along with the duration of the UFO's stay and description of the craft also lend themselves to predictions. Can we find patterns in their arrival times and durations? Do aliens work on weekends? Help defend the planet and learn about your fellow earthlings (and when they are most likely to see ET). | |
| Content | |
| Date_time - standardized date and time of sighting | |
| city - location of UFO sighting | |
| state/province - the US state or Canadian province, appears blank for other locations | |
| country - Country of UFO sighting | |
| UFO_shape - a one word description of the ""spacecraft"" | |
| length_of_encounter_seconds - standardized to seconds, length of the observation of the UFO | |
| described_duration _of_encounter - raw description of the length of the encounter (shows uncertainty to previous column) | |
| description - text description of the UFO encounter. Warning column is messy, with some curation it could lend itself to some natural language processing and sentiment analysis. | |
| date_documented - when was the UFO sighting reported | |
| latitude - latitude | |
| longitude - longitude | |
| Note there are missing data in the columns. I've left it as is because depending on what the user is interested in the missing values in any one column may or may not matter. | |
| Acknowledgements | |
| I found these data here: https://github.com/planetsig/ufo-reports Full credit to them for the curation, I added some column headers and just described what I've seen | |
| Inspiration | |
| Some great ways to use these data would be: | |
| A global plot of the locations of recorded UFO sightings. | |
| Can the duration of the UFO visit be predicted from the other data? | |
| Is there a pattern to the appearances? At certain times of day, on certain days of the week or days of the year? (i.e. are people on their way home from the pub more likely to see little green men?) | |
| Are certain shapes of UFO more likely to be seen in different geographical regions." | |
| Association of Tennis Professionals Matches,ATP tournament results from 2000 to 2017,GMAdevs,26,"Version 2,2017-02-07|Version 1,2017-02-05","tennis | |
| sports",CSV,11 MB,CC4,"5,598 views","1,250 downloads",11 kernels,0 topics,https://www.kaggle.com/gmadevs/atp-matches-dataset,"Context | |
| A dataset of ATP matches including individual statistics. | |
| Content | |
| In these datasets there are individual csv files for ATP tournament from 2000 to 2017. | |
| The numbers in the last columns are absolute values, using them you can calculate percentages. | |
| Dataset legend | |
| All the match statistics are in absolute number format, you can convert to percentages using the total point number | |
| ace = absolute number of aces | |
| df = number of double faults | |
| svpt = total serve points | |
| 1stin = 1st serve in | |
| 1st won = points won on 1st serve | |
| 2ndwon = points won on 2nd serve | |
| SvGms = serve games | |
| bpSaved = break point saved | |
| bpFaced = break point faced | |
| Acknowledgement | |
| Thanks to Jeff Sackmann for the excellent work. Be sure to visit his github profile | |
| https://github.com/JeffSackmann/tennis_atp | |
| Inspiration | |
| This dataset would be likely used to develop predictive modeling of tennis matches and to do statistic research. I'm planning to add historical odds and injuries data as soon as I have the time to get them." | |
| U.S. Educational Finances,"Revenues and expenditures for U.S. grade schools, by year and state",Roy Garrard,26,"Version 4,2018-02-04|Version 3,2018-01-16|Version 2,2017-12-01|Version 1,2017-11-29","united states | |
| education | |
| finance",Other,81 MB,CC0,"4,670 views",671 downloads,8 kernels,0 topics,https://www.kaggle.com/noriuk/us-educational-finances,"Context | |
| The United States Census Bureau conducts annual surveys to assess the finances of elementary and high schools. The attached CSV file contains a summary of revenue and expenditure for the years 1992-2015, organized by state. | |
| Content | |
| [elsec_main.csv] A comma-separated spreadsheet containing revenues and expenditures for all U.S. school districts, 1993-2015. | |
| STATE,ENROLL,NAME,YRDATA,TOTALREV,TFEDREV,TSTREV,TLOCREV,TOTALEXP,TCURINST,TCURSSVC,TCURONON,TCAPOUT | |
| Alabama,7568,AUTAUGA CO SCH DIST,1995,31827,2821,21389,7617,27457,15228,7123,2575,2176 | |
| Alabama,19961,BALDWIN CO SCH DIST,1995,93379,6655,55108,31616,87973,48750,22961,6927,6795 | |
| [elsec_summary.csv] A comma-separated spreadsheet containing state summaries of revenues and expenditures, organized by year. | |
| STATE,YEAR,ENROLL,TOTAL_REVENUE,FEDERAL_REVENUE,STATE_REVENUE,LOCAL_REVENUE,TOTAL_EXPENDITURE,INSTRUCTION_EXPENDITURE,SUPPORT_SERVICES_EXPENDITURE,OTHER_EXPENDITURE,CAPITAL_OUTLAY_EXPENDITURE | |
| Alabama,1992,,2678885,304177,1659028,715680,2653798,1481703,735036,,174053 | |
| Alaska,1992,,1049591,106780,720711,222100,972488,498362,350902,,37451 | |
| Be warned, some data will be NaN's (most notably, the 1992 records contain no data for enrollment). | |
| Data was created from the spreadsheets in [elsec.zip] (taken from the U.S. Census Bureau site) using [chew_data.py] and [state_summary.py]. Column names are documented in [school15doc.pdf]. | |
| Sources | |
| https://www.census.gov/programs-surveys/school-finances/data/tables.html | |
| Changelog | |
| [v 0.2] Added data from 1993-2001. Data is now harvested from the main spreadsheets instead of the summary spreadsheets. Data by school district is now available. | |
| [v 0.3] Added 1992 data. Added enrollment data for all years except 1992 (unavailable). | |
| [v 0.4] Straightening a few things out as I play with the data in my own kernel. Changed ""program_other_expenditure"" to ""other_expenditure"" and fixed chew_data.py to properly pull that information. Removed ""non-elsec"" funding and ""program_current_expenditure"" columns." | |
| StarCraft II Replay Analysis,In-Depth Look at StarCraft II Replays,Simon Fraser University - Summit,26,"Version 1,2016-11-05",video games,CSV,532 KB,CC4,"7,535 views",447 downloads,27 kernels,,https://www.kaggle.com/sfu-summit/starcraft-ii-replay-analysis,"Context | |
| This dataset is an aggregate of the screen-fixations from screen movements of StarCraft 2 replay files. | |
| Content | |
| This dataset contains 21 variables: | |
| GameID: Unique ID for each game | |
| LeagueIndex: 1-8 for Bronze, Silver, Gold, Diamond, Master, GrandMaster, Professional leagues | |
| Age: Age of each player | |
| HoursPerWeek: Hours spent playing per week | |
| TotalHours: Total hours spent playing | |
| APM: Action per minute | |
| SelectByHotkeys: Number of unit selections made using hotkeys per timestamp | |
| AssignToHotkeys: Number of units assigned to hotkeys per timestamp | |
| UniqueHotkeys: Number of unique hotkeys used per timestamp | |
| MinimapAttacks: Number of attack actions on minimal per timestamp | |
| MinimapRightClicks: Number of right-clicks on minimal per timestamp | |
| NumberOfPACs: Number of PACs per timestamp | |
| GapBetweenPACs: Mean duration between PACs (milliseconds) | |
| ActionLatency: Mean latency from the onset of PACs to their first action (milliseconds) | |
| ActionsInPAC: Mean number of actions within each PAC | |
| TotalMapExplored: Number of 24x24 game coordinate grids viewed by player per timestamp | |
| WorkersMade: Number of SCVs, drones, probes trained per timestamp | |
| UniqueUnitsMade: Unique units made per timestamp | |
| ComplexUnitsMade: Number of ghosts, investors, and high templars trained per timestamp | |
| ComplexAbilityUsed: Abilities requiring specific targeting instructions used per timestamp | |
| MaxTimeStamp: Time stamp of game's last recorded event | |
| Inspiration | |
| Questions worth exploring: | |
| How do the replay attributes differ by level of player expertise? | |
| What are significant predictors of a player's league? | |
| Acknowledgements | |
| This dataset is from Simon Fraser University - Summit and can be found here. You must give attribution to the work; You may not use this work for commercial purposes; You may not alter, transform, or build upon this work. Any further uses require the permission of the rights holder." | |
| "Crime in Context, 1975-2015",Are violent crime rates rising or falling in American cities?,The Marshall Project,26,"Version 1,2017-02-10","history | |
| crime",CSV,258 KB,Other,"7,097 views","1,295 downloads",13 kernels,,https://www.kaggle.com/marshallproject/crime-rates,"Context | |
| Is crime in America rising or falling? The answer is not as simple as politicians make it out to be because of how the FBI collects crime data from the country’s more than 18,000 police agencies. National estimates can be inconsistent and out of date, as the FBI takes months or years to piece together reports from those agencies that choose to participate in the voluntary program. | |
| To try to fill this gap, The Marshall Project collected and analyzed more than 40 years of data on the four major crimes the FBI classifies as violent — homicide, rape, robbery and assault — in 68 police jurisdictions with populations of 250,000 or greater. We obtained 2015 reports, which have yet to be released by the FBI, directly from 61 of them. We calculated the rate of crime in each category and for all violent crime, per 100,000 residents in the jurisdiction, based on the FBI’s estimated population for that year. We used the 2014 estimated population to calculate 2015 crime rates per capita. | |
| Acknowledgements | |
| The crime data was acquired from the FBI Uniform Crime Reporting program's ""Offenses Known and Clearances by Arrest"" database for the year in question, held at the National Archives of Criminal Justice Data. The data was compiled and analyzed by Gabriel Dance, Tom Meagher, and Emily Hopkins of The Marshall Project; the analysis was published as Crime in Context on 18 August 2016." | |
| UNHCR Refugee Data,Data on Uprooted Populations and Asylum Processing,United Nations,26,"Version 1,2017-08-01","politics | |
| demographics",CSV,40 MB,CC4,"3,751 views",713 downloads,3 kernels,0 topics,https://www.kaggle.com/unitednations/refugee-data,"Context: | |
| The mass movement of uprooted people is a highly charged geopolitical issue. This data, gathered by the UN High Commissioner for Refugees (UNHCR), covers movement of displaced persons (asylum seekers, refugees, internally displaced persons (IDP), stateless). Also included are destination country responses to asylum petitions. | |
| Content: | |
| This dataset includes 6 csv files covering: | |
| Asylum monthly applications opened (asylum_seekers_monthly.csv) | |
| Yearly progress through the refugee system (asylum_seekers.csv) | |
| Refugee demographics (demographics.csv) | |
| Yearly time series data on UNHCR’s populations of concern (time_series.csv) | |
| Yearly population statistics on refugees by residence and destination (persons_of_concern.csv) | |
| Yearly data on resettlement arrivals, with or without UNHCR assistance (resettlement.csv) | |
| Acknowledgements: | |
| This dataset was gathered from UNHCR. Photo by Ali Tareq. | |
| Inspiration: | |
| What are the most frequent destination countries for refugees? How has refugee flow changed? Any trends that could predict future refugee patterns?" | |
| All Lending Club loan data,2007 through current Lending Club accepted and rejected loan data,NathanGeorge,25,"Version 6,2018-02-01|Version 5,2018-01-31|Version 4,2017-08-11|Version 3,2017-03-03|Version 2,2017-03-02|Version 1,2017-03-02","business | |
| finance | |
| lending",CSV,409 MB,CC0,"5,616 views",881 downloads,8 kernels,0 topics,https://www.kaggle.com/wordsforthewise/lending-club,"Context | |
| I wanted an easy way to share all the lending club data with others. Unfortunately, the data on their site is fragmented into many smaller files. There is another lending club dataset on Kaggle, but it hasn't been updated in years. It also doesn't include the rejected loans, which I put in here. | |
| I created a git repo for the code to create this data: https://github.com/nateGeorge/preprocess_lending_club_data | |
| Content | |
| The definitions for the fields are here, at the bottom of the page. | |
| Unfortunately, there is a limit of 500MB for dataset files (so lame!), so I had to compress the files with gzip in the Python pandas package. | |
| I cleaned the data a tiny bit: I removed %s from int_rate and revol_util, and deleted the url column. | |
| To load the data in Python: | |
| import pandas as pd | |
| accept_df = pd.read_csv('../input/accepted_2007_to_2016.csv.gz', compression='gzip') | |
| reject_df = pd.read_csv('../input/rejected_2007_to_2016.csv.gz', compression='gzip') | |
| # too many columns to print the info summary out, so we need to force it | |
| print(accept_df.info(verbose=True, null_counts=True)) | |
| In R: | |
| library(data.table) | |
| accepted_def <- read.csv(gzfile('rejected_2007_to_2016.csv.gz'), na.strings='') | |
| acc_dt <- as.data.table(accepted_def) | |
| rejected_def <- read.csv(gzfile('accepted_2007_to_2016.csv.gz'), na.strings='') | |
| rej_dt <- as.data.table(accepted_def) | |
| There are also separate csvs in the main input folder, but the only advantage over the lending club site is that the 2016 year is joined into one file instead of 4. | |
| Inspiration | |
| I wanted to make this dataset easily available for others to use." | |
| 101 Innovations - Research Tools Survey,Explore global research practices and opinions on scholarly communication,Bianca Kramer,25,"Version 2,2016-05-18|Version 1,2016-05-17",research,CSV,27 MB,CC0,"16,619 views","1,057 downloads",19 kernels,0 topics,https://www.kaggle.com/bmkramer/101-innovations-research-tools-survey,"Many new websites and online tools have come into existence to support scholarly communication in all phases of the research workflow. To what extent researchers are using these and more traditional tools has been largely unknown. This 2015-2016 survey aimed to fill that gap. | |
| The survey captured information on tool usage for 17 research activities, stance towards open access and open science, and expectations of the most important development in scholarly communication. Respondents’ demographics included research roles, country of affiliation, research discipline and year of first publication. The online survey employed an open, non-probability sample. A largely self-selected group of 20,663 researchers, librarians, editors, publishers and other groups involved in research took the survey, which was available in seven languages. The survey was open from May 10, 2015 to February 10, 2016. | |
| This data set contains: | |
| Full raw (anonymized) and cleaned data files (csv, each file containing 20,663 records and 178 variables) | |
| Variable lists for raw and cleaned data files (csv) | |
| Readme file (txt) | |
| The dataset is also deposited in Zenodo: http://dx.doi.org/10.5281/zenodo.49583 | |
| The full description of survey methodology is in a data publication in F1000 Research: http://dx.doi.org/10.12688/f1000research.8414.1 | |
| More information on the project this survey is part of can be found here: http://101innovations.wordpress.com | |
| [edited to add] For quick visual exploration of the data, check out the interactive dashboard on Silk: http://dashboard101innovations.silk.co/ | |
| Contact: | |
| Jeroen Bosman: http://orcid.org/0000-0001-5796-2727 / j.bosman@uu.nl | |
| Bianca Kramer: http://orcid.org/0000-0002-5965-6560 / b.m.r.kramer@uu.nl" | |
| Gun Deaths in the US: 2012-2014,"Information about gun-deaths from the CDC: Ages, gender, intent and more",Zurda,25,"Version 1,2017-01-25","death | |
| crime | |
| violence | |
| demographics",CSV,6 MB,Other,"10,227 views","1,828 downloads",58 kernels,0 topics,https://www.kaggle.com/hakabuk/gun-deaths-in-the-us,"Context | |
| This dataset includes information about gun-death in the US in the years 2012-2014. | |
| Content | |
| The data includes data regarding the victim's age, sex, race, education, intent, time (month and year) and place of death, and whether or not police was at the place of death. | |
| Acknowledgements | |
| I came across this thanks to FiveThirtyEight's Gun Deaths in America project. The data originated from the CDC, and can be found here." | |
| Bible Corpus,English Bible Translations Dataset for Text Mining and NLP,Oswin Rahadiyan Hartono,25,"Version 3,2017-06-16|Version 2,2017-06-15|Version 1,2017-06-07","faith and traditions | |
| linguistics",CSV,427 MB,CC0,"4,131 views",331 downloads,4 kernels,,https://www.kaggle.com/oswinrh/bible,"Context | |
| Bible (or Biblia in Greek) is a collection of sacred texts or scriptures that Jews and Christians consider to be a product of divine inspiration and a record of the relationship between God and humans (Wiki). And for data mining purpose, we could do many things using Bible scriptures as for NLP, Classification, Sentiment Analysis and other particular topics between Data Science and Theology perspective. | |
| Content | |
| Here you will find the following bible versions in sql, sqlite, xml, csv, and json format: | |
| American Standard-ASV1901 (ASV) | |
| Bible in Basic English (BBE) | |
| Darby English Bible (DARBY) | |
| King James Version (KJV) | |
| Webster's Bible (WBT) | |
| World English Bible (WEB) | |
| Young's Literal Translation (YLT) | |
| Each verse is accessed by a unique key, the combination of the BOOK+CHAPTER+VERSE id. | |
| Example: | |
| Genesis 1:1 (Genesis chapter 1, verse 1) = 01001001 (01 001 001) | |
| Exodus 2:3 (Exodus chapter 2, verse 3) = 02002003 (02 002 003) | |
| The verse-id system is used for faster, simplified queries. | |
| For instance: 01001001 - 02001005 would capture all verses between Genesis 1:1 through Exodus 1:5. | |
| Written simply: | |
| SELECT * FROM bible.t_asv WHERE id BETWEEN 01001001 AND 02001005 | |
| Coordinating Tables | |
| There is also a number-to-book key (key_english table), a cross-reference list (cross_reference table), and a bible key containing meta information about the included translations (bible_version_key table). See below SQL table layout. These tables work together providing you a great basis for a bible-reading and cross-referencing app. In addition, each book is marked with a particular genre, mapping in the number-to-genre key (key_genre_english table) and common abbreviations for each book can be looked up in the abbreviations list (key_abbreviations_english table). While its expected that your programs would use the verse-id system, book #, chapter #, and verse # columns have been included in the bible versions tables. | |
| A Valuable Cross-Reference Table | |
| A very special and valuable addition to these databases is the extensive cross-reference table. It was created from the project at http://www.openbible.info/labs/cross-references/. See .txt version included from http://www.openbible.info website. Its extremely useful in bible study for discovering related scriptures. For any given verse, you simply query vid (verse id), and a list of rows will be returned. Each of those rows has a rank (r) for relevance, start-verse (sv), and end verse (ev) if there is one. | |
| Basic Web Interaction | |
| The web folder contains two php files. Edit the first few lines of index.php to match your server's settings. Place these in a folder on your webserver. The references search box can be multiple comma separated values. (i.e. John 3:16, Rom 3:23, 1 Jn 1:9, Romans 10:9-10) You can also directly link to a verse by altering the URI: [http://localhost/index.php?b=John 3:16, Rom 3:23, 1 Jn 1:9, Romans 10:9-10](http://localhost/index.php?b=John 3:16, Rom 3:23, 1 Jn 1:9, Romans 10:9-10) | |
| bible-mysql.sql (MySQL) is the main database and most feature-oriented due to contributions from developers. It is suggested you use that for most things, or at least convert the information from it. | |
| cross_references-mysql.sql (MySQL) is the cross-reference table. It has been separated to become an optional feature. This is converted from the project at http://www.openbible.info/labs/cross-references/. | |
| bible-sqlite.db (SQLite) is a basic simplified database for simpler applications (includes cross-references too). | |
| cross_references.txt is the source cross-reference file obtained from http://www.openbible.info/labs/cross-references/ | |
| In CSV folder, you will find (same list order with the other formats): | |
| bible_version_key.csv | |
| key_abbreviations_english.csv | |
| key_english.csv | |
| key_genre_english.csv | |
| t_asv.csv, t_bbe.csv, t_dby.csv, t_wbt.csv, t_web.csv, t_ylt.csv | |
| Acknowledgements | |
| In behalf of the original contributors (Github) | |
| Inspirations | |
| WordNet as an additional semantic resource for NLP" | |
| Caravan Insurance Challenge,Identify potential purchasers of caravan insurance policies,UCI Machine Learning,25,"Version 1,2016-11-28","finance | |
| automobiles",CSV,2 MB,Other,"12,266 views","1,169 downloads",17 kernels,,https://www.kaggle.com/uciml/caravan-insurance-challenge,"This data set used in the CoIL 2000 Challenge contains information on customers of an insurance company. The data consists of 86 variables and includes product usage data and socio-demographic data derived from zip area codes. The data was collected to answer the following question: Can you predict who would be interested in buying a caravan insurance policy and give an explanation why? | |
| Acknowledgements | |
| DISCLAIMER | |
| This dataset is owned and supplied by the Dutch datamining company Sentient Machine Research, and is based on real world business data. You are allowed to use this dataset and accompanying information for non commercial research and education purposes only. It is explicitly not allowed to use this dataset for commercial education or demonstration purposes. For any other use, please contact Peter van der Putten, info@smr.nl. | |
| This dataset has been used in the CoIL Challenge 2000 datamining competition. For papers describing results on this dataset, see the TIC 2000 homepage: http://www.wi.leidenuniv.nl/~putten/library/cc2000/ | |
| Please cite/acknowledge: | |
| P. van der Putten and M. van Someren (eds) . CoIL Challenge 2000: The Insurance Company Case. Published by Sentient Machine Research, Amsterdam. Also a Leiden Institute of Advanced Computer Science Technical Report 2000-09. June 22, 2000. | |
| The Data | |
| Originally, this dataset was broken into two parts: the training set and the evaluation set. As this was a competition, the responses to the evaluation set were not given as part of the original release; they were, however, released after the end of the competition in a separate file. This dataset contains all three of these files, combined into one. | |
| The field ORIGIN in the caravan-insurance-challenge.csv file has the values train and test, corresponding to the training and evaluation sets, respectively. To simulate the original challenge, you can ignore the test rows, and test your model's prediction on those observations once you've trained only on the training set. | |
| Each observation corresponds to a postal code. Variables beginning with M refer to demographic statistics of the postal code, while variables beginning with P and A (as well as CARAVAN, the target variable) refer to product ownership and insurance statistics in the postal code. | |
| The data file contains the following fields: | |
| ORIGIN: train or test, as described above | |
| MOSTYPE: Customer Subtype; see L0 | |
| MAANTHUI: Number of houses 1 - 10 | |
| MGEMOMV: Avg size household 1 - 6 | |
| MGEMLEEF: Avg age; see L1 | |
| MOSHOOFD: Customer main type; see L2 | |
| ** Percentages in each group, per postal code (see L3)**: | |
| MGODRK: Roman catholic | |
| MGODPR: Protestant ... | |
| MGODOV: Other religion | |
| MGODGE: No religion | |
| MRELGE: Married | |
| MRELSA: Living together | |
| MRELOV: Other relation | |
| MFALLEEN: Singles | |
| MFGEKIND: Household without children | |
| MFWEKIND: Household with children | |
| MOPLHOOG: High level education | |
| MOPLMIDD: Medium level education | |
| MOPLLAAG: Lower level education | |
| MBERHOOG: High status | |
| MBERZELF: Entrepreneur | |
| MBERBOER: Farmer | |
| MBERMIDD: Middle management | |
| MBERARBG: Skilled labourers | |
| MBERARBO: Unskilled labourers | |
| MSKA: Social class A | |
| MSKB1: Social class B1 | |
| MSKB2: Social class B2 | |
| MSKC: Social class C | |
| MSKD: Social class D | |
| MHHUUR: Rented house | |
| MHKOOP: Home owners | |
| MAUT1: 1 car | |
| MAUT2: 2 cars | |
| MAUT0: No car | |
| MZFONDS: National Health Service | |
| MZPART: Private health insurance | |
| MINKM30: Income < 30.000 | |
| MINK3045: Income 30-45.000 | |
| MINK4575: Income 45-75.000 | |
| MINK7512: Income 75-122.000 | |
| MINK123M: Income >123.000 | |
| MINKGEM: Average income | |
| MKOOPKLA: Purchasing power class | |
| ** Total number of variable in postal code (see L4)**: | |
| PWAPART: Contribution private third party insurance | |
| PWABEDR: Contribution third party insurance (firms) ... | |
| PWALAND: Contribution third party insurane (agriculture) | |
| PPERSAUT: Contribution car policies | |
| PBESAUT: Contribution delivery van policies | |
| PMOTSCO: Contribution motorcycle/scooter policies | |
| PVRAAUT: Contribution lorry policies | |
| PAANHANG: Contribution trailer policies | |
| PTRACTOR: Contribution tractor policies | |
| PWERKT: Contribution agricultural machines policies | |
| PBROM: Contribution moped policies | |
| PLEVEN: Contribution life insurances | |
| PPERSONG: Contribution private accident insurance policies | |
| PGEZONG: Contribution family accidents insurance policies | |
| PWAOREG: Contribution disability insurance policies | |
| PBRAND: Contribution fire policies | |
| PZEILPL: Contribution surfboard policies | |
| PPLEZIER: Contribution boat policies | |
| PFIETS: Contribution bicycle policies | |
| PINBOED: Contribution property insurance policies | |
| PBYSTAND: Contribution social security insurance policies | |
| AWAPART: Number of private third party insurance 1 - 12 | |
| AWABEDR: Number of third party insurance (firms) ... | |
| AWALAND: Number of third party insurance (agriculture) | |
| APERSAUT: Number of car policies | |
| ABESAUT: Number of delivery van policies | |
| AMOTSCO: Number of motorcycle/scooter policies | |
| AVRAAUT: Number of lorry policies | |
| AAANHANG: Number of trailer policies | |
| ATRACTOR: Number of tractor policies | |
| AWERKT: Number of agricultural machines policies | |
| ABROM: Number of moped policies | |
| ALEVEN: Number of life insurances | |
| APERSONG: Number of private accident insurance policies | |
| AGEZONG: Number of family accidents insurance policies | |
| AWAOREG: Number of disability insurance policies | |
| ABRAND: Number of fire policies | |
| AZEILPL: Number of surfboard policies | |
| APLEZIER: Number of boat policies | |
| AFIETS: Number of bicycle policies | |
| AINBOED: Number of property insurance policies | |
| ABYSTAND: Number of social security insurance policies | |
| CARAVAN: Number of mobile home policies 0 - 1 | |
| Keys (L1 - L4) | |
| L0: Customer subtype | |
| 1: High Income, expensive child | |
| 2: Very Important Provincials | |
| 3: High status seniors | |
| 4: Affluent senior apartments | |
| 5: Mixed seniors | |
| 6: Career and childcare | |
| 7: Dinki's (double income no kids) | |
| 8: Middle class families | |
| 9: Modern, complete families | |
| 10: Stable family | |
| 11: Family starters | |
| 12: Affluent young families | |
| 13: Young all american family | |
| 14: Junior cosmopolitan | |
| 15: Senior cosmopolitans | |
| 16: Students in apartments | |
| 17: Fresh masters in the city | |
| 18: Single youth | |
| 19: Suburban youth | |
| 20: Etnically diverse | |
| 21: Young urban have-nots | |
| 22: Mixed apartment dwellers | |
| 23: Young and rising | |
| 24: Young, low educated | |
| 25: Young seniors in the city | |
| 26: Own home elderly | |
| 27: Seniors in apartments | |
| 28: Residential elderly | |
| 29: Porchless seniors: no front yard | |
| 30: Religious elderly singles | |
| 31: Low income catholics | |
| 32: Mixed seniors | |
| 33: Lower class large families | |
| 34: Large family, employed child | |
| 35: Village families | |
| 36: Couples with teens 'Married with children' | |
| 37: Mixed small town dwellers | |
| 38: Traditional families | |
| 39: Large religous families | |
| 40: Large family farms | |
| 41: Mixed rurals | |
| L1: average age keys: | |
| 1: 20-30 years 2: 30-40 years 3: 40-50 years 4: 50-60 years 5: 60-70 years 6: 70-80 years | |
| L2: customer main type keys: | |
| 1: Successful hedonists | |
| 2: Driven Growers | |
| 3: Average Family | |
| 4: Career Loners | |
| 5: Living well | |
| 6: Cruising Seniors | |
| 7: Retired and Religeous | |
| 8: Family with grown ups | |
| 9: Conservative families | |
| 10: Farmers | |
| L3: percentage keys: | |
| 0: 0% | |
| 1: 1 - 10% | |
| 2: 11 - 23% | |
| 3: 24 - 36% | |
| 4: 37 - 49% | |
| 5: 50 - 62% | |
| 6: 63 - 75% | |
| 7: 76 - 88% | |
| 8: 89 - 99% | |
| 9: 100% | |
| L4: total number keys: | |
| 0: 0 | |
| 1: 1 - 49 | |
| 2: 50 - 99 | |
| 3: 100 - 199 | |
| 4: 200 - 499 | |
| 5: 500 - 999 | |
| 6: 1000 - 4999 | |
| 7: 5000 - 9999 | |
| 8: 10,000 - 19,999 | |
| 9: >= 20,000" | |
| StackSample: 10% of Stack Overflow Q&A,Text from 10% of Stack Overflow questions and answers on programming topics,Stack Overflow,25,"Version 1,2016-10-21","internet | |
| programming languages",CSV,3 GB,Other,"10,271 views","1,146 downloads",17 kernels,0 topics,https://www.kaggle.com/stackoverflow/stacksample,"Dataset with the text of 10% of questions and answers from the Stack Overflow programming Q&A website. | |
| This is organized as three tables: | |
| Questions contains the title, body, creation date, closed date (if applicable), score, and owner ID for all non-deleted Stack Overflow questions whose Id is a multiple of 10. | |
| Answers contains the body, creation date, score, and owner ID for each of the answers to these questions. The ParentId column links back to the Questions table. | |
| Tags contains the tags on each of these questions | |
| Datasets of all R questions and all Python questions are also available on Kaggle, but this dataset is especially useful for analyses that span many languages. | |
| Example projects include: | |
| Identifying tags from question text | |
| Predicting whether questions will be upvoted, downvoted, or closed based on their text | |
| Predicting how long questions will take to answer | |
| License | |
| All Stack Overflow user contributions are licensed under CC-BY-SA 3.0 with attribution required." | |
| Hotel Reviews,"A list of 1,000 hotels and their online reviews.",Datafiniti,25,"Version 1,2017-05-29","databases | |
| hotels | |
| linguistics | |
| internet",CSV,16 MB,CC4,"10,556 views","1,476 downloads",24 kernels,3 topics,https://www.kaggle.com/datafiniti/hotel-reviews,"About This Data | |
| This is a list of 1,000 hotels and their reviews provided by Datafiniti's Business Database. The dataset includes hotel location, name, rating, review data, title, username, and more. | |
| What You Can Do With This Data | |
| You can use this data to compare hotel reviews on a state-by-state basis; experiment with sentiment scoring and other natural language processing techniques. The review data lets you correlate keywords in the review text with ratings. E.g.: | |
| What are the bottom and top states for hotel reviews by average rating? | |
| What is the correlation between a state’s population and their number of hotel reviews? | |
| What is the correlation between a state’s tourism budget and their number of hotel reviews? | |
| Data Schema | |
| A full schema for the data is available in our support documentation. | |
| About Datafiniti | |
| Datafiniti provides instant access to web data. We compile data from thousands of websites to create standardized databases of business, product, and property information. Learn more. | |
| Want More? | |
| You can get more data like this by joining Datafiniti or requesting a demo." | |
| Human Development Report 2015,"Countries ranked by human development, gender inequality, and poverty",United Nations Development Program,25,"Version 1,2017-01-25","demographics | |
| international relations",CSV,270 KB,CC4,"7,245 views","1,587 downloads",15 kernels,0 topics,https://www.kaggle.com/undp/human-development,"Content | |
| The Human Development Index (HDI) is a summary measure of achievements in key dimensions of human development: a long and healthy life, access to knowledge, and a decent standard of living. The HDI is the geometric mean of normalized indices for each of the three dimensions. The health dimension is assessed by life expectancy at birth, the education dimension is measured by mean of years of education for adults aged 25 years and more and expected years of education for children, and the standard of living dimension is measured by gross national income per capita. The Inequality-Adjusted Human Development Index (IHDI) adjusts the HDI for inequality in the distribution of each dimension across the population. | |
| The Gender Development Index (GDI) measures gender inequalities in achievement in three basic dimensions of human development: health, measured by female and male life expectancy at birth; education, measured by female and male expected years of education for children and female and male mean years of education for adults ages 25 and older; and command over economic resources, measured by female and male estimated earned income. The Gender Inequality Index (GII) reflects gender-based disadvantage in three dimensions—reproductive health, empowerment, and the labour market—for as many countries as data of reasonable quality allow. It shows the loss in potential human development due to inequality between female and male achievements in these dimensions. | |
| The Multidimensional Poverty Index (MPI) identifies multiple deprivations at the household level in education, health, and standard of living as indicators of poverty. It uses micro data from household surveys, and — unlike the IHDI — all the indicators needed to construct the measure must come from the same survey." | |
| 2016 Global Ecological Footprint,Does your country consume more resources than it produces in a year?,Global Footprint Network,25,"Version 1,2017-03-02",ecology,CSV,22 KB,CC4,"5,878 views",704 downloads,8 kernels,0 topics,https://www.kaggle.com/footprintnetwork/ecological-footprint,"Context | |
| The ecological footprint measures the ecological assets that a given population requires to produce the natural resources it consumes (including plant-based food and fiber products, livestock and fish products, timber and other forest products, space for urban infrastructure) and to absorb its waste, especially carbon emissions. The footprint tracks the use of six categories of productive surface areas: cropland, grazing land, fishing grounds, built-up (or urban) land, forest area, and carbon demand on land. | |
| A nation’s biocapacity represents the productivity of its ecological assets, including cropland, grazing land, forest land, fishing grounds, and built-up land. These areas, especially if left unharvested, can also absorb much of the waste we generate, especially our carbon emissions. | |
| Both the ecological footprint and biocapacity are expressed in global hectares — globally comparable, standardized hectares with world average productivity. | |
| If a population’s ecological footprint exceeds the region’s biocapacity, that region runs an ecological deficit. Its demand for the goods and services that its land and seas can provide — fruits and vegetables, meat, fish, wood, cotton for clothing, and carbon dioxide absorption — exceeds what the region’s ecosystems can renew. A region in ecological deficit meets demand by importing, liquidating its own ecological assets (such as overfishing), and/or emitting carbon dioxide into the atmosphere. If a region’s biocapacity exceeds its ecological footprint, it has an ecological reserve. | |
| Acknowledgements | |
| The ecological footprint measure was conceived by Mathis Wackernagel and William Rees at the University of British Columbia. Ecological footprint data was provided by the Global Footprint Network. | |
| Inspiration | |
| Is your country running an ecological deficit, consuming more resources than it can produce per year? Which countries have the greatest ecological deficits or reserves? Do they consume less or produce more than the average country? When will Earth Overshoot Day, the day on the calendar |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment