Exploring the Universe: Explored deep insights from The Photometric LSST Astronomical Time-Series data.

plasticc-eda.ipynb

{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"DATA_PATH='../input/'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9afc7cff6bbfb4b1736df76c74f1205ab81ed04d"},"cell_type":"code","source":"df_train = pd.read_csv(DATA_PATH + 'training_set.csv')\ndf_train_meta = pd.read_csv(DATA_PATH + 'training_set_metadata.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d736a8bab3899747e92acea14312b33edb3749e9"},"cell_type":"code","source":"df_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"22391fa5acc8c349e35d7f34a7da7629b3814c85"},"cell_type":"code","source":"df_train.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2b2c9dee03744f8927ddf3b17fbefc20ebb16e8e"},"cell_type":"code","source":"df_train.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"54fe657173606ab20b5df4347276d7a96ef37236"},"cell_type":"code","source":"df_train_meta.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"33838fa2ee4c6ccf880eb72866a9bc589321e16a"},"cell_type":"code","source":"df_train_meta.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"08d26ffec6bfaa209ddca7ede2118bbb2c6446ca"},"cell_type":"code","source":"df_train_meta.info()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"135552bcba8e4d5ae8048273b1ddcef2d8dddb90"},"cell_type":"markdown","source":"# EDA"},{"metadata":{"_uuid":"d8f95c68f1a2d66b16d96f070bf4decd372f03f1"},"cell_type":"markdown","source":" ### Competition Details\nWe've astonomical Time Series data. These simulated time series, or ‘light curves’, are measurements of\nan object’s brightness as a function of time - by measuring the photon flux in six differ-\nent astronomical filters (commonly referred to as passbands). \n\n**We've to classify each object into 15 classes (14 are in the training set while the 15th one is for *otherwise* category).**\n\nThere is a common error which occurs frequenctly in the observation known as *Redshift* error. Due to this error, the rate of arrival slows down which results in fainter(reddish) light. Redshift can cause either because of the dust in the path of the light or because of the doppler's effect i.e both the entities are moving opposite each other. It is necessary to take *Redshift* in consideration in classification\n\nIn this competition **weighted LogLoss Metrics** is used which means one cannot afford to make wrong predictions with high confidence which makes this problem more interesting. \n\n\n\n"},{"metadata":{"_uuid":"f11dcf875cc35f5cd23e865aad22f443de0b0e2a"},"cell_type":"markdown","source":"### Dataset Introduction\n\n**Training data**\n* `object_id` -> the Object ID, unique identifier (given as int32 numbers).\n* `mjd` -> The time in Modified Julian Date (MJD) of the observation.  The MJD is a unit of time introduced by the Smithsonian Astrophysical Observatory in 1957 to record the orbit of Sputnik.\n* `passband` -> The specific LSST passband integer, such that u,g,r,i,z,y = 0, 1, 2, 3, 4, 5 in which it was viewed. \n* `flux` -> the measured flux (brightness) in the passband of observation as listed in the passband column. The flux is corrected for MWEBV, but for large dust extinctions the uncertainty will be much larger in spite of the correction.\n* `flux_err` ->  the uncertainty on the measurement of the flux listed above, given as float32 number..\n* `detected` -> If detected = 1, the object’s brightness is significantly different at the 3σ level relative to the reference template. This is given as a Boolean flag.\n\n** Training Meta data **\n* `object_id` -> the Object ID, unique identifier\n* `ra` ->  right ascension, sky coordinate:  longitude, units are degrees (given as float32 numbers).\n* `decl` -> declination, sky coordinate: latitude, units are degrees \n* `gal_l` -> Galactic longitude, units are degrees (given as float32 numbers).\n* `gal_b` -> Galactic lattitude, units are degrees (given as float32 numbers).\n* `ddf` -> A Boolean flag to identify the object as coming from the DDF survey area (with value ddf = 1 for the DDF). Note that while the DDF fields are contained within the full WFD survey area, the DDF fields have significantly smaller uncertainties, given that the data are provided as additions of all observations in a given night.\n* `hostgal_specz` -> The spectroscopic redshift of the source.  This is an extremely ac-\ncurate measure of redshift, provided for the training set and a small fraction of the\ntest set.\n* `hostgal_photoz` ->  The photometric redshift of the host galaxy of the astronomical source.  While this is meant to be a proxy for `hostgal_specz`, there can be large differences between the two and `hostgal_photoz` should be regarded as a far less accurate version of `hostgal_specz`.  The `hostgal_photoz` is given as float32 numbers.\n* `hostgal_photoz_err` -> The uncertainty on the `hostgal_photoz` based on LSST survey projections, given as float32 numbers.\n* `distmod` -> The  distance  (modulus)  calculated  from  the `hostgal_photoz` since  this redshift is given for all objects (given as float32 numbers).  Computing the distance modulus requires knowledge of General Relativity, and assumed values of the dark energy and dark matter content of the Universe, as mentioned in the introduction section.\n* `MWEBV` -> is equivaltent to MW E(B-V). this ‘extinction’ of light is a property of the Milky Way (MW) dust along the line of sight to the astronomical source,  and is thus a function of the sky coordinates of the source `ra`, `decl`.  This is used to determine a passband dependent dimming and reddening of light from astronomical sources as described in subsection 2.1, and is given as float32 numbers.\n* `target` -> The class of the astronomical source. This is provided in the training data.\nCorrectly determining the target (correctly assigning classification probabilities to\nthe objects) is the goal of the classification challenge for the test data.  The\ntarget is given as int8 numbers\n\n** Caveats**\n* Galactic vs extragalactic\n* Data Gaps\n* Negative flux"},{"metadata":{"_uuid":"aef580f9a12dfedfac632247c110a013767c113b"},"cell_type":"markdown","source":"### Questions and Intutions\n* `flux` and `flux_err` distribution wrt passbands\n* How much passbands have `detected` True.\n* Does `detected` is True for each passband of an object.\n* Find transient and Variable targets.\n* `flux` - `mjd` distribution\n* `flux` and `flux_err` distribution wrt mjd and passbands\n* correlation b/w `ra`, 'decl` with `gal_l`, `gal_b`\n* Effect of ddf in `hostgal_photoz_err` and `flux_err`\n* correlation between `hostgal_specz`, `MWEBV` and `hostgal_photoz`\n* Checkout frequency of negative flux, distribution of -ve flux and corresponding targets frequncy.\n* Find galactic and extragalctic targets distribution\n* Detected-Target distribution"},{"metadata":{"trusted":true,"_uuid":"11f7bb365f56551ad5520940d090d642610fa142"},"cell_type":"markdown","source":"### Let the EDA begins"},{"metadata":{"trusted":true,"_uuid":"ed85b5af7bcf439030fa6bfc56cc8c328cd88d29"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(12, 6))\nsns.distplot(df_train['flux'], ax=ax)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f9505c758fda6f270c1697b90d525e69e942a2a6"},"cell_type":"code","source":"df_train.flux.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"696e2264994feec321988c479838ba242aef53fa"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(12, 6))\nsns.distplot(df_train['flux_err'], ax=ax)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a5acb4f55a101d004b8ae329dde1c948b839a339"},"cell_type":"code","source":"df_train.flux_err.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0b8fa12be66dca9162c2d10dad78fd16680b8cbd"},"cell_type":"code","source":"df_train[(df_train.flux> 1000) | (df_train.flux < -1000)].describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"54881c8e246ccf8ddd89f8a8f466f4368d1d9ffd"},"cell_type":"code","source":"sns.distplot(df_train[(df_train.flux < 1000) & (df_train.flux > -1000)].flux)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2f3784be32a6474b988cc66168754eb3fedef19a"},"cell_type":"code","source":"sns.countplot(df_train[(df_train.flux < 1000) & (df_train.flux > -1000)].detected)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"894e47448a16d980bfaca17e189e4d363721e706"},"cell_type":"code","source":"sns.countplot(df_train[(df_train.flux> 250) | (df_train.flux < -250)].detected)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"bdc0521cde4dac648fc076ffa32015961b50f05e"},"cell_type":"code","source":"sns.distplot(df_train[(df_train.flux < 100) & (df_train.flux > -100)].flux)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"90e17f6a2d09656720b113dc86102e450961efff"},"cell_type":"code","source":"sns.countplot(df_train[(df_train.flux_err >= 100) | (df_train.flux_err <= 100)].detected)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a65777ab615ca7f36fc774cef26cba01d33c3fe4"},"cell_type":"code","source":"sns.countplot(df_train[(df_train.flux_err > 100) | (df_train.flux_err < -100)].detected)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"91cba3339c1366bbed39b4dcdc1e2aa04b58e0ff"},"cell_type":"code","source":"sns.heatmap(df_train[['flux', 'flux_err']].corr(), annot=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5405eeef76a41d3f370f12f854bf3efaa18ac8ca"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"496e3aef3187e0583990e26433f1be8ac8c2ddeb"},"cell_type":"markdown","source":"1. Most of the flux lies between -250 to 250\n2. Majority of the flux greater 250 has detected = True and majority of flux less than 250 has detective = False. Hence detected-target distribution would bee facinating.\n3. Most of the flux lies between -100 to 100"},{"metadata":{"trusted":true,"_uuid":"9f261447348885fa32a5c47c4f4ead13a0372ece"},"cell_type":"code","source":"passbands = [0, 1,2,3,4,5]\nfig, ax = plt.subplots(nrows=2, ncols=3, figsize=(18,10))\ni = 0\nfor row in ax:\n    for col in row:\n        sns.distplot(df_train[(df_train.passband == passbands[i]) & (df_train.flux < 250) & (df_train.flux > -250)]['flux'], ax=col, axlabel='flux distribution of passband ' + str(i))\n        i += 1\nplt.show()\n        ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5408435690d44f37def6dcca8923018adb6d9e8f"},"cell_type":"code","source":"sns.countplot(x='detected', data=df_train, hue='passband')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a44960687c8e7964eb37f9ecd22da7ec42ab76bf"},"cell_type":"code","source":"df_train.groupby(['passband']).count()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"42ebf594f2734d3d619bb35a73db5ddff66586cf"},"cell_type":"markdown","source":"1. Passband 4 and 5 has comparitively wider distribution.\n2. Passband 2 has the most concentrated distribution\n3. Passband 1,3,4 are almost similar\n4. Total count of passbands are different which suggests that there would be data inconsistency in the training set.\n"},{"metadata":{"trusted":true,"_uuid":"73a3935416d7a9624675780a6a9e1aee779d660d"},"cell_type":"code","source":"ts_lens = df_train.groupby(['object_id', 'passband']).size()\nf, ax = plt.subplots(figsize=(12, 6))\nsns.distplot(ts_lens, ax=ax)\nax.set_title('distribution of time series lengths')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"826981f1549b852620328906bb3a816f40a0c7f3"},"cell_type":"code","source":"passbands = [0, 1,2,3,4,5]\nfig, ax = plt.subplots(nrows=2, ncols=3, figsize=(18,10))\ni = 0\nfor row in ax:\n    for col in row:\n        sns.distplot(df_train[df_train.passband == i].groupby(['object_id']).size(), ax=col, axlabel='timeseries distribution of passband ' + str(i))\n        i += 1\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"612118528db0d673ce942f305739b22c5e04c2f9"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(12, 6))\nsns.distplot(df_train['mjd'], ax=ax, bins=200)\nax.set_title('number of observations made at each time point')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"45251d6f983e79645b3d4d8f4df3d697fcb0102d"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(12, 6))\nsns.distplot(df_train[df_train['object_id'] == 713]['mjd'], ax=ax, bins=200)\nax.set_title('number of observations made at each time point')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3ac15b91fe2171c34e32f7585a6d2d816026af45"},"cell_type":"code","source":"passbands = [0, 1,2,3,4,5]\nfig, ax = plt.subplots(nrows=2, ncols=3, figsize=(22,10))\ni = 0\nfor row in ax:\n    for col in row:\n        sns.distplot(df_train[(df_train['object_id'] == 713) & (df_train['passband'] == 3)]['mjd'], ax=col, bins=200)\n        col.set_title('number of observations made at each time point by passband ' +  str(i))\n        i += 1\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c870dd0177e989cf9eb19ca9f4403c4714ee8625"},"cell_type":"code","source":"\nf, ax = plt.subplots(figsize=(12, 9))\nax.scatter(x='mjd', y='flux', data=df_train.groupby(['mjd']).mean().reset_index())\nax.scatter(x='mjd', y='flux_err', data=df_train.groupby(['mjd']).mean().reset_index())\nax.legend(['flux', 'flux error'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"50e120a74719b809ecc0d39a596287449a2c6a90"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(12, 9))\nax.scatter(x='mjd', y='flux', data=df_train[df_train.object_id==713].groupby(['mjd']).mean().reset_index())\nax.scatter(x='mjd', y='flux_err', data=df_train[df_train.object_id==713].groupby(['mjd']).mean().reset_index())\nax.legend(['flux', 'flux error'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"48a4991791d00998492597d334a0094d96329a35"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(12, 9))\nax.scatter(x='mjd', y='flux', data=df_train[df_train.object_id==615].groupby(['mjd']).mean().reset_index())\nax.scatter(x='mjd', y='flux_err', data=df_train[df_train.object_id==615].groupby(['mjd']).mean().reset_index())\nax.legend(['flux', 'flux error'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f7fb93a9633e37a996de0346de2fa43da0123dc0"},"cell_type":"code","source":"objects = df_train.object_id.unique()\nrandom_id = np.random.randint(0, len(objects), 12)\nfig, ax = plt.subplots(nrows=4, ncols=3, figsize=(18,10))\ni = 0\nfor row in ax:\n    for col in row:\n        col.scatter(x='mjd', y='flux', data=df_train[df_train.object_id==objects[random_id[i]]].groupby(['mjd']).mean().reset_index())\n        col.scatter(x='mjd', y='flux_err', data=df_train[df_train.object_id==objects[random_id[i]]].groupby(['mjd']).mean().reset_index())\n        col.legend(['flux', 'flux error'])\n        i += 1\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1998394b5beea22c8563ff55ba4910f3844d7141"},"cell_type":"markdown","source":"1. Time series data is unevenly distributed.\n2. A significant amount of data gaps are present . \n3. Average flux of an object  either have clustered vertical distribution or well-scattered vertical distribution.\n4. Flux Errors are usually clustered and we might use the the distribution of flux and flux_error in later prediction."},{"metadata":{"trusted":true,"_uuid":"8e9125f056c864891724af25ec60a60b6f9a2a44"},"cell_type":"code","source":"sns.heatmap(df_train[['mjd', 'detected']].corr(), annot=True)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b53147d8fb1570b84a10bad6363ebf4e055c7099"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(12, 9))\nsns.heatmap(df_train_meta[['ra', 'decl', 'gal_l', 'gal_b']].corr(), annot=True, ax=ax)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e8e8635c2cb75f1aebf91eac79ad35ca347ab6bf"},"cell_type":"markdown","source":"Since four of them are coordinate it seems that `gal_l` is negatively correlated to `decl` and `ra` is positively correlated to `gal_b`"},{"metadata":{"trusted":true,"_uuid":"315200c169cf3308cd67e4c4c3096e172a12ab81"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(21, 9))\nax.scatter(df_train_meta.object_id, df_train_meta.hostgal_specz)\nax.scatter(df_train_meta.object_id, df_train_meta.hostgal_photoz)\nax.legend(['specz redshift', 'photoz redshift'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2b4ed7d9a0f13d50b77aab32c16b26a585a70a97"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(21, 9))\nax.scatter(df_train_meta.object_id, df_train_meta.hostgal_photoz_err)\nax.scatter(df_train_meta.object_id, df_train_meta.hostgal_photoz)\nax.legend(['photoz redshift error', 'photoz redshift'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"aeaf10d06f42bebb0eaac6e0772d9a36443b9ebd"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(12, 9))\nsns.distplot(df_train_meta.hostgal_photoz)\nsns.distplot(df_train_meta.hostgal_specz)\nax.legend(['photoz redshift', 'specz redshift'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"23a9f5e9012fe7e55d23513b7fa22d2a576b5bf0"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(21, 9))\nsns.distplot(df_train_meta.hostgal_photoz)\nsns.distplot(df_train_meta.hostgal_photoz_err)\nax.legend(['photoz redshift',  'photoz redshift error'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c29c6176b2456535649ba25889442c5531ab4aae"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(12, 9))\nsns.heatmap(df_train_meta[['hostgal_specz', 'hostgal_photoz', 'hostgal_photoz_err']].corr(), annot=True, ax=ax)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d750382766ae2b88ec04a19ad0664b9466a955a4"},"cell_type":"code","source":"sns.countplot(df_train_meta.target)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"14f0240de187990dced3a094719f9cd5f173a92e"},"cell_type":"code","source":"fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(12,6))\nsns.countplot(df_train_meta[df_train_meta['hostgal_photoz'] == 0].target, ax = ax[0])\nsns.countplot(df_train_meta[df_train_meta['hostgal_photoz'] != 0].target, ax = ax[1])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4a45f6887e61e2180cf079671b40b1a65d4e887a"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(12, 9))\nsns.countplot('target', hue='ddf', data=df_train_meta, ax=ax)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fdb5f2a55dfc9adcf0bd3a9835f9df9fc672a842"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(12, 6))\nsns.distplot(df_train_meta[df_train_meta.ddf==1].hostgal_photoz)\nsns.distplot(df_train_meta[df_train_meta.ddf==0].hostgal_photoz)\nax.legend(['Redshift on DDF Survey Area',  'Redshift outside DDF survey Area'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f7fa289f57bf7d38cffb8c9a105cc4bbf051e766"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(12, 6))\nsns.distplot(df_train_meta[df_train_meta.ddf==1].hostgal_photoz_err)\nsns.distplot(df_train_meta[df_train_meta.ddf==0].hostgal_photoz_err)\nax.legend(['Redshift error on DDF Survey Area',  'Redshift error outside DDF survey Area'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a75d83778797bd4a491ed02b03ad3affb4359167"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(12, 6))\nsns.distplot(df_train_meta.hostgal_photoz)\nsns.distplot(df_train_meta.mwebv)\nax.legend(['photoz redshift',  'MWEBV'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"06ea78861eb7c89a0b938507de4ffed7584e668b"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(12, 6))\nsns.distplot(df_train_meta[df_train_meta.hostgal_photoz==0].mwebv)\nsns.distplot(df_train_meta[df_train_meta.hostgal_photoz!=0].mwebv)\nax.legend(['Galactic MWEBV',  'Extragalactic MWEBV'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"327fe7bfcd9822f9814a1562f3f9d9c0393067d9"},"cell_type":"code","source":"sns.distplot(df_train_meta.distmod.dropna())\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4959c9fa72073d92be0303af0af23cbff5134eb7"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(12, 6))\nsns.distplot(df_train_meta[df_train_meta.hostgal_photoz == 0].dropna().distmod)\nsns.distplot(df_train_meta[df_train_meta.hostgal_photoz != 0].dropna().distmod)\nax.legend(['Galactic distmod',  'Extragalactic distmod'])\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8d10d428842c76c7d9038bd46b1b8640bda29d8d"},"cell_type":"code","source":"print(df_train_meta[df_train_meta.hostgal_photoz != 0].info())\nprint(df_train_meta[df_train_meta.hostgal_photoz == 0].info())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d5e4f1de92d81d91aa6c8dba33c11d27598fa125"},"cell_type":"code","source":"f, ax = plt.subplots(figsize=(12, 6))\nsns.violinplot(y='distmod', x='target', data=df_train_meta, ax=ax)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5e01ebe469438500ad2a89351d34f05c6bee3660"},"cell_type":"markdown","source":"1. Photoz redhift is more scattered than the specz redshift\n2. Error distribution is pretty concentrated.\n3. There are two types of targets: Galactic(``mwebv = 0``) and extragalcatic.\n4. Redshift outside DDF area are more scattered.\n5. distmod is ``NA`` of extragalactic objects.\n"},{"metadata":{"trusted":true,"_uuid":"4cda8d13f8ff7c0f3a2bb5bd17fc6c25184456aa"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"04af444e2b9990b6516b328b6bfb6abb6beb8dc4"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d501facf6cd19317f24366d7f582ebbf81f0fbbe"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"af72836fd068a90d1bfbb884a778aaf4a86346d3"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0b72434d79445a9eeb73b99dd0755fce3c63bc6a"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}