Skip to content

Instantly share code, notes, and snippets.

@suhaskv
Created December 23, 2020 11:32
Show Gist options
  • Save suhaskv/7c685afa854cb95d94c35b85a99867d2 to your computer and use it in GitHub Desktop.
Save suhaskv/7c685afa854cb95d94c35b85a99867d2 to your computer and use it in GitHub Desktop.
Check presence of Null and NaN in test data.
null_arr = []
nan_arr = []
for sig in metadata_test['signal_id'].values:
sig_data = pd.read_parquet('/content/test.parquet',
engine='fastparquet', columns=[str(sig)])
null_arr.append(sig_data.isnull().sum())
nan_arr.append(sig_data.isna().sum())
print(f"Number of Null values in test data: {np.sum(null_arr)}")
print(f"Number of NaN values in test data: {np.sum(nan_arr)}")
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment